fix: use enum value (#65)

- alt photos bug fix (#57)

README.md

@@ -1,24 +1,16 @@
 <img src="https://github.com/ZacharyHampton/HomeHarvest/assets/78247585/d1a2bf8b-09f5-4c57-b33a-0ada8a34f12d" width="400">
 
-**HomeHarvest** is a simple, yet comprehensive, real estate scraping library that extracts and formats data in the style of MLS listings.
+**HomeHarvest** is a real estate scraping library that extracts and formats data in the style of MLS listings.
-
-[![Try with Replit](https://replit.com/badge?caption=Try%20with%20Replit)](https://replit.com/@ZacharyHampton/HomeHarvestDemo)
 
 **Not technical?** Try out the web scraping tool on our site at [tryhomeharvest.com](https://tryhomeharvest.com).
 
 *Looking to build a data-focused software product?* **[Book a call](https://bunsly.com)** *to work with us.*
 
-Check out another project we wrote: ***[JobSpy](https://github.com/Bunsly/JobSpy)** – a Python package for job scraping*
-
 ## HomeHarvest Features
 
 - **Source**: Fetches properties directly from **Realtor.com**.
 - **Data Format**: Structures data to resemble MLS listings.
 - **Export Flexibility**: Options to save as either CSV or Excel.
-- **Usage Modes**:
-  - **Python**: For those who'd like to integrate scraping into their Python scripts.
-  - **CLI**: For users who prefer command-line operations.
-
 
 [Video Guide for HomeHarvest](https://youtu.be/J1qgNPgmSLI) - _updated for release v0.3.4_
 
@@ -27,7 +19,7 @@ Check out another project we wrote: ***[JobSpy](https://github.com/Bunsly/JobSpy
 ## Installation
 
 ```bash
-pip install homeharvest
+pip install -U homeharvest
 ```
   _Python version >= [3.10](https://www.python.org/downloads/release/python-3100/) required_ 
 
@@ -50,9 +42,9 @@ properties = scrape_property(
   
   # date_from="2023-05-01", # alternative to past_days 
   # date_to="2023-05-28", 
+  # foreclosure=True
   
   # mls_only=True,  # only fetch MLS listings
-  # proxy="http://user:pass@host:port"  # use a proxy to change your IP address
 )
 print(f"Number of properties: {len(properties)}")
 
@@ -61,7 +53,6 @@ properties.to_csv(filename, index=False)
 print(properties.head())
 ```
 
-
 ## Output
 ```plaintext
 >>> properties.head()
@@ -98,37 +89,9 @@ Optional
 │
 ├── mls_only (True/False): If set, fetches only MLS listings (mainly applicable to 'sold' listings)
 │
+├── foreclosure (True/False): If set, fetches only foreclosures
+│
 └── proxy (string): In format 'http://user:pass@host:port'
-
-
-```
-
-### CLI 
-
-```
-usage: homeharvest [-l {for_sale,for_rent,sold}] [-o {excel,csv}] [-f FILENAME] [-p PROXY] [-d DAYS] [-r RADIUS] [-m] [-c] location
-                                                                                                                             
-Home Harvest Property Scraper                                                                                                 
-                                                                                                                             
-positional arguments:                                                                                                         
-  location              Location to scrape (e.g., San Francisco, CA)                                                          
-                                                                                                                             
-options:                                                                                                                      
-  -l {for_sale,for_rent,sold,pending}, --listing_type {for_sale,for_rent,sold,pending}                                                        
-                        Listing type to scrape                                                                                
-  -o {excel,csv}, --output {excel,csv}                                                                                        
-                        Output format                                                                                         
-  -f FILENAME, --filename FILENAME                                                                                            
-                        Name of the output file (without extension)                                                           
-  -p PROXY, --proxy PROXY                                                                                                     
-                        Proxy to use for scraping                                                                             
-  -d DAYS, --days DAYS  Sold/listed in last _ days filter.                                                                           
-  -r RADIUS, --radius RADIUS                                                                                                  
-                        Get comparable properties within _ (e.g., 0.0) miles. Only applicable for individual addresses.        
-  -m, --mls_only        If set, fetches only MLS listings.                                                                    
-```
-```bash
-homeharvest "San Francisco, CA" -l for_rent -o excel -f HomeHarvest
 ```
 
 ### Property Schema
@@ -161,6 +124,7 @@ Property
 │ ├── days_on_mls
 │ ├── list_price
 │ ├── list_date
+│ ├── pending_date
 │ ├── sold_price
 │ ├── last_sold_date
 │ ├── price_per_sqft
@@ -180,20 +144,3 @@ The following exceptions may be raised when using HomeHarvest:
 - `InvalidListingType` - valid options: `for_sale`, `for_rent`, `sold`
 - `InvalidDate` - date_from or date_to is not in the format YYYY-MM-DD
   
-  
-## Frequently Asked Questions
----
-
-**Q: Encountering issues with your searches?**  
-**A:** Try to broaden the parameters you're using. If problems persist, [submit an issue](https://github.com/ZacharyHampton/HomeHarvest/issues).
-
----
-
-**Q: Received a Forbidden 403 response code?**  
-**A:** This indicates that you have been blocked by Realtor.com for sending too many requests. We recommend:
-
-- Waiting a few seconds between requests.
-- Trying a VPN or useing a proxy as a parameter to scrape_property() to change your IP address.
-
----
-

homeharvest/__init__.py

@@ -15,6 +15,7 @@ def scrape_property(
     proxy: str = None,
     date_from: str = None,
     date_to: str = None,
+    foreclosure: bool = None,
 ) -> pd.DataFrame:
     """
     Scrape properties from Realtor.com based on a given location and listing type.
@@ -38,6 +39,7 @@ def scrape_property(
         last_x_days=past_days,
         date_from=date_from,
         date_to=date_to,
+        foreclosure=foreclosure,
     )
 
     site = RealtorScraper(scraper_input)

homeharvest/core/scrapers/__init__.py

@@ -13,6 +13,7 @@ class ScraperInput:
     last_x_days: int | None = None
     date_from: str | None = None
     date_to: str | None = None
+    foreclosure: bool | None = None
 
 
 class Scraper:
@@ -40,6 +41,7 @@ class Scraper:
         self.mls_only = scraper_input.mls_only
         self.date_from = scraper_input.date_from
         self.date_to = scraper_input.date_to
+        self.foreclosure = scraper_input.foreclosure
 
     def search(self) -> list[Property]:
         ...

homeharvest/core/scrapers/models.py

@@ -23,6 +23,27 @@ class ListingType(Enum):
     SOLD = "SOLD"
 
 
+class PropertyType(Enum):
+    APARTMENT = "APARTMENT"
+    BUILDING = "BUILDING"
+    COMMERCIAL = "COMMERCIAL"
+    CONDO_TOWNHOME = "CONDO_TOWNHOME"
+    CONDO_TOWNHOME_ROWHOME_COOP = "CONDO_TOWNHOME_ROWHOME_COOP"
+    CONDO = "CONDO"
+    CONDOS = "CONDOS"
+    COOP = "COOP"
+    DUPLEX_TRIPLEX = "DUPLEX_TRIPLEX"
+    FARM = "FARM"
+    INVESTMENT = "INVESTMENT"
+    LAND = "LAND"
+    MOBILE = "MOBILE"
+    MULTI_FAMILY = "MULTI_FAMILY"
+    RENTAL = "RENTAL"
+    SINGLE_FAMILY = "SINGLE_FAMILY"
+    TOWNHOMES = "TOWNHOMES"
+    OTHER = "OTHER"
+
+
 @dataclass
 class Address:
     street: str | None = None
@@ -36,7 +57,7 @@ class Address:
 class Description:
     primary_photo: str | None = None
     alt_photos: list[str] | None = None
-    style: str | None = None
+    style: PropertyType | None = None
     beds: int | None = None
     baths_full: int | None = None
     baths_half: int | None = None
@@ -58,6 +79,7 @@ class Property:
 
     list_price: int | None = None
     list_date: str | None = None
+    pending_date: str | None = None
     last_sold_date: str | None = None
     prc_sqft: int | None = None
     hoa_fee: int | None = None

homeharvest/core/scrapers/realtor/__init__.py

@@ -9,7 +9,7 @@ from typing import Dict, Union, Optional
 from concurrent.futures import ThreadPoolExecutor, as_completed
 
 from .. import Scraper
-from ..models import Property, Address, ListingType, Description
+from ..models import Property, Address, ListingType, Description, PropertyType
 
 
 class RealtorScraper(Scraper):
@@ -84,11 +84,10 @@ class RealtorScraper(Scraper):
                             garage
                             permalink
                         }
-                        primary_photo {
+                        media {
-                            href
+                            photos {
-                        }
+                                href
-                        photos {
+                            }
-                            href
                         }
                     }
                 }"""
@@ -120,9 +119,11 @@ class RealtorScraper(Scraper):
             "list_date") else None
         last_sold_date_str = property_info["basic"]["sold_date"].split("T")[0] if property_info["basic"].get(
             "sold_date") else None
+        pending_date_str = property_info["pending_date"].split("T")[0] if property_info.get("pending_date") else None
 
         list_date = datetime.strptime(list_date_str, "%Y-%m-%d") if list_date_str else None
         last_sold_date = datetime.strptime(last_sold_date_str, "%Y-%m-%d") if last_sold_date_str else None
+        pending_date = datetime.strptime(pending_date_str, "%Y-%m-%d") if pending_date_str else None
         today = datetime.now()
 
         days_on_mls = None
@@ -150,6 +151,7 @@ class RealtorScraper(Scraper):
                and property_info["basic"].get("sqft")
             else None,
             last_sold_date=last_sold_date,
+            pending_date=pending_date,
             latitude=property_info["address"]["location"]["coordinate"].get("lat")
             if able_to_get_lat_long
             else None,
@@ -158,8 +160,7 @@ class RealtorScraper(Scraper):
             else None,
             address=self._parse_address(property_info, search_type="handle_listing"),
             description=Description(
-                primary_photo=property_info["primary_photo"].get("href", "").replace("s.jpg", "od-w480_h360_x2.webp?w=1080&q=75"),
+                alt_photos=self.process_alt_photos(property_info.get("media", {}).get("photos", [])),
-                alt_photos=self.process_alt_photos(property_info.get("photos", [])),
                 style=property_info["basic"].get("type", "").upper(),
                 beds=property_info["basic"].get("beds"),
                 baths_full=property_info["basic"].get("baths_full"),
@@ -288,7 +289,7 @@ class RealtorScraper(Scraper):
         ]
 
     def general_search(
-        self, variables: dict, search_type: str
+            self, variables: dict, search_type: str
     ) -> Dict[str, Union[int, list[Property]]]:
         """
         Handles a location area & returns a list of properties
@@ -297,6 +298,7 @@ class RealtorScraper(Scraper):
                             count
                             total
                             results {
+                                pending_date
                                 property_id
                                 list_date
                                 status
@@ -309,6 +311,7 @@ class RealtorScraper(Scraper):
                                     is_pending
                                 }
                                 description {
+                                    type
                                     sqft
                                     beds
                                     baths_full
@@ -383,6 +386,12 @@ class RealtorScraper(Scraper):
         )
 
         listing_type = ListingType.FOR_SALE if self.listing_type == ListingType.PENDING else self.listing_type
+        is_foreclosure = ""
+
+        if variables.get('foreclosure') is True:
+            is_foreclosure = "foreclosure: true"
+        elif variables.get('foreclosure') is False:
+            is_foreclosure = "foreclosure: false"
 
         if search_type == "comps":  #: comps search, came from an address
             query = """query Property_search(
@@ -392,6 +401,7 @@ class RealtorScraper(Scraper):
                     ) {
                         home_search(
                             query: { 
+                                %s
                                 nearby: {
                                     coordinates: $coordinates
                                     radius: $radius 
@@ -404,6 +414,7 @@ class RealtorScraper(Scraper):
                             limit: 200
                             offset: $offset
                     ) %s""" % (
+                is_foreclosure,
                 listing_type.value.lower(),
                 date_param,
                 pending_or_contingent_param,
@@ -420,6 +431,7 @@ class RealtorScraper(Scraper):
                             ) {
                                 home_search(
                                     query: {
+                                        %s
                                         city: $city
                                         county: $county
                                         postal_code: $postal_code
@@ -432,6 +444,7 @@ class RealtorScraper(Scraper):
                                     limit: 200
                                     offset: $offset
                                 ) %s""" % (
+                is_foreclosure,
                 listing_type.value.lower(),
                 date_param,
                 pending_or_contingent_param,
@@ -440,7 +453,7 @@ class RealtorScraper(Scraper):
             )
         else:  #: general search, came from an address
             query = (
-                """query Property_search(
+                    """query Property_search(
                         $property_id: [ID]!
                         $offset: Int!,
                     ) {
@@ -451,7 +464,7 @@ class RealtorScraper(Scraper):
                             limit: 1
                             offset: $offset
                         ) %s"""
-                % results_query
+                    % results_query
             )
 
         payload = {
@@ -467,12 +480,12 @@ class RealtorScraper(Scraper):
         properties: list[Property] = []
 
         if (
-            response_json is None
+                response_json is None
-            or "data" not in response_json
+                or "data" not in response_json
-            or response_json["data"] is None
+                or response_json["data"] is None
-            or search_key not in response_json["data"]
+                or search_key not in response_json["data"]
-            or response_json["data"][search_key] is None
+                or response_json["data"][search_key] is None
-            or "results" not in response_json["data"][search_key]
+                or "results" not in response_json["data"][search_key]
         ):
             return {"total": 0, "properties": []}
 
@@ -487,10 +500,10 @@ class RealtorScraper(Scraper):
                 continue
 
             able_to_get_lat_long = (
-                result
+                    result
-                and result.get("location")
+                    and result.get("location")
-                and result["location"].get("address")
+                    and result["location"].get("address")
-                and result["location"]["address"].get("coordinate")
+                    and result["location"]["address"].get("coordinate")
             )
 
             is_pending = result["flags"].get("is_pending") or result["flags"].get("is_contingent")
@@ -503,7 +516,7 @@ class RealtorScraper(Scraper):
                 mls_id=result["source"].get("listing_id")
                 if "source" in result and isinstance(result["source"], dict)
                 else None,
-                property_url=f"{self.PROPERTY_URL}{result['property_id']}",
+                property_url=f"{self.PROPERTY_URL}{result['property_id']}" if self.listing_type != ListingType.FOR_RENT else f"{self.PROPERTY_URL}M{result['property_id']}?listing_status=rental",
                 status="PENDING" if is_pending else result["status"].upper(),
                 list_price=result["list_price"],
                 list_date=result["list_date"].split("T")[0]
@@ -567,12 +580,20 @@ class RealtorScraper(Scraper):
                     return gql_results["properties"]
 
             else:  #: general search, comps (radius)
+                if not location_info.get("centroid"):
+                    return []
+
                 coordinates = list(location_info["centroid"].values())
                 search_variables |= {
                     "coordinates": coordinates,
                     "radius": "{}mi".format(self.radius),
                 }
 
+        elif location_type == "postal_code":
+            search_variables |= {
+                "postal_code": location_info.get("postal_code"),
+            }
+
         else:  #: general search, location
             search_variables |= {
                 "city": location_info.get("city"),
@@ -581,6 +602,9 @@ class RealtorScraper(Scraper):
                 "postal_code": location_info.get("postal_code"),
             }
 
+        if self.foreclosure:
+            search_variables['foreclosure'] = self.foreclosure
+
         result = self.general_search(search_variables, search_type=search_type)
         total = result["total"]
         homes = result["properties"]
@@ -641,7 +665,6 @@ class RealtorScraper(Scraper):
 
     @staticmethod
     def _parse_description(result: dict) -> Description:
-
         description_data = result.get("description", {})
 
         if description_data is None or not isinstance(description_data, dict):
@@ -661,7 +684,7 @@ class RealtorScraper(Scraper):
         return Description(
             primary_photo=primary_photo,
             alt_photos=RealtorScraper.process_alt_photos(result.get("photos")),
-            style=style,
+            style=PropertyType(style) if style else None,
             beds=description_data.get("beds"),
             baths_full=description_data.get("baths_full"),
             baths_half=description_data.get("baths_half"),
@@ -673,7 +696,6 @@ class RealtorScraper(Scraper):
             stories=description_data.get("stories"),
         )
 
-
     @staticmethod
     def calculate_days_on_mls(result: dict) -> Optional[int]:
         list_date_str = result.get("list_date")

homeharvest/utils.py

@@ -5,8 +5,6 @@ from .exceptions import InvalidListingType, InvalidDate
 
 ordered_properties = [
     "property_url",
-    "primary_photo",
-    "alt_photos",
     "mls",
     "mls_id",
     "status",
@@ -33,6 +31,8 @@ ordered_properties = [
     "stories",
     "hoa_fee",
     "parking_garage",
+    "primary_photo",
+    "alt_photos",
 ]
 
 
@@ -53,7 +53,7 @@ def process_result(result: Property) -> pd.DataFrame:
     description = result.description
     prop_data["primary_photo"] = description.primary_photo
     prop_data["alt_photos"] = ", ".join(description.alt_photos)
-    prop_data["style"] = description.style
+    prop_data["style"] = description.style.value
     prop_data["beds"] = description.beds
     prop_data["full_baths"] = description.baths_full
     prop_data["half_baths"] = description.baths_half

poetry.lock

@@ -1,4 +1,4 @@
-# This file is automatically @generated by Poetry 1.6.1 and should not be changed by hand.
+# This file is automatically @generated by Poetry 1.8.2 and should not be changed by hand.
 
 [[package]]
 name = "certifi"
@@ -121,17 +121,6 @@ files = [
     {file = "colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44"},
 ]
 
-[[package]]
-name = "et-xmlfile"
-version = "1.1.0"
-description = "An implementation of lxml.xmlfile for the standard library"
-optional = false
-python-versions = ">=3.6"
-files = [
-    {file = "et_xmlfile-1.1.0-py3-none-any.whl", hash = "sha256:a2ba85d1d6a74ef63837eed693bcb89c3f752169b0e3e7ae5b16ca5e1b3deada"},
-    {file = "et_xmlfile-1.1.0.tar.gz", hash = "sha256:8eb9e2bc2f8c97e37a2dc85a09ecdcdec9d8a396530a6d5a33b30b9a92da0c5c"},
-]
-
 [[package]]
 name = "exceptiongroup"
 version = "1.1.3"
@@ -209,20 +198,6 @@ files = [
     {file = "numpy-1.26.0.tar.gz", hash = "sha256:f93fc78fe8bf15afe2b8d6b6499f1c73953169fad1e9a8dd086cdff3190e7fdf"},
 ]
 
-[[package]]
-name = "openpyxl"
-version = "3.1.2"
-description = "A Python library to read/write Excel 2010 xlsx/xlsm files"
-optional = false
-python-versions = ">=3.6"
-files = [
-    {file = "openpyxl-3.1.2-py2.py3-none-any.whl", hash = "sha256:f91456ead12ab3c6c2e9491cf33ba6d08357d802192379bb482f1033ade496f5"},
-    {file = "openpyxl-3.1.2.tar.gz", hash = "sha256:a6f5977418eff3b2d5500d54d9db50c8277a368436f4e4f8ddb1be3422870184"},
-]
-
-[package.dependencies]
-et-xmlfile = "*"
-
 [[package]]
 name = "packaging"
 version = "23.2"
@@ -438,4 +413,4 @@ zstd = ["zstandard (>=0.18.0)"]
 [metadata]
 lock-version = "2.0"
 python-versions = ">=3.10,<3.13"
-content-hash = "09ad811d74a42363ff4c3ccd012d8f73c89d7d978e5a6445b0f3d2e231922f1b"
+content-hash = "018a1a6afb2d7f4c764b9e1926145d7d8d630ffa43f7786e062cbfd9a9a845a0"

pyproject.toml

@@ -1,8 +1,8 @@
 [tool.poetry]
 name = "homeharvest"
-version = "0.3.10"
+version = "0.3.15"
-description = "Real estate scraping library supporting Zillow, Realtor.com & Redfin."
+description = "Real estate scraping library"
-authors = ["Zachary Hampton <zachary@zacharysproducts.com>", "Cullen Watson <cullen@cullen.ai>"]
+authors = ["Zachary Hampton <zachary@bunsly.com>", "Cullen Watson <cullen@bunsly.com>"]
 homepage = "https://github.com/Bunsly/HomeHarvest"
 readme = "README.md"
 
@@ -13,7 +13,6 @@ homeharvest = "homeharvest.cli:main"
 python = ">=3.10,<3.13"
 requests = "^2.31.0"
 pandas = "^2.1.1"
-openpyxl = "^3.1.2"
 
 
 [tool.poetry.group.dev.dependencies]

tests/test_realtor.py

@@ -131,6 +131,15 @@ def test_realtor():
     assert all([result is not None for result in results])
 
 
+def test_realtor_city():
+    results = scrape_property(
+        location="Atlanta, GA",
+        listing_type="for_sale",
+    )
+
+    assert results is not None and len(results) > 0
+
+
 def test_realtor_bad_address():
     bad_results = scrape_property(
             location="abceefg ju098ot498hh9",
@@ -139,3 +148,15 @@ def test_realtor_bad_address():
     if len(bad_results) == 0:
         assert True
 
+
+def test_realtor_foreclosed():
+    foreclosed = scrape_property(
+        location="Dallas, TX", listing_type="for_sale", past_days=100, foreclosure=True
+    )
+
+    not_foreclosed = scrape_property(
+        location="Dallas, TX", listing_type="for_sale", past_days=100, foreclosure=False
+    )
+
+    assert len(foreclosed) != len(not_foreclosed)
+