Zachary Hampton
GitHub
commit
1f0c351974
131
README.md
131
README.md
|
|
@ -1,26 +1,44 @@
|
|||
# HomeHarvest
|
||||
|
||||
**HomeHarvest** aims to be the top Python real estate scraping library.
|
||||
|
||||
_**Under Consideration**: We're looking into the possibility of an Excel plugin to cater to a broader audience._
|
||||
**HomeHarvest** is a simple but comprehensive real estate scraping library.
|
||||
|
||||
[](https://replit.com/@ZacharyHampton/HomeHarvestDemo)
|
||||
|
||||
|
||||
*Looking to build a data-focused software product?* **[Book a call](https://calendly.com/zachary-products/15min)** *to work with us.*
|
||||
## Features
|
||||
|
||||
|
||||
- Scrapes properties from **Zillow**, **Realtor.com** & **Redfin** simultaneously
|
||||
- Aggregates the properties in a Pandas DataFrame
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install --upgrade homeharvest
|
||||
```
|
||||
_Python version >= [3.10](https://www.python.org/downloads/release/python-3100/) required_
|
||||
|
||||
## Example Usage
|
||||
## Usage
|
||||
```py
|
||||
>>> from homeharvest import scrape_property
|
||||
... properties = scrape_property(
|
||||
... location="85281", site_name="zillow", listing_type="for_rent"
|
||||
... )
|
||||
from homeharvest import scrape_property
|
||||
import pandas as pd
|
||||
|
||||
properties: pd.DataFrame = scrape_property(
|
||||
site_name=["zillow", "realtor.com", "redfin"],
|
||||
location="85281",
|
||||
listing_type="for_rent" # for_sale / sold
|
||||
)
|
||||
|
||||
#: Note, to export to CSV or Excel, use properties.to_csv() or properties.to_excel().
|
||||
print(properties)
|
||||
|
||||
|
||||
```
|
||||
## Output
|
||||
```py
|
||||
>>> properties.head()
|
||||
address_one city ... mls_id description
|
||||
street city ... mls_id description
|
||||
0 420 N Scottsdale Rd Tempe ... NaN NaN
|
||||
1 1255 E University Dr Tempe ... NaN NaN
|
||||
2 1979 E Rio Salado Pkwy Tempe ... NaN NaN
|
||||
|
|
@ -29,14 +47,93 @@ pip install --upgrade homeharvest
|
|||
[5 rows x 23 columns]
|
||||
```
|
||||
|
||||
### Site Name Options
|
||||
### Parameters for `scrape_properties()`
|
||||
```plaintext
|
||||
Required
|
||||
├── location (str): address in various formats e.g. just zip, full address, city/state, etc.
|
||||
└── listing_type (enum): for_rent, for_sale, sold
|
||||
Optional
|
||||
├── site_name (List[enum], default=all three sites): zillow, realtor.com, redfin
|
||||
```
|
||||
|
||||
- `zillow`
|
||||
- `redfin`
|
||||
- `realtor.com`
|
||||
### Property Schema
|
||||
```plaintext
|
||||
Property
|
||||
├── Basic Information:
|
||||
│ ├── property_url (str)
|
||||
│ ├── site_name (enum): zillow, redfin, realtor.com
|
||||
│ ├── listing_type (enum: ListingType)
|
||||
│ └── property_type (enum): house, apartment, condo, townhouse, single_family, multi_family, building
|
||||
|
||||
### Listing Types
|
||||
├── Address Details:
|
||||
│ ├── street_address (str)
|
||||
│ ├── city (str)
|
||||
│ ├── state (str)
|
||||
│ ├── zip_code (str)
|
||||
│ ├── unit (str)
|
||||
│ └── country (str)
|
||||
|
||||
├── Property Features:
|
||||
│ ├── price (int)
|
||||
│ ├── tax_assessed_value (int)
|
||||
│ ├── currency (str)
|
||||
│ ├── square_feet (int)
|
||||
│ ├── beds (int)
|
||||
│ ├── baths (float)
|
||||
│ ├── lot_area_value (float)
|
||||
│ ├── lot_area_unit (str)
|
||||
│ ├── stories (int)
|
||||
│ └── year_built (int)
|
||||
|
||||
├── Miscellaneous Details:
|
||||
│ ├── price_per_sqft (int)
|
||||
│ ├── mls_id (str)
|
||||
│ ├── agent_name (str)
|
||||
│ ├── img_src (str)
|
||||
│ ├── description (str)
|
||||
│ ├── status_text (str)
|
||||
│ ├── latitude (float)
|
||||
│ ├── longitude (float)
|
||||
│ └── posted_time (str) [Only for Zillow]
|
||||
|
||||
├── Building Details (for property_type: building):
|
||||
│ ├── bldg_name (str)
|
||||
│ ├── bldg_unit_count (int)
|
||||
│ ├── bldg_min_beds (int)
|
||||
│ ├── bldg_min_baths (float)
|
||||
│ └── bldg_min_area (int)
|
||||
|
||||
└── Apartment Details (for property type: apartment):
|
||||
└── apt_min_price (int)
|
||||
```
|
||||
## Supported Countries for Property Scraping
|
||||
|
||||
* **Zillow**: contains listings in the **US** & **Canada**
|
||||
* **Realtor.com**: mainly from the **US** but also has international listings
|
||||
* **Redfin**: listings mainly in the **US**, **Canada**, & has expanded to some areas in **Mexico**
|
||||
|
||||
### Exceptions
|
||||
The following exceptions may be raised when using HomeHarvest:
|
||||
|
||||
- `InvalidSite` - valid options: `zillow`, `redfin`, `realtor.com`
|
||||
- `InvalidListingType` - valid options: `for_sale`, `for_rent`, `sold`
|
||||
- `NoResultsFound` - no properties found from your input
|
||||
- `GeoCoordsNotFound` - if Zillow scraper is not able to create geo-coordinates from the location you input
|
||||
|
||||
## Frequently Asked Questions
|
||||
|
||||
---
|
||||
|
||||
**Q: Encountering issues with your queries?**
|
||||
**A:** Try a single site and/or broaden the location. If problems persist, [submit an issue](https://github.com/ZacharyHampton/HomeHarvest/issues).
|
||||
|
||||
---
|
||||
|
||||
**Q: Received a Forbidden 403 response code?**
|
||||
**A:** This indicates that you have been blocked by the real estate site for sending too many requests. Currently, **Zillow** is particularly aggressive with blocking. We recommend:
|
||||
|
||||
- Waiting a few seconds between requests.
|
||||
- Trying a VPN to change your IP address.
|
||||
|
||||
---
|
||||
|
||||
- `for_rent`
|
||||
- `for_sale`
|
||||
- `sold`
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@ import re
|
|||
import json
|
||||
from .. import Scraper
|
||||
from ....utils import parse_address_two, parse_unit
|
||||
from ....exceptions import NoResultsFound, PropertyNotFound
|
||||
from ....exceptions import GeoCoordsNotFound
|
||||
from ..models import Property, Address, ListingType, PropertyType, SiteName
|
||||
|
||||
|
||||
|
|
@ -45,7 +45,7 @@ class ZillowScraper(Scraper):
|
|||
return self._fetch_properties_backend(coords)
|
||||
|
||||
else:
|
||||
raise BoxBoundsNotFound("Box bounds could not be located.")
|
||||
raise GeoCoordsNotFound("Box bounds could not be located.")
|
||||
|
||||
elif "gdpClientCache" in data["props"]["pageProps"]:
|
||||
gdp_client_cache = json.loads(data["props"]["pageProps"]["gdpClientCache"])
|
||||
|
|
@ -55,7 +55,7 @@ class ZillowScraper(Scraper):
|
|||
property = self._get_single_property_page(property_data)
|
||||
|
||||
return [property]
|
||||
raise PropertyNotFound("Specific property data not found in the response.")
|
||||
raise NoResultsFound("Specific property data not found in the response.")
|
||||
|
||||
def _fetch_properties_backend(self, coords):
|
||||
url = "https://www.zillow.com/async-create-search-page-state"
|
||||
|
|
|
|||
|
|
@ -10,9 +10,5 @@ class NoResultsFound(Exception):
|
|||
"""Raised when no results are found for the given location"""
|
||||
|
||||
|
||||
class PropertyNotFound(Exception):
|
||||
"""Raised when no property is found for the given address"""
|
||||
|
||||
|
||||
class BoxBoundsNotFound(Exception):
|
||||
class GeoCoordsNotFound(Exception):
|
||||
"""Raised when no property is found for the given address"""
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
[tool.poetry]
|
||||
name = "homeharvest"
|
||||
version = "0.1.4"
|
||||
description = "Real estate scraping library"
|
||||
version = "0.2.0"
|
||||
description = "Real estate scraping library supporting Zillow, Realtor.com & Redfin."
|
||||
authors = ["Zachary Hampton <zachary@zacharysproducts.com>", "Cullen Watson <cullen@cullen.ai>"]
|
||||
homepage = "https://github.com/ZacharyHampton/HomeHarvest"
|
||||
readme = "README.md"
|
||||
|
|
|
|||
Loading…
Reference in New Issue