mirror of
https://github.com/Bunsly/HomeHarvest.git
synced 2026-03-04 19:44:29 -08:00
[chore]: clean up
This commit is contained in:
Cullen Watson
155
README.md
155
README.md
@@ -1,6 +1,6 @@
|
||||
<img src="https://github.com/ZacharyHampton/HomeHarvest/assets/78247585/d1a2bf8b-09f5-4c57-b33a-0ada8a34f12d" width="400">
|
||||
|
||||
**HomeHarvest** is a simple, yet comprehensive, real estate scraping library.
|
||||
**HomeHarvest** is a simple, yet comprehensive, real estate scraping library that extracts and formats data in the style of MLS listings.
|
||||
|
||||
[](https://replit.com/@ZacharyHampton/HomeHarvestDemo)
|
||||
|
||||
@@ -11,10 +11,14 @@
|
||||
|
||||
Check out another project we wrote: ***[JobSpy](https://github.com/cullenwatson/JobSpy)** – a Python package for job scraping*
|
||||
|
||||
## Features
|
||||
## HomeHarvest Features
|
||||
|
||||
- Scrapes properties from **Zillow**, **Realtor.com** & **Redfin** simultaneously
|
||||
- Aggregates the properties in a Pandas DataFrame
|
||||
- **Source**: Fetches properties directly from **Realtor.com**.
|
||||
- **Data Format**: Structures data to resemble MLS listings.
|
||||
- **Export Flexibility**: Options to save as either CSV or Excel.
|
||||
- **Usage Modes**:
|
||||
- **CLI**: For users who prefer command-line operations.
|
||||
- **Python**: For those who'd like to integrate scraping into their Python scripts.
|
||||
|
||||
[Video Guide for HomeHarvest](https://youtu.be/JnV7eR2Ve2o) - _updated for release v0.2.7_
|
||||
|
||||
@@ -29,21 +33,6 @@ pip install homeharvest
|
||||
|
||||
## Usage
|
||||
|
||||
### Python
|
||||
|
||||
```py
|
||||
from homeharvest import scrape_property
|
||||
import pandas as pd
|
||||
|
||||
properties: pd.DataFrame = scrape_property(
|
||||
location="85281",
|
||||
listing_type="for_rent" # for_sale / sold
|
||||
)
|
||||
|
||||
#: Note, to export to CSV or Excel, use properties.to_csv() or properties.to_excel().
|
||||
print(properties)
|
||||
```
|
||||
|
||||
### CLI
|
||||
|
||||
```
|
||||
@@ -55,7 +44,6 @@ positional arguments:
|
||||
location Location to scrape (e.g., San Francisco, CA)
|
||||
|
||||
options:
|
||||
-h, --help show this help message and exit
|
||||
-l {for_sale,for_rent,sold}, --listing_type {for_sale,for_rent,sold}
|
||||
Listing type to scrape
|
||||
-o {excel,csv}, --output {excel,csv}
|
||||
@@ -72,104 +60,107 @@ options:
|
||||
> homeharvest "San Francisco, CA" -l for_rent -o excel -f HomeHarvest
|
||||
```
|
||||
|
||||
## Output
|
||||
### Python
|
||||
|
||||
```py
|
||||
>>> properties.head()
|
||||
property_url site_name listing_type apt_min_price apt_max_price ...
|
||||
0 https://www.redfin.com/AZ/Tempe/1003-W-Washing... redfin for_rent 1666.0 2750.0 ...
|
||||
1 https://www.redfin.com/AZ/Tempe/VELA-at-Town-L... redfin for_rent 1665.0 3763.0 ...
|
||||
2 https://www.redfin.com/AZ/Tempe/Camden-Tempe/a... redfin for_rent 1939.0 3109.0 ...
|
||||
3 https://www.redfin.com/AZ/Tempe/Emerson-Park/a... redfin for_rent 1185.0 1817.0 ...
|
||||
4 https://www.redfin.com/AZ/Tempe/Rio-Paradiso-A... redfin for_rent 1470.0 2235.0 ...
|
||||
[5 rows x 41 columns]
|
||||
from homeharvest import scrape_property
|
||||
from datetime import datetime
|
||||
|
||||
# Generate filename based on current timestamp
|
||||
current_timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
filename = f"output/{current_timestamp}.csv"
|
||||
|
||||
properties = scrape_property(
|
||||
location="San Diego, CA",
|
||||
listing_type="sold", # for_sale, for_rent
|
||||
)
|
||||
print(f"Number of properties: {len(properties)}")
|
||||
properties.to_csv(filename, index=False)
|
||||
```
|
||||
|
||||
### Parameters for `scrape_properties()`
|
||||
|
||||
## Output
|
||||
```plaintext
|
||||
>>> properties.head()
|
||||
MLS MLS # Status Style ... COEDate LotSFApx PrcSqft Stories
|
||||
0 SDCA 230018348 SOLD CONDOS ... 2023-10-03 290110 803 2
|
||||
1 SDCA 230016614 SOLD TOWNHOMES ... 2023-10-03 None 838 3
|
||||
2 SDCA 230016367 SOLD CONDOS ... 2023-10-03 30056 649 1
|
||||
3 MRCA NDP2306335 SOLD SINGLE_FAMILY ... 2023-10-03 7519 661 2
|
||||
4 SDCA 230014532 SOLD CONDOS ... 2023-10-03 None 752 1
|
||||
[5 rows x 22 columns]
|
||||
```
|
||||
|
||||
### Parameters for `scrape_property()`
|
||||
```
|
||||
Required
|
||||
├── location (str): address in various formats e.g. just zip, full address, city/state, etc.
|
||||
└── listing_type (enum): for_rent, for_sale, sold
|
||||
Optional
|
||||
├── site_name (list[enum], default=all three sites): zillow, realtor.com, redfin
|
||||
├── proxy (str): in format 'http://user:pass@host:port' or [https, socks]
|
||||
└── keep_duplicates (bool, default=False): whether to keep or remove duplicate properties based on address
|
||||
├── radius_for_comps (float): Radius in miles to find comparable properties based on individual addresses.
|
||||
├── sold_last_x_days (int): Number of past days to filter sold properties.
|
||||
├── proxy (str): in format 'http://user:pass@host:port'
|
||||
```
|
||||
|
||||
### Property Schema
|
||||
```plaintext
|
||||
Property
|
||||
├── Basic Information:
|
||||
│ ├── property_url (str)
|
||||
│ ├── site_name (enum): zillow, redfin, realtor.com
|
||||
│ ├── listing_type (enum): for_sale, for_rent, sold
|
||||
│ └── property_type (enum): house, apartment, condo, townhouse, single_family, multi_family, building
|
||||
│ ├── property_url (str)
|
||||
│ ├── mls (str)
|
||||
│ ├── mls_id (str)
|
||||
│ └── status (str)
|
||||
|
||||
├── Address Details:
|
||||
│ ├── street_address (str)
|
||||
│ ├── city (str)
|
||||
│ ├── state (str)
|
||||
│ ├── zip_code (str)
|
||||
│ ├── unit (str)
|
||||
│ └── country (str)
|
||||
│ ├── street (str)
|
||||
│ ├── unit (str)
|
||||
│ ├── city (str)
|
||||
│ ├── state (str)
|
||||
│ └── zip (str)
|
||||
|
||||
├── House for Sale Features:
|
||||
│ ├── tax_assessed_value (int)
|
||||
│ ├── lot_area_value (float)
|
||||
│ ├── lot_area_unit (str)
|
||||
│ ├── stories (int)
|
||||
│ ├── year_built (int)
|
||||
│ └── price_per_sqft (int)
|
||||
├── Property Description:
|
||||
│ ├── style (str)
|
||||
│ ├── beds (int)
|
||||
│ ├── baths_full (int)
|
||||
│ ├── baths_half (int)
|
||||
│ ├── sqft (int)
|
||||
│ ├── lot_sqft (int)
|
||||
│ ├── sold_price (int)
|
||||
│ ├── year_built (int)
|
||||
│ ├── garage (float)
|
||||
│ └── stories (int)
|
||||
|
||||
├── Building for Sale and Apartment Details:
|
||||
│ ├── bldg_name (str)
|
||||
│ ├── beds_min (int)
|
||||
│ ├── beds_max (int)
|
||||
│ ├── baths_min (float)
|
||||
│ ├── baths_max (float)
|
||||
│ ├── sqft_min (int)
|
||||
│ ├── sqft_max (int)
|
||||
│ ├── price_min (int)
|
||||
│ ├── price_max (int)
|
||||
│ ├── area_min (int)
|
||||
│ └── unit_count (int)
|
||||
├── Property Listing Details:
|
||||
│ ├── list_price (int)
|
||||
│ ├── list_date (str)
|
||||
│ ├── last_sold_date (str)
|
||||
│ ├── prc_sqft (int)
|
||||
│ └── hoa_fee (int)
|
||||
|
||||
├── Miscellaneous Details:
|
||||
│ ├── mls_id (str)
|
||||
│ ├── agent_name (str)
|
||||
│ ├── img_src (str)
|
||||
│ ├── description (str)
|
||||
│ ├── status_text (str)
|
||||
│ └── posted_time (str)
|
||||
|
||||
└── Location Details:
|
||||
├── latitude (float)
|
||||
└── longitude (float)
|
||||
├── Location Details:
|
||||
│ ├── latitude (float)
|
||||
│ ├── longitude (float)
|
||||
│ └── neighborhoods (str)
|
||||
```
|
||||
## Supported Countries for Property Scraping
|
||||
|
||||
* **Zillow**: contains listings in the **US** & **Canada**
|
||||
* **Realtor.com**: mainly from the **US** but also has international listings
|
||||
* **Redfin**: listings mainly in the **US**, **Canada**, & has expanded to some areas in **Mexico**
|
||||
|
||||
### Exceptions
|
||||
The following exceptions may be raised when using HomeHarvest:
|
||||
|
||||
- `InvalidSite` - valid options: `zillow`, `redfin`, `realtor.com`
|
||||
- `InvalidListingType` - valid options: `for_sale`, `for_rent`, `sold`
|
||||
- `NoResultsFound` - no properties found from your input
|
||||
- `GeoCoordsNotFound` - if Zillow scraper is not able to derive geo-coordinates from the location you input
|
||||
|
||||
## Frequently Asked Questions
|
||||
|
||||
---
|
||||
|
||||
**Q: Encountering issues with your queries?**
|
||||
**A:** Try a single site and/or broaden the location. If problems persist, [submit an issue](https://github.com/ZacharyHampton/HomeHarvest/issues).
|
||||
**Q: Encountering issues with your searches?**
|
||||
**A:** Try to broaden the location. If problems persist, [submit an issue](https://github.com/ZacharyHampton/HomeHarvest/issues).
|
||||
|
||||
---
|
||||
|
||||
**Q: Received a Forbidden 403 response code?**
|
||||
**A:** This indicates that you have been blocked by the real estate site for sending too many requests. Currently, **Zillow** is particularly aggressive with blocking. We recommend:
|
||||
**A:** This indicates that you have been blocked by Realtor.com for sending too many requests. We recommend:
|
||||
|
||||
- Waiting a few seconds between requests.
|
||||
- Trying a VPN to change your IP address.
|
||||
|
||||
Reference in New Issue
Block a user