Merge pull request #14 from ZacharyHampton/keep_duplicates_flag

Keep duplicates flag

README.md

@@ -10,7 +10,7 @@
 - Scrapes properties from **Zillow**, **Realtor.com** & **Redfin** simultaneously
 - Aggregates the properties in a Pandas DataFrame
 
-[Video Guide for HomeHarvest](https://www.youtube.com/watch?v=HCoHoiJdWQY)
+[Video Guide for HomeHarvest](https://youtu.be/JnV7eR2Ve2o) - _updated for release v0.2.7_
 
 ![homeharvest](https://github.com/ZacharyHampton/HomeHarvest/assets/78247585/b3d5d727-e67b-4a9f-85d8-1e65fd18620a)
 
@@ -37,6 +37,7 @@ By default:
 - The `-o` or `--output` default format is `excel`. Options are `csv` or `excel`.
 - If `-f` or `--filename` is left blank, the default is `HomeHarvest_<current_timestamp>`.
 - If `-p` or `--proxy` is not provided, the scraper uses the local IP.
+- Use `-k` or `--keep_duplicates` to keep duplicate properties based on address. If not provided, duplicates will be removed.
 ### Python 
 
 ```py
@@ -71,8 +72,9 @@ Required
 ├── location (str): address in various formats e.g. just zip, full address, city/state, etc.
 └── listing_type (enum): for_rent, for_sale, sold
 Optional
-├── site_name (List[enum], default=all three sites): zillow, realtor.com, redfin
+├── site_name (list[enum], default=all three sites): zillow, realtor.com, redfin
 ├── proxy (str): in format 'http://user:pass@host:port' or [https, socks]
+└── keep_duplicates (bool, default=False): whether to keep or remove duplicate properties based on address
 ```
 
 ### Property Schema
@@ -81,7 +83,7 @@ Property
 ├── Basic Information:
 │   ├── property_url (str)
 │   ├── site_name (enum): zillow, redfin, realtor.com
-│   ├── listing_type (enum: ListingType)
+│   ├── listing_type (enum): for_sale, for_rent, sold
 │   └── property_type (enum): house, apartment, condo, townhouse, single_family, multi_family, building
 
 ├── Address Details:
@@ -92,45 +94,38 @@ Property
 │   ├── unit (str)
 │   └── country (str)
 
-├── Property Features:
+├── House for Sale Features:
-│   ├── price (int)
 │   ├── tax_assessed_value (int)
-│   ├── currency (str)
-│   ├── square_feet (int)
-│   ├── beds (int)
-│   ├── baths (float)
 │   ├── lot_area_value (float)
 │   ├── lot_area_unit (str)
 │   ├── stories (int)
-│   └── year_built (int)
+│   ├── year_built (int)
+│   └── price_per_sqft (int)
+
+├── Building for Sale and Apartment Details:
+│   ├── bldg_name (str)
+│   ├── beds_min (int)
+│   ├── beds_max (int)
+│   ├── baths_min (float)
+│   ├── baths_max (float)
+│   ├── sqft_min (int)
+│   ├── sqft_max (int)
+│   ├── price_min (int)
+│   ├── price_max (int)
+│   ├── area_min (int)
+│   └── unit_count (int)
 
 ├── Miscellaneous Details:
-│   ├── price_per_sqft (int)
 │   ├── mls_id (str)
 │   ├── agent_name (str)
 │   ├── img_src (str)
 │   ├── description (str)
 │   ├── status_text (str)
-│   ├── latitude (float)
+│   └── posted_time (str)
-│   ├── longitude (float)
-│   └── posted_time (str) [Only for Zillow]
 
-├── Building Details (for property_type: building):
+└── Location Details:
-│   ├── bldg_name (str)
+    ├── latitude (float)
-│   ├── bldg_unit_count (int)
+    └── longitude (float)
-│   ├── bldg_min_beds (int)
-│   ├── bldg_min_baths (float)
-│   └── bldg_min_area (int)
-
-└── Apartment Details (for property type: apartment):
-    ├── apt_min_beds: int
-    ├── apt_max_beds: int
-    ├── apt_min_baths: float
-    ├── apt_max_baths: float
-    ├── apt_min_price: int
-    ├── apt_max_price: int
-    ├── apt_min_sqft: int
-    ├── apt_max_sqft: int
 ```
 ## Supported Countries for Property Scraping
 
@@ -144,7 +139,7 @@ The following exceptions may be raised when using HomeHarvest:
 - `InvalidSite` - valid options: `zillow`, `redfin`, `realtor.com`
 - `InvalidListingType` - valid options: `for_sale`, `for_rent`, `sold`
 - `NoResultsFound` - no properties found from your input
-- `GeoCoordsNotFound` - if Zillow scraper is not able to create geo-coordinates from the location you input
+- `GeoCoordsNotFound` - if Zillow scraper is not able to derive geo-coordinates from the location you input
 
 ## Frequently Asked Questions
 

homeharvest/__init__.py

@@ -119,6 +119,7 @@ def scrape_property(
     site_name: Union[str, list[str]] = None,
     listing_type: str = "for_sale",
     proxy: str = None,
+    keep_duplicates: bool = False
 ) -> pd.DataFrame:
     """
     Scrape property from various sites from a given location and listing type.
@@ -165,5 +166,6 @@ def scrape_property(
         if col not in final_df.columns:
             final_df[col] = None
 
-    final_df = final_df.drop_duplicates(subset=columns_to_track, keep="first")
+    if not keep_duplicates:
+        final_df = final_df.drop_duplicates(subset=columns_to_track, keep="first")
     return final_df

homeharvest/cli.py

@@ -42,11 +42,18 @@ def main():
         help="Name of the output file (without extension)",
     )
 
+    parser.add_argument(
+        "-k",
+        "--keep_duplicates",
+        action="store_true",
+        help="Keep duplicate properties based on address"
+    )
+
     parser.add_argument("-p", "--proxy", type=str, default=None, help="Proxy to use for scraping")
 
     args = parser.parse_args()
 
-    result = scrape_property(args.location, args.site_name, args.listing_type, proxy=args.proxy)
+    result = scrape_property(args.location, args.site_name, args.listing_type, proxy=args.proxy, keep_duplicates=args.keep_duplicates)
 
     if not args.filename:
         timestamp = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "homeharvest"
-version = "0.2.7"
+version = "0.2.8"
 description = "Real estate scraping library supporting Zillow, Realtor.com & Redfin."
 authors = ["Zachary Hampton <zachary@zacharysproducts.com>", "Cullen Watson <cullen@cullen.ai>"]
 homepage = "https://github.com/ZacharyHampton/HomeHarvest"