Add comprehensive filtering system with hour-based datetime and property filters

This major enhancement addresses user needs for more precise filtering and introduces
powerful new capabilities for property searches:

Key Features:
- Hour-based date filtering (past_hours, datetime_from/to with ISO 8601 support)
- Server-side property filters (beds, baths, sqft, price, lot_sqft, year_built)
- Sorting support (list_date, sold_date, list_price, sqft, beds, baths)
- Full timestamp preservation (YYYY-MM-DD HH:MM:SS instead of date-only)
- Comprehensive validation with helpful error messages

Technical Changes:
- Preserve full datetime precision in processors.py and parsers.py
- Implement client-side hour-based post-filtering for all listing types
- Add server-side GraphQL filters for property characteristics
- Generalize filtering to work across SOLD, PENDING, FOR_SALE, FOR_RENT
- Add 15 comprehensive tests covering all new features
- Maintain full backward compatibility with existing parameters

Fixes #113 (sorting support)

Version bump to 0.7.0 reflects significant new functionality while maintaining
backward compatibility.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Zachary Hampton
2025-10-20 14:21:12 -07:00
parent c9b05ebd9d
commit 18815e4207
9 changed files with 1009 additions and 34 deletions

View File

@@ -27,6 +27,29 @@ class ScraperInput(BaseModel):
limit: int = 10000
return_type: ReturnType = ReturnType.pandas
# New date/time filtering parameters
past_hours: int | None = None
datetime_from: str | None = None
datetime_to: str | None = None
# New property filtering parameters
beds_min: int | None = None
beds_max: int | None = None
baths_min: float | None = None
baths_max: float | None = None
sqft_min: int | None = None
sqft_max: int | None = None
price_min: int | None = None
price_max: int | None = None
lot_sqft_min: int | None = None
lot_sqft_max: int | None = None
year_built_min: int | None = None
year_built_max: int | None = None
# New sorting parameters
sort_by: str | None = None
sort_direction: str = "desc"
class Scraper:
session = None
@@ -85,6 +108,29 @@ class Scraper:
self.limit = scraper_input.limit
self.return_type = scraper_input.return_type
# New date/time filtering
self.past_hours = scraper_input.past_hours
self.datetime_from = scraper_input.datetime_from
self.datetime_to = scraper_input.datetime_to
# New property filtering
self.beds_min = scraper_input.beds_min
self.beds_max = scraper_input.beds_max
self.baths_min = scraper_input.baths_min
self.baths_max = scraper_input.baths_max
self.sqft_min = scraper_input.sqft_min
self.sqft_max = scraper_input.sqft_max
self.price_min = scraper_input.price_min
self.price_max = scraper_input.price_max
self.lot_sqft_min = scraper_input.lot_sqft_min
self.lot_sqft_max = scraper_input.lot_sqft_max
self.year_built_min = scraper_input.year_built_min
self.year_built_max = scraper_input.year_built_max
# New sorting
self.sort_by = scraper_input.sort_by
self.sort_direction = scraper_input.sort_direction
def search(self) -> list[Union[Property | dict]]: ...
@staticmethod