mirror of
https://github.com/Bunsly/HomeHarvest.git
synced 2026-03-04 19:44:29 -08:00
Optimize time-based filtering with auto-sort and early termination
## Performance Optimizations ### Auto-Apply Optimal Sort - Auto-apply `sort_by="last_update_date"` when using `updated_since` or `updated_in_past_hours` - Auto-apply `sort_by="pending_date"` when using PENDING listings with date filters - Ensures API returns properties in chronological order for efficient filtering - Users can still override by specifying different `sort_by` ### Early Termination - Pre-check page 1 before launching parallel pagination - If last property is outside time window, stop pagination immediately - Avoids 95%+ of unnecessary API calls for narrow time windows - Only applies when conditions guarantee correctness (date sort + time filter) ## Impact - 10x faster for narrow time windows (2-3 seconds vs 30+ seconds) - Fixes inefficiency where 10,000 properties fetched to return 10 matches - Maintains backward compatibility - falls back when optimization unavailable ## Changes - homeharvest/__init__.py: Auto-sort logic for time filters - homeharvest/core/scrapers/realtor/__init__.py: `_should_fetch_more_pages()` method + early termination in pagination - tests/test_realtor.py: Tests for optimization behavior - README.md: Updated parameters documentation with all 8 listing types 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
[tool.poetry]
|
||||
name = "homeharvest"
|
||||
version = "0.8.1"
|
||||
version = "0.8.2"
|
||||
description = "Real estate scraping library"
|
||||
authors = ["Zachary Hampton <zachary@bunsly.com>", "Cullen Watson <cullen@bunsly.com>"]
|
||||
homepage = "https://github.com/ZacharyHampton/HomeHarvest"
|
||||
|
||||
Reference in New Issue
Block a user