mirror of
https://github.com/Bunsly/HomeHarvest.git
synced 2026-03-04 19:44:29 -08:00
Add configurable parallel/sequential pagination with parallel parameter
- Add `parallel: bool = True` parameter to control pagination strategy - Parallel mode (default): Fetches all pages in parallel for maximum speed - Sequential mode: Fetches pages one-by-one with early termination checks - Early termination stops pagination when time-based filters indicate no more matches - Useful for rate limiting and narrow time windows - Simplified pagination logic by removing hybrid first-page pre-check - Updated README with usage example and parameter documentation - Version bump to 0.8.4 - All 54 tests passing 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
15
README.md
15
README.md
@@ -94,6 +94,17 @@ properties = scrape_property(
|
||||
)
|
||||
```
|
||||
|
||||
#### Pagination Control
|
||||
```py
|
||||
# Sequential mode with early termination (more efficient for narrow filters)
|
||||
properties = scrape_property(
|
||||
location="Los Angeles, CA",
|
||||
listing_type="for_sale",
|
||||
updated_in_past_hours=2, # Narrow time window
|
||||
parallel=False # Fetch pages sequentially, stop when filters no longer match
|
||||
)
|
||||
```
|
||||
|
||||
## Output
|
||||
```plaintext
|
||||
>>> properties.head()
|
||||
@@ -234,7 +245,9 @@ Optional
|
||||
│
|
||||
├── limit (integer): Limit the number of properties to fetch. Max & default is 10000.
|
||||
│
|
||||
└── offset (integer): Starting position for pagination within the 10k limit. Use with limit to fetch results in chunks.
|
||||
├── offset (integer): Starting position for pagination within the 10k limit. Use with limit to fetch results in chunks.
|
||||
│
|
||||
└── parallel (True/False): Controls pagination strategy. Default is True (fetch pages in parallel for speed). Set to False for sequential fetching with early termination (useful for rate limiting or narrow time windows).
|
||||
```
|
||||
|
||||
### Property Schema
|
||||
|
||||
Reference in New Issue
Block a user