Add configurable parallel/sequential pagination with parallel parameter

- Add `parallel: bool = True` parameter to control pagination strategy
- Parallel mode (default): Fetches all pages in parallel for maximum speed
- Sequential mode: Fetches pages one-by-one with early termination checks
- Early termination stops pagination when time-based filters indicate no more matches
- Useful for rate limiting and narrow time windows
- Simplified pagination logic by removing hybrid first-page pre-check
- Updated README with usage example and parameter documentation
- Version bump to 0.8.4
- All 54 tests passing

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Zachary Hampton
2025-11-13 10:36:47 -08:00
parent 9b61a89c77
commit c2f01df1ad
5 changed files with 68 additions and 32 deletions

View File

@@ -94,6 +94,17 @@ properties = scrape_property(
)
```
#### Pagination Control
```py
# Sequential mode with early termination (more efficient for narrow filters)
properties = scrape_property(
location="Los Angeles, CA",
listing_type="for_sale",
updated_in_past_hours=2, # Narrow time window
parallel=False # Fetch pages sequentially, stop when filters no longer match
)
```
## Output
```plaintext
>>> properties.head()
@@ -234,7 +245,9 @@ Optional
├── limit (integer): Limit the number of properties to fetch. Max & default is 10000.
── offset (integer): Starting position for pagination within the 10k limit. Use with limit to fetch results in chunks.
── offset (integer): Starting position for pagination within the 10k limit. Use with limit to fetch results in chunks.
└── parallel (True/False): Controls pagination strategy. Default is True (fetch pages in parallel for speed). Set to False for sequential fetching with early termination (useful for rate limiting or narrow time windows).
```
### Property Schema