101 Commits

Author SHA1 Message Date
Zachary Hampton
ef6db606fd Version bump to 0.8.10
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-16 18:32:33 -08:00
zacharyhampton
9406c92a66 Version bump to 0.8.9
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-14 17:55:33 -08:00
zacharyhampton
fefacdd264 Version bump to 0.8.8
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-14 17:32:06 -08:00
Zachary Hampton
f5784e0191 Update to iOS mobile app headers for improved API stability
- Replace browser-based headers with iOS mobile app headers
- Update GraphQL query names to match iOS app conventions (1:1 alignment)
- Add _graphql_post() wrapper to centralize GraphQL calls with dynamic operation names
- Simplify session management by removing unnecessary thread-local complexity
- Add test_parallel_search_consistency test to verify concurrent request stability
- Bump version from 0.8.6b to 0.8.7

Changes fix API flakiness under concurrent load - parallel consistency test now passes 100% (5/5 runs).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-05 19:27:47 -08:00
zacharyhampton
406ff97260 - version bump 2025-12-04 23:08:37 -08:00
Zachary Hampton
79b2b648f5 Fix sold listings not included when listing_type=None (issue #142)
When listing_type=None, sold listings were excluded despite documentation stating all types should be returned. This fix includes two changes:

1. Explicitly include common listing types (for_sale, for_rent, sold, pending, off_market) when listing_type=None instead of sending empty status parameter
2. Fix or_filters logic to only apply for PENDING when not mixed with other types like SOLD, preventing unintended filtering

Updated README documentation to accurately reflect that None returns common listing types rather than all 8 types.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-14 13:30:54 -08:00
Zachary Hampton
c2f01df1ad Add configurable parallel/sequential pagination with parallel parameter
- Add `parallel: bool = True` parameter to control pagination strategy
- Parallel mode (default): Fetches all pages in parallel for maximum speed
- Sequential mode: Fetches pages one-by-one with early termination checks
- Early termination stops pagination when time-based filters indicate no more matches
- Useful for rate limiting and narrow time windows
- Simplified pagination logic by removing hybrid first-page pre-check
- Updated README with usage example and parameter documentation
- Version bump to 0.8.4
- All 54 tests passing

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-13 10:36:47 -08:00
Zachary Hampton
9b61a89c77 Fix timezone handling for all date parameters
- Treat naive datetimes as local time and convert to UTC automatically
- Support both naive and timezone-aware datetimes for updated_since, date_from, date_to
- Fix timezone comparison bug that caused incorrect filtering with naive datetimes
- Update documentation with clear timezone handling examples
- Add comprehensive timezone tests for naive and aware datetimes
- Bump version to 0.8.3

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-11 17:40:21 -08:00
Zachary Hampton
7065f8a0d4 Optimize time-based filtering with auto-sort and early termination
## Performance Optimizations

### Auto-Apply Optimal Sort
- Auto-apply `sort_by="last_update_date"` when using `updated_since` or `updated_in_past_hours`
- Auto-apply `sort_by="pending_date"` when using PENDING listings with date filters
- Ensures API returns properties in chronological order for efficient filtering
- Users can still override by specifying different `sort_by`

### Early Termination
- Pre-check page 1 before launching parallel pagination
- If last property is outside time window, stop pagination immediately
- Avoids 95%+ of unnecessary API calls for narrow time windows
- Only applies when conditions guarantee correctness (date sort + time filter)

## Impact
- 10x faster for narrow time windows (2-3 seconds vs 30+ seconds)
- Fixes inefficiency where 10,000 properties fetched to return 10 matches
- Maintains backward compatibility - falls back when optimization unavailable

## Changes
- homeharvest/__init__.py: Auto-sort logic for time filters
- homeharvest/core/scrapers/realtor/__init__.py: `_should_fetch_more_pages()` method + early termination in pagination
- tests/test_realtor.py: Tests for optimization behavior
- README.md: Updated parameters documentation with all 8 listing types

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-11 16:52:49 -08:00
Zachary Hampton
282064d8be - readme 2025-11-11 15:21:08 -08:00
Zachary Hampton
2326d8cee9 - delete cli & version bump 2025-11-11 12:20:29 -08:00
Zachary Hampton
4e6e144617 Fix exclude_pending and mls_only filters not working with raw return type
When return_type="raw" was specified, the exclude_pending and mls_only
parameters were ignored because these filters only existed in
process_property(), which is bypassed for raw data returns.

Changes:
- Added _apply_raw_data_filters() method to handle client-side filtering
  for raw data
- Applied the filter in search() method after sorting but before returning
- Fixed exclude_pending to check flags.is_pending and flags.is_contingent
- Fixed mls_only to check source.id (not mls.id which doesn't exist in raw data)
- Added comprehensive tests for both filters with raw data

Fixes #140

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-10 11:21:28 -08:00
Zachary Hampton
21b6ba44f4 Add pagination offset support for API queries
Implements offset parameter to enable pagination within the 10k API limit. Users can now fetch results in chunks (e.g., offset=200, limit=200 for results 200-399). Includes validation to ensure offset + limit doesn't exceed API maximum. Also fixes multi-page result sorting to preserve correct order across page boundaries.

Fixes #139

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-05 10:57:01 -08:00
Zachary Hampton
1608020b69 Add last_status_change_date field for hour-level precision in date filtering
Enhances pending_date and last_sold_date with hour-level precision by introducing the last_status_change_date field. This allows for more accurate filtering of PENDING and SOLD properties when using past_hours parameter. Includes comprehensive tests and version bump to 0.7.1.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-05 10:09:58 -08:00
Zachary Hampton
18815e4207 Add comprehensive filtering system with hour-based datetime and property filters
This major enhancement addresses user needs for more precise filtering and introduces
powerful new capabilities for property searches:

Key Features:
- Hour-based date filtering (past_hours, datetime_from/to with ISO 8601 support)
- Server-side property filters (beds, baths, sqft, price, lot_sqft, year_built)
- Sorting support (list_date, sold_date, list_price, sqft, beds, baths)
- Full timestamp preservation (YYYY-MM-DD HH:MM:SS instead of date-only)
- Comprehensive validation with helpful error messages

Technical Changes:
- Preserve full datetime precision in processors.py and parsers.py
- Implement client-side hour-based post-filtering for all listing types
- Add server-side GraphQL filters for property characteristics
- Generalize filtering to work across SOLD, PENDING, FOR_SALE, FOR_RENT
- Add 15 comprehensive tests covering all new features
- Maintain full backward compatibility with existing parameters

Fixes #113 (sorting support)

Version bump to 0.7.0 reflects significant new functionality while maintaining
backward compatibility.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-20 14:21:12 -07:00
Zachary Hampton
e9bfd66986 Update pyproject.toml 2025-09-08 23:19:15 -07:00
Zachary Hampton
23a8fd6a77 Update pyproject.toml 2025-09-08 22:15:52 -07:00
Zachary Hampton
75c245cde7 implement client-side pending_date filtering for PENDING properties
- Fix PENDING properties to filter by pending_date instead of list_date
- Add client-side filtering for PENDING as server-side pending_date filter is broken
- Include contingent properties without pending_date for comprehensive results
- Enhance documentation to clarify past_days behavior per listing type
- Add property_history field to GraphQL queries for future enhancements
- Add comprehensive test for pending date filtering functionality
- Optimize filtering logic with helper methods for better maintainability

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-09-08 16:36:48 -07:00
Zachary Hampton
44e6a43cc4 - fix none type error 2025-07-21 17:33:58 -07:00
Zachary Hampton
3013786355 Update pyproject.toml 2025-07-19 11:46:55 -07:00
Zachary Hampton
6c6243eba4 - add all new data fields 2025-07-15 13:21:48 -07:00
Zachary Hampton
0d85100091 - update dependencies 2025-07-14 17:08:27 -07:00
Zachary Hampton
0fdc309262 Update pyproject.toml 2025-07-03 10:28:14 -07:00
Zachary Hampton
b4f05b254a Update pyproject.toml 2025-07-03 09:43:10 -07:00
zachary
68a3438c6e - single home return type bug fix 2025-05-05 12:29:36 -07:00
zachary
d06595fe56 - updated queries 2025-05-03 13:28:12 -07:00
zachary
e378feeefe - bug fixes 2025-04-12 18:34:35 -07:00
zachary
8a5683fe79 - return type parameter
- optimized get extra fields with query clustering
2025-04-12 17:55:52 -07:00
Cullen Watson
0de916e590 enh:tax history 2025-01-06 05:28:36 -06:00
Cullen Watson
8e04f6b117 enh: property type (#102) 2024-11-03 17:23:07 -06:00
Zachary Hampton
1f717bd9e3 - switch eps
- new hrefs
- property_id, listing_id data points
2024-09-06 15:49:07 -07:00
Zachary Hampton
8cfe056f79 - office mls set 2024-08-23 10:54:43 -07:00
Zachary Hampton
1010c743b6 - agent mls set and nrds id 2024-08-23 10:47:45 -07:00
Zachary Hampton
32fdc281e3 - rewrote & optimized flow
- new_construction data point
- renamed "agent" & "broker" to "agent_name" & "broker_name"
- added builder & office data
- added entity uuids
2024-08-20 05:19:15 -07:00
Zachary Hampton
6d14b8df5a - fix limit parameter
- fix specific for_rent apartment listing prices
2024-08-13 10:44:11 -07:00
Zachary Hampton
3f44744d61 - primary photo bug fix
- limit parameter
2024-07-15 07:19:57 -07:00
Zachary Hampton
ac0cad62a7 - optimizations 2024-06-14 21:50:23 -07:00
Cullen Watson
beb885cc8d fix: govt type (#82) 2024-06-12 17:34:34 -05:00
Zachary Hampton
011680f7d8 - style error bug fix 2024-06-06 15:24:12 -07:00
Zachary Hampton
93e6778a48 - exclude_pending parameter 2024-05-31 22:17:29 -07:00
Zachary Hampton
ec036bb989 - optimizations & updated realtor headers 2024-05-20 12:13:30 -07:00
Zachary Hampton
aacd168545 - alt photos bug fix 2024-05-18 17:47:55 -07:00
Zachary Hampton
0d70007000 - alt photos bug fix 2024-05-16 23:04:07 -07:00
Zachary Hampton
018d3fbac4 - Python 3.9 support (tested) (could potentially work for lower versions, but I have not validated such) 2024-05-14 19:13:04 -07:00
Zachary Hampton
803fd618e9 - data cleaning & CONDOP bug fixes 2024-05-12 21:12:12 -07:00
Zachary Hampton
b23b55ca80 - full street line (data quality improvement) 2024-05-12 18:49:44 -07:00
Zachary Hampton
3458a08383 - broker data 2024-05-11 21:35:29 -07:00
Zachary Hampton
46985dcee4 - various data quality fixes (including #70) 2024-05-02 08:48:53 -07:00
Cullen Watson
04ae968716 enh: assessed/estimated value (#77) 2024-04-30 15:29:54 -05:00
Cullen
c5b15e9be5 chore: version 2024-04-20 17:45:29 -05:00