enh: indeed more fields (#126)

* fix(ziprecruiter): pagination

* chore: version

README.md

@@ -4,17 +4,14 @@
 
 **Not technical?** Try out the web scraping tool on our site at [usejobspy.com](https://usejobspy.com).
 
-*Looking to build a data-focused software product?* **[Book a call](https://calendly.com/zachary-products/15min)** *to
+*Looking to build a data-focused software product?* **[Book a call](https://bunsly.com/)** *to
-work with us.*  
+work with us.*
-\
-Check out another project we wrote: ***[HomeHarvest](https://github.com/ZacharyHampton/HomeHarvest)** – a Python package
-for real estate scraping*
 
 ## Features
 
-- Scrapes job postings from **LinkedIn**, **Indeed** & **ZipRecruiter** simultaneously
+- Scrapes job postings from **LinkedIn**, **Indeed**, **Glassdoor**, & **ZipRecruiter** simultaneously
 - Aggregates the job postings in a Pandas DataFrame
-- Proxy support (HTTP/S, SOCKS)
+- Proxy support
 
 [Video Guide for JobSpy](https://www.youtube.com/watch?v=RuP1HrAZnxs&pp=ygUgam9icyBzY3JhcGVyIGJvdCBsaW5rZWRpbiBpbmRlZWQ%3D) -
 Updated for release v1.1.3
@@ -24,7 +21,7 @@ Updated for release v1.1.3
 ### Installation
 
 ```
-pip install --upgrade python-jobspy
+pip install -U python-jobspy
 ```
 
 _Python version >= [3.10](https://www.python.org/downloads/release/python-3100/) required_
@@ -32,40 +29,20 @@ _Python version >= [3.10](https://www.python.org/downloads/release/python-3100/)
 ### Usage
 
 ```python
+import csv
 from jobspy import scrape_jobs
-import pandas as pd
 
-jobs: pd.DataFrame = scrape_jobs(
+jobs = scrape_jobs(
-    site_name=["indeed", "linkedin", "zip_recruiter"],
+    site_name=["indeed", "linkedin", "zip_recruiter", "glassdoor"],
     search_term="software engineer",
     location="Dallas, TX",
-    results_wanted=10,
+    results_wanted=20,
-
+    hours_old=72, # (only linkedin is hour specific, others round up to days old)
-    country_indeed='USA'  # only needed for indeed
+    country_indeed='USA'  # only needed for indeed / glassdoor
-
-    # use if you want to use a proxy
-    # proxy="http://jobspy:5a4vpWtj8EeJ2hoYzk@ca.smartproxy.com:20001",
-    # offset=25 # use if you want to start at a specific offset
 )
-
+print(f"Found {len(jobs)} jobs")
-# formatting for pandas
+print(jobs.head())
-pd.set_option('display.max_columns', None)
+jobs.to_csv("jobs.csv", quoting=csv.QUOTE_NONNUMERIC, escapechar="\\", index=False) # to_xlsx
-pd.set_option('display.max_rows', None)
-pd.set_option('display.width', None)
-pd.set_option('display.max_colwidth', 50)  # set to 0 to see full job url / desc
-
-# 1 output to console
-print(jobs)
-
-# 2 display in Jupyter Notebook (1. pip install jupyter 2. jupyter notebook)
-# display(jobs)
-
-# 3 output to .csv
-# jobs.to_csv('jobs.csv', index=False)
-
-# 4 output to .xlsx
-# jobs.to_xlsx('jobs.xlsx', index=False)
-
 ```
 
 ### Output
@@ -84,18 +61,22 @@ zip_recruiter Software Developer                 TEKsystems        Phoenix
 
 ```plaintext
 Required
-├── site_type (List[enum]): linkedin, zip_recruiter, indeed
+├── site_type (List[enum]): linkedin, zip_recruiter, indeed, glassdoor
 └── search_term (str)
 Optional
-├── location (int)
+├── location (str)
-├── distance (int): in miles
+├── distance (int): in miles, default 50
 ├── job_type (enum): fulltime, parttime, internship, contract
-├── proxy (str): in format 'http://user:pass@host:port' or [https, socks]
+├── proxy (str): in format 'http://user:pass@host:port'
 ├── is_remote (bool)
+├── linkedin_fetch_description (bool): fetches full description for LinkedIn (slower)
 ├── results_wanted (int): number of job results to retrieve for each site specified in 'site_type'
-├── easy_apply (bool): filters for jobs that are hosted on LinkedIn
+├── easy_apply (bool): filters for jobs that are hosted on the job board site
+├── linkedin_company_ids (list[int): searches for linkedin jobs with specific company ids
+├── description_format (enum): markdown, html (format type of the job descriptions)
 ├── country_indeed (enum): filters the country on Indeed (see below for correct spelling)
-├── offset (enum): starts the search from an offset (e.g. 25 will start the search from the 25th result)
+├── offset (num): starts the search from an offset (e.g. 25 will start the search from the 25th result)
+├── hours_old (int): filters jobs by the number of hours since the job was posted (ZipRecruiter and Glassdoor round up to next day. If you use this on Indeed, it will not filter by job_type or is_remote)
 ```
 
 ### JobPost Schema
@@ -104,89 +85,92 @@ Optional
 JobPost
 ├── title (str)
 ├── company (str)
+├── company_url (str)
 ├── job_url (str)
 ├── location (object)
 │   ├── country (str)
 │   ├── city (str)
 │   ├── state (str)
 ├── description (str)
-├── job_type (enum): fulltime, parttime, internship, contract
+├── job_type (str): fulltime, parttime, internship, contract
 ├── compensation (object)
-│   ├── interval (enum): yearly, monthly, weekly, daily, hourly
+│   ├── interval (str): yearly, monthly, weekly, daily, hourly
 │   ├── min_amount (int)
 │   ├── max_amount (int)
 │   └── currency (enum)
 └── date_posted (date)
+└── emails (str)
+└── is_remote (bool)
+
+Indeed specific
+├── company_country (str)
+└── company_addresses (str)
+└── company_industry (str)
+└── company_employees_label (str)
+└── company_revenue_label (str)
+└── company_description (str)
+└── ceo_name (str)
+└── ceo_photo_url (str)
+└── logo_photo_url (str)
+└── banner_photo_url (str)
 ```
 
-### Exceptions
-
-The following exceptions may be raised when using JobSpy:
-
-* `LinkedInException`
-* `IndeedException`
-* `ZipRecruiterException`
-
 ## Supported Countries for Job Searching
 
 ### **LinkedIn**
 
-LinkedIn searches globally & uses only the `location` parameter.
+LinkedIn searches globally & uses only the `location` parameter. You can only fetch 1000 jobs max from the LinkedIn endpoint we are using
 
 ### **ZipRecruiter**
 
 ZipRecruiter searches for jobs in **US/Canada** & uses only the `location` parameter.
 
-### **Indeed**
+### **Indeed / Glassdoor**
 
-Indeed supports most countries, but the `country_indeed` parameter is required. Additionally, use the `location`
+Indeed & Glassdoor supports most countries, but the `country_indeed` parameter is required. Additionally, use the `location`
-parameter to narrow down the location, e.g. city & state if necessary.
+parameter to narrow down the location, e.g. city & state if necessary. 
 
-You can specify the following countries when searching on Indeed (use the exact name):
+You can specify the following countries when searching on Indeed (use the exact name, * indicates support for Glassdoor):
 
 |                      |              |            |                |
 |----------------------|--------------|------------|----------------|
-| Argentina            | Australia    | Austria    | Bahrain        |
+| Argentina            | Australia*   | Austria*   | Bahrain        |
-| Belgium              | Brazil       | Canada     | Chile          |
+| Belgium*             | Brazil*      | Canada*    | Chile          |
 | China                | Colombia     | Costa Rica | Czech Republic |
 | Denmark              | Ecuador      | Egypt      | Finland        |
-| France               | Germany      | Greece     | Hong Kong      |
+| France*              | Germany*     | Greece     | Hong Kong*     |
-| Hungary              | India        | Indonesia  | Ireland        |
+| Hungary              | India*       | Indonesia  | Ireland*       |
-| Israel               | Italy        | Japan      | Kuwait         |
+| Israel               | Italy*       | Japan      | Kuwait         |
-| Luxembourg           | Malaysia     | Mexico     | Morocco        |
+| Luxembourg           | Malaysia     | Mexico*    | Morocco        |
-| Netherlands          | New Zealand  | Nigeria    | Norway         |
+| Netherlands*         | New Zealand* | Nigeria    | Norway         |
 | Oman                 | Pakistan     | Panama     | Peru           |
 | Philippines          | Poland       | Portugal   | Qatar          |
-| Romania              | Saudi Arabia | Singapore  | South Africa   |
+| Romania              | Saudi Arabia | Singapore* | South Africa   |
-| South Korea          | Spain        | Sweden     | Switzerland    |
+| South Korea          | Spain*       | Sweden     | Switzerland*   |
 | Taiwan               | Thailand     | Turkey     | Ukraine        |
-| United Arab Emirates | UK           | USA        | Uruguay        |
+| United Arab Emirates | UK*          | USA*       | Uruguay        |
-| Venezuela            | Vietnam      |            |                |
+| Venezuela            | Vietnam*     |            |                |
 
+
+## Notes
+* Indeed is the best scraper currently with no rate limiting.  
+* Glassdoor can only fetch 900 jobs from the endpoint we're using on a given search.  
+* LinkedIn is the most restrictive and usually rate limits on around the 10th page  
+* ZipRecruiter is okay but has a 5 second delay in between each page to avoid rate limiting.
 ## Frequently Asked Questions
 
 ---
 
 **Q: Encountering issues with your queries?**  
 **A:** Try reducing the number of `results_wanted` and/or broadening the filters. If problems
-persist, [submit an issue](https://github.com/cullenwatson/JobSpy/issues).
+persist, [submit an issue](https://github.com/Bunsly/JobSpy/issues).
 
 ---
 
 **Q: Received a response code 429?**  
-**A:** This indicates that you have been blocked by the job board site for sending too many requests. Currently, *
+**A:** This indicates that you have been blocked by the job board site for sending too many requests. All of the job board sites are aggressive with blocking. We recommend:
-*LinkedIn** is particularly aggressive with blocking. We recommend:
 
-- Waiting a few seconds between requests.
+- Waiting some time between scrapes (site-dependent).
 - Trying a VPN or proxy to change your IP address.
 
----
+---
-
-**Q: Experiencing a "Segmentation fault: 11" on macOS Catalina?**  
-**A:** This is due to `tls_client` dependency not supporting your architecture. Solutions and workarounds include:
-
-- Upgrade to a newer version of MacOS
-- Reach out to the maintainers of [tls_client](https://github.com/bogdanfinn/tls-client) for fixes
-
-
-  

examples/JobSpy_AllSites.py

@@ -0,0 +1,30 @@
+from jobspy import scrape_jobs
+import pandas as pd
+
+jobs: pd.DataFrame = scrape_jobs(
+    site_name=["indeed", "linkedin", "zip_recruiter", "glassdoor"],
+    search_term="software engineer",
+    location="Dallas, TX",
+    results_wanted=25,  # be wary the higher it is, the more likey you'll get blocked (rotating proxy can help tho)
+    country_indeed="USA",
+    # proxy="http://jobspy:5a4vpWtj8EeJ2hoYzk@ca.smartproxy.com:20001",
+)
+
+# formatting for pandas
+pd.set_option("display.max_columns", None)
+pd.set_option("display.max_rows", None)
+pd.set_option("display.width", None)
+pd.set_option("display.max_colwidth", 50)  # set to 0 to see full job url / desc
+
+# 1: output to console
+print(jobs)
+
+# 2: output to .csv
+jobs.to_csv("./jobs.csv", index=False)
+print("outputted to jobs.csv")
+
+# 3: output to .xlsx
+# jobs.to_xlsx('jobs.xlsx', index=False)
+
+# 4: display in Jupyter Notebook (1. pip install jupyter 2. jupyter notebook)
+# display(jobs)

examples/JobSpy_Demo.py

@@ -1,31 +0,0 @@
-from jobspy import scrape_jobs
-import pandas as pd
-
-jobs: pd.DataFrame = scrape_jobs(
-    site_name=["indeed", "linkedin", "zip_recruiter"],
-    search_term="software engineer",
-    location="Dallas, TX",
-    results_wanted=50,  # be wary the higher it is, the more likey you'll get blocked (rotating proxy should work tho)
-    country_indeed='USA',
-    offset=25  # start jobs from an offset (use if search failed and want to continue)
-    # proxy="http://jobspy:5a4vpWtj8EeJ2hoYzk@ca.smartproxy.com:20001",
-)
-
-# formatting for pandas
-pd.set_option('display.max_columns', None)
-pd.set_option('display.max_rows', None)
-pd.set_option('display.width', None)
-pd.set_option('display.max_colwidth', 50)  # set to 0 to see full job url / desc
-
-# 1: output to console
-print(jobs)
-
-# 2: output to .csv
-jobs.to_csv('./jobs.csv', index=False)
-print('outputted to jobs.csv')
-
-# 3: output to .xlsx
-# jobs.to_xlsx('jobs.xlsx', index=False)
-
-# 4: display in Jupyter Notebook (1. pip install jupyter 2. jupyter notebook)
-# display(jobs)

examples/JobSpy_LongScrape.py

@@ -0,0 +1,77 @@
+from jobspy import scrape_jobs
+import pandas as pd
+import os
+import time
+
+# creates csv a new filename if the jobs.csv already exists.
+csv_filename = "jobs.csv"
+counter = 1
+while os.path.exists(csv_filename):
+    csv_filename = f"jobs_{counter}.csv"
+    counter += 1
+
+# results wanted and offset
+results_wanted = 1000
+offset = 0
+
+all_jobs = []
+
+# max retries
+max_retries = 3
+
+# nuumber of results at each iteration
+results_in_each_iteration = 30
+
+while len(all_jobs) < results_wanted:
+    retry_count = 0
+    while retry_count < max_retries:
+        print("Doing from", offset, "to", offset + results_in_each_iteration, "jobs")
+        try:
+            jobs = scrape_jobs(
+                site_name=["indeed"],
+                search_term="software engineer",
+                # New York, NY
+                # Dallas, TX
+
+                # Los Angeles, CA
+                location="Los Angeles, CA",
+                results_wanted=min(results_in_each_iteration, results_wanted - len(all_jobs)),
+                country_indeed="USA",
+                offset=offset,
+                # proxy="http://jobspy:5a4vpWtj8EeJ2hoYzk@ca.smartproxy.com:20001",
+            )
+
+            # Add the scraped jobs to the list
+            all_jobs.extend(jobs.to_dict('records'))
+
+            # Increment the offset for the next page of results
+            offset += results_in_each_iteration
+
+            # Add a delay to avoid rate limiting (you can adjust the delay time as needed)
+            print(f"Scraped {len(all_jobs)} jobs")
+            print("Sleeping secs", 100 * (retry_count + 1))
+            time.sleep(100 * (retry_count + 1))  # Sleep for 2 seconds between requests
+
+            break  # Break out of the retry loop if successful
+        except Exception as e:
+            print(f"Error: {e}")
+            retry_count += 1
+            print("Sleeping secs before retry", 100 * (retry_count + 1))
+            time.sleep(100 * (retry_count + 1))
+            if retry_count >= max_retries:
+                print("Max retries reached. Exiting.")
+                break
+
+# DataFrame from the collected job data
+jobs_df = pd.DataFrame(all_jobs)
+
+# Formatting
+pd.set_option("display.max_columns", None)
+pd.set_option("display.max_rows", None)
+pd.set_option("display.width", None)
+pd.set_option("display.max_colwidth", 50)
+
+print(jobs_df)
+
+jobs_df.to_csv(csv_filename, index=False)
+print(f"Outputted to {csv_filename}")

pyproject.toml

@@ -1,9 +1,9 @@
 [tool.poetry]
 name = "python-jobspy"
-version = "1.1.9"
+version = "1.1.48"
-description = "Job scraper for LinkedIn, Indeed & ZipRecruiter"
+description = "Job scraper for LinkedIn, Indeed, Glassdoor & ZipRecruiter"
-authors = ["Zachary Hampton <zachary@zacharysproducts.com>", "Cullen Watson <cullen@cullen.ai>"]
+authors = ["Zachary Hampton <zachary@bunsly.com>", "Cullen Watson <cullen@bunsly.com>"]
-homepage = "https://github.com/cullenwatson/JobSpy"
+homepage = "https://github.com/Bunsly/JobSpy"
 readme = "README.md"
 
 packages = [
@@ -13,10 +13,12 @@ packages = [
 [tool.poetry.dependencies]
 python = "^3.10"
 requests = "^2.31.0"
-tls-client = "^0.2.1"
 beautifulsoup4 = "^4.12.2"
 pandas = "^2.1.0"
+NUMPY = "1.24.2"
 pydantic = "^2.3.0"
+tls-client = "^1.0.1"
+markdownify = "^0.11.6"
 
 
 [tool.poetry.group.dev.dependencies]

src/jobspy/__init__.py

@@ -1,48 +1,54 @@
 import pandas as pd
-import concurrent.futures
+from typing import Tuple
-from concurrent.futures import ThreadPoolExecutor
+from concurrent.futures import ThreadPoolExecutor, as_completed
-from typing import List, Tuple, Optional
 
 from .jobs import JobType, Location
+from .scrapers.utils import logger
 from .scrapers.indeed import IndeedScraper
 from .scrapers.ziprecruiter import ZipRecruiterScraper
+from .scrapers.glassdoor import GlassdoorScraper
 from .scrapers.linkedin import LinkedInScraper
 from .scrapers import ScraperInput, Site, JobResponse, Country
 from .scrapers.exceptions import (
     LinkedInException,
     IndeedException,
     ZipRecruiterException,
+    GlassdoorException,
 )
 
-SCRAPER_MAPPING = {
-    Site.LINKEDIN: LinkedInScraper,
-    Site.INDEED: IndeedScraper,
-    Site.ZIP_RECRUITER: ZipRecruiterScraper,
-}
-
-
-def _map_str_to_site(site_name: str) -> Site:
-    return Site[site_name.upper()]
-
 
 def scrape_jobs(
-        site_name: str | List[str] | Site | List[Site],
+    site_name: str | list[str] | Site | list[Site] | None = None,
-        search_term: str,
+    search_term: str | None = None,
-        location: str = "",
+    location: str | None = None,
-        distance: int = None,
+    distance: int | None = 50,
-        is_remote: bool = False,
+    is_remote: bool = False,
-        job_type: str = None,
+    job_type: str | None = None,
-        easy_apply: bool = False,  # linkedin
+    easy_apply: bool | None = None,
-        results_wanted: int = 15,
+    results_wanted: int = 15,
-        country_indeed: str = "usa",
+    country_indeed: str = "usa",
-        hyperlinks: bool = False,
+    hyperlinks: bool = False,
-        proxy: Optional[str] = None,
+    proxy: str | None = None,
-        offset: Optional[int] = 0
+    description_format: str = "markdown",
+    linkedin_fetch_description: bool | None = False,
+    linkedin_company_ids: list[int] | None = None,
+    offset: int | None = 0,
+    hours_old: int = None,
+    **kwargs,
 ) -> pd.DataFrame:
     """
     Simultaneously scrapes job data from multiple job sites.
     :return: results_wanted: pandas dataframe containing job data
     """
+    SCRAPER_MAPPING = {
+        Site.LINKEDIN: LinkedInScraper,
+        Site.INDEED: IndeedScraper,
+        Site.ZIP_RECRUITER: ZipRecruiterScraper,
+        Site.GLASSDOOR: GlassdoorScraper,
+    }
+
+    def map_str_to_site(site_name: str) -> Site:
+        return Site[site_name.upper()]
 
     def get_enum_from_value(value_str):
         for job_type in JobType:
@@ -52,18 +58,22 @@ def scrape_jobs(
 
     job_type = get_enum_from_value(job_type) if job_type else None
 
-    if type(site_name) == str:
+    def get_site_type():
-        site_type = [_map_str_to_site(site_name)]
+        site_types = list(Site)
-    else:  #: if type(site_name) == list
+        if isinstance(site_name, str):
-        site_type = [
+            site_types = [map_str_to_site(site_name)]
-            _map_str_to_site(site) if type(site) == str else site_name
+        elif isinstance(site_name, Site):
-            for site in site_name
+            site_types = [site_name]
-        ]
+        elif isinstance(site_name, list):
-
+            site_types = [
+                map_str_to_site(site) if isinstance(site, str) else site
+                for site in site_name
+            ]
+        return site_types
     country_enum = Country.from_string(country_indeed)
 
     scraper_input = ScraperInput(
-        site_type=site_type,
+        site_type=get_site_type(),
         country=country_enum,
         search_term=search_term,
         location=location,
@@ -71,46 +81,38 @@ def scrape_jobs(
         is_remote=is_remote,
         job_type=job_type,
         easy_apply=easy_apply,
+        description_format=description_format,
+        linkedin_fetch_description=linkedin_fetch_description,
         results_wanted=results_wanted,
-        offset=offset
+        linkedin_company_ids=linkedin_company_ids,
+        offset=offset,
+        hours_old=hours_old
     )
 
     def scrape_site(site: Site) -> Tuple[str, JobResponse]:
         scraper_class = SCRAPER_MAPPING[site]
         scraper = scraper_class(proxy=proxy)
-
+        scraped_data: JobResponse = scraper.scrape(scraper_input)
-        try:
+        site_name = 'ZipRecruiter' if site.value.capitalize() == 'Zip_recruiter' else site.value.capitalize()
-            scraped_data: JobResponse = scraper.scrape(scraper_input)
+        logger.info(f"{site_name} finished scraping")
-        except (LinkedInException, IndeedException, ZipRecruiterException) as lie:
-            raise lie
-        except Exception as e:
-            # unhandled exceptions
-            if site == Site.LINKEDIN:
-                raise LinkedInException()
-            if site == Site.INDEED:
-                raise IndeedException()
-            if site == Site.ZIP_RECRUITER:
-                raise ZipRecruiterException()
-            else:
-                raise e
         return site.value, scraped_data
 
     site_to_jobs_dict = {}
 
     def worker(site):
-        site_value, scraped_data = scrape_site(site)
+        site_val, scraped_info = scrape_site(site)
-        return site_value, scraped_data
+        return site_val, scraped_info
 
     with ThreadPoolExecutor() as executor:
         future_to_site = {
             executor.submit(worker, site): site for site in scraper_input.site_type
         }
 
-        for future in concurrent.futures.as_completed(future_to_site):
+        for future in as_completed(future_to_site):
             site_value, scraped_data = future.result()
             site_to_jobs_dict[site_value] = scraped_data
 
-    jobs_dfs: List[pd.DataFrame] = []
+    jobs_dfs: list[pd.DataFrame] = []
 
     for site, job_response in site_to_jobs_dict.items():
         for job in job_response.jobs:
@@ -120,13 +122,18 @@ def scrape_jobs(
             ] = f'<a href="{job_data["job_url"]}">{job_data["job_url"]}</a>'
             job_data["site"] = site
             job_data["company"] = job_data["company_name"]
-            if job_data["job_type"]:
+            job_data["job_type"] = (
-                # Take the first value from the job type tuple
+                ", ".join(job_type.value[0] for job_type in job_data["job_type"])
-                job_data["job_type"] = job_data["job_type"].value[0]
+                if job_data["job_type"]
-            else:
+                else None
-                job_data["job_type"] = None
+            )
-
+            job_data["emails"] = (
-            job_data["location"] = Location(**job_data["location"]).display_location()
+                ", ".join(job_data["emails"]) if job_data["emails"] else None
+            )
+            if job_data["location"]:
+                job_data["location"] = Location(
+                    **job_data["location"]
+                ).display_location()
 
             compensation_obj = job_data.get("compensation")
             if compensation_obj and isinstance(compensation_obj, dict):
@@ -148,25 +155,52 @@ def scrape_jobs(
             jobs_dfs.append(job_df)
 
     if jobs_dfs:
-        jobs_df = pd.concat(jobs_dfs, ignore_index=True)
+        # Step 1: Filter out all-NA columns from each DataFrame before concatenation
-        desired_order: List[str] = [
+        filtered_dfs = [df.dropna(axis=1, how='all') for df in jobs_dfs]
-            "job_url_hyper" if hyperlinks else "job_url",
+        
+        # Step 2: Concatenate the filtered DataFrames
+        jobs_df = pd.concat(filtered_dfs, ignore_index=True)
+        
+        # Desired column order
+        desired_order = [
             "site",
+            "job_url_hyper" if hyperlinks else "job_url",
+            "job_url_direct",
             "title",
             "company",
             "location",
             "job_type",
             "date_posted",
             "interval",
-            "benefits",
             "min_amount",
             "max_amount",
             "currency",
+            "is_remote",
             "emails",
             "description",
-        ]
-        jobs_formatted_df = jobs_df[desired_order]
-    else:
-        jobs_formatted_df = pd.DataFrame()
 
-    return jobs_formatted_df
+            "company_url",
+            "company_url_direct",
+            "company_addresses",
+            "company_industry",
+            "company_num_employees",
+            "company_revenue",
+            "company_description",
+            "logo_photo_url",
+            "banner_photo_url",
+            "ceo_name",
+            "ceo_photo_url",
+        ]
+        
+        # Step 3: Ensure all desired columns are present, adding missing ones as empty
+        for column in desired_order:
+            if column not in jobs_df.columns:
+                jobs_df[column] = None  # Add missing columns as empty
+        
+        # Reorder the DataFrame according to the desired order
+        jobs_df = jobs_df[desired_order]
+        
+        # Step 4: Sort the DataFrame as required
+        return jobs_df.sort_values(by=['site', 'date_posted'], ascending=[True, False])
+    else:
+        return pd.DataFrame()

src/jobspy/jobs/__init__.py

@@ -1,8 +1,7 @@
-from typing import Union, Optional
+from typing import Optional
 from datetime import date
 from enum import Enum
-
+from pydantic import BaseModel
-from pydantic import BaseModel, validator
 
 
 class JobType(Enum):
@@ -37,10 +36,16 @@ class JobType(Enum):
         "повназайнятість",
         "toànthờigian",
     )
-    PART_TIME = ("parttime", "teilzeit")
+    PART_TIME = ("parttime", "teilzeit", "částečnýúvazek", "deltid")
     CONTRACT = ("contract", "contractor")
     TEMPORARY = ("temporary",)
-    INTERNSHIP = ("internship", "prácticas", "ojt(onthejobtraining)", "praktikum")
+    INTERNSHIP = (
+        "internship",
+        "prácticas",
+        "ojt(onthejobtraining)",
+        "praktikum",
+        "praktik",
+    )
 
     PER_DIEM = ("perdiem",)
     NIGHTS = ("nights",)
@@ -50,40 +55,46 @@ class JobType(Enum):
 
 
 class Country(Enum):
-    ARGENTINA = ("argentina", "ar")
+    """
-    AUSTRALIA = ("australia", "au")
+    Gets the subdomain for Indeed and Glassdoor.
-    AUSTRIA = ("austria", "at")
+    The second item in the tuple is the subdomain (and API country code if there's a ':' separator) for Indeed
+    The third item in the tuple is the subdomain (and tld if there's a ':' separator) for Glassdoor
+    """
+
+    ARGENTINA = ("argentina", "ar", "com.ar")
+    AUSTRALIA = ("australia", "au", "com.au")
+    AUSTRIA = ("austria", "at", "at")
     BAHRAIN = ("bahrain", "bh")
-    BELGIUM = ("belgium", "be")
+    BELGIUM = ("belgium", "be", "fr:be")
-    BRAZIL = ("brazil", "br")
+    BRAZIL = ("brazil", "br", "com.br")
-    CANADA = ("canada", "ca")
+    CANADA = ("canada", "ca", "ca")
     CHILE = ("chile", "cl")
     CHINA = ("china", "cn")
     COLOMBIA = ("colombia", "co")
     COSTARICA = ("costa rica", "cr")
-    CZECHREPUBLIC = ("czech republic", "cz")
+    CZECHREPUBLIC = ("czech republic,czechia", "cz")
     DENMARK = ("denmark", "dk")
     ECUADOR = ("ecuador", "ec")
     EGYPT = ("egypt", "eg")
     FINLAND = ("finland", "fi")
-    FRANCE = ("france", "fr")
+    FRANCE = ("france", "fr", "fr")
-    GERMANY = ("germany", "de")
+    GERMANY = ("germany", "de", "de")
     GREECE = ("greece", "gr")
-    HONGKONG = ("hong kong", "hk")
+    HONGKONG = ("hong kong", "hk", "com.hk")
     HUNGARY = ("hungary", "hu")
-    INDIA = ("india", "in")
+    INDIA = ("india", "in", "co.in")
     INDONESIA = ("indonesia", "id")
-    IRELAND = ("ireland", "ie")
+    IRELAND = ("ireland", "ie", "ie")
     ISRAEL = ("israel", "il")
-    ITALY = ("italy", "it")
+    ITALY = ("italy", "it", "it")
     JAPAN = ("japan", "jp")
     KUWAIT = ("kuwait", "kw")
     LUXEMBOURG = ("luxembourg", "lu")
     MALAYSIA = ("malaysia", "malaysia")
-    MEXICO = ("mexico", "mx")
+    MEXICO = ("mexico", "mx", "com.mx")
     MOROCCO = ("morocco", "ma")
-    NETHERLANDS = ("netherlands", "nl")
+    NETHERLANDS = ("netherlands", "nl", "nl")
-    NEWZEALAND = ("new zealand", "nz")
+    NEWZEALAND = ("new zealand", "nz", "co.nz")
     NIGERIA = ("nigeria", "ng")
     NORWAY = ("norway", "no")
     OMAN = ("oman", "om")
@@ -96,54 +107,66 @@ class Country(Enum):
     QATAR = ("qatar", "qa")
     ROMANIA = ("romania", "ro")
     SAUDIARABIA = ("saudi arabia", "sa")
-    SINGAPORE = ("singapore", "sg")
+    SINGAPORE = ("singapore", "sg", "sg")
     SOUTHAFRICA = ("south africa", "za")
     SOUTHKOREA = ("south korea", "kr")
-    SPAIN = ("spain", "es")
+    SPAIN = ("spain", "es", "es")
     SWEDEN = ("sweden", "se")
-    SWITZERLAND = ("switzerland", "ch")
+    SWITZERLAND = ("switzerland", "ch", "de:ch")
     TAIWAN = ("taiwan", "tw")
     THAILAND = ("thailand", "th")
     TURKEY = ("turkey", "tr")
     UKRAINE = ("ukraine", "ua")
     UNITEDARABEMIRATES = ("united arab emirates", "ae")
-    UK = ("uk", "uk")
+    UK = ("uk,united kingdom", "uk:gb", "co.uk")
-    USA = ("usa", "www")
+    USA = ("usa,us,united states", "www:us", "com")
     URUGUAY = ("uruguay", "uy")
     VENEZUELA = ("venezuela", "ve")
-    VIETNAM = ("vietnam", "vn")
+    VIETNAM = ("vietnam", "vn", "com")
 
     # internal for ziprecruiter
     US_CANADA = ("usa/ca", "www")
 
-    # internal for linkeind
+    # internal for linkedin
     WORLDWIDE = ("worldwide", "www")
 
-    def __new__(cls, country, domain):
+    @property
-        obj = object.__new__(cls)
+    def indeed_domain_value(self):
-        obj._value_ = country
+        subdomain, _, api_country_code = self.value[1].partition(":")
-        obj.domain = domain
+        if subdomain and api_country_code:
-        return obj
+            return subdomain, api_country_code.upper()
+        return self.value[1], self.value[1].upper()
 
     @property
-    def domain_value(self):
+    def glassdoor_domain_value(self):
-        return self.domain
+        if len(self.value) == 3:
+            subdomain, _, domain = self.value[2].partition(":")
+            if subdomain and domain:
+                return f"{subdomain}.glassdoor.{domain}"
+            else:
+                return f"www.glassdoor.{self.value[2]}"
+        else:
+            raise Exception(f"Glassdoor is not available for {self.name}")
+
+    def get_glassdoor_url(self):
+        return f"https://{self.glassdoor_domain_value}/"
 
     @classmethod
     def from_string(cls, country_str: str):
         """Convert a string to the corresponding Country enum."""
         country_str = country_str.strip().lower()
         for country in cls:
-            if country.value == country_str:
+            country_names = country.value[0].split(',')
+            if country_str in country_names:
                 return country
         valid_countries = [country.value for country in cls]
         raise ValueError(
-            f"Invalid country string: '{country_str}'. Valid countries (only include this param for Indeed) are: {', '.join(valid_countries)}"
+            f"Invalid country string: '{country_str}'. Valid countries are: {', '.join([country[0] for country in valid_countries])}"
         )
 
 
 class Location(BaseModel):
-    country: Country = None
+    country: Country | str | None = None
     city: Optional[str] = None
     state: Optional[str] = None
 
@@ -153,11 +176,16 @@ class Location(BaseModel):
             location_parts.append(self.city)
         if self.state:
             location_parts.append(self.state)
-        if self.country and self.country not in (Country.US_CANADA, Country.WORLDWIDE):
+        if isinstance(self.country, str):
-            if self.country.value in ("usa", "uk"):
+            location_parts.append(self.country)
-                location_parts.append(self.country.value.upper())
+        elif self.country and self.country not in (Country.US_CANADA, Country.WORLDWIDE):
+            country_name = self.country.value[0]
+            if "," in country_name:
+                country_name = country_name.split(",")[0]
+            if country_name in ("usa", "uk"):
+                location_parts.append(country_name.upper())
             else:
-                location_parts.append(self.country.value.title())
+                location_parts.append(country_name.title())
         return ", ".join(location_parts)
 
 
@@ -168,26 +196,57 @@ class CompensationInterval(Enum):
     DAILY = "daily"
     HOURLY = "hourly"
 
+    @classmethod
+    def get_interval(cls, pay_period):
+        interval_mapping = {
+            "YEAR": cls.YEARLY,
+            "HOUR": cls.HOURLY,
+        }
+        if pay_period in interval_mapping:
+            return interval_mapping[pay_period].value
+        else:
+            return cls[pay_period].value if pay_period in cls.__members__ else None
+
 
 class Compensation(BaseModel):
     interval: Optional[CompensationInterval] = None
-    min_amount: int = None
+    min_amount: float | None = None
-    max_amount: int = None
+    max_amount: float | None = None
     currency: Optional[str] = "USD"
 
 
+class DescriptionFormat(Enum):
+    MARKDOWN = "markdown"
+    HTML = "html"
+
+
 class JobPost(BaseModel):
     title: str
-    company_name: str
+    company_name: str | None
     job_url: str
+    job_url_direct: str | None = None
     location: Optional[Location]
 
-    description: Optional[str] = None
+    description: str | None = None
-    job_type: Optional[JobType] = None
+    company_url: str | None = None
-    compensation: Optional[Compensation] = None
+    company_url_direct: str | None = None
-    date_posted: Optional[date] = None
+
-    benefits: Optional[str] = None
+    job_type: list[JobType] | None = None
-    emails: Optional[list[str]] = None
+    compensation: Compensation | None = None
+    date_posted: date | None = None
+    emails: list[str] | None = None
+    is_remote: bool | None = None
+
+    # indeed specific
+    company_addresses: str | None = None
+    company_industry: str | None = None
+    company_num_employees: str | None = None
+    company_revenue: str | None = None
+    company_description: str | None = None
+    ceo_name: str | None = None
+    ceo_photo_url: str | None = None
+    logo_photo_url: str | None = None
+    banner_photo_url: str | None = None
 
 
 class JobResponse(BaseModel):

src/jobspy/scrapers/__init__.py

@@ -1,32 +1,42 @@
-from ..jobs import Enum, BaseModel, JobType, JobResponse, Country
+from ..jobs import (
-from typing import List, Optional, Any
+    Enum,
+    BaseModel,
+    JobType,
+    JobResponse,
+    Country,
+    DescriptionFormat
+)
 
 
 class Site(Enum):
     LINKEDIN = "linkedin"
     INDEED = "indeed"
     ZIP_RECRUITER = "zip_recruiter"
+    GLASSDOOR = "glassdoor"
 
 
 class ScraperInput(BaseModel):
-    site_type: List[Site]
+    site_type: list[Site]
-    search_term: str
+    search_term: str | None = None
 
-    location: str = None
+    location: str | None = None
-    country: Optional[Country] = Country.USA
+    country: Country | None = Country.USA
-    distance: Optional[int] = None
+    distance: int | None = None
     is_remote: bool = False
-    job_type: Optional[JobType] = None
+    job_type: JobType | None = None
-    easy_apply: bool = None  # linkedin
+    easy_apply: bool | None = None
     offset: int = 0
+    linkedin_fetch_description: bool = False
+    linkedin_company_ids: list[int] | None = None
+    description_format: DescriptionFormat | None = DescriptionFormat.MARKDOWN
 
     results_wanted: int = 15
+    hours_old: int | None = None
 
 
 class Scraper:
-    def __init__(self, site: Site, proxy: Optional[List[str]] = None):
+    def __init__(self, site: Site, proxy: list[str] | None = None):
         self.site = site
         self.proxy = (lambda p: {"http": p, "https": p} if p else None)(proxy)
 
-    def scrape(self, scraper_input: ScraperInput) -> JobResponse:
+    def scrape(self, scraper_input: ScraperInput) -> JobResponse: ...
-        ...

src/jobspy/scrapers/exceptions.py

@@ -7,12 +7,20 @@ This module contains the set of Scrapers' exceptions.
 
 
 class LinkedInException(Exception):
-    """Failed to scrape LinkedIn"""
+    def __init__(self, message=None):
+        super().__init__(message or "An error occurred with LinkedIn")
 
 
 class IndeedException(Exception):
-    """Failed to scrape Indeed"""
+    def __init__(self, message=None):
+        super().__init__(message or "An error occurred with Indeed")
 
 
 class ZipRecruiterException(Exception):
-    """Failed to scrape ZipRecruiter"""
+    def __init__(self, message=None):
+        super().__init__(message or "An error occurred with ZipRecruiter")
+
+
+class GlassdoorException(Exception):
+    def __init__(self, message=None):
+        super().__init__(message or "An error occurred with Glassdoor")

src/jobspy/scrapers/glassdoor/__init__.py

@@ -0,0 +1,515 @@
+"""
+jobspy.scrapers.glassdoor
+~~~~~~~~~~~~~~~~~~~
+
+This module contains routines to scrape Glassdoor.
+"""
+import json
+import re
+
+import requests
+from typing import Optional
+from datetime import datetime, timedelta
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from ..utils import extract_emails_from_text
+
+from .. import Scraper, ScraperInput, Site
+from ..exceptions import GlassdoorException
+from ..utils import (
+    create_session,
+    markdown_converter,
+    logger
+)
+from ...jobs import (
+    JobPost,
+    Compensation,
+    CompensationInterval,
+    Location,
+    JobResponse,
+    JobType,
+    DescriptionFormat
+)
+
+
+class GlassdoorScraper(Scraper):
+    def __init__(self, proxy: Optional[str] = None):
+        """
+        Initializes GlassdoorScraper with the Glassdoor job search url
+        """
+        site = Site(Site.GLASSDOOR)
+        super().__init__(site, proxy=proxy)
+
+        self.base_url = None
+        self.country = None
+        self.session = None
+        self.scraper_input = None
+        self.jobs_per_page = 30
+        self.max_pages = 30
+        self.seen_urls = set()
+
+    def scrape(self, scraper_input: ScraperInput) -> JobResponse:
+        """
+        Scrapes Glassdoor for jobs with scraper_input criteria.
+        :param scraper_input: Information about job search criteria.
+        :return: JobResponse containing a list of jobs.
+        """
+        self.scraper_input = scraper_input
+        self.scraper_input.results_wanted = min(900, scraper_input.results_wanted)
+        self.base_url = self.scraper_input.country.get_glassdoor_url()
+
+        self.session = create_session(self.proxy, is_tls=True, has_retry=True)
+        token = self._get_csrf_token()
+        self.headers['gd-csrf-token'] = token if token else self.fallback_token
+
+        location_id, location_type = self._get_location(
+            scraper_input.location, scraper_input.is_remote
+        )
+        if location_type is None:
+            logger.error('Glassdoor: location not parsed')
+            return JobResponse(jobs=[])
+        all_jobs: list[JobPost] = []
+        cursor = None
+
+        for page in range(
+            1 + (scraper_input.offset // self.jobs_per_page),
+            min(
+                (scraper_input.results_wanted // self.jobs_per_page) + 2,
+                self.max_pages + 1,
+            ),
+        ):
+            logger.info(f'Glassdoor search page: {page}')
+            try:
+                jobs, cursor = self._fetch_jobs_page(
+                    scraper_input, location_id, location_type, page, cursor
+                )
+                all_jobs.extend(jobs)
+                if not jobs or len(all_jobs) >= scraper_input.results_wanted:
+                    all_jobs = all_jobs[: scraper_input.results_wanted]
+                    break
+            except Exception as e:
+                logger.error(f'Glassdoor: {str(e)}')
+                break
+        return JobResponse(jobs=all_jobs)
+
+    def _fetch_jobs_page(
+        self,
+        scraper_input: ScraperInput,
+        location_id: int,
+        location_type: str,
+        page_num: int,
+        cursor: str | None,
+    ) -> (list[JobPost], str | None):
+        """
+        Scrapes a page of Glassdoor for jobs with scraper_input criteria
+        """
+        jobs = []
+        self.scraper_input = scraper_input
+        try:
+            payload = self._add_payload(
+                location_id, location_type, page_num, cursor
+            )
+            response = self.session.post(
+                f"{self.base_url}/graph", headers=self.headers, timeout_seconds=15, data=payload
+            )
+            if response.status_code != 200:
+                raise GlassdoorException(f"bad response status code: {response.status_code}")
+            res_json = response.json()[0]
+            if "errors" in res_json:
+                raise ValueError("Error encountered in API response")
+        except (requests.exceptions.ReadTimeout, GlassdoorException, ValueError, Exception) as e:
+            logger.error(f'Glassdoor: {str(e)}')
+            return jobs, None
+
+        jobs_data = res_json["data"]["jobListings"]["jobListings"]
+
+        with ThreadPoolExecutor(max_workers=self.jobs_per_page) as executor:
+            future_to_job_data = {executor.submit(self._process_job, job): job for job in jobs_data}
+            for future in as_completed(future_to_job_data):
+                try:
+                    job_post = future.result()
+                    if job_post:
+                        jobs.append(job_post)
+                except Exception as exc:
+                    raise GlassdoorException(f'Glassdoor generated an exception: {exc}')
+
+        return jobs, self.get_cursor_for_page(
+            res_json["data"]["jobListings"]["paginationCursors"], page_num + 1
+        )
+
+    def _get_csrf_token(self):
+        """
+        Fetches csrf token needed for API by visiting a generic page
+        """
+        res = self.session.get(f'{self.base_url}/Job/computer-science-jobs.htm', headers=self.headers)
+        pattern = r'"token":\s*"([^"]+)"'
+        matches = re.findall(pattern, res.text)
+        token = None
+        if matches:
+            token = matches[0]
+        return token
+
+    def _process_job(self, job_data):
+        """
+        Processes a single job and fetches its description.
+        """
+        job_id = job_data["jobview"]["job"]["listingId"]
+        job_url = f'{self.base_url}job-listing/j?jl={job_id}'
+        if job_url in self.seen_urls:
+            return None
+        self.seen_urls.add(job_url)
+        job = job_data["jobview"]
+        title = job["job"]["jobTitleText"]
+        company_name = job["header"]["employerNameFromSearch"]
+        company_id = job_data['jobview']['header']['employer']['id']
+        location_name = job["header"].get("locationName", "")
+        location_type = job["header"].get("locationType", "")
+        age_in_days = job["header"].get("ageInDays")
+        is_remote, location = False, None
+        date_posted = (datetime.now() - timedelta(days=age_in_days)).date() if age_in_days is not None else None
+
+        if location_type == "S":
+            is_remote = True
+        else:
+            location = self.parse_location(location_name)
+
+        compensation = self.parse_compensation(job["header"])
+        try:
+            description = self._fetch_job_description(job_id)
+        except:
+            description = None
+        return JobPost(
+            title=title,
+            company_url=f"{self.base_url}Overview/W-EI_IE{company_id}.htm" if company_id else None,
+            company_name=company_name,
+            date_posted=date_posted,
+            job_url=job_url,
+            location=location,
+            compensation=compensation,
+            is_remote=is_remote,
+            description=description,
+            emails=extract_emails_from_text(description) if description else None,
+        )
+
+    def _fetch_job_description(self, job_id):
+        """
+        Fetches the job description for a single job ID.
+        """
+        url = f"{self.base_url}/graph"
+        body = [
+            {
+                "operationName": "JobDetailQuery",
+                "variables": {
+                    "jl": job_id,
+                    "queryString": "q",
+                    "pageTypeEnum": "SERP"
+                },
+                "query": """
+                query JobDetailQuery($jl: Long!, $queryString: String, $pageTypeEnum: PageTypeEnum) {
+                    jobview: jobView(
+                        listingId: $jl
+                        contextHolder: {queryString: $queryString, pageTypeEnum: $pageTypeEnum}
+                    ) {
+                        job {
+                            description
+                            __typename
+                        }
+                        __typename
+                    }
+                }
+                """
+            }
+        ]
+        res = requests.post(url, json=body, headers=self.headers)
+        if res.status_code != 200:
+            return None
+        data = res.json()[0]
+        desc = data['data']['jobview']['job']['description']
+        return markdown_converter(desc) if self.scraper_input.description_format == DescriptionFormat.MARKDOWN else desc
+
+    def _get_location(self, location: str, is_remote: bool) -> (int, str):
+        if not location or is_remote:
+            return "11047", "STATE"  # remote options
+        url = f"{self.base_url}/findPopularLocationAjax.htm?maxLocationsToReturn=10&term={location}"
+        session = create_session(self.proxy, has_retry=True)
+        res = self.session.get(url, headers=self.headers)
+        if res.status_code != 200:
+            if res.status_code == 429:
+                logger.error(f'429 Response - Blocked by Glassdoor for too many requests')
+                return None, None
+            else:
+                logger.error(f'Glassdoor response status code {res.status_code}')
+                return None, None
+        items = res.json()
+
+        if not items:
+            raise ValueError(f"Location '{location}' not found on Glassdoor")
+        location_type = items[0]["locationType"]
+        if location_type == "C":
+            location_type = "CITY"
+        elif location_type == "S":
+            location_type = "STATE"
+        elif location_type == 'N':
+            location_type = "COUNTRY"
+        return int(items[0]["locationId"]), location_type
+
+    def _add_payload(
+        self,
+        location_id: int,
+        location_type: str,
+        page_num: int,
+        cursor: str | None = None,
+    ) -> str:
+        fromage = max(self.scraper_input.hours_old // 24, 1) if self.scraper_input.hours_old else None
+        filter_params = []
+        if self.scraper_input.easy_apply:
+            filter_params.append({"filterKey": "applicationType", "values": "1"})
+        if fromage:
+            filter_params.append({"filterKey": "fromAge", "values": str(fromage)})
+        payload = {
+            "operationName": "JobSearchResultsQuery",
+            "variables": {
+                "excludeJobListingIds": [],
+                "filterParams": filter_params,
+                "keyword": self.scraper_input.search_term,
+                "numJobsToShow": 30,
+                "locationType": location_type,
+                "locationId": int(location_id),
+                "parameterUrlInput": f"IL.0,12_I{location_type}{location_id}",
+                "pageNumber": page_num,
+                "pageCursor": cursor,
+                "fromage": fromage,
+                "sort": "date"
+            },
+            "query": self.query_template
+        }
+        if self.scraper_input.job_type:
+            payload["variables"]["filterParams"].append(
+                {"filterKey": "jobType", "values": self.scraper_input.job_type.value[0]}
+            )
+        return json.dumps([payload])
+
+    @staticmethod
+    def parse_compensation(data: dict) -> Optional[Compensation]:
+        pay_period = data.get("payPeriod")
+        adjusted_pay = data.get("payPeriodAdjustedPay")
+        currency = data.get("payCurrency", "USD")
+        if not pay_period or not adjusted_pay:
+            return None
+
+        interval = None
+        if pay_period == "ANNUAL":
+            interval = CompensationInterval.YEARLY
+        elif pay_period:
+            interval = CompensationInterval.get_interval(pay_period)
+        min_amount = int(adjusted_pay.get("p10") // 1)
+        max_amount = int(adjusted_pay.get("p90") // 1)
+        return Compensation(
+            interval=interval,
+            min_amount=min_amount,
+            max_amount=max_amount,
+            currency=currency,
+        )
+
+    @staticmethod
+    def get_job_type_enum(job_type_str: str) -> list[JobType] | None:
+        for job_type in JobType:
+            if job_type_str in job_type.value:
+                return [job_type]
+
+    @staticmethod
+    def parse_location(location_name: str) -> Location | None:
+        if not location_name or location_name == "Remote":
+            return
+        city, _, state = location_name.partition(", ")
+        return Location(city=city, state=state)
+
+    @staticmethod
+    def get_cursor_for_page(pagination_cursors, page_num):
+        for cursor_data in pagination_cursors:
+            if cursor_data["pageNumber"] == page_num:
+                return cursor_data["cursor"]
+
+    fallback_token = "Ft6oHEWlRZrxDww95Cpazw:0pGUrkb2y3TyOpAIqF2vbPmUXoXVkD3oEGDVkvfeCerceQ5-n8mBg3BovySUIjmCPHCaW0H2nQVdqzbtsYqf4Q:wcqRqeegRUa9MVLJGyujVXB7vWFPjdaS1CtrrzJq-ok"
+    headers = {
+        "authority": "www.glassdoor.com",
+        "accept": "*/*",
+        "accept-language": "en-US,en;q=0.9",
+        "apollographql-client-name": "job-search-next",
+        "apollographql-client-version": "4.65.5",
+        "content-type": "application/json",
+        "origin": "https://www.glassdoor.com",
+        "referer": "https://www.glassdoor.com/",
+        "sec-ch-ua": '"Chromium";v="118", "Google Chrome";v="118", "Not=A?Brand";v="99"',
+        "sec-ch-ua-mobile": "?0",
+        "sec-ch-ua-platform": '"macOS"',
+        "sec-fetch-dest": "empty",
+        "sec-fetch-mode": "cors",
+        "sec-fetch-site": "same-origin",
+        "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
+    }
+    query_template = """
+            query JobSearchResultsQuery(
+                $excludeJobListingIds: [Long!], 
+                $keyword: String, 
+                $locationId: Int, 
+                $locationType: LocationTypeEnum, 
+                $numJobsToShow: Int!, 
+                $pageCursor: String, 
+                $pageNumber: Int, 
+                $filterParams: [FilterParams], 
+                $originalPageUrl: String, 
+                $seoFriendlyUrlInput: String, 
+                $parameterUrlInput: String, 
+                $seoUrl: Boolean
+            ) {
+                jobListings(
+                    contextHolder: {
+                        searchParams: {
+                            excludeJobListingIds: $excludeJobListingIds, 
+                            keyword: $keyword, 
+                            locationId: $locationId, 
+                            locationType: $locationType, 
+                            numPerPage: $numJobsToShow, 
+                            pageCursor: $pageCursor, 
+                            pageNumber: $pageNumber, 
+                            filterParams: $filterParams, 
+                            originalPageUrl: $originalPageUrl, 
+                            seoFriendlyUrlInput: $seoFriendlyUrlInput, 
+                            parameterUrlInput: $parameterUrlInput, 
+                            seoUrl: $seoUrl, 
+                            searchType: SR
+                        }
+                    }
+                ) {
+                    companyFilterOptions {
+                        id
+                        shortName
+                        __typename
+                    }
+                    filterOptions
+                    indeedCtk
+                    jobListings {
+                        ...JobView
+                        __typename
+                    }
+                    jobListingSeoLinks {
+                        linkItems {
+                            position
+                            url
+                            __typename
+                        }
+                        __typename
+                    }
+                    jobSearchTrackingKey
+                    jobsPageSeoData {
+                        pageMetaDescription
+                        pageTitle
+                        __typename
+                    }
+                    paginationCursors {
+                        cursor
+                        pageNumber
+                        __typename
+                    }
+                    indexablePageForSeo
+                    searchResultsMetadata {
+                        searchCriteria {
+                            implicitLocation {
+                                id
+                                localizedDisplayName
+                                type
+                                __typename
+                            }
+                            keyword
+                            location {
+                                id
+                                shortName
+                                localizedShortName
+                                localizedDisplayName
+                                type
+                                __typename
+                            }
+                            __typename
+                        }
+                        helpCenterDomain
+                        helpCenterLocale
+                        jobSerpJobOutlook {
+                            occupation
+                            paragraph
+                            __typename
+                        }
+                        showMachineReadableJobs
+                        __typename
+                    }
+                    totalJobsCount
+                    __typename
+                }
+            }
+
+            fragment JobView on JobListingSearchResult {
+                jobview {
+                    header {
+                        adOrderId
+                        advertiserType
+                        adOrderSponsorshipLevel
+                        ageInDays
+                        divisionEmployerName
+                        easyApply
+                        employer {
+                            id
+                            name
+                            shortName
+                            __typename
+                        }
+                        employerNameFromSearch
+                        goc
+                        gocConfidence
+                        gocId
+                        jobCountryId
+                        jobLink
+                        jobResultTrackingKey
+                        jobTitleText
+                        locationName
+                        locationType
+                        locId
+                        needsCommission
+                        payCurrency
+                        payPeriod
+                        payPeriodAdjustedPay {
+                            p10
+                            p50
+                            p90
+                            __typename
+                        }
+                        rating
+                        salarySource
+                        savedJobId
+                        sponsored
+                        __typename
+                    }
+                    job {
+                        description
+                        importConfigId
+                        jobTitleId
+                        jobTitleText
+                        listingId
+                        __typename
+                    }
+                    jobListingAdminDetails {
+                        cpcVal
+                        importConfigId
+                        jobListingId
+                        jobSourceId
+                        userEligibleForAdminJobDetails
+                        __typename
+                    }
+                    overview {
+                        shortName
+                        squareLogoUrl
+                        __typename
+                    }
+                    __typename
+                }
+                __typename
+            }
+    """

src/jobspy/scrapers/indeed/__init__.py

@@ -4,20 +4,19 @@ jobspy.scrapers.indeed
 
 This module contains routines to scrape Indeed.
 """
-import re
 import math
-import io
-import json
-from datetime import datetime
-from typing import Optional
-
-import tls_client
-import urllib.parse
-from bs4 import BeautifulSoup
-from bs4.element import Tag
 from concurrent.futures import ThreadPoolExecutor, Future
+from datetime import datetime
 
-from ..exceptions import IndeedException
+import requests
+
+from .. import Scraper, ScraperInput, Site
+from ..utils import (
+    extract_emails_from_text,
+    get_enum_from_job_type,
+    markdown_converter,
+    logger
+)
 from ...jobs import (
     JobPost,
     Compensation,
@@ -25,303 +24,351 @@ from ...jobs import (
     Location,
     JobResponse,
     JobType,
+    DescriptionFormat
 )
-from .. import Scraper, ScraperInput, Site
-from ...utils import extract_emails_from_text
 
 
 class IndeedScraper(Scraper):
-    def __init__(self, proxy: Optional[str] = None):
+    def __init__(self, proxy: str | None = None):
         """
-        Initializes IndeedScraper with the Indeed job search url
+        Initializes IndeedScraper with the Indeed API url
         """
-        self.url = None
+        self.scraper_input = None
-        self.country = None
+        self.jobs_per_page = 100
+        self.num_workers = 10
+        self.seen_urls = set()
+        self.headers = None
+        self.api_country_code = None
+        self.base_url = None
+        self.api_url = "https://apis.indeed.com/graphql"
         site = Site(Site.INDEED)
         super().__init__(site, proxy=proxy)
 
-        self.jobs_per_page = 15
-        self.seen_urls = set()
-
-    def scrape_page(
-            self, scraper_input: ScraperInput, page: int, session: tls_client.Session
-    ) -> tuple[list[JobPost], int]:
-        """
-        Scrapes a page of Indeed for jobs with scraper_input criteria
-        :param scraper_input:
-        :param page:
-        :param session:
-        :return: jobs found on page, total number of jobs found for search
-        """
-        self.country = scraper_input.country
-        domain = self.country.domain_value
-        self.url = f"https://{domain}.indeed.com"
-
-        job_list: list[JobPost] = []
-
-        params = {
-            "q": scraper_input.search_term,
-            "l": scraper_input.location,
-            "filter": 0,
-            "start": scraper_input.offset + page * 10,
-        }
-        if scraper_input.distance:
-            params["radius"] = scraper_input.distance
-
-        sc_values = []
-        if scraper_input.is_remote:
-            sc_values.append("attr(DSQF7)")
-        if scraper_input.job_type:
-            sc_values.append("jt({})".format(scraper_input.job_type.value))
-
-        if sc_values:
-            params["sc"] = "0kf:" + "".join(sc_values) + ";"
-        try:
-            response = session.get(
-                f"{self.url}/jobs",
-                params=params,
-                allow_redirects=True,
-                proxy=self.proxy,
-                timeout_seconds=10,
-            )
-            if response.status_code not in range(200, 400):
-                raise IndeedException(
-                    f"bad response with status code: {response.status_code}"
-                )
-        except Exception as e:
-            if "Proxy responded with" in str(e):
-                raise IndeedException("bad proxy")
-            raise IndeedException(str(e))
-
-        soup = BeautifulSoup(response.content, "html.parser")
-        if "did not match any jobs" in response.text:
-            raise IndeedException("Parsing exception: Search did not match any jobs")
-
-        jobs = IndeedScraper.parse_jobs(
-            soup
-        )  #: can raise exception, handled by main scrape function
-        total_num_jobs = IndeedScraper.total_jobs(soup)
-
-        if (
-                not jobs.get("metaData", {})
-                        .get("mosaicProviderJobCardsModel", {})
-                        .get("results")
-        ):
-            raise IndeedException("No jobs found.")
-
-        def process_job(job) -> Optional[JobPost]:
-            job_url = f'{self.url}/jobs/viewjob?jk={job["jobkey"]}'
-            job_url_client = f'{self.url}/viewjob?jk={job["jobkey"]}'
-            if job_url in self.seen_urls:
-                return None
-
-            extracted_salary = job.get("extractedSalary")
-            compensation = None
-            if extracted_salary:
-                salary_snippet = job.get("salarySnippet")
-                currency = salary_snippet.get("currency") if salary_snippet else None
-                interval = (extracted_salary.get("type"),)
-                if isinstance(interval, tuple):
-                    interval = interval[0]
-
-                interval = interval.upper()
-                if interval in CompensationInterval.__members__:
-                    compensation = Compensation(
-                        interval=CompensationInterval[interval],
-                        min_amount=int(extracted_salary.get("max")),
-                        max_amount=int(extracted_salary.get("min")),
-                        currency=currency,
-                    )
-
-            job_type = IndeedScraper.get_job_type(job)
-            timestamp_seconds = job["pubDate"] / 1000
-            date_posted = datetime.fromtimestamp(timestamp_seconds)
-            date_posted = date_posted.strftime("%Y-%m-%d")
-
-            description = self.get_description(job_url, session)
-            emails = extract_emails_from_text(description)
-            with io.StringIO(job["snippet"]) as f:
-                soup_io = BeautifulSoup(f, "html.parser")
-                li_elements = soup_io.find_all("li")
-                if description is None and li_elements:
-                    description = " ".join(li.text for li in li_elements)
-
-            job_post = JobPost(
-                title=job["normTitle"],
-                description=description,
-                company_name=job["company"],
-                location=Location(
-                    city=job.get("jobLocationCity"),
-                    state=job.get("jobLocationState"),
-                    country=self.country,
-                ),
-                emails=extract_emails_from_text(description),
-                job_type=job_type,
-                compensation=compensation,
-                date_posted=date_posted,
-                job_url=job_url_client,
-            )
-            return job_post
-
-        with ThreadPoolExecutor(max_workers=1) as executor:
-            job_results: list[Future] = [
-                executor.submit(process_job, job)
-                for job in jobs["metaData"]["mosaicProviderJobCardsModel"]["results"]
-            ]
-
-        job_list = [result.result() for result in job_results if result.result()]
-
-        return job_list, total_num_jobs
-
     def scrape(self, scraper_input: ScraperInput) -> JobResponse:
         """
         Scrapes Indeed for jobs with scraper_input criteria
         :param scraper_input:
         :return: job_response
         """
-        session = tls_client.Session(
+        self.scraper_input = scraper_input
-            client_identifier="chrome112", random_tls_extension_order=True
+        domain, self.api_country_code = self.scraper_input.country.indeed_domain_value
-        )
+        self.base_url = f"https://{domain}.indeed.com"
+        self.headers = self.api_headers.copy()
+        self.headers['indeed-co'] = self.scraper_input.country.indeed_domain_value
+        job_list = []
+        page = 1
 
-        pages_to_process = (
+        cursor = None
-                math.ceil(scraper_input.results_wanted / self.jobs_per_page) - 1
+        offset_pages = math.ceil(self.scraper_input.offset / 100)
-        )
+        for _ in range(offset_pages):
+            logger.info(f'Indeed skipping search page: {page}')
+            __, cursor = self._scrape_page(cursor)
+            if not __:
+                logger.info(f'Indeed found no jobs on page: {page}')
+                break
 
-        #: get first page to initialize session
+        while len(self.seen_urls) < scraper_input.results_wanted:
-        job_list, total_results = self.scrape_page(scraper_input, 0, session)
+            logger.info(f'Indeed search page: {page}')
+            jobs, cursor = self._scrape_page(cursor)
+            if not jobs:
+                logger.info(f'Indeed found no jobs on page: {page}')
+                break
+            job_list += jobs
+            page += 1
+        return JobResponse(jobs=job_list[:scraper_input.results_wanted])
 
-        with ThreadPoolExecutor(max_workers=1) as executor:
+    def _scrape_page(self, cursor: str | None) -> (list[JobPost], str | None):
-            futures: list[Future] = [
-                executor.submit(self.scrape_page, scraper_input, page, session)
-                for page in range(1, pages_to_process + 1)
-            ]
-
-            for future in futures:
-                jobs, _ = future.result()
-
-                job_list += jobs
-
-        if len(job_list) > scraper_input.results_wanted:
-            job_list = job_list[: scraper_input.results_wanted]
-
-        job_response = JobResponse(
-            jobs=job_list,
-            total_results=total_results,
-        )
-        return job_response
-
-    def get_description(self, job_page_url: str, session: tls_client.Session) -> Optional[str]:
         """
-        Retrieves job description by going to the job page url
+        Scrapes a page of Indeed for jobs with scraper_input criteria
-        :param job_page_url:
+        :param cursor:
-        :param session:
+        :return: jobs found on page, next page cursor
-        :return: description
         """
-        parsed_url = urllib.parse.urlparse(job_page_url)
+        jobs = []
-        params = urllib.parse.parse_qs(parsed_url.query)
+        new_cursor = None
-        jk_value = params.get("jk", [None])[0]
+        filters = self._build_filters()
-        formatted_url = f"{self.url}/viewjob?jk={jk_value}&spa=1"
+        query = self.job_search_query.format(
+            what=self.scraper_input.search_term,
+            location=self.scraper_input.location if self.scraper_input.location else self.scraper_input.country.value[0].split(',')[-1],
+            radius=self.scraper_input.distance,
+            dateOnIndeed=self.scraper_input.hours_old,
+            cursor=f'cursor: "{cursor}"' if cursor else '',
+            filters=filters
+        )
+        payload = {
+            'query': query,
+        }
+        api_headers = self.api_headers.copy()
+        api_headers['indeed-co'] = self.api_country_code
+        response = requests.post(self.api_url, headers=api_headers, json=payload, proxies=self.proxy, timeout=10)
+        if response.status_code != 200:
+            logger.info(f'Indeed responded with status code: {response.status_code} (submit GitHub issue if this appears to be a beg)')
+            return jobs, new_cursor
+        data = response.json()
+        jobs = data['data']['jobSearch']['results']
+        new_cursor = data['data']['jobSearch']['pageInfo']['nextCursor']
 
-        try:
+        with ThreadPoolExecutor(max_workers=self.num_workers) as executor:
-            response = session.get(
+            job_results: list[Future] = [
-                formatted_url, allow_redirects=True, timeout_seconds=5, proxy=self.proxy
+                executor.submit(self._process_job, job['job']) for job in jobs
-            )
+           ]
-        except Exception as e:
+        job_list = [result.result() for result in job_results if result.result()]
-            return None
+        return job_list, new_cursor
 
-        if response.status_code not in range(200, 400):
+    def _build_filters(self):
-            return None
+        """
+        Builds the filters dict for job type/is_remote. If hours_old is provided, composite filter for job_type/is_remote is not possible.
+        IndeedApply: filters: { keyword: { field: "indeedApplyScope", keys: ["DESKTOP"] } }
+        """
+        filters_str = ""
+        if self.scraper_input.hours_old:
+            filters_str = """
+            filters: {{
+                date: {{
+                  field: "dateOnIndeed",
+                  start: "{start}h"
+                }}
+            }}
+            """.format(start=self.scraper_input.hours_old)
+        elif self.scraper_input.job_type or self.scraper_input.is_remote:
+            job_type_key_mapping = {
+                JobType.FULL_TIME: "CF3CP",
+                JobType.PART_TIME: "75GKK",
+                JobType.CONTRACT: "NJXCK",
+                JobType.INTERNSHIP: "VDTG7",
+            }
 
-        raw_description = response.json()["body"]["jobInfoWrapperModel"][
+            keys = []
-            "jobInfoModel"
+            if self.scraper_input.job_type:
-        ]["sanitizedJobDescription"]
+                key = job_type_key_mapping[self.scraper_input.job_type]
-        with io.StringIO(raw_description) as f:
+                keys.append(key)
-            soup = BeautifulSoup(f, "html.parser")
+
-            text_content = " ".join(soup.get_text().split()).strip()
+            if self.scraper_input.is_remote:
-            return text_content
+                keys.append("DSQF7")
+
+            if keys:
+                keys_str = '", "'.join(keys)  # Prepare your keys string
+                filters_str = f"""
+                filters: {{
+                  composite: {{
+                    filters: [{{
+                      keyword: {{
+                        field: "attributes",
+                        keys: ["{keys_str}"]
+                      }}
+                    }}]
+                  }}
+                }}
+                """
+        return filters_str
+
+    def _process_job(self, job: dict) -> JobPost | None:
+        """
+        Parses the job dict into JobPost model
+        :param job: dict to parse
+        :return: JobPost if it's a new job
+        """
+        job_url = f'{self.base_url}/viewjob?jk={job["key"]}'
+        if job_url in self.seen_urls:
+            return
+        self.seen_urls.add(job_url)
+        description = job['description']['html']
+        description = markdown_converter(description) if self.scraper_input.description_format == DescriptionFormat.MARKDOWN else description
+
+        job_type = self._get_job_type(job['attributes'])
+        timestamp_seconds = job["datePublished"] / 1000
+        date_posted = datetime.fromtimestamp(timestamp_seconds).strftime("%Y-%m-%d")
+        employer = job['employer'].get('dossier') if job['employer'] else None
+        employer_details = employer.get('employerDetails', {}) if employer else {}
+        return JobPost(
+            title=job["title"],
+            description=description,
+            company_name=job['employer'].get("name") if job.get('employer') else None,
+            company_url=f"{self.base_url}{job['employer']['relativeCompanyPageUrl']}" if job[
+                'employer'] else None,
+            company_url_direct=employer['links']['corporateWebsite'] if employer else None,
+
+            location=Location(
+                city=job.get("location", {}).get("city"),
+                state=job.get("location", {}).get("admin1Code"),
+                country=job.get("location", {}).get("countryCode"),
+            ),
+            job_type=job_type,
+            compensation=self._get_compensation(job),
+            date_posted=date_posted,
+            job_url=job_url,
+            job_url_direct=job['recruit'].get('viewJobUrl') if job.get('recruit') else None,
+            emails=extract_emails_from_text(description) if description else None,
+            is_remote=self._is_job_remote(job, description),
+
+            company_addresses=employer_details['addresses'][0] if employer_details.get('addresses') else None,
+            company_industry=employer_details['industry'].replace('Iv1', '').replace('_', ' ').title() if employer_details.get('industry') else None,
+            company_num_employees=employer_details.get('employeesLocalizedLabel'),
+            company_revenue=employer_details.get('revenueLocalizedLabel'),
+            company_description=employer_details.get('briefDescription'),
+            ceo_name=employer_details.get('ceoName'),
+            ceo_photo_url=employer_details.get('ceoPhotoUrl'),
+
+            logo_photo_url=employer['images'].get('squareLogoUrl') if employer and employer.get('images') else None,
+            banner_photo_url=employer['images'].get('headerImageUrl') if employer and employer.get('images') else None,
+        )
 
     @staticmethod
-    def get_job_type(job: dict) -> Optional[JobType]:
+    def _get_job_type(attributes: list) -> list[JobType]:
         """
-        Parses the job to get JobTypeIndeed
+        Parses the attributes to get list of job types
+        :param attributes:
+        :return: list of JobType
+        """
+        job_types: list[JobType] = []
+        for attribute in attributes:
+            job_type_str = attribute['label'].replace("-", "").replace(" ", "").lower()
+            job_type = get_enum_from_job_type(job_type_str)
+            if job_type:
+                job_types.append(job_type)
+        return job_types
+
+    @staticmethod
+    def _get_compensation(job: dict) -> Compensation | None:
+        """
+        Parses the job to get compensation
         :param job:
-        :return:
+        :param job:
+        :return: compensation object
         """
-        for taxonomy in job["taxonomyAttributes"]:
+        comp = job['compensation']['baseSalary']
-            if taxonomy["label"] == "job-types":
+        if comp:
-                if len(taxonomy["attributes"]) > 0:
+            interval = IndeedScraper._get_compensation_interval(comp['unitOfWork'])
-                    label = taxonomy["attributes"][0].get("label")
+            if interval:
-                    if label:
+                return Compensation(
-                        job_type_str = label.replace("-", "").replace(" ", "").lower()
+                    interval=interval,
-                        return IndeedScraper.get_enum_from_job_type(job_type_str)
+                    min_amount=round(comp['range'].get('min'), 2) if comp['range'].get('min') is not None else None,
-        return None
+                    max_amount=round(comp['range'].get('max'), 2) if comp['range'].get('max') is not None else None,
+                    currency=job['compensation']['currencyCode']
+                )
 
     @staticmethod
-    def get_enum_from_job_type(job_type_str):
+    def _is_job_remote(job: dict, description: str) -> bool:
         """
-        Given a string, returns the corresponding JobType enum member if a match is found.
+        Searches the description, location, and attributes to check if job is remote
-        for job_type in JobType:
         """
-        for job_type in JobType:
+        remote_keywords = ['remote', 'work from home', 'wfh']
-            if job_type_str in job_type.value:
+        is_remote_in_attributes = any(
-                return job_type
+            any(keyword in attr['label'].lower() for keyword in remote_keywords)
-        return None
+            for attr in job['attributes']
+        )
+        is_remote_in_description = any(keyword in description.lower() for keyword in remote_keywords)
+        is_remote_in_location = any(
+            keyword in job['location']['formatted']['long'].lower()
+            for keyword in remote_keywords
+        )
+        return is_remote_in_attributes or is_remote_in_description or is_remote_in_location
 
     @staticmethod
-    def parse_jobs(soup: BeautifulSoup) -> dict:
+    def _get_compensation_interval(interval: str) -> CompensationInterval:
-        """
+        interval_mapping = {
-        Parses the jobs from the soup object
+            "DAY": "DAILY",
-        :param soup:
+            "YEAR": "YEARLY",
-        :return: jobs
+            "HOUR": "HOURLY",
-        """
+            "WEEK": "WEEKLY",
-
+            "MONTH": "MONTHLY"
-        def find_mosaic_script() -> Optional[Tag]:
+        }
-            """
+        mapped_interval = interval_mapping.get(interval.upper(), None)
-            Finds jobcards script tag
+        if mapped_interval and mapped_interval in CompensationInterval.__members__:
-            :return: script_tag
+            return CompensationInterval[mapped_interval]
-            """
-            script_tags = soup.find_all("script")
-
-            for tag in script_tags:
-                if (
-                        tag.string
-                        and "mosaic.providerData" in tag.string
-                        and "mosaic-provider-jobcards" in tag.string
-                ):
-                    return tag
-            return None
-
-        script_tag = find_mosaic_script()
-
-        if script_tag:
-            script_str = script_tag.string
-            pattern = r'window.mosaic.providerData\["mosaic-provider-jobcards"\]\s*=\s*({.*?});'
-            p = re.compile(pattern, re.DOTALL)
-            m = p.search(script_str)
-            if m:
-                jobs = json.loads(m.group(1).strip())
-                return jobs
-            else:
-                raise IndeedException("Could not find mosaic provider job cards data")
         else:
-            raise IndeedException(
+            raise ValueError(f"Unsupported interval: {interval}")
-                "Could not find a script tag containing mosaic provider data"
-            )
 
-    @staticmethod
+    api_headers = {
-    def total_jobs(soup: BeautifulSoup) -> int:
+        'Host': 'apis.indeed.com',
+        'content-type': 'application/json',
+        'indeed-api-key': '161092c2017b5bbab13edb12461a62d5a833871e7cad6d9d475304573de67ac8',
+        'accept': 'application/json',
+        'indeed-locale': 'en-US',
+        'accept-language': 'en-US,en;q=0.9',
+        'user-agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 16_6_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 Indeed App 193.1',
+        'indeed-app-info': 'appv=193.1; appid=com.indeed.jobsearch; osv=16.6.1; os=ios; dtype=phone',
+    }
+    job_search_query = """
+        query GetJobData {{
+          jobSearch(
+            what: "{what}"
+            location: {{ where: "{location}", radius: {radius}, radiusUnit: MILES }}
+            includeSponsoredResults: NONE
+            limit: 100
+            sort: DATE
+            {cursor}
+            {filters}
+          ) {{
+            pageInfo {{
+              nextCursor
+            }}
+            results {{
+              trackingKey
+              job {{
+                key
+                title
+                datePublished
+                dateOnIndeed
+                description {{
+                  html
+                }}
+                location {{
+                  countryName
+                  countryCode
+                  admin1Code
+                  city
+                  postalCode
+                  streetAddress
+                  formatted {{
+                    short
+                    long
+                  }}
+                }}
+                compensation {{
+                  baseSalary {{
+                    unitOfWork
+                    range {{
+                      ... on Range {{
+                        min
+                        max
+                      }}
+                    }}
+                  }}
+                  currencyCode
+                }}
+                attributes {{
+                  key
+                  label
+                }}
+                employer {{
+                  relativeCompanyPageUrl
+                  name
+                  dossier {{
+                      employerDetails {{
+                        addresses
+                        industry
+                        employeesLocalizedLabel
+                        revenueLocalizedLabel
+                        briefDescription
+                        ceoName
+                        ceoPhotoUrl
+                      }}
+                      images {{
+                            headerImageUrl
+                            squareLogoUrl
+                      }}
+                      links {{
+                        corporateWebsite
+                    }}
+                  }}
+                }}
+                recruit {{
+                  viewJobUrl
+                  detailedSalary
+                  workSchedule
+                }}
+              }}
+            }}
+          }}
+        }}
         """
-        Parses the total jobs for that search from soup object
-        :param soup:
-        :return: total_num_jobs
-        """
-        script = soup.find("script", string=lambda t: t and "window._initialData" in t)
-
-        pattern = re.compile(r"window._initialData\s*=\s*({.*})\s*;", re.DOTALL)
-        match = pattern.search(script.string)
-        total_num_jobs = 0
-        if match:
-            json_str = match.group(1)
-            data = json.loads(json_str)
-            total_num_jobs = int(data["searchTitleBarModel"]["totalNumResults"])
-        return total_num_jobs

src/jobspy/scrapers/linkedin/__init__.py

@@ -4,40 +4,50 @@ jobspy.scrapers.linkedin
 
 This module contains routines to scrape LinkedIn.
 """
+import time
+import random
 from typing import Optional
 from datetime import datetime
 
-import requests
-import time
-from requests.exceptions import ProxyError
-from concurrent.futures import ThreadPoolExecutor, as_completed
-from bs4 import BeautifulSoup
-from bs4.element import Tag
 from threading import Lock
+from bs4.element import Tag
+from bs4 import BeautifulSoup
+from urllib.parse import urlparse, urlunparse
 
 from .. import Scraper, ScraperInput, Site
 from ..exceptions import LinkedInException
+from ..utils import create_session
 from ...jobs import (
     JobPost,
     Location,
     JobResponse,
     JobType,
+    Country,
+    Compensation,
+    DescriptionFormat
+)
+from ..utils import (
+    logger,
+    extract_emails_from_text,
+    get_enum_from_job_type,
+    currency_parser,
+    markdown_converter
 )
-from ...utils import extract_emails_from_text
 
 
 class LinkedInScraper(Scraper):
-    MAX_RETRIES = 3
+    base_url = "https://www.linkedin.com"
-    DELAY = 10
+    delay = 3
+    band_delay = 4
+    jobs_per_page = 25
 
     def __init__(self, proxy: Optional[str] = None):
         """
         Initializes LinkedInScraper with the LinkedIn job search url
         """
-        site = Site(Site.LINKEDIN)
+        super().__init__(Site(Site.LINKEDIN), proxy=proxy)
+        self.scraper_input = None
         self.country = "worldwide"
-        self.url = "https://www.linkedin.com"
-        super().__init__(site, proxy=proxy)
 
     def scrape(self, scraper_input: ScraperInput) -> JobResponse:
         """
@@ -45,202 +55,193 @@ class LinkedInScraper(Scraper):
         :param scraper_input:
         :return: job_response
         """
+        self.scraper_input = scraper_input
         job_list: list[JobPost] = []
         seen_urls = set()
         url_lock = Lock()
         page = scraper_input.offset // 25 + 25 if scraper_input.offset else 0
-
+        seconds_old = (
-        def job_type_code(job_type_enum):
+            scraper_input.hours_old * 3600
-            mapping = {
+            if scraper_input.hours_old
-                JobType.FULL_TIME: "F",
+            else None
-                JobType.PART_TIME: "P",
+        )
-                JobType.INTERNSHIP: "I",
+        continue_search = lambda: len(job_list) < scraper_input.results_wanted and page < 1000
-                JobType.CONTRACT: "C",
+        while continue_search():
-                JobType.TEMPORARY: "T",
+            logger.info(f'LinkedIn search page: {page // 25 + 1}')
-            }
+            session = create_session(is_tls=False, has_retry=True, delay=5)
-
-            return mapping.get(job_type_enum, "")
-
-        while len(job_list) < scraper_input.results_wanted and page < 1000:
             params = {
                 "keywords": scraper_input.search_term,
                 "location": scraper_input.location,
                 "distance": scraper_input.distance,
                 "f_WT": 2 if scraper_input.is_remote else None,
-                "f_JT": job_type_code(scraper_input.job_type)
+                "f_JT": self.job_type_code(scraper_input.job_type)
                 if scraper_input.job_type
                 else None,
                 "pageNum": 0,
-                page: page + scraper_input.offset,
+                "start": page + scraper_input.offset,
                 "f_AL": "true" if scraper_input.easy_apply else None,
+                "f_C": ','.join(map(str, scraper_input.linkedin_company_ids)) if scraper_input.linkedin_company_ids else None,
             }
+            if seconds_old is not None:
+                params["f_TPR"] = f"r{seconds_old}"
 
             params = {k: v for k, v in params.items() if v is not None}
-
+            try:
-            params = {k: v for k, v in params.items() if v is not None}
+                response = session.get(
-            retries = 0
+                    f"{self.base_url}/jobs-guest/jobs/api/seeMoreJobPostings/search?",
-            while retries < self.MAX_RETRIES:
+                    params=params,
-                try:
+                    allow_redirects=True,
-                    response = requests.get(
+                    proxies=self.proxy,
-                        f"{self.url}/jobs-guest/jobs/api/seeMoreJobPostings/search?",
+                    headers=self.headers,
-                        params=params,
+                    timeout=10,
-                        allow_redirects=True,
+                )
-                        proxies=self.proxy,
+                if response.status_code not in range(200, 400):
-                        timeout=10,
+                    if response.status_code == 429:
-                    )
+                        logger.error(f'429 Response - Blocked by LinkedIn for too many requests')
-                    response.raise_for_status()
-
-                    break
-                except requests.HTTPError as e:
-                    if hasattr(e, 'response') and e.response is not None:
-                        if e.response.status_code == 429:
-                            time.sleep(self.DELAY)
-                            retries += 1
-                            continue
-                        else:
-                            raise LinkedInException(f"bad response status code: {e.response.status_code}")
                     else:
-                        raise
+                        logger.error(f'LinkedIn response status code {response.status_code}')
-                except ProxyError as e:
+                    return JobResponse(jobs=job_list)
-                    raise LinkedInException("bad proxy")
+            except Exception as e:
-                except Exception as e:
+                if "Proxy responded with" in str(e):
-                    raise LinkedInException(str(e))
+                    logger.error(f'LinkedIn: Bad proxy')
-            else:
+                else:
-                # Raise an exception if the maximum number of retries is reached
+                    logger.error(f'LinkedIn: {str(e)}')
-                raise LinkedInException("Max retries reached, failed to get a valid response")
+                return JobResponse(jobs=job_list)
 
             soup = BeautifulSoup(response.text, "html.parser")
+            job_cards = soup.find_all("div", class_="base-search-card")
+            if len(job_cards) == 0:
+                return JobResponse(jobs=job_list)
 
-            with ThreadPoolExecutor(max_workers=5) as executor:
+            for job_card in job_cards:
-                futures = []
+                job_url = None
-                for job_card in soup.find_all("div", class_="base-search-card"):
+                href_tag = job_card.find("a", class_="base-card__full-link")
-                    job_url = None
+                if href_tag and "href" in href_tag.attrs:
-                    href_tag = job_card.find("a", class_="base-card__full-link")
+                    href = href_tag.attrs["href"].split("?")[0]
-                    if href_tag and "href" in href_tag.attrs:
+                    job_id = href.split("-")[-1]
-                        href = href_tag.attrs["href"].split("?")[0]
+                    job_url = f"{self.base_url}/jobs/view/{job_id}"
-                        job_id = href.split("-")[-1]
-                        job_url = f"{self.url}/jobs/view/{job_id}"
 
-                    with url_lock:
+                with url_lock:
-                        if job_url in seen_urls:
+                    if job_url in seen_urls:
-                            continue
+                        continue
-                        seen_urls.add(job_url)
+                    seen_urls.add(job_url)
+                try:
+                    job_post = self._process_job(job_card, job_url, scraper_input.linkedin_fetch_description)
+                    if job_post:
+                        job_list.append(job_post)
+                    if not continue_search():
+                       break
+                except Exception as e:
+                    raise LinkedInException(str(e))
 
-                    futures.append(executor.submit(self.process_job, job_card, job_url))
+            if continue_search():
-
+                time.sleep(random.uniform(self.delay, self.delay + self.band_delay))
-                for future in as_completed(futures):
+                page += self.jobs_per_page
-                    try:
-                        job_post = future.result()
-                        if job_post:
-                            job_list.append(job_post)
-                    except Exception as e:
-                        raise LinkedInException("Exception occurred while processing jobs")
-            page += 25
 
         job_list = job_list[: scraper_input.results_wanted]
         return JobResponse(jobs=job_list)
 
-    def process_job(self, job_card: Tag, job_url: str) -> Optional[JobPost]:
+    def _process_job(self, job_card: Tag, job_url: str, full_descr: bool) -> Optional[JobPost]:
+        salary_tag = job_card.find('span', class_='job-search-card__salary-info')
+
+        compensation = None
+        if salary_tag:
+            salary_text = salary_tag.get_text(separator=" ").strip()
+            salary_values = [currency_parser(value) for value in salary_text.split("-")]
+            salary_min = salary_values[0]
+            salary_max = salary_values[1]
+            currency = salary_text[0] if salary_text[0] != "$" else "USD"
+
+            compensation = Compensation(
+                min_amount=int(salary_min),
+                max_amount=int(salary_max),
+                currency=currency,
+            )
+
         title_tag = job_card.find("span", class_="sr-only")
         title = title_tag.get_text(strip=True) if title_tag else "N/A"
 
         company_tag = job_card.find("h4", class_="base-search-card__subtitle")
         company_a_tag = company_tag.find("a") if company_tag else None
+        company_url = (
+            urlunparse(urlparse(company_a_tag.get("href"))._replace(query=""))
+            if company_a_tag and company_a_tag.has_attr("href")
+            else ""
+        )
         company = company_a_tag.get_text(strip=True) if company_a_tag else "N/A"
 
         metadata_card = job_card.find("div", class_="base-search-card__metadata")
-        location = self.get_location(metadata_card)
+        location = self._get_location(metadata_card)
 
-        datetime_tag = metadata_card.find("time", class_="job-search-card__listdate") if metadata_card else None
+        datetime_tag = (
-        date_posted = None
+            metadata_card.find("time", class_="job-search-card__listdate")
+            if metadata_card
+            else None
+        )
+        date_posted = description = job_type = None
         if datetime_tag and "datetime" in datetime_tag.attrs:
             datetime_str = datetime_tag["datetime"]
             try:
                 date_posted = datetime.strptime(datetime_str, "%Y-%m-%d")
-            except Exception as e:
+            except:
                 date_posted = None
         benefits_tag = job_card.find("span", class_="result-benefits__text")
-        benefits = " ".join(benefits_tag.get_text().split()) if benefits_tag else None
+        if full_descr:
-
+            description, job_type = self._get_job_description(job_url)
-        description, job_type = self.get_job_description(job_url)
 
         return JobPost(
             title=title,
-            description=description,
             company_name=company,
+            company_url=company_url,
             location=location,
             date_posted=date_posted,
             job_url=job_url,
+            compensation=compensation,
             job_type=job_type,
-            benefits=benefits,
+            description=description,
-            emails=extract_emails_from_text(description)
+            emails=extract_emails_from_text(description) if description else None,
         )
 
-    def get_job_description(self, job_page_url: str) -> tuple[None, None] | tuple[
+    def _get_job_description(
-        str | None, tuple[str | None, JobType | None]]:
+        self, job_page_url: str
+    ) -> tuple[None, None] | tuple[str | None, tuple[str | None, JobType | None]]:
         """
         Retrieves job description by going to the job page url
         :param job_page_url:
         :return: description or None
         """
         try:
-            response = requests.get(job_page_url, timeout=5, proxies=self.proxy)
+            session = create_session(is_tls=False, has_retry=True)
+            response = session.get(job_page_url, headers=self.headers, timeout=5, proxies=self.proxy)
             response.raise_for_status()
-        except Exception as e:
+        except:
+            return None, None
+        if response.url == "https://www.linkedin.com/signup":
             return None, None
 
         soup = BeautifulSoup(response.text, "html.parser")
         div_content = soup.find(
             "div", class_=lambda x: x and "show-more-less-html__markup" in x
         )
-
         description = None
-        if div_content:
+        if div_content is not None:
-            description = " ".join(div_content.get_text().split()).strip()
+            def remove_attributes(tag):
+                for attr in list(tag.attrs):
+                    del tag[attr]
+                return tag
+            div_content = remove_attributes(div_content)
+            description = div_content.prettify(formatter="html")
+            if self.scraper_input.description_format == DescriptionFormat.MARKDOWN:
+                description = markdown_converter(description)
+        return description, self._parse_job_type(soup)
 
-        def get_job_type(
+    def _get_location(self, metadata_card: Optional[Tag]) -> Location:
-                soup_job_type: BeautifulSoup,
-        ) -> JobType | None:
-            """
-            Gets the job type from job page
-            :param soup_job_type:
-            :return: JobType
-            """
-            h3_tag = soup_job_type.find(
-                "h3",
-                class_="description__job-criteria-subheader",
-                string=lambda text: "Employment type" in text,
-            )
-
-            employment_type = None
-            if h3_tag:
-                employment_type_span = h3_tag.find_next_sibling(
-                    "span",
-                    class_="description__job-criteria-text description__job-criteria-text--criteria",
-                )
-                if employment_type_span:
-                    employment_type = employment_type_span.get_text(strip=True)
-                    employment_type = employment_type.lower()
-                    employment_type = employment_type.replace("-", "")
-
-            return LinkedInScraper.get_enum_from_value(employment_type)
-
-        return description, get_job_type(soup)
-
-    @staticmethod
-    def get_enum_from_value(value_str):
-        for job_type in JobType:
-            if value_str in job_type.value:
-                return job_type
-        return None
-
-    def get_location(self, metadata_card: Optional[Tag]) -> Location:
         """
         Extracts the location data from the job metadata card.
         :param metadata_card
         :return: location
         """
-        location = Location(country=self.country)
+        location = Location(country=Country.from_string(self.country))
         if metadata_card is not None:
             location_tag = metadata_card.find(
                 "span", class_="job-search-card__location"
@@ -252,7 +253,57 @@ class LinkedInScraper(Scraper):
                 location = Location(
                     city=city,
                     state=state,
-                    country=self.country,
+                    country=Country.from_string(self.country),
+                )
+            elif len(parts) == 3:
+                city, state, country = parts
+                location = Location(
+                    city=city,
+                    state=state,
+                    country=Country.from_string(country)
                 )
-
         return location
+
+    @staticmethod
+    def _parse_job_type(soup_job_type: BeautifulSoup) -> list[JobType] | None:
+        """
+        Gets the job type from job page
+        :param soup_job_type:
+        :return: JobType
+        """
+        h3_tag = soup_job_type.find(
+            "h3",
+            class_="description__job-criteria-subheader",
+            string=lambda text: "Employment type" in text,
+        )
+        employment_type = None
+        if h3_tag:
+            employment_type_span = h3_tag.find_next_sibling(
+                "span",
+                class_="description__job-criteria-text description__job-criteria-text--criteria",
+            )
+            if employment_type_span:
+                employment_type = employment_type_span.get_text(strip=True)
+                employment_type = employment_type.lower()
+                employment_type = employment_type.replace("-", "")
+
+        return [get_enum_from_job_type(employment_type)] if employment_type else []
+
+    @staticmethod
+    def job_type_code(job_type_enum: JobType) -> str:
+        return {
+            JobType.FULL_TIME: "F",
+            JobType.PART_TIME: "P",
+            JobType.INTERNSHIP: "I",
+            JobType.CONTRACT: "C",
+            JobType.TEMPORARY: "T",
+        }.get(job_type_enum, "")
+
+    headers = {
+        "authority": "www.linkedin.com",
+        "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
+        "accept-language": "en-US,en;q=0.9",
+        "cache-control": "max-age=0",
+        "upgrade-insecure-requests": "1",
+        "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
+    }

src/jobspy/scrapers/utils.py

@@ -0,0 +1,89 @@
+import logging
+import re
+
+import numpy as np
+import requests
+import tls_client
+from markdownify import markdownify as md
+from requests.adapters import HTTPAdapter, Retry
+
+from ..jobs import JobType
+
+logger = logging.getLogger("JobSpy")
+logger.propagate = False
+if not logger.handlers:
+    logger.setLevel(logging.INFO)
+    console_handler = logging.StreamHandler()
+    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
+    console_handler.setFormatter(formatter)
+    logger.addHandler(console_handler)
+
+
+def markdown_converter(description_html: str):
+    if description_html is None:
+        return None
+    markdown = md(description_html)
+    return markdown.strip()
+
+
+def extract_emails_from_text(text: str) -> list[str] | None:
+    if not text:
+        return None
+    email_regex = re.compile(r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}")
+    return email_regex.findall(text)
+
+
+def create_session(proxy: dict | None = None, is_tls: bool = True, has_retry: bool = False, delay: int = 1) -> requests.Session:
+    """
+    Creates a requests session with optional tls, proxy, and retry settings.
+    :return: A session object
+    """
+    if is_tls:
+        session = tls_client.Session(random_tls_extension_order=True)
+        session.proxies = proxy
+    else:
+        session = requests.Session()
+        session.allow_redirects = True
+        if proxy:
+            session.proxies.update(proxy)
+        if has_retry:
+            retries = Retry(total=3,
+                            connect=3,
+                            status=3,
+                            status_forcelist=[500, 502, 503, 504, 429],
+                            backoff_factor=delay)
+            adapter = HTTPAdapter(max_retries=retries)
+
+            session.mount('http://', adapter)
+            session.mount('https://', adapter)
+    return session
+
+
+def get_enum_from_job_type(job_type_str: str) -> JobType | None:
+    """
+    Given a string, returns the corresponding JobType enum member if a match is found.
+    """
+    res = None
+    for job_type in JobType:
+        if job_type_str in job_type.value:
+            res = job_type
+    return res
+
+
+def currency_parser(cur_str):
+    # Remove any non-numerical characters
+    # except for ',' '.' or '-' (e.g. EUR)
+    cur_str = re.sub("[^-0-9.,]", '', cur_str)
+    # Remove any 000s separators (either , or .)
+    cur_str = re.sub("[.,]", '', cur_str[:-3]) + cur_str[-3:]
+
+    if '.' in list(cur_str[-3:]):
+        num = float(cur_str)
+    elif ',' in list(cur_str[-3:]):
+        num = float(cur_str.replace(',', '.'))
+    else:
+        num = float(cur_str)
+
+    return np.round(num, 2)
+
+

src/jobspy/scrapers/ziprecruiter/__init__.py

@@ -5,465 +5,204 @@ jobspy.scrapers.ziprecruiter
 This module contains routines to scrape ZipRecruiter.
 """
 import math
-import json
+import time
-import re
+from datetime import datetime
-from datetime import datetime, date
 from typing import Optional, Tuple, Any
-from urllib.parse import urlparse, parse_qs, urlunparse
 
-import tls_client
+from concurrent.futures import ThreadPoolExecutor
-import requests
-from bs4 import BeautifulSoup
-from bs4.element import Tag
-from concurrent.futures import ThreadPoolExecutor, Future
 
 from .. import Scraper, ScraperInput, Site
-from ..exceptions import ZipRecruiterException
+from ..utils import (
+    logger,
+    extract_emails_from_text,
+    create_session,
+    markdown_converter
+)
 from ...jobs import (
     JobPost,
     Compensation,
-    CompensationInterval,
     Location,
     JobResponse,
     JobType,
     Country,
+    DescriptionFormat
 )
-from ...utils import extract_emails_from_text
 
 
 class ZipRecruiterScraper(Scraper):
+    base_url = "https://www.ziprecruiter.com"
+    api_url = "https://api.ziprecruiter.com"
+
     def __init__(self, proxy: Optional[str] = None):
         """
-        Initializes LinkedInScraper with the ZipRecruiter job search url
+        Initializes ZipRecruiterScraper with the ZipRecruiter job search url
         """
-        site = Site(Site.ZIP_RECRUITER)
+        self.scraper_input = None
-        self.url = "https://www.ziprecruiter.com"
+        self.session = create_session(proxy)
-        super().__init__(site, proxy=proxy)
+        self._get_cookies()
+        super().__init__(Site.ZIP_RECRUITER, proxy=proxy)
 
+        self.delay = 5
         self.jobs_per_page = 20
         self.seen_urls = set()
-        self.session = tls_client.Session(
-            client_identifier="chrome112", random_tls_extension_order=True
-        )
-
-    def find_jobs_in_page(
-            self, scraper_input: ScraperInput, page: int
-    ) -> list[JobPost]:
-        """
-        Scrapes a page of ZipRecruiter for jobs with scraper_input criteria
-        :param scraper_input:
-        :param page:
-        :return: jobs found on page
-        """
-        job_list: list[JobPost] = []
-        try:
-            response = self.session.get(
-                f"{self.url}/jobs-search",
-                headers=ZipRecruiterScraper.headers(),
-                params=ZipRecruiterScraper.add_params(scraper_input, page),
-                allow_redirects=True,
-                proxy=self.proxy,
-                timeout_seconds=10,
-            )
-            if response.status_code != 200:
-                raise ZipRecruiterException(
-                    f"bad response status code: {response.status_code}"
-                )
-        except Exception as e:
-            if "Proxy responded with non 200 code" in str(e):
-                raise ZipRecruiterException("bad proxy")
-            raise ZipRecruiterException(str(e))
-        else:
-            soup = BeautifulSoup(response.text, "html.parser")
-            js_tag = soup.find("script", {"id": "js_variables"})
-
-        if js_tag:
-            page_json = json.loads(js_tag.string)
-            jobs_list = page_json.get("jobList")
-            if jobs_list:
-                page_variant = "javascript"
-                # print('type javascript', len(jobs_list))
-            else:
-                page_variant = "html_2"
-                jobs_list = soup.find_all("div", {"class": "job_content"})
-                # print('type 2 html', len(jobs_list))
-        else:
-            page_variant = "html_1"
-            jobs_list = soup.find_all("li", {"class": "job-listing"})
-            # print('type 1 html', len(jobs_list))
-
-        with ThreadPoolExecutor(max_workers=10) as executor:
-            if page_variant == "javascript":
-                job_results = [
-                    executor.submit(self.process_job_javascript, job)
-                    for job in jobs_list
-                ]
-            elif page_variant == "html_1":
-                job_results = [
-                    executor.submit(self.process_job_html_1, job) for job in jobs_list
-                ]
-            elif page_variant == "html_2":
-                job_results = [
-                    executor.submit(self.process_job_html_2, job) for job in jobs_list
-                ]
-
-        job_list = [result.result() for result in job_results if result.result()]
-        return job_list
 
     def scrape(self, scraper_input: ScraperInput) -> JobResponse:
         """
-        Scrapes ZipRecruiter for jobs with scraper_input criteria
+        Scrapes ZipRecruiter for jobs with scraper_input criteria.
-        :param scraper_input:
+        :param scraper_input: Information about job search criteria.
-        :return: job_response
+        :return: JobResponse containing a list of jobs.
         """
-        start_page = (scraper_input.offset // self.jobs_per_page) + 1 if scraper_input.offset else 1
+        self.scraper_input = scraper_input
-        #: get first page to initialize session
+        job_list: list[JobPost] = []
-        job_list: list[JobPost] = self.find_jobs_in_page(scraper_input, start_page)
+        continue_token = None
-        pages_to_process = max(
-            3, math.ceil(scraper_input.results_wanted / self.jobs_per_page)
-        )
 
-        with ThreadPoolExecutor(max_workers=10) as executor:
+        max_pages = math.ceil(scraper_input.results_wanted / self.jobs_per_page)
-            futures: list[Future] = [
+        for page in range(1, max_pages + 1):
-                executor.submit(self.find_jobs_in_page, scraper_input, page)
+            if len(job_list) >= scraper_input.results_wanted:
-                for page in range(start_page + 1, start_page + pages_to_process + 2)
+                break
-            ]
+            if page > 1:
-
+                time.sleep(self.delay)
-            for future in futures:
+            logger.info(f'ZipRecruiter search page: {page}')
-                jobs = future.result()
+            jobs_on_page, continue_token = self._find_jobs_in_page(
-
+                scraper_input, continue_token
-                job_list += jobs
-
-        job_list = job_list[: scraper_input.results_wanted]
-        return JobResponse(jobs=job_list)
-
-    def process_job_html_1(self, job: Tag) -> Optional[JobPost]:
-        """
-        Parses a job from the job content tag
-        :param job: BeautifulSoup Tag for one job post
-        :return JobPost
-        TODO this method isnt finished due to not encountering this type of html often
-        """
-        job_url = self.cleanurl(job.find("a", {"class": "job_link"})["href"])
-        if job_url in self.seen_urls:
-            return None
-
-        title = job.find("h2", {"class": "title"}).text
-        company = job.find("a", {"class": "company_name"}).text.strip()
-
-        description, updated_job_url = self.get_description(job_url)
-        # job_url = updated_job_url if updated_job_url else job_url
-        if description is None:
-            description = job.find("p", {"class": "job_snippet"}).text.strip()
-
-        job_type_element = job.find("li", {"class": "perk_item perk_type"})
-        job_type = None
-        if job_type_element:
-            job_type_text = (
-                job_type_element.text.strip().lower().replace("_", "").replace(" ", "")
             )
-            job_type = ZipRecruiterScraper.get_job_type_enum(job_type_text)
+            if jobs_on_page:
-
+                job_list.extend(jobs_on_page)
-        date_posted = ZipRecruiterScraper.get_date_posted(job)
-
-        job_post = JobPost(
-            title=title,
-            description=description,
-            company_name=company,
-            location=ZipRecruiterScraper.get_location(job),
-            job_type=job_type,
-            compensation=ZipRecruiterScraper.get_compensation(job),
-            date_posted=date_posted,
-            job_url=job_url,
-            emails=extract_emails_from_text(description),
-        )
-        return job_post
-
-    def process_job_html_2(self, job: Tag) -> Optional[JobPost]:
-        """
-        Parses a job from the job content tag for a second variat of HTML that ZR uses
-        :param job: BeautifulSoup Tag for one job post
-        :return JobPost
-        """
-        job_url = self.cleanurl(job.find("a", class_="job_link")["href"])
-        title = job.find("h2", class_="title").text
-        company = job.find("a", class_="company_name").text.strip()
-
-        description, updated_job_url = self.get_description(job_url)
-        # job_url = updated_job_url if updated_job_url else job_url
-        if description is None:
-            description = job.find("p", class_="job_snippet").get_text().strip()
-
-        job_type_text = job.find("li", class_="perk_item perk_type")
-        job_type = None
-        if job_type_text:
-            job_type_text = (
-                job_type_text.get_text()
-                .strip()
-                .lower()
-                .replace("-", "")
-                .replace(" ", "")
-            )
-            job_type = ZipRecruiterScraper.get_job_type_enum(job_type_text)
-        date_posted = ZipRecruiterScraper.get_date_posted(job)
-
-        job_post = JobPost(
-            title=title,
-            description=description,
-            company_name=company,
-            location=ZipRecruiterScraper.get_location(job),
-            job_type=job_type,
-            compensation=ZipRecruiterScraper.get_compensation(job),
-            date_posted=date_posted,
-            job_url=job_url,
-        )
-        return job_post
-
-    def process_job_javascript(self, job: dict) -> JobPost:
-        title = job.get("Title")
-        job_url = self.cleanurl(job.get("JobURL"))
-
-        description, updated_job_url = self.get_description(job_url)
-        # job_url = updated_job_url if updated_job_url else job_url
-        if description is None:
-            description = BeautifulSoup(
-                job.get("Snippet", "").strip(), "html.parser"
-            ).get_text()
-
-        company = job.get("OrgName")
-        location = Location(
-            city=job.get("City"), state=job.get("State"), country=Country.US_CANADA
-        )
-        job_type = ZipRecruiterScraper.get_job_type_enum(
-            job.get("EmploymentType", "").replace("-", "").lower()
-        )
-
-        formatted_salary = job.get("FormattedSalaryShort", "")
-        salary_parts = formatted_salary.split(" ")
-
-        min_salary_str = salary_parts[0][1:].replace(",", "")
-        if "." in min_salary_str:
-            min_amount = int(float(min_salary_str) * 1000)
-        else:
-            min_amount = int(min_salary_str.replace("K", "000"))
-
-        if len(salary_parts) >= 3 and salary_parts[2].startswith("$"):
-            max_salary_str = salary_parts[2][1:].replace(",", "")
-            if "." in max_salary_str:
-                max_amount = int(float(max_salary_str) * 1000)
             else:
-                max_amount = int(max_salary_str.replace("K", "000"))
+                break
-        else:
+            if not continue_token:
-            max_amount = 0
+                break
+        return JobResponse(jobs=job_list[: scraper_input.results_wanted])
 
-        compensation = Compensation(
+    def _find_jobs_in_page(
-            interval=CompensationInterval.YEARLY,
+        self, scraper_input: ScraperInput, continue_token: str | None = None
-            min_amount=min_amount,
+    ) -> Tuple[list[JobPost], Optional[str]]:
-            max_amount=max_amount,
+        """
-            currency="USD/CAD",
+        Scrapes a page of ZipRecruiter for jobs with scraper_input criteria
-        )
+        :param scraper_input:
-        save_job_url = job.get("SaveJobURL", "")
+        :param continue_token:
-        posted_time_match = re.search(
+        :return: jobs found on page
-            r"posted_time=(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z)", save_job_url
+        """
-        )
+        jobs_list = []
-        if posted_time_match:
+        params = self._add_params(scraper_input)
-            date_time_str = posted_time_match.group(1)
+        if continue_token:
-            date_posted_obj = datetime.strptime(date_time_str, "%Y-%m-%dT%H:%M:%SZ")
+            params["continue_from"] = continue_token
-            date_posted = date_posted_obj.date()
+        try:
-        else:
+            res= self.session.get(
-            date_posted = date.today()
+                f"{self.api_url}/jobs-app/jobs",
+                headers=self.headers,
+                params=params
+            )
+            if res.status_code not in range(200, 400):
+                if res.status_code == 429:
+                    logger.error(f'429 Response - Blocked by ZipRecruiter for too many requests')
+                else:
+                    logger.error(f'ZipRecruiter response status code {res.status_code}')
+                return jobs_list, ""
+        except Exception as e:
+            if "Proxy responded with" in str(e):
+                logger.error(f'Indeed: Bad proxy')
+            else:
+                logger.error(f'Indeed: {str(e)}')
+            return jobs_list, ""
 
+
+        res_data = res.json()
+        jobs_list = res_data.get("jobs", [])
+        next_continue_token = res_data.get("continue", None)
+        with ThreadPoolExecutor(max_workers=self.jobs_per_page) as executor:
+            job_results = [executor.submit(self._process_job, job) for job in jobs_list]
+
+        job_list = list(filter(None, (result.result() for result in job_results)))
+        return job_list, next_continue_token
+
+    def _process_job(self, job: dict) -> JobPost | None:
+        """
+        Processes an individual job dict from the response
+        """
+        title = job.get("name")
+        job_url = f"{self.base_url}/jobs//j?lvk={job['listing_key']}"
+        if job_url in self.seen_urls:
+            return
+        self.seen_urls.add(job_url)
+
+        description = job.get("job_description", "").strip()
+        description = markdown_converter(description) if self.scraper_input.description_format == DescriptionFormat.MARKDOWN else description
+        company = job.get("hiring_company", {}).get("name")
+        country_value = "usa" if job.get("job_country") == "US" else "canada"
+        country_enum = Country.from_string(country_value)
+
+        location = Location(
+            city=job.get("job_city"), state=job.get("job_state"), country=country_enum
+        )
+        job_type = self._get_job_type_enum(
+            job.get("employment_type", "").replace("_", "").lower()
+        )
+        date_posted = datetime.fromisoformat(job['posted_time'].rstrip("Z")).date()
         return JobPost(
             title=title,
-            description=description,
             company_name=company,
             location=location,
             job_type=job_type,
-            compensation=compensation,
+            compensation=Compensation(
+                interval="yearly"
+                if job.get("compensation_interval") == "annual"
+                else job.get("compensation_interval"),
+                min_amount=int(job["compensation_min"])
+                if "compensation_min" in job
+                else None,
+                max_amount=int(job["compensation_max"])
+                if "compensation_max" in job
+                else None,
+                currency=job.get("compensation_currency"),
+            ),
             date_posted=date_posted,
             job_url=job_url,
+            description=description,
+            emails=extract_emails_from_text(description) if description else None,
         )
-        return job_post
+
+    def _get_cookies(self):
+        data="event_type=session&logged_in=false&number_of_retry=1&property=model%3AiPhone&property=os%3AiOS&property=locale%3Aen_us&property=app_build_number%3A4734&property=app_version%3A91.0&property=manufacturer%3AApple&property=timestamp%3A2024-01-12T12%3A04%3A42-06%3A00&property=screen_height%3A852&property=os_version%3A16.6.1&property=source%3Ainstall&property=screen_width%3A393&property=device_model%3AiPhone%2014%20Pro&property=brand%3AApple"
+        self.session.post(f"{self.api_url}/jobs-app/event", data=data, headers=self.headers)
 
     @staticmethod
-    def get_job_type_enum(job_type_str: str) -> Optional[JobType]:
+    def _get_job_type_enum(job_type_str: str) -> list[JobType] | None:
         for job_type in JobType:
             if job_type_str in job_type.value:
-                a = True
+                return [job_type]
-                return job_type
         return None
 
-    def get_description(self, job_page_url: str) -> Tuple[Optional[str], Optional[str]]:
-        """
-        Retrieves job description by going to the job page url
-        :param job_page_url:
-        :param session:
-        :return: description or None, response url
-        """
-        try:
-            response = requests.get(
-                job_page_url,
-                headers=ZipRecruiterScraper.headers(),
-                allow_redirects=True,
-                timeout=5,
-                proxies=self.proxy,
-            )
-            if response.status_code not in range(200, 400):
-                return None, None
-        except Exception as e:
-            return None, None
-
-        html_string = response.content
-        soup_job = BeautifulSoup(html_string, "html.parser")
-
-        job_description_div = soup_job.find("div", {"class": "job_description"})
-        if job_description_div:
-            return job_description_div.text.strip(), response.url
-        return None, response.url
-
     @staticmethod
-    def add_params(scraper_input, page) -> dict[str, str | Any]:
+    def _add_params(scraper_input) -> dict[str, str | Any]:
         params = {
             "search": scraper_input.search_term,
             "location": scraper_input.location,
-            "page": page,
-            "form": "jobs-landing",
         }
-        job_type_value = None
+        if scraper_input.hours_old:
+            fromage = max(scraper_input.hours_old // 24, 1) if scraper_input.hours_old else None
+            params['days'] = fromage
+        job_type_map = {
+            JobType.FULL_TIME: 'full_time',
+            JobType.PART_TIME: 'part_time'
+        }
         if scraper_input.job_type:
-            if scraper_input.job_type.value == "fulltime":
+            params['employment_type'] = job_type_map[scraper_input.job_type] if scraper_input.job_type in job_type_map else scraper_input.job_type.value[0]
-                job_type_value = "full_time"
+        if scraper_input.easy_apply:
-            elif scraper_input.job_type.value == "parttime":
+            params['zipapply'] = 1
-                job_type_value = "part_time"
-            else:
-                job_type_value = scraper_input.job_type.value
-
-        if job_type_value:
-            params[
-                "refine_by_employment"
-            ] = f"employment_type:employment_type:{job_type_value}"
-
         if scraper_input.is_remote:
-            params["refine_by_location_type"] = "only_remote"
+            params["remote"] = 1
-
         if scraper_input.distance:
             params["radius"] = scraper_input.distance
+        return {k: v for k, v in params.items() if v is not None}
 
-        return params
+    headers = {
-
+        "Host": "api.ziprecruiter.com",
-    @staticmethod
+        "accept": "*/*",
-    def get_interval(interval_str: str):
+        "x-zr-zva-override": "100000000;vid:ZT1huzm_EQlDTVEc",
-        """
+        "x-pushnotificationid": "0ff4983d38d7fc5b3370297f2bcffcf4b3321c418f5c22dd152a0264707602a0",
-         Maps the interval alias to its appropriate CompensationInterval.
+        "x-deviceid": "D77B3A92-E589-46A4-8A39-6EF6F1D86006",
-        :param interval_str
+        "user-agent": "Job Search/87.0 (iPhone; CPU iOS 16_6_1 like Mac OS X)",
-        :return: CompensationInterval
+        "authorization": "Basic YTBlZjMyZDYtN2I0Yy00MWVkLWEyODMtYTI1NDAzMzI0YTcyOg==",
-        """
+        "accept-language": "en-US,en;q=0.9",
-        interval_alias = {"annually": CompensationInterval.YEARLY}
+    }
-        interval_str = interval_str.lower()
-
-        if interval_str in interval_alias:
-            return interval_alias[interval_str]
-
-        return CompensationInterval(interval_str)
-
-    @staticmethod
-    def get_date_posted(job: Tag) -> Optional[datetime.date]:
-        """
-        Extracts the date a job was posted
-        :param job
-        :return: date the job was posted or None
-        """
-        button = job.find(
-            "button", {"class": "action_input save_job zrs_btn_secondary_200"}
-        )
-        if not button:
-            return None
-
-        url_time = button.get("data-href", "")
-        url_components = urlparse(url_time)
-        params = parse_qs(url_components.query)
-        posted_time_str = params.get("posted_time", [None])[0]
-
-        if posted_time_str:
-            posted_date = datetime.strptime(
-                posted_time_str, "%Y-%m-%dT%H:%M:%SZ"
-            ).date()
-            return posted_date
-
-        return None
-
-    @staticmethod
-    def get_compensation(job: Tag) -> Optional[Compensation]:
-        """
-        Parses the compensation tag from the job BeautifulSoup object
-        :param job
-        :return: Compensation object or None
-        """
-        pay_element = job.find("li", {"class": "perk_item perk_pay"})
-        if pay_element is None:
-            return None
-        pay = pay_element.find("div", {"class": "value"}).find("span").text.strip()
-
-        def create_compensation_object(pay_string: str) -> Compensation:
-            """
-            Creates a Compensation object from a pay_string
-            :param pay_string
-            :return: compensation
-            """
-            interval = ZipRecruiterScraper.get_interval(pay_string.split()[-1])
-
-            amounts = []
-            for amount in pay_string.split("to"):
-                amount = amount.replace(",", "").strip("$ ").split(" ")[0]
-                if "K" in amount:
-                    amount = amount.replace("K", "")
-                    amount = int(float(amount)) * 1000
-                else:
-                    amount = int(float(amount))
-                amounts.append(amount)
-
-            compensation = Compensation(
-                interval=interval,
-                min_amount=min(amounts),
-                max_amount=max(amounts),
-                currency="USD/CAD",
-            )
-
-            return compensation
-
-        return create_compensation_object(pay)
-
-    @staticmethod
-    def get_location(job: Tag) -> Location:
-        """
-        Extracts the job location from BeatifulSoup object
-        :param job:
-        :return: location
-        """
-        location_link = job.find("a", {"class": "company_location"})
-        if location_link is not None:
-            location_string = location_link.text.strip()
-            parts = location_string.split(", ")
-            if len(parts) == 2:
-                city, state = parts
-            else:
-                city, state = None, None
-        else:
-            city, state = None, None
-        return Location(city=city, state=state, country=Country.US_CANADA)
-
-    @staticmethod
-    def headers() -> dict:
-        """
-        Returns headers needed for requests
-        :return: dict - Dictionary containing headers
-        """
-        return {
-            "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
-        }
-
-    @staticmethod
-    def cleanurl(url):
-        parsed_url = urlparse(url)
-
-        return urlunparse((parsed_url.scheme, parsed_url.netloc, parsed_url.path, parsed_url.params, '', ''))

src/jobspy/utils.py

@@ -1,9 +0,0 @@
-import re
-from typing import Optional
-
-
-def extract_emails_from_text(text: str) -> Optional[list[str]]:
-    if not text:
-        return None
-    email_regex = re.compile(r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}")
-    return email_regex.findall(text)

src/tests/test_all.py

@@ -4,9 +4,11 @@ import pandas as pd
 
 def test_all():
     result = scrape_jobs(
-        site_name=["linkedin", "indeed", "zip_recruiter"],
+        site_name=["linkedin", "indeed", "zip_recruiter", "glassdoor"],
         search_term="software engineer",
         results_wanted=5,
     )
 
-    assert isinstance(result, pd.DataFrame) and not result.empty, "Result should be a non-empty DataFrame"
+    assert (
+        isinstance(result, pd.DataFrame) and not result.empty
+    ), "Result should be a non-empty DataFrame"

src/tests/test_glassdoor.py

@@ -0,0 +1,11 @@
+from ..jobspy import scrape_jobs
+import pandas as pd
+
+
+def test_indeed():
+    result = scrape_jobs(
+        site_name="glassdoor", search_term="software engineer", country_indeed="USA"
+    )
+    assert (
+        isinstance(result, pd.DataFrame) and not result.empty
+    ), "Result should be a non-empty DataFrame"

src/tests/test_indeed.py

@@ -4,7 +4,8 @@ import pandas as pd
 
 def test_indeed():
     result = scrape_jobs(
-        site_name="indeed",
+        site_name="indeed", search_term="software engineer", country_indeed="usa"
-        search_term="software engineer",
     )
-    assert isinstance(result, pd.DataFrame) and not result.empty, "Result should be a non-empty DataFrame"
+    assert (
+        isinstance(result, pd.DataFrame) and not result.empty
+    ), "Result should be a non-empty DataFrame"

src/tests/test_linkedin.py

@@ -7,4 +7,6 @@ def test_linkedin():
         site_name="linkedin",
         search_term="software engineer",
     )
-    assert isinstance(result, pd.DataFrame) and not result.empty, "Result should be a non-empty DataFrame"
+    assert (
+        isinstance(result, pd.DataFrame) and not result.empty
+    ), "Result should be a non-empty DataFrame"

src/tests/test_ziprecruiter.py

@@ -8,4 +8,6 @@ def test_ziprecruiter():
         search_term="software engineer",
     )
 
-    assert isinstance(result, pd.DataFrame) and not result.empty, "Result should be a non-empty DataFrame"
+    assert (
+        isinstance(result, pd.DataFrame) and not result.empty
+    ), "Result should be a non-empty DataFrame"