feat: utils.py

* enh(indeed): mult job types

* feat(jobs):  urgent kws

* fix(indeed): use new session obj per request

* fix: emails as comma separated in output

* fix: put num urgent words in output

* chore: readme

.gitignore

@@ -1,10 +1,10 @@
-/.idea
-**/.DS_Store
 /venv/
-/ven/
+/.idea
 **/__pycache__/
 **/.pytest_cache/
+/.ipynb_checkpoints/
+**/output/
+**/.DS_Store
 *.pyc
 .env
 dist
-/.ipynb_checkpoints/

README.md

@@ -4,64 +4,54 @@
 
 **Not technical?** Try out the web scraping tool on our site at [usejobspy.com](https://usejobspy.com).
 
-*Looking to build a data-focused software product?* **[Book a call](https://calendly.com/zachary-products/15min)** *to work with us.*  
+*Looking to build a data-focused software product?* **[Book a call](https://calendly.com/zachary-products/15min)** *to
+work with us.*  
 \
-Check out another project we wrote: ***[HomeHarvest](https://github.com/ZacharyHampton/HomeHarvest)** – a Python package for real estate scraping*
+Check out another project we wrote: ***[HomeHarvest](https://github.com/ZacharyHampton/HomeHarvest)** – a Python package
-## Features
+for real estate scraping*
 
+## Features
 
 - Scrapes job postings from **LinkedIn**, **Indeed** & **ZipRecruiter** simultaneously
 - Aggregates the job postings in a Pandas DataFrame
 - Proxy support (HTTP/S, SOCKS)
 
-[Video Guide for JobSpy](https://www.youtube.com/watch?v=RuP1HrAZnxs&pp=ygUgam9icyBzY3JhcGVyIGJvdCBsaW5rZWRpbiBpbmRlZWQ%3D) - Updated for release v1.1.3
+[Video Guide for JobSpy](https://www.youtube.com/watch?v=RuP1HrAZnxs&pp=ygUgam9icyBzY3JhcGVyIGJvdCBsaW5rZWRpbiBpbmRlZWQ%3D) -
+Updated for release v1.1.3
 
 ![jobspy](https://github.com/cullenwatson/JobSpy/assets/78247585/ec7ef355-05f6-4fd3-8161-a817e31c5c57)
 
 ### Installation
+
 ```
 pip install --upgrade python-jobspy
 ```
 
-  _Python version >= [3.10](https://www.python.org/downloads/release/python-3100/) required_ 
+_Python version >= [3.10](https://www.python.org/downloads/release/python-3100/) required_
 
 ### Usage
 
 ```python
 from jobspy import scrape_jobs
-import pandas as pd
 
-jobs: pd.DataFrame = scrape_jobs(
+jobs = scrape_jobs(
     site_name=["indeed", "linkedin", "zip_recruiter"],
     search_term="software engineer",
     location="Dallas, TX",
     results_wanted=10,
-    
     country_indeed='USA'  # only needed for indeed
-    
-    # use if you want to use a proxy (3 types)
-    # proxy="socks5://jobspy:5a4vpWtj8EeJ2hoYzk@ca.smartproxy.com:20001",
-    # proxy="http://jobspy:5a4vpWtj8EeJ2hoYzk@ca.smartproxy.com:20001",
-    # proxy="https://jobspy:5a4vpWtj8EeJ2hoYzk@ca.smartproxy.com:20001",
 )
+print(f"Found {len(jobs)} jobs")
+print(jobs.head())
+jobs.to_csv("jobs.csv", index=False)
 
-# formatting for pandas
+# output to .xlsx
-pd.set_option('display.max_columns', None)
+# jobs.to_xlsx('jobs.xlsx', index=False)
-pd.set_option('display.max_rows', None)
-pd.set_option('display.width', None)
-pd.set_option('display.max_colwidth', 50)  # set to 0 to see full job url / desc
 
-#1 display in Jupyter Notebook (1. pip install jupyter 2. jupyter notebook)
-display(jobs)
-
-#2 output to console
-#print(jobs)
-
-#3 output to .csv
-#jobs.to_csv('jobs.csv', index=False)
 ```
 
 ### Output
+
 ```
 SITE           TITLE                             COMPANY_NAME      CITY          STATE  JOB_TYPE  INTERVAL  MIN_AMOUNT  MAX_AMOUNT  JOB_URL                                            DESCRIPTION
 indeed         Software Engineer                 AMERICAN SYSTEMS  Arlington     VA     None      yearly    200000      150000      https://www.indeed.com/viewjob?jk=5e409e577046...  THIS POSITION COMES WITH A 10K SIGNING BONUS!...
@@ -71,7 +61,9 @@ linkedin       Full-Stack Software Engineer      Rain              New York
 zip_recruiter Software Engineer - New Grad       ZipRecruiter      Santa Monica  CA     fulltime  yearly    130000      150000      https://www.ziprecruiter.com/jobs/ziprecruiter...  We offer a hybrid work environment. Most US-ba...
 zip_recruiter Software Developer                 TEKsystems        Phoenix       AZ     fulltime  hourly    65          75          https://www.ziprecruiter.com/jobs/teksystems-0...  Top Skills' Details• 6 years of Java developme...
 ```
+
 ### Parameters for `scrape_jobs()`
+
 ```plaintext
 Required
 ├── site_type (List[enum]): linkedin, zip_recruiter, indeed
@@ -85,10 +77,11 @@ Optional
 ├── results_wanted (int): number of job results to retrieve for each site specified in 'site_type'
 ├── easy_apply (bool): filters for jobs that are hosted on LinkedIn
 ├── country_indeed (enum): filters the country on Indeed (see below for correct spelling)
+├── offset (num): starts the search from an offset (e.g. 25 will start the search from the 25th result)
 ```
 
- 
 ### JobPost Schema
+
 ```plaintext
 JobPost
 ├── title (str)
@@ -106,17 +99,21 @@ JobPost
 │   ├── max_amount (int)
 │   └── currency (enum)
 └── date_posted (date)
+└── emails (str)
+└── num_urgent_words (int)
+└── is_remote (bool) - just for Indeed at the momen
 ```
 
 ### Exceptions
+
 The following exceptions may be raised when using JobSpy:
+
 * `LinkedInException`
 * `IndeedException`
 * `ZipRecruiterException`
 
 ## Supported Countries for Job Searching
 
-
 ### **LinkedIn**
 
 LinkedIn searches globally & uses only the `location` parameter.
@@ -125,15 +122,15 @@ LinkedIn searches globally & uses only the `location` parameter.
 
 ZipRecruiter searches for jobs in **US/Canada** & uses only the `location` parameter.
 
-
 ### **Indeed**
-Indeed supports most countries, but the `country_indeed` parameter is required. Additionally, use the `location` parameter to narrow down the location, e.g. city & state if necessary.
+
+Indeed supports most countries, but the `country_indeed` parameter is required. Additionally, use the `location`
+parameter to narrow down the location, e.g. city & state if necessary.
 
 You can specify the following countries when searching on Indeed (use the exact name):
 
-
 |                      |              |            |                |
-|------|------|------|------|
+|----------------------|--------------|------------|----------------|
 | Argentina            | Australia    | Austria    | Bahrain        |
 | Belgium              | Brazil       | Canada     | Chile          |
 | China                | Colombia     | Costa Rica | Czech Republic |
@@ -156,12 +153,14 @@ You can specify the following countries when searching on Indeed (use the exact
 ---
 
 **Q: Encountering issues with your queries?**  
-**A:** Try reducing the number of `results_wanted` and/or broadening the filters. If problems persist, [submit an issue](https://github.com/cullenwatson/JobSpy/issues).
+**A:** Try reducing the number of `results_wanted` and/or broadening the filters. If problems
+persist, [submit an issue](https://github.com/cullenwatson/JobSpy/issues).
 
 ---
 
 **Q: Received a response code 429?**  
-**A:** This indicates that you have been blocked by the job board site for sending too many requests. Currently, **LinkedIn** is particularly aggressive with blocking. We recommend:
+**A:** This indicates that you have been blocked by the job board site for sending too many requests. Currently, *
+*LinkedIn** is particularly aggressive with blocking. We recommend:
 
 - Waiting a few seconds between requests.
 - Trying a VPN or proxy to change your IP address.
@@ -170,6 +169,7 @@ You can specify the following countries when searching on Indeed (use the exact
 
 **Q: Experiencing a "Segmentation fault: 11" on macOS Catalina?**  
 **A:** This is due to `tls_client` dependency not supporting your architecture. Solutions and workarounds include:
+
 - Upgrade to a newer version of MacOS
 - Reach out to the maintainers of [tls_client](https://github.com/bogdanfinn/tls-client) for fixes
 

examples/JobSpy_Demo.py

@@ -0,0 +1,31 @@
+from jobspy import scrape_jobs
+import pandas as pd
+
+jobs: pd.DataFrame = scrape_jobs(
+    site_name=["indeed", "linkedin", "zip_recruiter"],
+    search_term="software engineer",
+    location="Dallas, TX",
+    results_wanted=50,  # be wary the higher it is, the more likey you'll get blocked (rotating proxy should work tho)
+    country_indeed="USA",
+    offset=25  # start jobs from an offset (use if search failed and want to continue)
+    # proxy="http://jobspy:5a4vpWtj8EeJ2hoYzk@ca.smartproxy.com:20001",
+)
+
+# formatting for pandas
+pd.set_option("display.max_columns", None)
+pd.set_option("display.max_rows", None)
+pd.set_option("display.width", None)
+pd.set_option("display.max_colwidth", 50)  # set to 0 to see full job url / desc
+
+# 1: output to console
+print(jobs)
+
+# 2: output to .csv
+jobs.to_csv("./jobs.csv", index=False)
+print("outputted to jobs.csv")
+
+# 3: output to .xlsx
+# jobs.to_xlsx('jobs.xlsx', index=False)
+
+# 4: display in Jupyter Notebook (1. pip install jupyter 2. jupyter notebook)
+# display(jobs)

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "python-jobspy"
-version = "1.1.7"
+version = "1.1.13"
 description = "Job scraper for LinkedIn, Indeed & ZipRecruiter"
 authors = ["Zachary Hampton <zachary@zacharysproducts.com>", "Cullen Watson <cullen@cullen.ai>"]
 homepage = "https://github.com/cullenwatson/JobSpy"

src/jobspy/__init__.py

@@ -1,8 +1,7 @@
 import pandas as pd
 import concurrent.futures
 from concurrent.futures import ThreadPoolExecutor
-from typing import List, Tuple, NamedTuple, Dict, Optional
+from typing import Tuple, Optional
-import traceback
 
 from .jobs import JobType, Location
 from .scrapers.indeed import IndeedScraper
@@ -27,7 +26,7 @@ def _map_str_to_site(site_name: str) -> Site:
 
 
 def scrape_jobs(
-    site_name: str | List[str] | Site | List[Site],
+    site_name: str | list[str] | Site | list[Site],
     search_term: str,
     location: str = "",
     distance: int = None,
@@ -38,6 +37,7 @@ def scrape_jobs(
     country_indeed: str = "usa",
     hyperlinks: bool = False,
     proxy: Optional[str] = None,
+    offset: Optional[int] = 0,
 ) -> pd.DataFrame:
     """
     Simultaneously scrapes job data from multiple job sites.
@@ -49,8 +49,8 @@ def scrape_jobs(
             if value_str in job_type.value:
                 return job_type
         raise Exception(f"Invalid job type: {value_str}")
-    job_type = get_enum_from_value(job_type) if job_type else None
 
+    job_type = get_enum_from_value(job_type) if job_type else None
 
     if type(site_name) == str:
         site_type = [_map_str_to_site(site_name)]
@@ -72,6 +72,7 @@ def scrape_jobs(
         job_type=job_type,
         easy_apply=easy_apply,
         results_wanted=results_wanted,
+        offset=offset,
     )
 
     def scrape_site(site: Site) -> Tuple[str, JobResponse]:
@@ -97,8 +98,8 @@ def scrape_jobs(
     site_to_jobs_dict = {}
 
     def worker(site):
-        site_value, scraped_data = scrape_site(site)
+        site_val, scraped_info = scrape_site(site)
-        return site_value, scraped_data
+        return site_val, scraped_info
 
     with ThreadPoolExecutor() as executor:
         future_to_site = {
@@ -109,7 +110,7 @@ def scrape_jobs(
             site_value, scraped_data = future.result()
             site_to_jobs_dict[site_value] = scraped_data
 
-    jobs_dfs: List[pd.DataFrame] = []
+    jobs_dfs: list[pd.DataFrame] = []
 
     for site, job_response in site_to_jobs_dict.items():
         for job in job_response.jobs:
@@ -119,12 +120,14 @@ def scrape_jobs(
             ] = f'<a href="{job_data["job_url"]}">{job_data["job_url"]}</a>'
             job_data["site"] = site
             job_data["company"] = job_data["company_name"]
-            if job_data["job_type"]:
+            job_data["job_type"] = (
-                # Take the first value from the job type tuple
+                ", ".join(job_type.value[0] for job_type in job_data["job_type"])
-                job_data["job_type"] = job_data["job_type"].value[0]
+                if job_data["job_type"]
-            else:
+                else None
-                job_data["job_type"] = None
+            )
-
+            job_data["emails"] = (
+                ", ".join(job_data["emails"]) if job_data["emails"] else None
+            )
             job_data["location"] = Location(**job_data["location"]).display_location()
 
             compensation_obj = job_data.get("compensation")
@@ -148,18 +151,22 @@ def scrape_jobs(
 
     if jobs_dfs:
         jobs_df = pd.concat(jobs_dfs, ignore_index=True)
-        desired_order: List[str] = [
+        desired_order: list[str] = [
+            "job_url_hyper" if hyperlinks else "job_url",
             "site",
             "title",
             "company",
             "location",
-            "date_posted",
             "job_type",
+            "date_posted",
             "interval",
             "min_amount",
             "max_amount",
             "currency",
-            "job_url_hyper" if hyperlinks else "job_url",
+            "is_remote",
+            "num_urgent_words",
+            "benefits",
+            "emails",
             "description",
         ]
         jobs_formatted_df = jobs_df[desired_order]

src/jobspy/jobs/__init__.py

@@ -170,7 +170,7 @@ class CompensationInterval(Enum):
 
 
 class Compensation(BaseModel):
-    interval: CompensationInterval
+    interval: Optional[CompensationInterval] = None
     min_amount: int = None
     max_amount: int = None
     currency: Optional[str] = "USD"
@@ -182,10 +182,15 @@ class JobPost(BaseModel):
     job_url: str
     location: Optional[Location]
 
-    description: Optional[str] = None
+    description: str | None = None
-    job_type: Optional[JobType] = None
+    job_type: list[JobType] | None = None
-    compensation: Optional[Compensation] = None
+    compensation: Compensation | None = None
-    date_posted: Optional[date] = None
+    date_posted: date | None = None
+    benefits: str | None = None
+    emails: list[str] | None = None
+    num_urgent_words: int | None = None
+    is_remote: bool | None = None
+    # company_industry: str | None = None
 
 
 class JobResponse(BaseModel):

src/jobspy/scrapers/__init__.py

@@ -18,6 +18,7 @@ class ScraperInput(BaseModel):
     is_remote: bool = False
     job_type: Optional[JobType] = None
     easy_apply: bool = None  # linkedin
+    offset: int = 0
 
     results_wanted: int = 15
 

src/jobspy/scrapers/indeed/__init__.py

@@ -8,17 +8,15 @@ import re
 import math
 import io
 import json
-import traceback
 from datetime import datetime
-from typing import Optional
 
-import tls_client
 import urllib.parse
 from bs4 import BeautifulSoup
 from bs4.element import Tag
 from concurrent.futures import ThreadPoolExecutor, Future
 
 from ..exceptions import IndeedException
+from ..utils import count_urgent_words, extract_emails_from_text, create_session
 from ...jobs import (
     JobPost,
     Compensation,
@@ -27,14 +25,16 @@ from ...jobs import (
     JobResponse,
     JobType,
 )
-from .. import Scraper, ScraperInput, Site, Country
+from .. import Scraper, ScraperInput, Site
 
 
 class IndeedScraper(Scraper):
-    def __init__(self, proxy: Optional[str] = None):
+    def __init__(self, proxy: str | None = None):
         """
         Initializes IndeedScraper with the Indeed job search url
         """
+        self.url = None
+        self.country = None
         site = Site(Site.INDEED)
         super().__init__(site, proxy=proxy)
 
@@ -42,26 +42,24 @@ class IndeedScraper(Scraper):
         self.seen_urls = set()
 
     def scrape_page(
-        self, scraper_input: ScraperInput, page: int, session: tls_client.Session
+        self, scraper_input: ScraperInput, page: int
     ) -> tuple[list[JobPost], int]:
         """
         Scrapes a page of Indeed for jobs with scraper_input criteria
         :param scraper_input:
         :param page:
-        :param session:
         :return: jobs found on page, total number of jobs found for search
         """
         self.country = scraper_input.country
         domain = self.country.domain_value
         self.url = f"https://{domain}.indeed.com"
-
+        session = create_session(self.proxy)
-        job_list: list[JobPost] = []
 
         params = {
             "q": scraper_input.search_term,
             "l": scraper_input.location,
             "filter": 0,
-            "start": 0 + page * 10,
+            "start": scraper_input.offset + page * 10,
         }
         if scraper_input.distance:
             params["radius"] = scraper_input.distance
@@ -76,10 +74,10 @@ class IndeedScraper(Scraper):
             params["sc"] = "0kf:" + "".join(sc_values) + ";"
         try:
             response = session.get(
-                self.url + "/jobs",
+                f"{self.url}/jobs",
+                headers=self.get_headers(),
                 params=params,
                 allow_redirects=True,
-                proxy=self.proxy,
                 timeout_seconds=10,
             )
             if response.status_code not in range(200, 400):
@@ -107,7 +105,7 @@ class IndeedScraper(Scraper):
         ):
             raise IndeedException("No jobs found.")
 
-        def process_job(job) -> Optional[JobPost]:
+        def process_job(job) -> JobPost | None:
             job_url = f'{self.url}/jobs/viewjob?jk={job["jobkey"]}'
             job_url_client = f'{self.url}/viewjob?jk={job["jobkey"]}'
             if job_url in self.seen_urls:
@@ -126,8 +124,8 @@ class IndeedScraper(Scraper):
                 if interval in CompensationInterval.__members__:
                     compensation = Compensation(
                         interval=CompensationInterval[interval],
-                        min_amount=int(extracted_salary.get("max")),
+                        min_amount=int(extracted_salary.get("min")),
-                        max_amount=int(extracted_salary.get("min")),
+                        max_amount=int(extracted_salary.get("max")),
                         currency=currency,
                     )
 
@@ -136,10 +134,10 @@ class IndeedScraper(Scraper):
             date_posted = datetime.fromtimestamp(timestamp_seconds)
             date_posted = date_posted.strftime("%Y-%m-%d")
 
-            description = self.get_description(job_url, session)
+            description = self.get_description(job_url)
             with io.StringIO(job["snippet"]) as f:
-                soup = BeautifulSoup(f, "html.parser")
+                soup_io = BeautifulSoup(f, "html.parser")
-                li_elements = soup.find_all("li")
+                li_elements = soup_io.find_all("li")
                 if description is None and li_elements:
                     description = " ".join(li.text for li in li_elements)
 
@@ -156,6 +154,11 @@ class IndeedScraper(Scraper):
                 compensation=compensation,
                 date_posted=date_posted,
                 job_url=job_url_client,
+                emails=extract_emails_from_text(description) if description else None,
+                num_urgent_words=count_urgent_words(description)
+                if description
+                else None,
+                is_remote=self.is_remote_job(job),
             )
             return job_post
 
@@ -175,20 +178,16 @@ class IndeedScraper(Scraper):
         :param scraper_input:
         :return: job_response
         """
-        session = tls_client.Session(
-            client_identifier="chrome112", random_tls_extension_order=True
-        )
-
         pages_to_process = (
             math.ceil(scraper_input.results_wanted / self.jobs_per_page) - 1
         )
 
         #: get first page to initialize session
-        job_list, total_results = self.scrape_page(scraper_input, 0, session)
+        job_list, total_results = self.scrape_page(scraper_input, 0)
 
         with ThreadPoolExecutor(max_workers=1) as executor:
             futures: list[Future] = [
-                executor.submit(self.scrape_page, scraper_input, page, session)
+                executor.submit(self.scrape_page, scraper_input, page)
                 for page in range(1, pages_to_process + 1)
             ]
 
@@ -206,21 +205,24 @@ class IndeedScraper(Scraper):
         )
         return job_response
 
-    def get_description(self, job_page_url: str, session: tls_client.Session) -> str:
+    def get_description(self, job_page_url: str) -> str | None:
         """
         Retrieves job description by going to the job page url
         :param job_page_url:
-        :param session:
         :return: description
         """
         parsed_url = urllib.parse.urlparse(job_page_url)
         params = urllib.parse.parse_qs(parsed_url.query)
         jk_value = params.get("jk", [None])[0]
         formatted_url = f"{self.url}/viewjob?jk={jk_value}&spa=1"
+        session = create_session(self.proxy)
 
         try:
             response = session.get(
-                formatted_url, allow_redirects=True, timeout_seconds=5, proxy=self.proxy
+                formatted_url,
+                headers=self.get_headers(),
+                allow_redirects=True,
+                timeout_seconds=5,
             )
         except Exception as e:
             return None
@@ -237,25 +239,32 @@ class IndeedScraper(Scraper):
             return text_content
 
     @staticmethod
-    def get_job_type(job: dict) -> Optional[JobType]:
+    def get_job_type(job: dict) -> list[JobType] | None:
         """
-        Parses the job to get JobTypeIndeed
+        Parses the job to get list of job types
         :param job:
         :return:
         """
+        job_types: list[JobType] = []
         for taxonomy in job["taxonomyAttributes"]:
             if taxonomy["label"] == "job-types":
-                if len(taxonomy["attributes"]) > 0:
+                for i in range(len(taxonomy["attributes"])):
-                    label = taxonomy["attributes"][0].get("label")
+                    label = taxonomy["attributes"][i].get("label")
                     if label:
                         job_type_str = label.replace("-", "").replace(" ", "").lower()
-                        return IndeedScraper.get_enum_from_value(job_type_str)
+                        job_types.append(
-        return None
+                            IndeedScraper.get_enum_from_job_type(job_type_str)
+                        )
+        return job_types
 
     @staticmethod
-    def get_enum_from_value(value_str):
+    def get_enum_from_job_type(job_type_str):
+        """
+        Given a string, returns the corresponding JobType enum member if a match is found.
         for job_type in JobType:
-            if value_str in job_type.value:
+        """
+        for job_type in JobType:
+            if job_type_str in job_type.value:
                 return job_type
         return None
 
@@ -267,7 +276,7 @@ class IndeedScraper(Scraper):
         :return: jobs
         """
 
-        def find_mosaic_script() -> Optional[Tag]:
+        def find_mosaic_script() -> Tag | None:
             """
             Finds jobcards script tag
             :return: script_tag
@@ -317,3 +326,30 @@ class IndeedScraper(Scraper):
             data = json.loads(json_str)
             total_num_jobs = int(data["searchTitleBarModel"]["totalNumResults"])
         return total_num_jobs
+
+    @staticmethod
+    def get_headers():
+        return {
+            "authority": "www.indeed.com",
+            "accept": "*/*",
+            "accept-language": "en-US,en;q=0.9",
+            "referer": "https://www.indeed.com/viewjob?jk=fe6182337d72c7b1&tk=1hcbfcmd0k62t802&from=serp&vjs=3&advn=8132938064490989&adid=408692607&ad=-6NYlbfkN0A3Osc99MJFDKjquSk4WOGT28ALb_ad4QMtrHreCb9ICg6MiSVy9oDAp3evvOrI7Q-O9qOtQTg1EPbthP9xWtBN2cOuVeHQijxHjHpJC65TjDtftH3AXeINjBvAyDrE8DrRaAXl8LD3Fs1e_xuDHQIssdZ2Mlzcav8m5jHrA0fA64ZaqJV77myldaNlM7-qyQpy4AsJQfvg9iR2MY7qeC5_FnjIgjKIy_lNi9OPMOjGRWXA94CuvC7zC6WeiJmBQCHISl8IOBxf7EdJZlYdtzgae3593TFxbkd6LUwbijAfjax39aAuuCXy3s9C4YgcEP3TwEFGQoTpYu9Pmle-Ae1tHGPgsjxwXkgMm7Cz5mBBdJioglRCj9pssn-1u1blHZM4uL1nK9p1Y6HoFgPUU9xvKQTHjKGdH8d4y4ETyCMoNF4hAIyUaysCKdJKitC8PXoYaWhDqFtSMR4Jys8UPqUV&xkcb=SoDD-_M3JLQfWnQTDh0LbzkdCdPP&xpse=SoBa6_I3JLW9FlWZlB0PbzkdCdPP&sjdu=i6xVERweJM_pVUvgf-MzuaunBTY7G71J5eEX6t4DrDs5EMPQdODrX7Nn-WIPMezoqr5wA_l7Of-3CtoiUawcHw",
+            "sec-ch-ua": '"Google Chrome";v="119", "Chromium";v="119", "Not?A_Brand";v="24"',
+            "sec-ch-ua-mobile": "?0",
+            "sec-ch-ua-platform": '"Windows"',
+            "sec-fetch-dest": "empty",
+            "sec-fetch-mode": "cors",
+            "sec-fetch-site": "same-origin",
+            "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36",
+        }
+
+    @staticmethod
+    def is_remote_job(job: dict) -> bool:
+        """
+        :param job:
+        :return: bool
+        """
+        for taxonomy in job.get("taxonomyAttributes", []):
+            if taxonomy["label"] == "remote" and len(taxonomy["attributes"]) > 0:
+                return True
+        return False

src/jobspy/scrapers/linkedin/__init__.py

@@ -4,33 +4,39 @@ jobspy.scrapers.linkedin
 
 This module contains routines to scrape LinkedIn.
 """
-from typing import Optional, Tuple
+from typing import Optional
 from datetime import datetime
-import traceback
 
 import requests
-from requests.exceptions import Timeout, ProxyError
+import time
+import re
+from requests.exceptions import ProxyError
+from concurrent.futures import ThreadPoolExecutor, as_completed
 from bs4 import BeautifulSoup
 from bs4.element import Tag
+from threading import Lock
 
 from .. import Scraper, ScraperInput, Site
+from ..utils import count_urgent_words, extract_emails_from_text
 from ..exceptions import LinkedInException
 from ...jobs import (
     JobPost,
     Location,
     JobResponse,
     JobType,
-    Compensation,
-    CompensationInterval,
 )
 
 
 class LinkedInScraper(Scraper):
+    MAX_RETRIES = 3
+    DELAY = 10
+
     def __init__(self, proxy: Optional[str] = None):
         """
         Initializes LinkedInScraper with the LinkedIn job search url
         """
         site = Site(Site.LINKEDIN)
+        self.country = "worldwide"
         self.url = "https://www.linkedin.com"
         super().__init__(site, proxy=proxy)
 
@@ -40,12 +46,12 @@ class LinkedInScraper(Scraper):
         :param scraper_input:
         :return: job_response
         """
-        self.country = "worldwide"
         job_list: list[JobPost] = []
         seen_urls = set()
-        page, processed_jobs, job_count = 0, 0, 0
+        url_lock = Lock()
+        page = scraper_input.offset // 25 + 25 if scraper_input.offset else 0
 
-        def job_type_code(job_type):
+        def job_type_code(job_type_enum):
             mapping = {
                 JobType.FULL_TIME: "F",
                 JobType.PART_TIME: "P",
@@ -54,10 +60,9 @@ class LinkedInScraper(Scraper):
                 JobType.TEMPORARY: "T",
             }
 
-            return mapping.get(job_type, "")
+            return mapping.get(job_type_enum, "")
 
-        with requests.Session() as session:
+        while len(job_list) < scraper_input.results_wanted and page < 1000:
-            while len(job_list) < scraper_input.results_wanted:
             params = {
                 "keywords": scraper_input.search_term,
                 "location": scraper_input.location,
@@ -66,105 +71,127 @@ class LinkedInScraper(Scraper):
                 "f_JT": job_type_code(scraper_input.job_type)
                 if scraper_input.job_type
                 else None,
-                    "pageNum": page,
+                "pageNum": 0,
+                page: page + scraper_input.offset,
                 "f_AL": "true" if scraper_input.easy_apply else None,
             }
 
             params = {k: v for k, v in params.items() if v is not None}
+
+            params = {k: v for k, v in params.items() if v is not None}
+            retries = 0
+            while retries < self.MAX_RETRIES:
                 try:
-                    response = session.get(
+                    response = requests.get(
-                        f"{self.url}/jobs/search",
+                        f"{self.url}/jobs-guest/jobs/api/seeMoreJobPostings/search?",
                         params=params,
                         allow_redirects=True,
                         proxies=self.proxy,
                         timeout=10,
                     )
                     response.raise_for_status()
+
+                    break
                 except requests.HTTPError as e:
+                    if hasattr(e, "response") and e.response is not None:
+                        if e.response.status_code == 429:
+                            time.sleep(self.DELAY)
+                            retries += 1
+                            continue
+                        else:
                             raise LinkedInException(
-                        f"bad response status code: {response.status_code}"
+                                f"bad response status code: {e.response.status_code}"
                             )
+                    else:
+                        raise
                 except ProxyError as e:
                     raise LinkedInException("bad proxy")
-                except (ProxyError, Exception) as e:
+                except Exception as e:
                     raise LinkedInException(str(e))
+            else:
+                # Raise an exception if the maximum number of retries is reached
+                raise LinkedInException(
+                    "Max retries reached, failed to get a valid response"
+                )
 
             soup = BeautifulSoup(response.text, "html.parser")
 
-                if page == 0:
+            with ThreadPoolExecutor(max_workers=5) as executor:
-                    job_count_text = soup.find(
+                futures = []
-                        "span", class_="results-context-header__job-count"
+                for job_card in soup.find_all("div", class_="base-search-card"):
-                    ).text
+                    job_url = None
-                    job_count = int("".join(filter(str.isdigit, job_count_text)))
+                    href_tag = job_card.find("a", class_="base-card__full-link")
-
+                    if href_tag and "href" in href_tag.attrs:
-                for job_card in soup.find_all(
+                        href = href_tag.attrs["href"].split("?")[0]
-                    "div",
+                        job_id = href.split("-")[-1]
-                    class_="base-card relative w-full hover:no-underline focus:no-underline base-card--link base-search-card base-search-card--link job-search-card",
-                ):
-                    processed_jobs += 1
-                    data_entity_urn = job_card.get("data-entity-urn", "")
-                    job_id = (
-                        data_entity_urn.split(":")[-1] if data_entity_urn else "N/A"
-                    )
                         job_url = f"{self.url}/jobs/view/{job_id}"
+
+                    with url_lock:
                         if job_url in seen_urls:
                             continue
                         seen_urls.add(job_url)
-                    job_info = job_card.find("div", class_="base-search-card__info")
-                    if job_info is None:
-                        continue
-                    title_tag = job_info.find("h3", class_="base-search-card__title")
-                    title = title_tag.text.strip() if title_tag else "N/A"
 
-                    company_tag = job_info.find("a", class_="hidden-nested-link")
+                    futures.append(executor.submit(self.process_job, job_card, job_url))
-                    company = company_tag.text.strip() if company_tag else "N/A"
 
-                    metadata_card = job_info.find(
+                for future in as_completed(futures):
-                        "div", class_="base-search-card__metadata"
+                    try:
+                        job_post = future.result()
+                        if job_post:
+                            job_list.append(job_post)
+                    except Exception as e:
+                        raise LinkedInException(
+                            "Exception occurred while processing jobs"
                         )
-                    location: Location = self.get_location(metadata_card)
+            page += 25
 
-                    datetime_tag = metadata_card.find(
+        job_list = job_list[: scraper_input.results_wanted]
-                        "time", class_="job-search-card__listdate"
+        return JobResponse(jobs=job_list)
+
+    def process_job(self, job_card: Tag, job_url: str) -> Optional[JobPost]:
+        title_tag = job_card.find("span", class_="sr-only")
+        title = title_tag.get_text(strip=True) if title_tag else "N/A"
+
+        company_tag = job_card.find("h4", class_="base-search-card__subtitle")
+        company_a_tag = company_tag.find("a") if company_tag else None
+        company = company_a_tag.get_text(strip=True) if company_a_tag else "N/A"
+
+        metadata_card = job_card.find("div", class_="base-search-card__metadata")
+        location = self.get_location(metadata_card)
+
+        datetime_tag = (
+            metadata_card.find("time", class_="job-search-card__listdate")
+            if metadata_card
+            else None
         )
-                    description, job_type = self.get_description(job_url)
+        date_posted = None
-                    if datetime_tag:
+        if datetime_tag and "datetime" in datetime_tag.attrs:
             datetime_str = datetime_tag["datetime"]
             try:
                 date_posted = datetime.strptime(datetime_str, "%Y-%m-%d")
             except Exception as e:
                 date_posted = None
-                    else:
+        benefits_tag = job_card.find("span", class_="result-benefits__text")
-                        date_posted = None
+        benefits = " ".join(benefits_tag.get_text().split()) if benefits_tag else None
 
-                    job_post = JobPost(
+        description, job_type = self.get_job_description(job_url)
+
+        return JobPost(
             title=title,
             description=description,
             company_name=company,
             location=location,
             date_posted=date_posted,
             job_url=job_url,
+            # job_type=[JobType.FULL_TIME],
             job_type=job_type,
-                        compensation=Compensation(
+            benefits=benefits,
-                            interval=CompensationInterval.YEARLY, currency=None
+            emails=extract_emails_from_text(description) if description else None,
-                        ),
+            num_urgent_words=count_urgent_words(description) if description else None,
         )
-                    job_list.append(job_post)
-                    if processed_jobs >= job_count:
-                        break
-                    if len(job_list) >= scraper_input.results_wanted:
-                        break
-                if processed_jobs >= job_count:
-                    break
-                if len(job_list) >= scraper_input.results_wanted:
-                    break
 
-                page += 1
+    def get_job_description(
-
+        self, job_page_url: str
-        job_list = job_list[: scraper_input.results_wanted]
+    ) -> tuple[None, None] | tuple[str | None, tuple[str | None, JobType | None]]:
-        return JobResponse(jobs=job_list)
-
-    def get_description(self, job_page_url: str) -> Optional[str]:
         """
         Retrieves job description by going to the job page url
         :param job_page_url:
@@ -181,19 +208,19 @@ class LinkedInScraper(Scraper):
             "div", class_=lambda x: x and "show-more-less-html__markup" in x
         )
 
-        text_content = None
+        description = None
         if div_content:
-            text_content = " ".join(div_content.get_text().split()).strip()
+            description = " ".join(div_content.get_text().split()).strip()
 
         def get_job_type(
-            soup: BeautifulSoup,
+            soup_job_type: BeautifulSoup,
-        ) -> Tuple[Optional[str], Optional[JobType]]:
+        ) -> list[JobType] | None:
             """
             Gets the job type from job page
-            :param soup:
+            :param soup_job_type:
             :return: JobType
             """
-            h3_tag = soup.find(
+            h3_tag = soup_job_type.find(
                 "h3",
                 class_="description__job-criteria-subheader",
                 string=lambda text: "Employment type" in text,
@@ -212,13 +239,13 @@ class LinkedInScraper(Scraper):
 
             return LinkedInScraper.get_enum_from_value(employment_type)
 
-        return text_content, get_job_type(soup)
+        return description, get_job_type(soup)
 
     @staticmethod
     def get_enum_from_value(value_str):
         for job_type in JobType:
             if value_str in job_type.value:
-                return job_type
+                return [job_type]
         return None
 
     def get_location(self, metadata_card: Optional[Tag]) -> Location:

src/jobspy/scrapers/utils.py

@@ -0,0 +1,44 @@
+import re
+import tls_client
+
+
+def count_urgent_words(description: str) -> int:
+    """
+    Count the number of urgent words or phrases in a job description.
+    """
+    urgent_patterns = re.compile(
+        r"\burgen(t|cy)|\bimmediate(ly)?\b|start asap|\bhiring (now|immediate(ly)?)\b",
+        re.IGNORECASE,
+    )
+    matches = re.findall(urgent_patterns, description)
+    count = len(matches)
+
+    return count
+
+
+def extract_emails_from_text(text: str) -> list[str] | None:
+    if not text:
+        return None
+    email_regex = re.compile(r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}")
+    return email_regex.findall(text)
+
+
+def create_session(proxy: str | None = None):
+    """
+    Creates a tls client session
+
+    :return: A session object with or without proxies.
+    """
+    session = tls_client.Session(
+        client_identifier="chrome112",
+        random_tls_extension_order=True,
+    )
+    session.proxies = proxy
+    # TODO multiple proxies
+    # if self.proxies:
+    #     session.proxies = {
+    #         "http": random.choice(self.proxies),
+    #         "https": random.choice(self.proxies),
+    #     }
+
+    return session

src/jobspy/scrapers/ziprecruiter/__init__.py

@@ -7,12 +7,10 @@ This module contains routines to scrape ZipRecruiter.
 import math
 import json
 import re
-import traceback
+from datetime import datetime, date
-from datetime import datetime
+from typing import Optional, Tuple, Any
-from typing import Optional, Tuple
+from urllib.parse import urlparse, parse_qs, urlunparse
-from urllib.parse import urlparse, parse_qs
 
-import tls_client
 import requests
 from bs4 import BeautifulSoup
 from bs4.element import Tag
@@ -20,6 +18,7 @@ from concurrent.futures import ThreadPoolExecutor, Future
 
 from .. import Scraper, ScraperInput, Site
 from ..exceptions import ZipRecruiterException
+from ..utils import count_urgent_words, extract_emails_from_text, create_session
 from ...jobs import (
     JobPost,
     Compensation,
@@ -42,28 +41,23 @@ class ZipRecruiterScraper(Scraper):
 
         self.jobs_per_page = 20
         self.seen_urls = set()
-        self.session = tls_client.Session(
-            client_identifier="chrome112", random_tls_extension_order=True
-        )
 
     def find_jobs_in_page(
         self, scraper_input: ScraperInput, page: int
-    ) -> tuple[list[JobPost], int | None]:
+    ) -> list[JobPost]:
         """
         Scrapes a page of ZipRecruiter for jobs with scraper_input criteria
         :param scraper_input:
         :param page:
-        :param session:
+        :return: jobs found on page
-        :return: jobs found on page, total number of jobs found for search
         """
-        job_list: list[JobPost] = []
+        session = create_session(self.proxy)
         try:
-            response = self.session.get(
+            response = session.get(
-                self.url + "/jobs-search",
+                f"{self.url}/jobs-search",
-                headers=ZipRecruiterScraper.headers(),
+                headers=self.headers(),
-                params=ZipRecruiterScraper.add_params(scraper_input, page),
+                params=self.add_params(scraper_input, page),
                 allow_redirects=True,
-                proxy=self.proxy,
                 timeout_seconds=10,
             )
             if response.status_code != 200:
@@ -92,8 +86,6 @@ class ZipRecruiterScraper(Scraper):
             page_variant = "html_1"
             jobs_list = soup.find_all("li", {"class": "job-listing"})
             # print('type 1 html', len(jobs_list))
-            # with open("zip_method_8.html", "w") as f:
-            #     f.write(soup.prettify())
 
         with ThreadPoolExecutor(max_workers=10) as executor:
             if page_variant == "javascript":
@@ -119,8 +111,13 @@ class ZipRecruiterScraper(Scraper):
         :param scraper_input:
         :return: job_response
         """
+        start_page = (
+            (scraper_input.offset // self.jobs_per_page) + 1
+            if scraper_input.offset
+            else 1
+        )
         #: get first page to initialize session
-        job_list: list[JobPost] = self.find_jobs_in_page(scraper_input, 1)
+        job_list: list[JobPost] = self.find_jobs_in_page(scraper_input, start_page)
         pages_to_process = max(
             3, math.ceil(scraper_input.results_wanted / self.jobs_per_page)
         )
@@ -128,7 +125,7 @@ class ZipRecruiterScraper(Scraper):
         with ThreadPoolExecutor(max_workers=10) as executor:
             futures: list[Future] = [
                 executor.submit(self.find_jobs_in_page, scraper_input, page)
-                for page in range(2, pages_to_process + 1)
+                for page in range(start_page + 1, start_page + pages_to_process + 2)
             ]
 
             for future in futures:
@@ -139,92 +136,13 @@ class ZipRecruiterScraper(Scraper):
         job_list = job_list[: scraper_input.results_wanted]
         return JobResponse(jobs=job_list)
 
-    def process_job_html_1(self, job: Tag) -> Optional[JobPost]:
-        """
-        Parses a job from the job content tag
-        :param job: BeautifulSoup Tag for one job post
-        :return JobPost
-        """
-        job_url = job.find("a", {"class": "job_link"})["href"]
-        if job_url in self.seen_urls:
-            return None
-
-        title = job.find("h2", {"class": "title"}).text
-        company = job.find("a", {"class": "company_name"}).text.strip()
-
-        description, updated_job_url = self.get_description(job_url)
-        job_url = updated_job_url if updated_job_url else job_url
-        if description is None:
-            description = job.find("p", {"class": "job_snippet"}).text.strip()
-
-        job_type_element = job.find("li", {"class": "perk_item perk_type"})
-        job_type = None
-        if job_type_element:
-            job_type_text = (
-                job_type_element.text.strip().lower().replace("_", "").replace(" ", "")
-            )
-            job_type = ZipRecruiterScraper.get_job_type_enum(job_type_text)
-
-        date_posted = ZipRecruiterScraper.get_date_posted(job)
-
-        job_post = JobPost(
-            title=title,
-            description=description,
-            company_name=company,
-            location=ZipRecruiterScraper.get_location(job),
-            job_type=job_type,
-            compensation=ZipRecruiterScraper.get_compensation(job),
-            date_posted=date_posted,
-            job_url=job_url,
-        )
-        return job_post
-
-    def process_job_html_2(self, job: Tag) -> Optional[JobPost]:
-        """
-        Parses a job from the job content tag for a second variat of HTML that ZR uses
-        :param job: BeautifulSoup Tag for one job post
-        :return JobPost
-        """
-        job_url = job.find("a", class_="job_link")["href"]
-        title = job.find("h2", class_="title").text
-        company = job.find("a", class_="company_name").text.strip()
-
-        description, updated_job_url = self.get_description(job_url)
-        job_url = updated_job_url if updated_job_url else job_url
-        if description is None:
-            description = job.find("p", class_="job_snippet").get_text().strip()
-
-        job_type_text = job.find("li", class_="perk_item perk_type")
-        job_type = None
-        if job_type_text:
-            job_type_text = (
-                job_type_text.get_text()
-                .strip()
-                .lower()
-                .replace("-", "")
-                .replace(" ", "")
-            )
-            job_type = ZipRecruiterScraper.get_job_type_enum(job_type_text)
-        date_posted = ZipRecruiterScraper.get_date_posted(job)
-
-        job_post = JobPost(
-            title=title,
-            description=description,
-            company_name=company,
-            location=ZipRecruiterScraper.get_location(job),
-            job_type=job_type,
-            compensation=ZipRecruiterScraper.get_compensation(job),
-            date_posted=date_posted,
-            job_url=job_url,
-        )
-        return job_post
-
     def process_job_javascript(self, job: dict) -> JobPost:
+        """the most common type of jobs page on ZR"""
         title = job.get("Title")
         job_url = job.get("JobURL")
 
         description, updated_job_url = self.get_description(job_url)
-        job_url = updated_job_url if updated_job_url else job_url
+        # job_url = updated_job_url if updated_job_url else job_url
         if description is None:
             description = BeautifulSoup(
                 job.get("Snippet", "").strip(), "html.parser"
@@ -272,42 +190,129 @@ class ZipRecruiterScraper(Scraper):
             date_posted = date_posted_obj.date()
         else:
             date_posted = date.today()
-        job_url = job.get("JobURL")
 
         return JobPost(
             title=title,
-            description=description,
             company_name=company,
             location=location,
             job_type=job_type,
             compensation=compensation,
             date_posted=date_posted,
             job_url=job_url,
+            description=description,
+            emails=extract_emails_from_text(description) if description else None,
+            num_urgent_words=count_urgent_words(description) if description else None,
+        )
+
+    def process_job_html_2(self, job: Tag) -> Optional[JobPost]:
+        """
+        second most common type of jobs page on ZR after process_job_javascript()
+        Parses a job from the job content tag for a second variat of HTML that ZR uses
+        :param job: BeautifulSoup Tag for one job post
+        :return JobPost
+        """
+        job_url = job.find("a", class_="job_link")["href"]
+        title = job.find("h2", class_="title").text
+        company = job.find("a", class_="company_name").text.strip()
+
+        description, updated_job_url = self.get_description(job_url)
+        # job_url = updated_job_url if updated_job_url else job_url
+        if description is None:
+            description = job.find("p", class_="job_snippet").get_text().strip()
+
+        job_type_text = job.find("li", class_="perk_item perk_type")
+        job_type = None
+        if job_type_text:
+            job_type_text = (
+                job_type_text.get_text()
+                .strip()
+                .lower()
+                .replace("-", "")
+                .replace(" ", "")
+            )
+            job_type = ZipRecruiterScraper.get_job_type_enum(job_type_text)
+        date_posted = ZipRecruiterScraper.get_date_posted(job)
+
+        job_post = JobPost(
+            title=title,
+            company_name=company,
+            location=ZipRecruiterScraper.get_location(job),
+            job_type=job_type,
+            compensation=ZipRecruiterScraper.get_compensation(job),
+            date_posted=date_posted,
+            job_url=job_url,
+            description=description,
+            emails=extract_emails_from_text(description) if description else None,
+            num_urgent_words=count_urgent_words(description) if description else None,
+        )
+        return job_post
+
+    def process_job_html_1(self, job: Tag) -> Optional[JobPost]:
+        """
+        TODO this method isnt finished due to not encountering this type of html often
+        least common type of jobs page on ZR (rarely found)
+        Parses a job from the job content tag
+        :param job: BeautifulSoup Tag for one job post
+        :return JobPost
+        """
+        job_url = job.find("a", {"class": "job_link"})["href"]
+        # job_url = self.cleanurl(job.find("a", {"class": "job_link"})["href"])
+        if job_url in self.seen_urls:
+            return None
+
+        title = job.find("h2", {"class": "title"}).text
+        company = job.find("a", {"class": "company_name"}).text.strip()
+
+        description, _ = self.get_description(job_url)
+        # job_url = updated_job_url if updated_job_url else job_url
+        # get description from jobs listing page if get_description from the specific job page fails
+        if description is None:
+            description = job.find("p", {"class": "job_snippet"}).text.strip()
+
+        job_type_element = job.find("li", {"class": "perk_item perk_type"})
+        job_type = None
+        if job_type_element:
+            job_type_text = (
+                job_type_element.text.strip().lower().replace("_", "").replace(" ", "")
+            )
+            job_type = ZipRecruiterScraper.get_job_type_enum(job_type_text)
+
+        date_posted = ZipRecruiterScraper.get_date_posted(job)
+
+        job_post = JobPost(
+            title=title,
+            description=description,
+            company_name=company,
+            location=ZipRecruiterScraper.get_location(job),
+            job_type=job_type,
+            compensation=ZipRecruiterScraper.get_compensation(job),
+            date_posted=date_posted,
+            job_url=job_url,
+            emails=extract_emails_from_text(description),
+            num_urgent_words=count_urgent_words(description),
         )
         return job_post
 
     @staticmethod
-    def get_job_type_enum(job_type_str: str) -> Optional[JobType]:
+    def get_job_type_enum(job_type_str: str) -> list[JobType] | None:
         for job_type in JobType:
             if job_type_str in job_type.value:
-                a = True
+                return [job_type]
-                return job_type
         return None
 
-    def get_description(self, job_page_url: str) -> Tuple[Optional[str], Optional[str]]:
+    def get_description(self, job_page_url: str) -> Tuple[str | None, str | None]:
         """
         Retrieves job description by going to the job page url
         :param job_page_url:
-        :param session:
         :return: description or None, response url
         """
         try:
-            response = requests.get(
+            session = create_session(self.proxy)
+            response = session.get(
                 job_page_url,
-                headers=ZipRecruiterScraper.headers(),
+                headers=self.headers(),
                 allow_redirects=True,
-                timeout=5,
+                timeout_seconds=5,
-                proxies=self.proxy,
             )
             if response.status_code not in range(200, 400):
                 return None, None
@@ -323,7 +328,7 @@ class ZipRecruiterScraper(Scraper):
         return None, response.url
 
     @staticmethod
-    def add_params(scraper_input, page) -> Optional[str]:
+    def add_params(scraper_input, page) -> dict[str, str | Any]:
         params = {
             "search": scraper_input.search_term,
             "location": scraper_input.location,
@@ -368,7 +373,7 @@ class ZipRecruiterScraper(Scraper):
         return CompensationInterval(interval_str)
 
     @staticmethod
-    def get_date_posted(job: BeautifulSoup) -> Optional[datetime.date]:
+    def get_date_posted(job: Tag) -> Optional[datetime.date]:
         """
         Extracts the date a job was posted
         :param job
@@ -394,7 +399,7 @@ class ZipRecruiterScraper(Scraper):
         return None
 
     @staticmethod
-    def get_compensation(job: BeautifulSoup) -> Optional[Compensation]:
+    def get_compensation(job: Tag) -> Optional[Compensation]:
         """
         Parses the compensation tag from the job BeautifulSoup object
         :param job
@@ -435,7 +440,7 @@ class ZipRecruiterScraper(Scraper):
         return create_compensation_object(pay)
 
     @staticmethod
-    def get_location(job: BeautifulSoup) -> Location:
+    def get_location(job: Tag) -> Location:
         """
         Extracts the job location from BeatifulSoup object
         :param job:
@@ -462,3 +467,9 @@ class ZipRecruiterScraper(Scraper):
         return {
             "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
         }
+
+    # @staticmethod
+    # def cleanurl(url) -> str:
+    #     parsed_url = urlparse(url)
+    #
+    #     return urlunparse((parsed_url.scheme, parsed_url.netloc, parsed_url.path, parsed_url.params, '', ''))

src/tests/test_all.py

@@ -1,4 +1,5 @@
 from ..jobspy import scrape_jobs
+import pandas as pd
 
 
 def test_all():
@@ -7,4 +8,7 @@ def test_all():
         search_term="software engineer",
         results_wanted=5,
     )
-    assert result is not None and result.errors.empty is True
+
+    assert (
+        isinstance(result, pd.DataFrame) and not result.empty
+    ), "Result should be a non-empty DataFrame"

src/tests/test_indeed.py

@@ -1,4 +1,5 @@
 from ..jobspy import scrape_jobs
+import pandas as pd
 
 
 def test_indeed():
@@ -6,4 +7,6 @@ def test_indeed():
         site_name="indeed",
         search_term="software engineer",
     )
-    assert result is not None and result.errors.empty is True
+    assert (
+        isinstance(result, pd.DataFrame) and not result.empty
+    ), "Result should be a non-empty DataFrame"

src/tests/test_linkedin.py

@@ -1,4 +1,5 @@
 from ..jobspy import scrape_jobs
+import pandas as pd
 
 
 def test_linkedin():
@@ -6,4 +7,6 @@ def test_linkedin():
         site_name="linkedin",
         search_term="software engineer",
     )
-    assert result is not None and result.errors.empty is True
+    assert (
+        isinstance(result, pd.DataFrame) and not result.empty
+    ), "Result should be a non-empty DataFrame"

src/tests/test_ziprecruiter.py

@@ -1,4 +1,5 @@
 from ..jobspy import scrape_jobs
+import pandas as pd
 
 
 def test_ziprecruiter():
@@ -7,4 +8,6 @@ def test_ziprecruiter():
         search_term="software engineer",
     )
 
-    assert result is not None and result.errors.empty is True
+    assert (
+        isinstance(result, pd.DataFrame) and not result.empty
+    ), "Result should be a non-empty DataFrame"