pass test

indeed:remove tpe
2026-03-05 03:54:31 -08:00 · 2024-10-19 18:01:02 -05:00 · 2024-10-19 17:58:26 -05:00 · 2024-10-19 17:55:36 -05:00 · 2024-10-19 16:56:38 -05:00 · 2024-10-08 17:49:06 -05:00
32 changed files with 2330 additions and 1939 deletions
--- a/.github/workflows/python-test.yml
+++ b/.github/workflows/python-test.yml
@@ -0,0 +1,22 @@
+name: Python Tests
+
+on:
+  pull_request:
+    branches:
+      - main
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up Python
+        uses: actions/setup-python@v2
+        with:
+          python-version: '3.8'
+      - name: Install dependencies
+        run: |
+          pip install poetry
+          poetry install
+      - name: Run tests
+        run: poetry run pytest tests/test_all.py
--- a/README.md
+++ b/README.md
@@ -11,10 +11,7 @@ work with us.*

 - Scrapes job postings from **LinkedIn**, **Indeed**, **Glassdoor**, & **ZipRecruiter** simultaneously
 - Aggregates the job postings in a Pandas DataFrame
- Proxy support
-
-[Video Guide for JobSpy](https://www.youtube.com/watch?v=RuP1HrAZnxs&pp=ygUgam9icyBzY3JhcGVyIGJvdCBsaW5rZWRpbiBpbmRlZWQ%3D) -
-Updated for release v1.1.3
+- Proxies support

 ![jobspy](https://github.com/cullenwatson/JobSpy/assets/78247585/ec7ef355-05f6-4fd3-8161-a817e31c5c57)

@@ -38,17 +35,21 @@ jobs = scrape_jobs(
    location="Dallas, TX",
    results_wanted=20,
    hours_old=72, # (only Linkedin/Indeed is hour specific, others round up to days old)
-    country_indeed='USA'  # only needed for indeed / glassdoor
+    country_indeed='USA',  # only needed for indeed / glassdoor
+    
+    # linkedin_fetch_description=True # get more info such as full description, direct job url for linkedin (slower)
+    # proxies=["208.195.175.46:65095", "208.195.175.45:65095", "localhost"],
+    
 )
 print(f"Found {len(jobs)} jobs")
 print(jobs.head())
-jobs.to_csv("jobs.csv", quoting=csv.QUOTE_NONNUMERIC, escapechar="\\", index=False) # to_xlsx
+jobs.to_csv("jobs.csv", quoting=csv.QUOTE_NONNUMERIC, escapechar="\\", index=False) # to_excel
 ```

 ### Output

 ```
-SITE           TITLE                             COMPANY_NAME      CITY          STATE  JOB_TYPE  INTERVAL  MIN_AMOUNT  MAX_AMOUNT  JOB_URL                                            DESCRIPTION
+SITE           TITLE                             COMPANY           CITY          STATE  JOB_TYPE  INTERVAL  MIN_AMOUNT  MAX_AMOUNT  JOB_URL                                            DESCRIPTION
 indeed         Software Engineer                 AMERICAN SYSTEMS  Arlington     VA     None      yearly    200000      150000      https://www.indeed.com/viewjob?jk=5e409e577046...  THIS POSITION COMES WITH A 10K SIGNING BONUS!...
 indeed         Senior Software Engineer          TherapyNotes.com  Philadelphia  PA     fulltime  yearly    135000      110000      https://www.indeed.com/viewjob?jk=da39574a40cb...  About Us TherapyNotes is the national leader i...
 linkedin       Software Engineer - Early Career  Lockheed Martin   Sunnyvale     CA     fulltime  yearly    None        None        https://www.linkedin.com/jobs/view/3693012711      Description:By bringing together people that u...
@@ -60,66 +61,121 @@ zip_recruiter Software Developer                 TEKsystems        Phoenix
 ### Parameters for `scrape_jobs()`

 ```plaintext
-Required
-├── site_type (List[enum]): linkedin, zip_recruiter, indeed, glassdoor
-└── search_term (str)
 Optional
+├── site_name (list|str): 
+|    linkedin, zip_recruiter, indeed, glassdoor 
+|    (default is all four)
+│
+├── search_term (str)
+│
 ├── location (str)
-├── distance (int): in miles, default 50
-├── job_type (enum): fulltime, parttime, internship, contract
-├── proxy (str): in format 'http://user:pass@host:port'
+│
+├── distance (int): 
+|    in miles, default 50
+│
+├── job_type (str): 
+|    fulltime, parttime, internship, contract
+│
+├── proxies (list): 
+|    in format ['user:pass@host:port', 'localhost']
+|    each job board scraper will round robin through the proxies
+|
+├── ca_cert (str)
+|    path to CA Certificate file for proxies
+│
 ├── is_remote (bool)
-├── linkedin_fetch_description (bool): fetches full description for LinkedIn (slower)
-├── results_wanted (int): number of job results to retrieve for each site specified in 'site_type'
-├── easy_apply (bool): filters for jobs that are hosted on the job board site (not supported on Indeed)
-├── linkedin_company_ids (list[int): searches for linkedin jobs with specific company ids
-├── description_format (enum): markdown, html (format type of the job descriptions)
-├── country_indeed (enum): filters the country on Indeed (see below for correct spelling)
-├── offset (num): starts the search from an offset (e.g. 25 will start the search from the 25th result)
-├── hours_old (int): filters jobs by the number of hours since the job was posted (ZipRecruiter and Glassdoor round up to next day. If you use this on Indeed, it will not filter by job_type or is_remote)
+│
+├── results_wanted (int): 
+|    number of job results to retrieve for each site specified in 'site_name'
+│
+├── easy_apply (bool): 
+|    filters for jobs that are hosted on the job board site
+│
+├── description_format (str): 
+|    markdown, html (Format type of the job descriptions. Default is markdown.)
+│
+├── offset (int): 
+|    starts the search from an offset (e.g. 25 will start the search from the 25th result)
+│
+├── hours_old (int): 
+|    filters jobs by the number of hours since the job was posted 
+|    (ZipRecruiter and Glassdoor round up to next day.)
+│
+├── verbose (int) {0, 1, 2}: 
+|    Controls the verbosity of the runtime printouts 
+|    (0 prints only errors, 1 is errors+warnings, 2 is all logs. Default is 2.)
+
+├── linkedin_fetch_description (bool): 
+|    fetches full description and direct job url for LinkedIn (Increases requests by O(n))
+│
+├── linkedin_company_ids (list[int]): 
+|    searches for linkedin jobs with specific company ids
+|
+├── country_indeed (str): 
+|    filters the country on Indeed & Glassdoor (see below for correct spelling)
+|
+├── enforce_annual_salary (bool): 
+|    converts wages to annual salary
 ```

+```
+├── Indeed limitations:
+|    Only one from this list can be used in a search:
+|    - hours_old
+|    - job_type & is_remote
+|    - easy_apply
+│
+└── LinkedIn limitations:
+|    Only one from this list can be used in a search:
+|    - hours_old
+|    - easy_apply
+```
+
+
 ### JobPost Schema

 ```plaintext
 JobPost
-├── title (str)
-├── company (str)
-├── company_url (str)
-├── job_url (str)
-├── location (object)
-│   ├── country (str)
-│   ├── city (str)
-│   ├── state (str)
-├── description (str)
-├── job_type (str): fulltime, parttime, internship, contract
-├── compensation (object)
-│   ├── interval (str): yearly, monthly, weekly, daily, hourly
-│   ├── min_amount (int)
-│   ├── max_amount (int)
-│   └── currency (enum)
-└── date_posted (date)
-└── emails (str)
-└── is_remote (bool)
+├── title
+├── company
+├── company_url
+├── job_url
+├── location
+│   ├── country
+│   ├── city
+│   ├── state
+├── description
+├── job_type: fulltime, parttime, internship, contract
+├── job_function
+│   ├── interval: yearly, monthly, weekly, daily, hourly
+│   ├── min_amount
+│   ├── max_amount
+│   ├── currency
+│   └── salary_source: direct_data, description (parsed from posting)
+├── date_posted
+├── emails
+└── is_remote
+
+Linkedin specific
+└── job_level
+
+Linkedin & Indeed specific
+└── company_industry

 Indeed specific
-├── company_country (str)
-└── company_addresses (str)
-└── company_industry (str)
-└── company_employees_label (str)
-└── company_revenue_label (str)
-└── company_description (str)
-└── ceo_name (str)
-└── ceo_photo_url (str)
-└── logo_photo_url (str)
-└── banner_photo_url (str)
+├── company_country
+├── company_addresses
+├── company_employees_label
+├── company_revenue_label
+├── company_description
+└── logo_photo_url
 ```

 ## Supported Countries for Job Searching

 ### **LinkedIn**

-LinkedIn searches globally & uses only the `location` parameter. You can only fetch 1000 jobs max from the LinkedIn endpoint we are using
+LinkedIn searches globally & uses only the `location` parameter. 

 ### **ZipRecruiter**

@@ -154,11 +210,23 @@ You can specify the following countries when searching on Indeed (use the exact

 ## Notes
 * Indeed is the best scraper currently with no rate limiting.  
-* Glassdoor/Ziprecruiter can only fetch 900/1000 jobs from the endpoints we are using on a given search.  
-* LinkedIn is the most restrictive and usually rate limits around the 10th page.
+* All the job board endpoints are capped at around 1000 jobs on a given search.  
+* LinkedIn is the most restrictive and usually rate limits around the 10th page with one ip. Proxies are a must basically.

 ## Frequently Asked Questions

+---
+**Q: Why is Indeed giving unrelated roles?**  
+**A:** Indeed is searching each one of your terms e.g. software intern, it searches software OR intern. Try search_term='"software intern"' in quotes for stricter searching
+
+---
+
+**Q: Received a response code 429?**  
+**A:** This indicates that you have been blocked by the job board site for sending too many requests. All of the job board sites are aggressive with blocking. We recommend:
+
+- Wait some time between scrapes (site-dependent).
+- Try using the proxies param to change your IP address.
+
 ---

 **Q: Encountering issues with your queries?**  
@@ -166,11 +234,3 @@ You can specify the following countries when searching on Indeed (use the exact
 persist, [submit an issue](https://github.com/Bunsly/JobSpy/issues).

 ---
-
-**Q: Received a response code 429?**  
-**A:** This indicates that you have been blocked by the job board site for sending too many requests. All of the job board sites are aggressive with blocking. We recommend:
-
- Waiting some time between scrapes (site-dependent).
- Trying a VPN or proxy to change your IP address.
-
---
--- a/examples/JobSpy_AllSites.py
+++ b/examples/JobSpy_AllSites.py
@@ -1,30 +0,0 @@
-from jobspy import scrape_jobs
-import pandas as pd
-
-jobs: pd.DataFrame = scrape_jobs(
-    site_name=["indeed", "linkedin", "zip_recruiter", "glassdoor"],
-    search_term="software engineer",
-    location="Dallas, TX",
-    results_wanted=25,  # be wary the higher it is, the more likey you'll get blocked (rotating proxy can help tho)
-    country_indeed="USA",
-    # proxy="http://jobspy:5a4vpWtj8EeJ2hoYzk@ca.smartproxy.com:20001",
-)
-
-# formatting for pandas
-pd.set_option("display.max_columns", None)
-pd.set_option("display.max_rows", None)
-pd.set_option("display.width", None)
-pd.set_option("display.max_colwidth", 50)  # set to 0 to see full job url / desc
-
-# 1: output to console
-print(jobs)
-
-# 2: output to .csv
-jobs.to_csv("./jobs.csv", index=False)
-print("outputted to jobs.csv")
-
-# 3: output to .xlsx
-# jobs.to_xlsx('jobs.xlsx', index=False)
-
-# 4: display in Jupyter Notebook (1. pip install jupyter 2. jupyter notebook)
-# display(jobs)
--- a/examples/JobSpy_Demo.ipynb
+++ b/examples/JobSpy_Demo.ipynb
@@ -1,167 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "00a94b47-f47b-420f-ba7e-714ef219c006",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from jobspy import scrape_jobs\n",
-    "import pandas as pd\n",
-    "from IPython.display import display, HTML"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "9f773e6c-d9fc-42cc-b0ef-63b739e78435",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "pd.set_option('display.max_columns', None)\n",
-    "pd.set_option('display.max_rows', None)\n",
-    "pd.set_option('display.width', None)\n",
-    "pd.set_option('display.max_colwidth', 50)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "1253c1f8-9437-492e-9dd3-e7fe51099420",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# example 1 (no hyperlinks, USA)\n",
-    "jobs = scrape_jobs(\n",
-    "    site_name=[\"linkedin\"],\n",
-    "    location='san francisco',\n",
-    "    search_term=\"engineer\",\n",
-    "    results_wanted=5,\n",
-    "\n",
-    "    # use if you want to use a proxy\n",
-    "    # proxy=\"socks5://jobspy:5a4vpWtj4EeJ2hoYzk@us.smartproxy.com:10001\",\n",
-    "    proxy=\"http://jobspy:5a4vpWtj4EeJ2hoYzk@us.smartproxy.com:10001\",\n",
-    "    #proxy=\"https://jobspy:5a4vpWtj4EeJ2hoYzk@us.smartproxy.com:10001\",\n",
-    ")\n",
-    "display(jobs)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "6a581b2d-f7da-4fac-868d-9efe143ee20a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# example 2 - remote USA & hyperlinks\n",
-    "jobs = scrape_jobs(\n",
-    "    site_name=[\"linkedin\", \"zip_recruiter\", \"indeed\"],\n",
-    "    # location='san francisco',\n",
-    "    search_term=\"software engineer\",\n",
-    "    country_indeed=\"USA\",\n",
-    "    hyperlinks=True,\n",
-    "    is_remote=True,\n",
-    "    results_wanted=5, \n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "fe8289bc-5b64-4202-9a64-7c117c83fd9a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# use if hyperlinks=True\n",
-    "html = jobs.to_html(escape=False)\n",
-    "# change max-width: 200px to show more or less of the content\n",
-    "truncate_width = f'<style>.dataframe td {{ max-width: 200px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }}</style>{html}'\n",
-    "display(HTML(truncate_width))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "951c2fe1-52ff-407d-8bb1-068049b36777",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# example 3 - with hyperlinks, international - linkedin (no zip_recruiter)\n",
-    "jobs = scrape_jobs(\n",
-    "    site_name=[\"linkedin\"],\n",
-    "    location='berlin',\n",
-    "    search_term=\"engineer\",\n",
-    "    hyperlinks=True,\n",
-    "    results_wanted=5,\n",
-    "    easy_apply=True\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "1e37a521-caef-441c-8fc2-2eb5b2e7da62",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# use if hyperlinks=True\n",
-    "html = jobs.to_html(escape=False)\n",
-    "# change max-width: 200px to show more or less of the content\n",
-    "truncate_width = f'<style>.dataframe td {{ max-width: 200px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }}</style>{html}'\n",
-    "display(HTML(truncate_width))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "0650e608-0b58-4bf5-ae86-68348035b16a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# example 4 - international indeed (no zip_recruiter)\n",
-    "jobs = scrape_jobs(\n",
-    "    site_name=[\"indeed\"],\n",
-    "    search_term=\"engineer\",\n",
-    "    country_indeed = \"China\",\n",
-    "    hyperlinks=True\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "40913ac8-3f8a-4d7e-ac47-afb88316432b",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# use if hyperlinks=True\n",
-    "html = jobs.to_html(escape=False)\n",
-    "# change max-width: 200px to show more or less of the content\n",
-    "truncate_width = f'<style>.dataframe td {{ max-width: 200px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }}</style>{html}'\n",
-    "display(HTML(truncate_width))"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.5"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
--- a/examples/JobSpy_LongScrape.py
+++ b/examples/JobSpy_LongScrape.py
@@ -1,77 +0,0 @@
-from jobspy import scrape_jobs
-import pandas as pd
-import os
-import time
-
-# creates csv a new filename if the jobs.csv already exists.
-csv_filename = "jobs.csv"
-counter = 1
-while os.path.exists(csv_filename):
-    csv_filename = f"jobs_{counter}.csv"
-    counter += 1
-
-# results wanted and offset
-results_wanted = 1000
-offset = 0
-
-all_jobs = []
-
-# max retries
-max_retries = 3
-
-# nuumber of results at each iteration
-results_in_each_iteration = 30
-
-while len(all_jobs) < results_wanted:
-    retry_count = 0
-    while retry_count < max_retries:
-        print("Doing from", offset, "to", offset + results_in_each_iteration, "jobs")
-        try:
-            jobs = scrape_jobs(
-                site_name=["indeed"],
-                search_term="software engineer",
-                # New York, NY
-                # Dallas, TX
-
-                # Los Angeles, CA
-                location="Los Angeles, CA",
-                results_wanted=min(results_in_each_iteration, results_wanted - len(all_jobs)),
-                country_indeed="USA",
-                offset=offset,
-                # proxy="http://jobspy:5a4vpWtj8EeJ2hoYzk@ca.smartproxy.com:20001",
-            )
-
-            # Add the scraped jobs to the list
-            all_jobs.extend(jobs.to_dict('records'))
-
-            # Increment the offset for the next page of results
-            offset += results_in_each_iteration
-
-            # Add a delay to avoid rate limiting (you can adjust the delay time as needed)
-            print(f"Scraped {len(all_jobs)} jobs")
-            print("Sleeping secs", 100 * (retry_count + 1))
-            time.sleep(100 * (retry_count + 1))  # Sleep for 2 seconds between requests
-
-            break  # Break out of the retry loop if successful
-        except Exception as e:
-            print(f"Error: {e}")
-            retry_count += 1
-            print("Sleeping secs before retry", 100 * (retry_count + 1))
-            time.sleep(100 * (retry_count + 1))
-            if retry_count >= max_retries:
-                print("Max retries reached. Exiting.")
-                break
-
-# DataFrame from the collected job data
-jobs_df = pd.DataFrame(all_jobs)
-
-# Formatting
-pd.set_option("display.max_columns", None)
-pd.set_option("display.max_rows", None)
-pd.set_option("display.width", None)
-pd.set_option("display.max_colwidth", 50)
-
-print(jobs_df)
-
-jobs_df.to_csv(csv_filename, index=False)
-print(f"Outputted to {csv_filename}")
--- a/poetry.lock
+++ b/poetry.lock
--- a/poetry.toml
+++ b/poetry.toml
@@ -0,0 +1,2 @@
+[virtualenvs]
+in-project = true
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,10 +1,11 @@
 [tool.poetry]
 name = "python-jobspy"
-version = "1.1.50"
+version = "1.1.70"
 description = "Job scraper for LinkedIn, Indeed, Glassdoor & ZipRecruiter"
 authors = ["Zachary Hampton <zachary@bunsly.com>", "Cullen Watson <cullen@bunsly.com>"]
 homepage = "https://github.com/Bunsly/JobSpy"
 readme = "README.md"
+keywords = ['jobs-scraper', 'linkedin', 'indeed', 'glassdoor', 'ziprecruiter']

 packages = [
    { include = "jobspy", from = "src" }
@@ -15,17 +16,18 @@ python = "^3.10"
 requests = "^2.31.0"
 beautifulsoup4 = "^4.12.2"
 pandas = "^2.1.0"
-NUMPY = "1.24.2"
+NUMPY = "1.26.3"
 pydantic = "^2.3.0"
 tls-client = "^1.0.1"
 markdownify = "^0.11.6"
+regex = "^2024.4.28"


 [tool.poetry.group.dev.dependencies]
 pytest = "^7.4.1"
 jupyter = "^1.0.0"
-black = "^24.2.0"
-pre-commit = "^3.6.2"
+black = "*"
+pre-commit = "*"

 [build-system]
 requires = ["poetry-core"]
--- a/src/jobspy/init.py
+++ b/src/jobspy/init.py
@@ -5,12 +5,12 @@ from typing import Tuple
 from concurrent.futures import ThreadPoolExecutor, as_completed

 from .jobs import JobType, Location
-from .scrapers.utils import logger
+from .scrapers.utils import set_logger_level, extract_salary, create_logger
 from .scrapers.indeed import IndeedScraper
 from .scrapers.ziprecruiter import ZipRecruiterScraper
 from .scrapers.glassdoor import GlassdoorScraper
 from .scrapers.linkedin import LinkedInScraper
-from .scrapers import ScraperInput, Site, JobResponse, Country
+from .scrapers import SalarySource, ScraperInput, Site, JobResponse, Country
 from .scrapers.exceptions import (
    LinkedInException,
    IndeedException,
@@ -30,17 +30,20 @@ def scrape_jobs(
    results_wanted: int = 15,
    country_indeed: str = "usa",
    hyperlinks: bool = False,
-    proxy: str | None = None,
+    proxies: list[str] | str | None = None,
+    ca_cert: str | None = None,
    description_format: str = "markdown",
    linkedin_fetch_description: bool | None = False,
    linkedin_company_ids: list[int] | None = None,
    offset: int | None = 0,
    hours_old: int = None,
+    enforce_annual_salary: bool = False,
+    verbose: int = 2,
    **kwargs,
 ) -> pd.DataFrame:
    """
    Simultaneously scrapes job data from multiple job sites.
-    :return: results_wanted: pandas dataframe containing job data
+    :return: pandas dataframe containing job data
    """
    SCRAPER_MAPPING = {
        Site.LINKEDIN: LinkedInScraper,
@@ -48,6 +51,7 @@ def scrape_jobs(
        Site.ZIP_RECRUITER: ZipRecruiterScraper,
        Site.GLASSDOOR: GlassdoorScraper,
    }
+    set_logger_level(verbose)

    def map_str_to_site(site_name: str) -> Site:
        return Site[site_name.upper()]
@@ -94,11 +98,11 @@ def scrape_jobs(

    def scrape_site(site: Site) -> Tuple[str, JobResponse]:
        scraper_class = SCRAPER_MAPPING[site]
-        scraper = scraper_class(proxy=proxy)
+        scraper = scraper_class(proxies=proxies, ca_cert=ca_cert)
        scraped_data: JobResponse = scraper.scrape(scraper_input)
        cap_name = site.value.capitalize()
        site_name = "ZipRecruiter" if cap_name == "Zip_recruiter" else cap_name
-        logger.info(f"{site_name} finished scraping")
+        create_logger(site_name).info(f"finished scraping")
        return site.value, scraped_data

    site_to_jobs_dict = {}
@@ -116,6 +120,21 @@ def scrape_jobs(
            site_value, scraped_data = future.result()
            site_to_jobs_dict[site_value] = scraped_data

+    def convert_to_annual(job_data: dict):
+        if job_data["interval"] == "hourly":
+            job_data["min_amount"] *= 2080
+            job_data["max_amount"] *= 2080
+        if job_data["interval"] == "monthly":
+            job_data["min_amount"] *= 12
+            job_data["max_amount"] *= 12
+        if job_data["interval"] == "weekly":
+            job_data["min_amount"] *= 52
+            job_data["max_amount"] *= 52
+        if job_data["interval"] == "daily":
+            job_data["min_amount"] *= 260
+            job_data["max_amount"] *= 260
+        job_data["interval"] = "yearly"
+
    jobs_dfs: list[pd.DataFrame] = []

    for site, job_response in site_to_jobs_dict.items():
@@ -148,12 +167,33 @@ def scrape_jobs(
                job_data["min_amount"] = compensation_obj.get("min_amount")
                job_data["max_amount"] = compensation_obj.get("max_amount")
                job_data["currency"] = compensation_obj.get("currency", "USD")
-            else:
-                job_data["interval"] = None
-                job_data["min_amount"] = None
-                job_data["max_amount"] = None
-                job_data["currency"] = None
+                job_data["salary_source"] = SalarySource.DIRECT_DATA.value
+                if enforce_annual_salary and (
+                    job_data["interval"]
+                    and job_data["interval"] != "yearly"
+                    and job_data["min_amount"]
+                    and job_data["max_amount"]
+                ):
+                    convert_to_annual(job_data)

+            else:
+                if country_enum == Country.USA:
+                    (
+                        job_data["interval"],
+                        job_data["min_amount"],
+                        job_data["max_amount"],
+                        job_data["currency"],
+                    ) = extract_salary(
+                        job_data["description"],
+                        enforce_annual_salary=enforce_annual_salary,
+                    )
+                    job_data["salary_source"] = SalarySource.DESCRIPTION.value
+
+            job_data["salary_source"] = (
+                job_data["salary_source"]
+                if "min_amount" in job_data and job_data["min_amount"]
+                else None
+            )
            job_df = pd.DataFrame([job_data])
            jobs_dfs.append(job_df)

@@ -166,6 +206,7 @@ def scrape_jobs(

        # Desired column order
        desired_order = [
+            "id",
            "site",
            "job_url_hyper" if hyperlinks else "job_url",
            "job_url_direct",
@@ -174,24 +215,25 @@ def scrape_jobs(
            "location",
            "job_type",
            "date_posted",
+            "salary_source",
            "interval",
            "min_amount",
            "max_amount",
            "currency",
            "is_remote",
+            "job_level",
+            "job_function",
+            "company_industry",
+            "listing_type",
            "emails",
            "description",
            "company_url",
+            "logo_photo_url",
            "company_url_direct",
            "company_addresses",
-            "company_industry",
            "company_num_employees",
            "company_revenue",
            "company_description",
-            "logo_photo_url",
-            "banner_photo_url",
-            "ceo_name",
-            "ceo_photo_url",
        ]

        # Step 3: Ensure all desired columns are present, adding missing ones as empty
--- a/src/jobspy/jobs/init.py
+++ b/src/jobspy/jobs/init.py
@@ -92,7 +92,8 @@ class Country(Enum):
    JAPAN = ("japan", "jp")
    KUWAIT = ("kuwait", "kw")
    LUXEMBOURG = ("luxembourg", "lu")
-    MALAYSIA = ("malaysia", "malaysia")
+    MALAYSIA = ("malaysia", "malaysia:my", "com")
+    MALTA = ("malta", "malta:mt", "mt")
    MEXICO = ("mexico", "mx", "com.mx")
    MOROCCO = ("morocco", "ma")
    NETHERLANDS = ("netherlands", "nl", "nl")
@@ -117,7 +118,7 @@ class Country(Enum):
    SWITZERLAND = ("switzerland", "ch", "de:ch")
    TAIWAN = ("taiwan", "tw")
    THAILAND = ("thailand", "th")
-    TURKEY = ("turkey", "tr")
+    TURKEY = ("türkiye,turkey", "tr")
    UKRAINE = ("ukraine", "ua")
    UNITEDARABEMIRATES = ("united arab emirates", "ae")
    UK = ("uk,united kingdom", "uk:gb", "co.uk")
@@ -226,6 +227,7 @@ class DescriptionFormat(Enum):


 class JobPost(BaseModel):
+    id: str | None = None
    title: str
    company_name: str | None
    job_url: str
@@ -241,18 +243,25 @@ class JobPost(BaseModel):
    date_posted: date | None = None
    emails: list[str] | None = None
    is_remote: bool | None = None
+    listing_type: str | None = None
+
+    # linkedin specific
+    job_level: str | None = None
+
+    # linkedin and indeed specific
+    company_industry: str | None = None

    # indeed specific
    company_addresses: str | None = None
-    company_industry: str | None = None
    company_num_employees: str | None = None
    company_revenue: str | None = None
    company_description: str | None = None
-    ceo_name: str | None = None
-    ceo_photo_url: str | None = None
    logo_photo_url: str | None = None
    banner_photo_url: str | None = None

+    # linkedin only atm
+    job_function: str | None = None
+

 class JobResponse(BaseModel):
    jobs: list[JobPost] = []
--- a/src/jobspy/scrapers/init.py
+++ b/src/jobspy/scrapers/init.py
@@ -1,5 +1,7 @@
 from __future__ import annotations

+from abc import ABC, abstractmethod
+
 from ..jobs import (
    Enum,
    BaseModel,
@@ -16,6 +18,9 @@ class Site(Enum):
    ZIP_RECRUITER = "zip_recruiter"
    GLASSDOOR = "glassdoor"

+class SalarySource(Enum):
+    DIRECT_DATA = "direct_data"
+    DESCRIPTION = "description"

 class ScraperInput(BaseModel):
    site_type: list[Site]
@@ -36,9 +41,11 @@ class ScraperInput(BaseModel):
    hours_old: int | None = None


-class Scraper:
-    def __init__(self, site: Site, proxy: list[str] | None = None):
+class Scraper(ABC):
+    def __init__(self, site: Site, proxies: list[str] | None = None, ca_cert: str | None = None):
        self.site = site
-        self.proxy = (lambda p: {"http": p, "https": p} if p else None)(proxy)
+        self.proxies = proxies
+        self.ca_cert = ca_cert

+    @abstractmethod
    def scrape(self, scraper_input: ScraperInput) -> JobResponse: ...
--- a/src/jobspy/scrapers/glassdoor/init.py
+++ b/src/jobspy/scrapers/glassdoor/init.py
@@ -14,13 +14,13 @@ from typing import Optional, Tuple
 from datetime import datetime, timedelta
 from concurrent.futures import ThreadPoolExecutor, as_completed

+from .constants import fallback_token, query_template, headers
 from .. import Scraper, ScraperInput, Site
-from ..utils import extract_emails_from_text
+from ..utils import extract_emails_from_text, create_logger
 from ..exceptions import GlassdoorException
 from ..utils import (
    create_session,
    markdown_converter,
-    logger,
 )
 from ...jobs import (
    JobPost,
@@ -32,14 +32,18 @@ from ...jobs import (
    DescriptionFormat,
 )

+logger = create_logger("Glassdoor")
+

 class GlassdoorScraper(Scraper):
-    def __init__(self, proxy: Optional[str] = None):
+    def __init__(
+        self, proxies: list[str] | str | None = None, ca_cert: str | None = None
+    ):
        """
        Initializes GlassdoorScraper with the Glassdoor job search url
        """
        site = Site(Site.GLASSDOOR)
-        super().__init__(site, proxy=proxy)
+        super().__init__(site, proxies=proxies, ca_cert=ca_cert)

        self.base_url = None
        self.country = None
@@ -59,9 +63,12 @@ class GlassdoorScraper(Scraper):
        self.scraper_input.results_wanted = min(900, scraper_input.results_wanted)
        self.base_url = self.scraper_input.country.get_glassdoor_url()

-        self.session = create_session(self.proxy, is_tls=True, has_retry=True)
+        self.session = create_session(
+            proxies=self.proxies, ca_cert=self.ca_cert, is_tls=True, has_retry=True
+        )
        token = self._get_csrf_token()
-        self.headers["gd-csrf-token"] = token if token else self.fallback_token
+        headers["gd-csrf-token"] = token if token else fallback_token
+        self.session.headers.update(headers)

        location_id, location_type = self._get_location(
            scraper_input.location, scraper_input.is_remote
@@ -69,26 +76,26 @@ class GlassdoorScraper(Scraper):
        if location_type is None:
            logger.error("Glassdoor: location not parsed")
            return JobResponse(jobs=[])
-        all_jobs: list[JobPost] = []
+        job_list: list[JobPost] = []
        cursor = None

        range_start = 1 + (scraper_input.offset // self.jobs_per_page)
        tot_pages = (scraper_input.results_wanted // self.jobs_per_page) + 2
        range_end = min(tot_pages, self.max_pages + 1)
        for page in range(range_start, range_end):
-            logger.info(f"Glassdoor search page: {page}")
+            logger.info(f"search page: {page} / {range_end-1}")
            try:
                jobs, cursor = self._fetch_jobs_page(
                    scraper_input, location_id, location_type, page, cursor
                )
-                all_jobs.extend(jobs)
-                if not jobs or len(all_jobs) >= scraper_input.results_wanted:
-                    all_jobs = all_jobs[: scraper_input.results_wanted]
+                job_list.extend(jobs)
+                if not jobs or len(job_list) >= scraper_input.results_wanted:
+                    job_list = job_list[: scraper_input.results_wanted]
                    break
            except Exception as e:
                logger.error(f"Glassdoor: {str(e)}")
                break
-        return JobResponse(jobs=all_jobs)
+        return JobResponse(jobs=job_list)

    def _fetch_jobs_page(
        self,
@@ -107,7 +114,6 @@ class GlassdoorScraper(Scraper):
            payload = self._add_payload(location_id, location_type, page_num, cursor)
            response = self.session.post(
                f"{self.base_url}/graph",
-                headers=self.headers,
                timeout_seconds=15,
                data=payload,
            )
@@ -148,9 +154,7 @@ class GlassdoorScraper(Scraper):
        """
        Fetches csrf token needed for API by visiting a generic page
        """
-        res = self.session.get(
-            f"{self.base_url}/Job/computer-science-jobs.htm", headers=self.headers
-        )
+        res = self.session.get(f"{self.base_url}/Job/computer-science-jobs.htm")
        pattern = r'"token":\s*"([^"]+)"'
        matches = re.findall(pattern, res.text)
        token = None
@@ -189,7 +193,17 @@ class GlassdoorScraper(Scraper):
        except:
            description = None
        company_url = f"{self.base_url}Overview/W-EI_IE{company_id}.htm"
+        company_logo = (
+            job_data["jobview"].get("overview", {}).get("squareLogoUrl", None)
+        )
+        listing_type = (
+            job_data["jobview"]
+            .get("header", {})
+            .get("adOrderSponsorshipLevel", "")
+            .lower()
+        )
        return JobPost(
+            id=f"gd-{job_id}",
            title=title,
            company_url=company_url if company_id else None,
            company_name=company_name,
@@ -200,6 +214,8 @@ class GlassdoorScraper(Scraper):
            is_remote=is_remote,
            description=description,
            emails=extract_emails_from_text(description) if description else None,
+            logo_photo_url=company_logo,
+            listing_type=listing_type,
        )

    def _fetch_job_description(self, job_id):
@@ -231,7 +247,7 @@ class GlassdoorScraper(Scraper):
                """,
            }
        ]
-        res = requests.post(url, json=body, headers=self.headers)
+        res = requests.post(url, json=body, headers=headers)
        if res.status_code != 200:
            return None
        data = res.json()[0]
@@ -244,8 +260,7 @@ class GlassdoorScraper(Scraper):
        if not location or is_remote:
            return "11047", "STATE"  # remote options
        url = f"{self.base_url}/findPopularLocationAjax.htm?maxLocationsToReturn=10&term={location}"
-        session = create_session(self.proxy, has_retry=True)
-        res = self.session.get(url, headers=self.headers)
+        res = self.session.get(url)
        if res.status_code != 200:
            if res.status_code == 429:
                err = f"429 Response - Blocked by Glassdoor for too many requests"
@@ -299,7 +314,7 @@ class GlassdoorScraper(Scraper):
                "fromage": fromage,
                "sort": "date",
            },
-            "query": self.query_template,
+            "query": query_template,
        }
        if self.scraper_input.job_type:
            payload["variables"]["filterParams"].append(
@@ -347,188 +362,3 @@ class GlassdoorScraper(Scraper):
        for cursor_data in pagination_cursors:
            if cursor_data["pageNumber"] == page_num:
                return cursor_data["cursor"]
-
-    fallback_token = "Ft6oHEWlRZrxDww95Cpazw:0pGUrkb2y3TyOpAIqF2vbPmUXoXVkD3oEGDVkvfeCerceQ5-n8mBg3BovySUIjmCPHCaW0H2nQVdqzbtsYqf4Q:wcqRqeegRUa9MVLJGyujVXB7vWFPjdaS1CtrrzJq-ok"
-    headers = {
-        "authority": "www.glassdoor.com",
-        "accept": "*/*",
-        "accept-language": "en-US,en;q=0.9",
-        "apollographql-client-name": "job-search-next",
-        "apollographql-client-version": "4.65.5",
-        "content-type": "application/json",
-        "origin": "https://www.glassdoor.com",
-        "referer": "https://www.glassdoor.com/",
-        "sec-ch-ua": '"Chromium";v="118", "Google Chrome";v="118", "Not=A?Brand";v="99"',
-        "sec-ch-ua-mobile": "?0",
-        "sec-ch-ua-platform": '"macOS"',
-        "sec-fetch-dest": "empty",
-        "sec-fetch-mode": "cors",
-        "sec-fetch-site": "same-origin",
-        "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
-    }
-    query_template = """
-            query JobSearchResultsQuery(
-                $excludeJobListingIds: [Long!], 
-                $keyword: String, 
-                $locationId: Int, 
-                $locationType: LocationTypeEnum, 
-                $numJobsToShow: Int!, 
-                $pageCursor: String, 
-                $pageNumber: Int, 
-                $filterParams: [FilterParams], 
-                $originalPageUrl: String, 
-                $seoFriendlyUrlInput: String, 
-                $parameterUrlInput: String, 
-                $seoUrl: Boolean
-            ) {
-                jobListings(
-                    contextHolder: {
-                        searchParams: {
-                            excludeJobListingIds: $excludeJobListingIds, 
-                            keyword: $keyword, 
-                            locationId: $locationId, 
-                            locationType: $locationType, 
-                            numPerPage: $numJobsToShow, 
-                            pageCursor: $pageCursor, 
-                            pageNumber: $pageNumber, 
-                            filterParams: $filterParams, 
-                            originalPageUrl: $originalPageUrl, 
-                            seoFriendlyUrlInput: $seoFriendlyUrlInput, 
-                            parameterUrlInput: $parameterUrlInput, 
-                            seoUrl: $seoUrl, 
-                            searchType: SR
-                        }
-                    }
-                ) {
-                    companyFilterOptions {
-                        id
-                        shortName
-                        __typename
-                    }
-                    filterOptions
-                    indeedCtk
-                    jobListings {
-                        ...JobView
-                        __typename
-                    }
-                    jobListingSeoLinks {
-                        linkItems {
-                            position
-                            url
-                            __typename
-                        }
-                        __typename
-                    }
-                    jobSearchTrackingKey
-                    jobsPageSeoData {
-                        pageMetaDescription
-                        pageTitle
-                        __typename
-                    }
-                    paginationCursors {
-                        cursor
-                        pageNumber
-                        __typename
-                    }
-                    indexablePageForSeo
-                    searchResultsMetadata {
-                        searchCriteria {
-                            implicitLocation {
-                                id
-                                localizedDisplayName
-                                type
-                                __typename
-                            }
-                            keyword
-                            location {
-                                id
-                                shortName
-                                localizedShortName
-                                localizedDisplayName
-                                type
-                                __typename
-                            }
-                            __typename
-                        }
-                        helpCenterDomain
-                        helpCenterLocale
-                        jobSerpJobOutlook {
-                            occupation
-                            paragraph
-                            __typename
-                        }
-                        showMachineReadableJobs
-                        __typename
-                    }
-                    totalJobsCount
-                    __typename
-                }
-            }
-
-            fragment JobView on JobListingSearchResult {
-                jobview {
-                    header {
-                        adOrderId
-                        advertiserType
-                        adOrderSponsorshipLevel
-                        ageInDays
-                        divisionEmployerName
-                        easyApply
-                        employer {
-                            id
-                            name
-                            shortName
-                            __typename
-                        }
-                        employerNameFromSearch
-                        goc
-                        gocConfidence
-                        gocId
-                        jobCountryId
-                        jobLink
-                        jobResultTrackingKey
-                        jobTitleText
-                        locationName
-                        locationType
-                        locId
-                        needsCommission
-                        payCurrency
-                        payPeriod
-                        payPeriodAdjustedPay {
-                            p10
-                            p50
-                            p90
-                            __typename
-                        }
-                        rating
-                        salarySource
-                        savedJobId
-                        sponsored
-                        __typename
-                    }
-                    job {
-                        description
-                        importConfigId
-                        jobTitleId
-                        jobTitleText
-                        listingId
-                        __typename
-                    }
-                    jobListingAdminDetails {
-                        cpcVal
-                        importConfigId
-                        jobListingId
-                        jobSourceId
-                        userEligibleForAdminJobDetails
-                        __typename
-                    }
-                    overview {
-                        shortName
-                        squareLogoUrl
-                        __typename
-                    }
-                    __typename
-                }
-                __typename
-            }
-    """
--- a/src/jobspy/scrapers/glassdoor/constants.py
+++ b/src/jobspy/scrapers/glassdoor/constants.py
@@ -0,0 +1,184 @@
+headers = {
+    "authority": "www.glassdoor.com",
+    "accept": "*/*",
+    "accept-language": "en-US,en;q=0.9",
+    "apollographql-client-name": "job-search-next",
+    "apollographql-client-version": "4.65.5",
+    "content-type": "application/json",
+    "origin": "https://www.glassdoor.com",
+    "referer": "https://www.glassdoor.com/",
+    "sec-ch-ua": '"Chromium";v="118", "Google Chrome";v="118", "Not=A?Brand";v="99"',
+    "sec-ch-ua-mobile": "?0",
+    "sec-ch-ua-platform": '"macOS"',
+    "sec-fetch-dest": "empty",
+    "sec-fetch-mode": "cors",
+    "sec-fetch-site": "same-origin",
+    "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
+}
+query_template = """
+        query JobSearchResultsQuery(
+            $excludeJobListingIds: [Long!], 
+            $keyword: String, 
+            $locationId: Int, 
+            $locationType: LocationTypeEnum, 
+            $numJobsToShow: Int!, 
+            $pageCursor: String, 
+            $pageNumber: Int, 
+            $filterParams: [FilterParams], 
+            $originalPageUrl: String, 
+            $seoFriendlyUrlInput: String, 
+            $parameterUrlInput: String, 
+            $seoUrl: Boolean
+        ) {
+            jobListings(
+                contextHolder: {
+                    searchParams: {
+                        excludeJobListingIds: $excludeJobListingIds, 
+                        keyword: $keyword, 
+                        locationId: $locationId, 
+                        locationType: $locationType, 
+                        numPerPage: $numJobsToShow, 
+                        pageCursor: $pageCursor, 
+                        pageNumber: $pageNumber, 
+                        filterParams: $filterParams, 
+                        originalPageUrl: $originalPageUrl, 
+                        seoFriendlyUrlInput: $seoFriendlyUrlInput, 
+                        parameterUrlInput: $parameterUrlInput, 
+                        seoUrl: $seoUrl, 
+                        searchType: SR
+                    }
+                }
+            ) {
+                companyFilterOptions {
+                    id
+                    shortName
+                    __typename
+                }
+                filterOptions
+                indeedCtk
+                jobListings {
+                    ...JobView
+                    __typename
+                }
+                jobListingSeoLinks {
+                    linkItems {
+                        position
+                        url
+                        __typename
+                    }
+                    __typename
+                }
+                jobSearchTrackingKey
+                jobsPageSeoData {
+                    pageMetaDescription
+                    pageTitle
+                    __typename
+                }
+                paginationCursors {
+                    cursor
+                    pageNumber
+                    __typename
+                }
+                indexablePageForSeo
+                searchResultsMetadata {
+                    searchCriteria {
+                        implicitLocation {
+                            id
+                            localizedDisplayName
+                            type
+                            __typename
+                        }
+                        keyword
+                        location {
+                            id
+                            shortName
+                            localizedShortName
+                            localizedDisplayName
+                            type
+                            __typename
+                        }
+                        __typename
+                    }
+                    helpCenterDomain
+                    helpCenterLocale
+                    jobSerpJobOutlook {
+                        occupation
+                        paragraph
+                        __typename
+                    }
+                    showMachineReadableJobs
+                    __typename
+                }
+                totalJobsCount
+                __typename
+            }
+        }
+
+        fragment JobView on JobListingSearchResult {
+            jobview {
+                header {
+                    adOrderId
+                    advertiserType
+                    adOrderSponsorshipLevel
+                    ageInDays
+                    divisionEmployerName
+                    easyApply
+                    employer {
+                        id
+                        name
+                        shortName
+                        __typename
+                    }
+                    employerNameFromSearch
+                    goc
+                    gocConfidence
+                    gocId
+                    jobCountryId
+                    jobLink
+                    jobResultTrackingKey
+                    jobTitleText
+                    locationName
+                    locationType
+                    locId
+                    needsCommission
+                    payCurrency
+                    payPeriod
+                    payPeriodAdjustedPay {
+                        p10
+                        p50
+                        p90
+                        __typename
+                    }
+                    rating
+                    salarySource
+                    savedJobId
+                    sponsored
+                    __typename
+                }
+                job {
+                    description
+                    importConfigId
+                    jobTitleId
+                    jobTitleText
+                    listingId
+                    __typename
+                }
+                jobListingAdminDetails {
+                    cpcVal
+                    importConfigId
+                    jobListingId
+                    jobSourceId
+                    userEligibleForAdminJobDetails
+                    __typename
+                }
+                overview {
+                    shortName
+                    squareLogoUrl
+                    __typename
+                }
+                __typename
+            }
+            __typename
+        }
+"""
+fallback_token = "Ft6oHEWlRZrxDww95Cpazw:0pGUrkb2y3TyOpAIqF2vbPmUXoXVkD3oEGDVkvfeCerceQ5-n8mBg3BovySUIjmCPHCaW0H2nQVdqzbtsYqf4Q:wcqRqeegRUa9MVLJGyujVXB7vWFPjdaS1CtrrzJq-ok"
--- a/src/jobspy/scrapers/indeed/init.py
+++ b/src/jobspy/scrapers/indeed/init.py
@@ -10,16 +10,15 @@ from __future__ import annotations
 import math
 from typing import Tuple
 from datetime import datetime
-from concurrent.futures import ThreadPoolExecutor, Future
-
-import requests

+from .constants import job_search_query, api_headers
 from .. import Scraper, ScraperInput, Site
 from ..utils import (
    extract_emails_from_text,
    get_enum_from_job_type,
    markdown_converter,
-    logger,
+    create_session,
+    create_logger,
 )
 from ...jobs import (
    JobPost,
@@ -31,12 +30,21 @@ from ...jobs import (
    DescriptionFormat,
 )

+logger = create_logger("Indeed")
+

 class IndeedScraper(Scraper):
-    def __init__(self, proxy: str | None = None):
+    def __init__(
+        self, proxies: list[str] | str | None = None, ca_cert: str | None = None
+    ):
        """
        Initializes IndeedScraper with the Indeed API url
        """
+        super().__init__(Site.INDEED, proxies=proxies)
+
+        self.session = create_session(
+            proxies=self.proxies, ca_cert=ca_cert, is_tls=False
+        )
        self.scraper_input = None
        self.jobs_per_page = 100
        self.num_workers = 10
@@ -45,8 +53,6 @@ class IndeedScraper(Scraper):
        self.api_country_code = None
        self.base_url = None
        self.api_url = "https://apis.indeed.com/graphql"
-        site = Site(Site.INDEED)
-        super().__init__(site, proxy=proxy)

    def scrape(self, scraper_input: ScraperInput) -> JobResponse:
        """
@@ -57,7 +63,7 @@ class IndeedScraper(Scraper):
        self.scraper_input = scraper_input
        domain, self.api_country_code = self.scraper_input.country.indeed_domain_value
        self.base_url = f"https://{domain}.indeed.com"
-        self.headers = self.api_headers.copy()
+        self.headers = api_headers.copy()
        self.headers["indeed-co"] = self.scraper_input.country.indeed_domain_value
        job_list = []
        page = 1
@@ -65,17 +71,19 @@ class IndeedScraper(Scraper):
        cursor = None
        offset_pages = math.ceil(self.scraper_input.offset / 100)
        for _ in range(offset_pages):
-            logger.info(f"Indeed skipping search page: {page}")
+            logger.info(f"skipping search page: {page}")
            __, cursor = self._scrape_page(cursor)
            if not __:
-                logger.info(f"Indeed found no jobs on page: {page}")
+                logger.info(f"found no jobs on page: {page}")
                break

        while len(self.seen_urls) < scraper_input.results_wanted:
-            logger.info(f"Indeed search page: {page}")
+            logger.info(
+                f"search page: {page} / {math.ceil(scraper_input.results_wanted / 100)}"
+            )
            jobs, cursor = self._scrape_page(cursor)
            if not jobs:
-                logger.info(f"Indeed found no jobs on page: {page}")
+                logger.info(f"found no jobs on page: {page}")
                break
            job_list += jobs
            page += 1
@@ -90,12 +98,13 @@ class IndeedScraper(Scraper):
        jobs = []
        new_cursor = None
        filters = self._build_filters()
-        query = self.job_search_query.format(
-            what=(
-                f'what: "{self.scraper_input.search_term}"'
-                if self.scraper_input.search_term
-                else ""
-            ),
+        search_term = (
+            self.scraper_input.search_term.replace('"', '\\"')
+            if self.scraper_input.search_term
+            else ""
+        )
+        query = job_search_query.format(
+            what=(f'what: "{search_term}"' if search_term else ""),
            location=(
                f'location: {{where: "{self.scraper_input.location}", radius: {self.scraper_input.distance}, radiusUnit: MILES}}'
                if self.scraper_input.location
@@ -108,29 +117,29 @@ class IndeedScraper(Scraper):
        payload = {
            "query": query,
        }
-        api_headers = self.api_headers.copy()
-        api_headers["indeed-co"] = self.api_country_code
-        response = requests.post(
+        api_headers_temp = api_headers.copy()
+        api_headers_temp["indeed-co"] = self.api_country_code
+        response = self.session.post(
            self.api_url,
-            headers=api_headers,
+            headers=api_headers_temp,
            json=payload,
-            proxies=self.proxy,
            timeout=10,
        )
-        if response.status_code != 200:
+        if not response.ok:
            logger.info(
-                f"Indeed responded with status code: {response.status_code} (submit GitHub issue if this appears to be a beg)"
+                f"responded with status code: {response.status_code} (submit GitHub issue if this appears to be a bug)"
            )
            return jobs, new_cursor
        data = response.json()
        jobs = data["data"]["jobSearch"]["results"]
        new_cursor = data["data"]["jobSearch"]["pageInfo"]["nextCursor"]

-        with ThreadPoolExecutor(max_workers=self.num_workers) as executor:
-            job_results: list[Future] = [
-                executor.submit(self._process_job, job["job"]) for job in jobs
-            ]
-        job_list = [result.result() for result in job_results if result.result()]
+        job_list = []
+        for job in jobs:
+            processed_job = self._process_job(job["job"])
+            if processed_job:
+                job_list.append(processed_job)
+
        return job_list, new_cursor

    def _build_filters(self):
@@ -150,6 +159,15 @@ class IndeedScraper(Scraper):
            """.format(
                start=self.scraper_input.hours_old
            )
+        elif self.scraper_input.easy_apply:
+            filters_str = """
+            filters: {
+                keyword: {
+                  field: "indeedApplyScope",
+                  keys: ["DESKTOP"]
+                }
+            }
+            """
        elif self.scraper_input.job_type or self.scraper_input.is_remote:
            job_type_key_mapping = {
                JobType.FULL_TIME: "CF3CP",
@@ -167,7 +185,7 @@ class IndeedScraper(Scraper):
                keys.append("DSQF7")

            if keys:
-                keys_str = '", "'.join(keys)  # Prepare your keys string
+                keys_str = '", "'.join(keys)
                filters_str = f"""
                filters: {{
                  composite: {{
@@ -203,6 +221,7 @@ class IndeedScraper(Scraper):
        employer_details = employer.get("employerDetails", {}) if employer else {}
        rel_url = job["employer"]["relativeCompanyPageUrl"] if job["employer"] else None
        return JobPost(
+            id=f'in-{job["key"]}',
            title=job["title"],
            description=description,
            company_name=job["employer"].get("name") if job.get("employer") else None,
@@ -216,7 +235,7 @@ class IndeedScraper(Scraper):
                country=job.get("location", {}).get("countryCode"),
            ),
            job_type=job_type,
-            compensation=self._get_compensation(job),
+            compensation=self._get_compensation(job["compensation"]),
            date_posted=date_posted,
            job_url=job_url,
            job_url_direct=(
@@ -234,24 +253,18 @@ class IndeedScraper(Scraper):
                .replace("Iv1", "")
                .replace("_", " ")
                .title()
+                .strip()
                if employer_details.get("industry")
                else None
            ),
            company_num_employees=employer_details.get("employeesLocalizedLabel"),
            company_revenue=employer_details.get("revenueLocalizedLabel"),
            company_description=employer_details.get("briefDescription"),
-            ceo_name=employer_details.get("ceoName"),
-            ceo_photo_url=employer_details.get("ceoPhotoUrl"),
            logo_photo_url=(
                employer["images"].get("squareLogoUrl")
                if employer and employer.get("images")
                else None
            ),
-            banner_photo_url=(
-                employer["images"].get("headerImageUrl")
-                if employer and employer.get("images")
-                else None
-            ),
        )

    @staticmethod
@@ -270,14 +283,19 @@ class IndeedScraper(Scraper):
        return job_types

    @staticmethod
-    def _get_compensation(job: dict) -> Compensation | None:
+    def _get_compensation(compensation: dict) -> Compensation | None:
        """
        Parses the job to get compensation
        :param job:
-        :param job:
        :return: compensation object
        """
-        comp = job["compensation"]["baseSalary"]
+        if not compensation["baseSalary"] and not compensation["estimated"]:
+            return None
+        comp = (
+            compensation["baseSalary"]
+            if compensation["baseSalary"]
+            else compensation["estimated"]["baseSalary"]
+        )
        if not comp:
            return None
        interval = IndeedScraper._get_compensation_interval(comp["unitOfWork"])
@@ -287,9 +305,13 @@ class IndeedScraper(Scraper):
        max_range = comp["range"].get("max")
        return Compensation(
            interval=interval,
-            min_amount=round(min_range, 2) if min_range is not None else None,
-            max_amount=round(max_range, 2) if max_range is not None else None,
-            currency=job["compensation"]["currencyCode"],
+            min_amount=int(min_range) if min_range is not None else None,
+            max_amount=int(max_range) if max_range is not None else None,
+            currency=(
+                compensation["estimated"]["currencyCode"]
+                if compensation["estimated"]
+                else compensation["currencyCode"]
+            ),
        )

    @staticmethod
@@ -327,98 +349,3 @@ class IndeedScraper(Scraper):
            return CompensationInterval[mapped_interval]
        else:
            raise ValueError(f"Unsupported interval: {interval}")
-
-    api_headers = {
-        "Host": "apis.indeed.com",
-        "content-type": "application/json",
-        "indeed-api-key": "161092c2017b5bbab13edb12461a62d5a833871e7cad6d9d475304573de67ac8",
-        "accept": "application/json",
-        "indeed-locale": "en-US",
-        "accept-language": "en-US,en;q=0.9",
-        "user-agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 16_6_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 Indeed App 193.1",
-        "indeed-app-info": "appv=193.1; appid=com.indeed.jobsearch; osv=16.6.1; os=ios; dtype=phone",
-    }
-    job_search_query = """
-        query GetJobData {{
-          jobSearch(
-            {what}
-            {location}
-            includeSponsoredResults: NONE
-            limit: 100
-            sort: DATE
-            {cursor}
-            {filters}
-          ) {{
-            pageInfo {{
-              nextCursor
-            }}
-            results {{
-              trackingKey
-              job {{
-                key
-                title
-                datePublished
-                dateOnIndeed
-                description {{
-                  html
-                }}
-                location {{
-                  countryName
-                  countryCode
-                  admin1Code
-                  city
-                  postalCode
-                  streetAddress
-                  formatted {{
-                    short
-                    long
-                  }}
-                }}
-                compensation {{
-                  baseSalary {{
-                    unitOfWork
-                    range {{
-                      ... on Range {{
-                        min
-                        max
-                      }}
-                    }}
-                  }}
-                  currencyCode
-                }}
-                attributes {{
-                  key
-                  label
-                }}
-                employer {{
-                  relativeCompanyPageUrl
-                  name
-                  dossier {{
-                      employerDetails {{
-                        addresses
-                        industry
-                        employeesLocalizedLabel
-                        revenueLocalizedLabel
-                        briefDescription
-                        ceoName
-                        ceoPhotoUrl
-                      }}
-                      images {{
-                            headerImageUrl
-                            squareLogoUrl
-                      }}
-                      links {{
-                        corporateWebsite
-                    }}
-                  }}
-                }}
-                recruit {{
-                  viewJobUrl
-                  detailedSalary
-                  workSchedule
-                }}
-              }}
-            }}
-          }}
-        }}
-        """
--- a/src/jobspy/scrapers/indeed/constants.py
+++ b/src/jobspy/scrapers/indeed/constants.py
@@ -0,0 +1,109 @@
+job_search_query = """
+    query GetJobData {{
+        jobSearch(
+        {what}
+        {location}
+        limit: 100
+        {cursor}
+        sort: RELEVANCE
+        {filters}
+        ) {{
+        pageInfo {{
+            nextCursor
+        }}
+        results {{
+            trackingKey
+            job {{
+            source {{
+                name
+            }}
+            key
+            title
+            datePublished
+            dateOnIndeed
+            description {{
+                html
+            }}
+            location {{
+                countryName
+                countryCode
+                admin1Code
+                city
+                postalCode
+                streetAddress
+                formatted {{
+                short
+                long
+                }}
+            }}
+            compensation {{
+                estimated {{
+                currencyCode
+                baseSalary {{
+                    unitOfWork
+                    range {{
+                    ... on Range {{
+                        min
+                        max
+                    }}
+                    }}
+                }}
+                }}
+                baseSalary {{
+                unitOfWork
+                range {{
+                    ... on Range {{
+                    min
+                    max
+                    }}
+                }}
+                }}
+                currencyCode
+            }}
+            attributes {{
+                key
+                label
+            }}
+            employer {{
+                relativeCompanyPageUrl
+                name
+                dossier {{
+                    employerDetails {{
+                    addresses
+                    industry
+                    employeesLocalizedLabel
+                    revenueLocalizedLabel
+                    briefDescription
+                    ceoName
+                    ceoPhotoUrl
+                    }}
+                    images {{
+                        headerImageUrl
+                        squareLogoUrl
+                    }}
+                    links {{
+                    corporateWebsite
+                }}
+                }}
+            }}
+            recruit {{
+                viewJobUrl
+                detailedSalary
+                workSchedule
+            }}
+            }}
+        }}
+        }}
+    }}
+    """
+
+api_headers = {
+    "Host": "apis.indeed.com",
+    "content-type": "application/json",
+    "indeed-api-key": "161092c2017b5bbab13edb12461a62d5a833871e7cad6d9d475304573de67ac8",
+    "accept": "application/json",
+    "indeed-locale": "en-US",
+    "accept-language": "en-US,en;q=0.9",
+    "user-agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 16_6_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 Indeed App 193.1",
+    "indeed-app-info": "appv=193.1; appid=com.indeed.jobsearch; osv=16.6.1; os=ios; dtype=phone",
+}
--- a/src/jobspy/scrapers/linkedin/init.py
+++ b/src/jobspy/scrapers/linkedin/init.py
@@ -7,19 +7,21 @@ This module contains routines to scrape LinkedIn.

 from __future__ import annotations

+import math
 import time
 import random
+import regex as re
 from typing import Optional
 from datetime import datetime

-from threading import Lock
 from bs4.element import Tag
 from bs4 import BeautifulSoup
-from urllib.parse import urlparse, urlunparse
+from urllib.parse import urlparse, urlunparse, unquote

+from .constants import headers
 from .. import Scraper, ScraperInput, Site
 from ..exceptions import LinkedInException
-from ..utils import create_session
+from ..utils import create_session, remove_attributes, create_logger
 from ...jobs import (
    JobPost,
    Location,
@@ -30,13 +32,14 @@ from ...jobs import (
    DescriptionFormat,
 )
 from ..utils import (
-    logger,
    extract_emails_from_text,
    get_enum_from_job_type,
    currency_parser,
    markdown_converter,
 )

+logger = create_logger("LinkedIn")
+

 class LinkedInScraper(Scraper):
    base_url = "https://www.linkedin.com"
@@ -44,13 +47,25 @@ class LinkedInScraper(Scraper):
    band_delay = 4
    jobs_per_page = 25

-    def __init__(self, proxy: Optional[str] = None):
+    def __init__(
+        self, proxies: list[str] | str | None = None, ca_cert: str | None = None
+    ):
        """
        Initializes LinkedInScraper with the LinkedIn job search url
        """
-        super().__init__(Site(Site.LINKEDIN), proxy=proxy)
+        super().__init__(Site.LINKEDIN, proxies=proxies, ca_cert=ca_cert)
+        self.session = create_session(
+            proxies=self.proxies,
+            ca_cert=ca_cert,
+            is_tls=False,
+            has_retry=True,
+            delay=5,
+            clear_cookies=True,
+        )
+        self.session.headers.update(headers)
        self.scraper_input = None
        self.country = "worldwide"
+        self.job_url_direct_regex = re.compile(r'(?<=\?url=)[^"]+')

    def scrape(self, scraper_input: ScraperInput) -> JobResponse:
        """
@@ -60,18 +75,20 @@ class LinkedInScraper(Scraper):
        """
        self.scraper_input = scraper_input
        job_list: list[JobPost] = []
-        seen_urls = set()
-        url_lock = Lock()
-        page = scraper_input.offset // 25 + 25 if scraper_input.offset else 0
+        seen_ids = set()
+        start = scraper_input.offset // 10 * 10 if scraper_input.offset else 0
+        request_count = 0
        seconds_old = (
            scraper_input.hours_old * 3600 if scraper_input.hours_old else None
        )
        continue_search = (
-            lambda: len(job_list) < scraper_input.results_wanted and page < 1000
+            lambda: len(job_list) < scraper_input.results_wanted and start < 1000
        )
        while continue_search():
-            logger.info(f"LinkedIn search page: {page // 25 + 1}")
-            session = create_session(is_tls=False, has_retry=True, delay=5)
+            request_count += 1
+            logger.info(
+                f"search page: {request_count} / {math.ceil(scraper_input.results_wanted / 10)}"
+            )
            params = {
                "keywords": scraper_input.search_term,
                "location": scraper_input.location,
@@ -83,7 +100,7 @@ class LinkedInScraper(Scraper):
                    else None
                ),
                "pageNum": 0,
-                "start": page + scraper_input.offset,
+                "start": start,
                "f_AL": "true" if scraper_input.easy_apply else None,
                "f_C": (
                    ",".join(map(str, scraper_input.linkedin_company_ids))
@@ -96,12 +113,9 @@ class LinkedInScraper(Scraper):

            params = {k: v for k, v in params.items() if v is not None}
            try:
-                response = session.get(
+                response = self.session.get(
                    f"{self.base_url}/jobs-guest/jobs/api/seeMoreJobPostings/search?",
                    params=params,
-                    allow_redirects=True,
-                    proxies=self.proxy,
-                    headers=self.headers,
                    timeout=10,
                )
                if response.status_code not in range(200, 400):
@@ -127,36 +141,34 @@ class LinkedInScraper(Scraper):
                return JobResponse(jobs=job_list)

            for job_card in job_cards:
-                job_url = None
                href_tag = job_card.find("a", class_="base-card__full-link")
                if href_tag and "href" in href_tag.attrs:
                    href = href_tag.attrs["href"].split("?")[0]
                    job_id = href.split("-")[-1]
-                    job_url = f"{self.base_url}/jobs/view/{job_id}"

-                with url_lock:
-                    if job_url in seen_urls:
+                    if job_id in seen_ids:
                        continue
-                    seen_urls.add(job_url)
-                try:
-                    fetch_desc = scraper_input.linkedin_fetch_description
-                    job_post = self._process_job(job_card, job_url, fetch_desc)
-                    if job_post:
-                        job_list.append(job_post)
-                    if not continue_search():
-                        break
-                except Exception as e:
-                    raise LinkedInException(str(e))
+                    seen_ids.add(job_id)
+
+                    try:
+                        fetch_desc = scraper_input.linkedin_fetch_description
+                        job_post = self._process_job(job_card, job_id, fetch_desc)
+                        if job_post:
+                            job_list.append(job_post)
+                        if not continue_search():
+                            break
+                    except Exception as e:
+                        raise LinkedInException(str(e))

            if continue_search():
                time.sleep(random.uniform(self.delay, self.delay + self.band_delay))
-                page += self.jobs_per_page
+                start += len(job_list)

        job_list = job_list[: scraper_input.results_wanted]
        return JobResponse(jobs=job_list)

    def _process_job(
-        self, job_card: Tag, job_url: str, full_descr: bool
+        self, job_card: Tag, job_id: str, full_descr: bool
    ) -> Optional[JobPost]:
        salary_tag = job_card.find("span", class_="job-search-card__salary-info")

@@ -194,48 +206,51 @@ class LinkedInScraper(Scraper):
            if metadata_card
            else None
        )
-        date_posted = description = job_type = None
+        date_posted = None
        if datetime_tag and "datetime" in datetime_tag.attrs:
            datetime_str = datetime_tag["datetime"]
            try:
                date_posted = datetime.strptime(datetime_str, "%Y-%m-%d")
            except:
                date_posted = None
-        benefits_tag = job_card.find("span", class_="result-benefits__text")
+        job_details = {}
        if full_descr:
-            description, job_type = self._get_job_description(job_url)
+            job_details = self._get_job_details(job_id)

        return JobPost(
+            id=f"li-{job_id}",
            title=title,
            company_name=company,
            company_url=company_url,
            location=location,
            date_posted=date_posted,
-            job_url=job_url,
+            job_url=f"{self.base_url}/jobs/view/{job_id}",
            compensation=compensation,
-            job_type=job_type,
-            description=description,
-            emails=extract_emails_from_text(description) if description else None,
+            job_type=job_details.get("job_type"),
+            job_level=job_details.get("job_level", "").lower(),
+            company_industry=job_details.get("company_industry"),
+            description=job_details.get("description"),
+            job_url_direct=job_details.get("job_url_direct"),
+            emails=extract_emails_from_text(job_details.get("description")),
+            logo_photo_url=job_details.get("logo_photo_url"),
+            job_function=job_details.get("job_function"),
        )

-    def _get_job_description(
-        self, job_page_url: str
-    ) -> tuple[None, None] | tuple[str | None, tuple[str | None, JobType | None]]:
+    def _get_job_details(self, job_id: str) -> dict:
        """
-        Retrieves job description by going to the job page url
+        Retrieves job description and other job details by going to the job page url
        :param job_page_url:
-        :return: description or None
+        :return: dict
        """
        try:
-            session = create_session(is_tls=False, has_retry=True)
-            response = session.get(
-                job_page_url, headers=self.headers, timeout=5, proxies=self.proxy
+            response = self.session.get(
+                f"{self.base_url}/jobs/view/{job_id}", timeout=5
            )
            response.raise_for_status()
        except:
-            return None, None
-        if response.url == "https://www.linkedin.com/signup":
-            return None, None
+            return {}
+        if "linkedin.com/signup" in response.url:
+            return {}

        soup = BeautifulSoup(response.text, "html.parser")
        div_content = soup.find(
@@ -243,17 +258,37 @@ class LinkedInScraper(Scraper):
        )
        description = None
        if div_content is not None:
-
-            def remove_attributes(tag):
-                for attr in list(tag.attrs):
-                    del tag[attr]
-                return tag
-
            div_content = remove_attributes(div_content)
            description = div_content.prettify(formatter="html")
            if self.scraper_input.description_format == DescriptionFormat.MARKDOWN:
                description = markdown_converter(description)
-        return description, self._parse_job_type(soup)
+
+        h3_tag = soup.find(
+            "h3", text=lambda text: text and "Job function" in text.strip()
+        )
+
+        job_function = None
+        if h3_tag:
+            job_function_span = h3_tag.find_next(
+                "span", class_="description__job-criteria-text"
+            )
+            if job_function_span:
+                job_function = job_function_span.text.strip()
+
+        logo_photo_url = (
+            logo_image.get("data-delayed-url")
+            if (logo_image := soup.find("img", {"class": "artdeco-entity-image"}))
+            else None
+        )
+        return {
+            "description": description,
+            "job_level": self._parse_job_level(soup),
+            "company_industry": self._parse_company_industry(soup),
+            "job_type": self._parse_job_type(soup),
+            "job_url_direct": self._parse_job_url_direct(soup),
+            "logo_photo_url": logo_photo_url,
+            "job_function": job_function,
+        }

    def _get_location(self, metadata_card: Optional[Tag]) -> Location:
        """
@@ -306,6 +341,69 @@ class LinkedInScraper(Scraper):

        return [get_enum_from_job_type(employment_type)] if employment_type else []

+    @staticmethod
+    def _parse_job_level(soup_job_level: BeautifulSoup) -> str | None:
+        """
+        Gets the job level from job page
+        :param soup_job_level:
+        :return: str
+        """
+        h3_tag = soup_job_level.find(
+            "h3",
+            class_="description__job-criteria-subheader",
+            string=lambda text: "Seniority level" in text,
+        )
+        job_level = None
+        if h3_tag:
+            job_level_span = h3_tag.find_next_sibling(
+                "span",
+                class_="description__job-criteria-text description__job-criteria-text--criteria",
+            )
+            if job_level_span:
+                job_level = job_level_span.get_text(strip=True)
+
+        return job_level
+
+    @staticmethod
+    def _parse_company_industry(soup_industry: BeautifulSoup) -> str | None:
+        """
+        Gets the company industry from job page
+        :param soup_industry:
+        :return: str
+        """
+        h3_tag = soup_industry.find(
+            "h3",
+            class_="description__job-criteria-subheader",
+            string=lambda text: "Industries" in text,
+        )
+        industry = None
+        if h3_tag:
+            industry_span = h3_tag.find_next_sibling(
+                "span",
+                class_="description__job-criteria-text description__job-criteria-text--criteria",
+            )
+            if industry_span:
+                industry = industry_span.get_text(strip=True)
+
+        return industry
+
+    def _parse_job_url_direct(self, soup: BeautifulSoup) -> str | None:
+        """
+        Gets the job url direct from job page
+        :param soup:
+        :return: str
+        """
+        job_url_direct = None
+        job_url_direct_content = soup.find("code", id="applyUrl")
+        if job_url_direct_content:
+            job_url_direct_match = self.job_url_direct_regex.search(
+                job_url_direct_content.decode_contents().strip()
+            )
+            if job_url_direct_match:
+                job_url_direct = unquote(job_url_direct_match.group())
+
+        return job_url_direct
+
    @staticmethod
    def job_type_code(job_type_enum: JobType) -> str:
        return {
@@ -315,12 +413,3 @@ class LinkedInScraper(Scraper):
            JobType.CONTRACT: "C",
            JobType.TEMPORARY: "T",
        }.get(job_type_enum, "")
-
-    headers = {
-        "authority": "www.linkedin.com",
-        "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
-        "accept-language": "en-US,en;q=0.9",
-        "cache-control": "max-age=0",
-        "upgrade-insecure-requests": "1",
-        "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
-    }
--- a/src/jobspy/scrapers/linkedin/constants.py
+++ b/src/jobspy/scrapers/linkedin/constants.py
@@ -0,0 +1,8 @@
+headers = {
+    "authority": "www.linkedin.com",
+    "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
+    "accept-language": "en-US,en;q=0.9",
+    "cache-control": "max-age=0",
+    "upgrade-insecure-requests": "1",
+    "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
+}
--- a/src/jobspy/scrapers/utils.py
+++ b/src/jobspy/scrapers/utils.py
@@ -2,23 +2,150 @@ from __future__ import annotations

 import re
 import logging
+from itertools import cycle
+
 import requests
 import tls_client
 import numpy as np
 from markdownify import markdownify as md
 from requests.adapters import HTTPAdapter, Retry

-from ..jobs import JobType
+from ..jobs import CompensationInterval, JobType

-logger = logging.getLogger("JobSpy")
-logger.propagate = False
-if not logger.handlers:
-    logger.setLevel(logging.INFO)
-    console_handler = logging.StreamHandler()
-    format = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
-    formatter = logging.Formatter(format)
-    console_handler.setFormatter(formatter)
-    logger.addHandler(console_handler)
+
+def create_logger(name: str):
+    logger = logging.getLogger(f"JobSpy:{name}")
+    logger.propagate = False
+    if not logger.handlers:
+        logger.setLevel(logging.INFO)
+        console_handler = logging.StreamHandler()
+        format = "%(asctime)s - %(levelname)s - %(name)s - %(message)s"
+        formatter = logging.Formatter(format)
+        console_handler.setFormatter(formatter)
+        logger.addHandler(console_handler)
+    return logger
+
+
+class RotatingProxySession:
+    def __init__(self, proxies=None):
+        if isinstance(proxies, str):
+            self.proxy_cycle = cycle([self.format_proxy(proxies)])
+        elif isinstance(proxies, list):
+            self.proxy_cycle = (
+                cycle([self.format_proxy(proxy) for proxy in proxies])
+                if proxies
+                else None
+            )
+        else:
+            self.proxy_cycle = None
+
+    @staticmethod
+    def format_proxy(proxy):
+        """Utility method to format a proxy string into a dictionary."""
+        if proxy.startswith("http://") or proxy.startswith("https://"):
+            return {"http": proxy, "https": proxy}
+        return {"http": f"http://{proxy}", "https": f"http://{proxy}"}
+
+
+class RequestsRotating(RotatingProxySession, requests.Session):
+
+    def __init__(self, proxies=None, has_retry=False, delay=1, clear_cookies=False):
+        RotatingProxySession.__init__(self, proxies=proxies)
+        requests.Session.__init__(self)
+        self.clear_cookies = clear_cookies
+        self.allow_redirects = True
+        self.setup_session(has_retry, delay)
+
+    def setup_session(self, has_retry, delay):
+        if has_retry:
+            retries = Retry(
+                total=3,
+                connect=3,
+                status=3,
+                status_forcelist=[500, 502, 503, 504, 429],
+                backoff_factor=delay,
+            )
+            adapter = HTTPAdapter(max_retries=retries)
+            self.mount("http://", adapter)
+            self.mount("https://", adapter)
+
+    def request(self, method, url, **kwargs):
+        if self.clear_cookies:
+            self.cookies.clear()
+
+        if self.proxy_cycle:
+            next_proxy = next(self.proxy_cycle)
+            if next_proxy["http"] != "http://localhost":
+                self.proxies = next_proxy
+            else:
+                self.proxies = {}
+        return requests.Session.request(self, method, url, **kwargs)
+
+
+class TLSRotating(RotatingProxySession, tls_client.Session):
+
+    def __init__(self, proxies=None):
+        RotatingProxySession.__init__(self, proxies=proxies)
+        tls_client.Session.__init__(self, random_tls_extension_order=True)
+
+    def execute_request(self, *args, **kwargs):
+        if self.proxy_cycle:
+            next_proxy = next(self.proxy_cycle)
+            if next_proxy["http"] != "http://localhost":
+                self.proxies = next_proxy
+            else:
+                self.proxies = {}
+        response = tls_client.Session.execute_request(self, *args, **kwargs)
+        response.ok = response.status_code in range(200, 400)
+        return response
+
+
+def create_session(
+    *,
+    proxies: dict | str | None = None,
+    ca_cert: str | None = None,
+    is_tls: bool = True,
+    has_retry: bool = False,
+    delay: int = 1,
+    clear_cookies: bool = False,
+) -> requests.Session:
+    """
+    Creates a requests session with optional tls, proxy, and retry settings.
+    :return: A session object
+    """
+    if is_tls:
+        session = TLSRotating(proxies=proxies)
+    else:
+        session = RequestsRotating(
+            proxies=proxies,
+            has_retry=has_retry,
+            delay=delay,
+            clear_cookies=clear_cookies,
+        )
+
+    if ca_cert:
+        session.verify = ca_cert
+
+    return session
+
+
+def set_logger_level(verbose: int = 2):
+    """
+    Adjusts the logger's level. This function allows the logging level to be changed at runtime.
+
+    Parameters:
+    - verbose: int {0, 1, 2} (default=2, all logs)
+    """
+    if verbose is None:
+        return
+    level_name = {2: "INFO", 1: "WARNING", 0: "ERROR"}.get(verbose, "INFO")
+    level = getattr(logging, level_name.upper(), None)
+    if level is not None:
+        for logger_name in logging.root.manager.loggerDict:
+            if logger_name.startswith("JobSpy:"):
+                logging.getLogger(logger_name).setLevel(level)
+    else:
+        raise ValueError(f"Invalid log level: {level_name}")


 def markdown_converter(description_html: str):
@@ -35,39 +162,6 @@ def extract_emails_from_text(text: str) -> list[str] | None:
    return email_regex.findall(text)


-def create_session(
-    proxy: dict | None = None,
-    is_tls: bool = True,
-    has_retry: bool = False,
-    delay: int = 1,
-) -> requests.Session:
-    """
-    Creates a requests session with optional tls, proxy, and retry settings.
-    :return: A session object
-    """
-    if is_tls:
-        session = tls_client.Session(random_tls_extension_order=True)
-        session.proxies = proxy
-    else:
-        session = requests.Session()
-        session.allow_redirects = True
-        if proxy:
-            session.proxies.update(proxy)
-        if has_retry:
-            retries = Retry(
-                total=3,
-                connect=3,
-                status=3,
-                status_forcelist=[500, 502, 503, 504, 429],
-                backoff_factor=delay,
-            )
-            adapter = HTTPAdapter(max_retries=retries)
-
-            session.mount("http://", adapter)
-            session.mount("https://", adapter)
-    return session
-
-
 def get_enum_from_job_type(job_type_str: str) -> JobType | None:
    """
    Given a string, returns the corresponding JobType enum member if a match is found.
@@ -94,3 +188,79 @@ def currency_parser(cur_str):
        num = float(cur_str)

    return np.round(num, 2)
+
+
+def remove_attributes(tag):
+    for attr in list(tag.attrs):
+        del tag[attr]
+    return tag
+
+
+def extract_salary(
+    salary_str,
+    lower_limit=1000,
+    upper_limit=700000,
+    hourly_threshold=350,
+    monthly_threshold=30000,
+    enforce_annual_salary=False,
+):
+    """
+    Extracts salary information from a string and returns the salary interval, min and max salary values, and currency.
+    (TODO: Needs test cases as the regex is complicated and may not cover all edge cases)
+    """
+    if not salary_str:
+        return None, None, None, None
+
+    annual_max_salary = None
+    min_max_pattern = r"\$(\d+(?:,\d+)?(?:\.\d+)?)([kK]?)\s*[-—–]\s*(?:\$)?(\d+(?:,\d+)?(?:\.\d+)?)([kK]?)"
+
+    def to_int(s):
+        return int(float(s.replace(",", "")))
+
+    def convert_hourly_to_annual(hourly_wage):
+        return hourly_wage * 2080
+
+    def convert_monthly_to_annual(monthly_wage):
+        return monthly_wage * 12
+
+    match = re.search(min_max_pattern, salary_str)
+
+    if match:
+        min_salary = to_int(match.group(1))
+        max_salary = to_int(match.group(3))
+        # Handle 'k' suffix for min and max salaries independently
+        if "k" in match.group(2).lower() or "k" in match.group(4).lower():
+            min_salary *= 1000
+            max_salary *= 1000
+
+        # Convert to annual if less than the hourly threshold
+        if min_salary < hourly_threshold:
+            interval = CompensationInterval.HOURLY.value
+            annual_min_salary = convert_hourly_to_annual(min_salary)
+            if max_salary < hourly_threshold:
+                annual_max_salary = convert_hourly_to_annual(max_salary)
+
+        elif min_salary < monthly_threshold:
+            interval = CompensationInterval.MONTHLY.value
+            annual_min_salary = convert_monthly_to_annual(min_salary)
+            if max_salary < monthly_threshold:
+                annual_max_salary = convert_monthly_to_annual(max_salary)
+
+        else:
+            interval = CompensationInterval.YEARLY.value
+            annual_min_salary = min_salary
+            annual_max_salary = max_salary
+
+        # Ensure salary range is within specified limits
+        if not annual_max_salary:
+            return None, None, None, None
+        if (
+            lower_limit <= annual_min_salary <= upper_limit
+            and lower_limit <= annual_max_salary <= upper_limit
+            and annual_min_salary < annual_max_salary
+        ):
+            if enforce_annual_salary:
+                return interval, annual_min_salary, annual_max_salary, "USD"
+            else:
+                return interval, min_salary, max_salary, "USD"
+    return None, None, None, None
--- a/src/jobspy/scrapers/ziprecruiter/init.py
+++ b/src/jobspy/scrapers/ziprecruiter/init.py
@@ -7,19 +7,25 @@ This module contains routines to scrape ZipRecruiter.

 from __future__ import annotations

+import json
 import math
+import re
 import time
 from datetime import datetime
 from typing import Optional, Tuple, Any

 from concurrent.futures import ThreadPoolExecutor

+from bs4 import BeautifulSoup
+
+from .constants import headers
 from .. import Scraper, ScraperInput, Site
 from ..utils import (
-    logger,
    extract_emails_from_text,
    create_session,
    markdown_converter,
+    remove_attributes,
+    create_logger,
 )
 from ...jobs import (
    JobPost,
@@ -31,19 +37,25 @@ from ...jobs import (
    DescriptionFormat,
 )

+logger = create_logger("ZipRecruiter")
+

 class ZipRecruiterScraper(Scraper):
    base_url = "https://www.ziprecruiter.com"
    api_url = "https://api.ziprecruiter.com"

-    def __init__(self, proxy: Optional[str] = None):
+    def __init__(
+        self, proxies: list[str] | str | None = None, ca_cert: str | None = None
+    ):
        """
        Initializes ZipRecruiterScraper with the ZipRecruiter job search url
        """
+        super().__init__(Site.ZIP_RECRUITER, proxies=proxies)
+
        self.scraper_input = None
-        self.session = create_session(proxy)
+        self.session = create_session(proxies=proxies, ca_cert=ca_cert)
+        self.session.headers.update(headers)
        self._get_cookies()
-        super().__init__(Site.ZIP_RECRUITER, proxy=proxy)

        self.delay = 5
        self.jobs_per_page = 20
@@ -65,7 +77,7 @@ class ZipRecruiterScraper(Scraper):
                break
            if page > 1:
                time.sleep(self.delay)
-            logger.info(f"ZipRecruiter search page: {page}")
+            logger.info(f"search page: {page} / {max_pages}")
            jobs_on_page, continue_token = self._find_jobs_in_page(
                scraper_input, continue_token
            )
@@ -91,9 +103,7 @@ class ZipRecruiterScraper(Scraper):
        if continue_token:
            params["continue_from"] = continue_token
        try:
-            res = self.session.get(
-                f"{self.api_url}/jobs-app/jobs", headers=self.headers, params=params
-            )
+            res = self.session.get(f"{self.api_url}/jobs-app/jobs", params=params)
            if res.status_code not in range(200, 400):
                if res.status_code == 429:
                    err = "429 Response - Blocked by ZipRecruiter for too many requests"
@@ -129,6 +139,7 @@ class ZipRecruiterScraper(Scraper):
        self.seen_urls.add(job_url)

        description = job.get("job_description", "").strip()
+        listing_type = job.get("buyer_type", "")
        description = (
            markdown_converter(description)
            if self.scraper_input.description_format == DescriptionFormat.MARKDOWN
@@ -150,7 +161,10 @@ class ZipRecruiterScraper(Scraper):
        comp_min = int(job["compensation_min"]) if "compensation_min" in job else None
        comp_max = int(job["compensation_max"]) if "compensation_max" in job else None
        comp_currency = job.get("compensation_currency")
+        description_full, job_url_direct = self._get_descr(job_url)
+
        return JobPost(
+            id=f'zr-{job["listing_key"]}',
            title=title,
            company_name=company,
            location=location,
@@ -163,14 +177,47 @@ class ZipRecruiterScraper(Scraper):
            ),
            date_posted=date_posted,
            job_url=job_url,
-            description=description,
+            description=description_full if description_full else description,
            emails=extract_emails_from_text(description) if description else None,
+            job_url_direct=job_url_direct,
+            listing_type=listing_type,
        )

+    def _get_descr(self, job_url):
+        res = self.session.get(job_url, allow_redirects=True)
+        description_full = job_url_direct = None
+        if res.ok:
+            soup = BeautifulSoup(res.text, "html.parser")
+            job_descr_div = soup.find("div", class_="job_description")
+            company_descr_section = soup.find("section", class_="company_description")
+            job_description_clean = (
+                remove_attributes(job_descr_div).prettify(formatter="html")
+                if job_descr_div
+                else ""
+            )
+            company_description_clean = (
+                remove_attributes(company_descr_section).prettify(formatter="html")
+                if company_descr_section
+                else ""
+            )
+            description_full = job_description_clean + company_description_clean
+            script_tag = soup.find("script", type="application/json")
+            if script_tag:
+                job_json = json.loads(script_tag.string)
+                job_url_val = job_json["model"].get("saveJobURL", "")
+                m = re.search(r"job_url=(.+)", job_url_val)
+                if m:
+                    job_url_direct = m.group(1)
+
+            if self.scraper_input.description_format == DescriptionFormat.MARKDOWN:
+                description_full = markdown_converter(description_full)
+
+        return description_full, job_url_direct
+
    def _get_cookies(self):
        data = "event_type=session&logged_in=false&number_of_retry=1&property=model%3AiPhone&property=os%3AiOS&property=locale%3Aen_us&property=app_build_number%3A4734&property=app_version%3A91.0&property=manufacturer%3AApple&property=timestamp%3A2024-01-12T12%3A04%3A42-06%3A00&property=screen_height%3A852&property=os_version%3A16.6.1&property=source%3Ainstall&property=screen_width%3A393&property=device_model%3AiPhone%2014%20Pro&property=brand%3AApple"
        url = f"{self.api_url}/jobs-app/event"
-        self.session.post(url, data=data, headers=self.headers)
+        self.session.post(url, data=data)

    @staticmethod
    def _get_job_type_enum(job_type_str: str) -> list[JobType] | None:
@@ -198,14 +245,3 @@ class ZipRecruiterScraper(Scraper):
        if scraper_input.distance:
            params["radius"] = scraper_input.distance
        return {k: v for k, v in params.items() if v is not None}
-
-    headers = {
-        "Host": "api.ziprecruiter.com",
-        "accept": "*/*",
-        "x-zr-zva-override": "100000000;vid:ZT1huzm_EQlDTVEc",
-        "x-pushnotificationid": "0ff4983d38d7fc5b3370297f2bcffcf4b3321c418f5c22dd152a0264707602a0",
-        "x-deviceid": "D77B3A92-E589-46A4-8A39-6EF6F1D86006",
-        "user-agent": "Job Search/87.0 (iPhone; CPU iOS 16_6_1 like Mac OS X)",
-        "authorization": "Basic YTBlZjMyZDYtN2I0Yy00MWVkLWEyODMtYTI1NDAzMzI0YTcyOg==",
-        "accept-language": "en-US,en;q=0.9",
-    }
--- a/src/jobspy/scrapers/ziprecruiter/constants.py
+++ b/src/jobspy/scrapers/ziprecruiter/constants.py
@@ -0,0 +1,10 @@
+headers = {
+    "Host": "api.ziprecruiter.com",
+    "accept": "*/*",
+    "x-zr-zva-override": "100000000;vid:ZT1huzm_EQlDTVEc",
+    "x-pushnotificationid": "0ff4983d38d7fc5b3370297f2bcffcf4b3321c418f5c22dd152a0264707602a0",
+    "x-deviceid": "D77B3A92-E589-46A4-8A39-6EF6F1D86006",
+    "user-agent": "Job Search/87.0 (iPhone; CPU iOS 16_6_1 like Mac OS X)",
+    "authorization": "Basic YTBlZjMyZDYtN2I0Yy00MWVkLWEyODMtYTI1NDAzMzI0YTcyOg==",
+    "accept-language": "en-US,en;q=0.9",
+}
--- a/src/tests/init.py
+++ b/src/tests/init.py
--- a/src/tests/test_all.py
+++ b/src/tests/test_all.py
@@ -1,14 +0,0 @@
-from ..jobspy import scrape_jobs
-import pandas as pd
-
-
-def test_all():
-    result = scrape_jobs(
-        site_name=["linkedin", "indeed", "zip_recruiter", "glassdoor"],
-        search_term="software engineer",
-        results_wanted=5,
-    )
-
-    assert (
-        isinstance(result, pd.DataFrame) and not result.empty
-    ), "Result should be a non-empty DataFrame"
--- a/src/tests/test_glassdoor.py
+++ b/src/tests/test_glassdoor.py
@@ -1,11 +0,0 @@
-from ..jobspy import scrape_jobs
-import pandas as pd
-
-
-def test_indeed():
-    result = scrape_jobs(
-        site_name="glassdoor", search_term="software engineer", country_indeed="USA"
-    )
-    assert (
-        isinstance(result, pd.DataFrame) and not result.empty
-    ), "Result should be a non-empty DataFrame"
--- a/src/tests/test_indeed.py
+++ b/src/tests/test_indeed.py
@@ -1,11 +0,0 @@
-from ..jobspy import scrape_jobs
-import pandas as pd
-
-
-def test_indeed():
-    result = scrape_jobs(
-        site_name="indeed", search_term="software engineer", country_indeed="usa"
-    )
-    assert (
-        isinstance(result, pd.DataFrame) and not result.empty
-    ), "Result should be a non-empty DataFrame"
--- a/src/tests/test_linkedin.py
+++ b/src/tests/test_linkedin.py
@@ -1,12 +0,0 @@
-from ..jobspy import scrape_jobs
-import pandas as pd
-
-
-def test_linkedin():
-    result = scrape_jobs(
-        site_name="linkedin",
-        search_term="software engineer",
-    )
-    assert (
-        isinstance(result, pd.DataFrame) and not result.empty
-    ), "Result should be a non-empty DataFrame"
--- a/src/tests/test_ziprecruiter.py
+++ b/src/tests/test_ziprecruiter.py
@@ -1,13 +0,0 @@
-from ..jobspy import scrape_jobs
-import pandas as pd
-
-
-def test_ziprecruiter():
-    result = scrape_jobs(
-        site_name="zip_recruiter",
-        search_term="software engineer",
-    )
-
-    assert (
-        isinstance(result, pd.DataFrame) and not result.empty
-    ), "Result should be a non-empty DataFrame"
--- a/tests/init.py
+++ b/tests/init.py
--- a/tests/test_all.py
+++ b/tests/test_all.py
@@ -0,0 +1,18 @@
+from jobspy import scrape_jobs
+import pandas as pd
+
+
+def test_all():
+    sites = [
+        "indeed",
+        "glassdoor",
+    ]  # ziprecruiter/linkedin needs good ip, and temp fix to pass test on ci
+    result = scrape_jobs(
+        site_name=sites,
+        search_term="engineer",
+        results_wanted=5,
+    )
+
+    assert (
+        isinstance(result, pd.DataFrame) and len(result) == len(sites) * 5
+    ), "Result should be a non-empty DataFrame"
--- a/tests/test_glassdoor.py
+++ b/tests/test_glassdoor.py
@@ -0,0 +1,13 @@
+from jobspy import scrape_jobs
+import pandas as pd
+
+
+def test_glassdoor():
+    result = scrape_jobs(
+        site_name="glassdoor",
+        search_term="engineer",
+        results_wanted=5,
+    )
+    assert (
+        isinstance(result, pd.DataFrame) and len(result) == 5
+    ), "Result should be a non-empty DataFrame"
--- a/tests/test_indeed.py
+++ b/tests/test_indeed.py
@@ -0,0 +1,13 @@
+from jobspy import scrape_jobs
+import pandas as pd
+
+
+def test_indeed():
+    result = scrape_jobs(
+        site_name="indeed",
+        search_term="engineer",
+        results_wanted=5,
+    )
+    assert (
+        isinstance(result, pd.DataFrame) and len(result) == 5
+    ), "Result should be a non-empty DataFrame"
--- a/tests/test_linkedin.py
+++ b/tests/test_linkedin.py
@@ -0,0 +1,9 @@
+from jobspy import scrape_jobs
+import pandas as pd
+
+
+def test_linkedin():
+    result = scrape_jobs(site_name="linkedin", search_term="engineer", results_wanted=5)
+    assert (
+        isinstance(result, pd.DataFrame) and len(result) == 5
+    ), "Result should be a non-empty DataFrame"
--- a/tests/test_ziprecruiter.py
+++ b/tests/test_ziprecruiter.py
@@ -0,0 +1,12 @@
+from jobspy import scrape_jobs
+import pandas as pd
+
+
+def test_ziprecruiter():
+    result = scrape_jobs(
+        site_name="zip_recruiter", search_term="software engineer", results_wanted=5
+    )
+
+    assert (
+        isinstance(result, pd.DataFrame) and len(result) == 5
+    ), "Result should be a non-empty DataFrame"
Author	SHA1	Message	Date
Cullen Watson	9c43f82fb1	pass test	2024-10-19 18:01:02 -05:00
Cullen Watson	6ba571f5e4	pass test	2024-10-19 17:58:26 -05:00
Cullen Watson	b43289fa38	indeed:remove tpe	2024-10-19 17:55:36 -05:00
Olzhas Arystanov	9207ab56f6	fix: extract tests out of src (#209 )	2024-10-19 16:56:38 -05:00
Cullen Watson	757a94853e	chore:version	2024-10-08 17:49:06 -05:00
Marcel Gozalbo Baró	6bc191d5c7	FEATURE: Add the "ca_cert" setting for providing a Certification Authority certificate in order to use proxies requiring it. (#204 )	2024-10-08 17:46:46 -05:00
Cullen Watson	0cc34287f7	fix:turkey	2024-10-02 01:31:00 -05:00
Anton Pikhteryev	923979093b	Add Malta for linkedin country support (#198 )	2024-09-19 20:41:22 -05:00
Cullen Watson	286f0e4487	docs:readme	2024-09-18 18:49:41 -05:00
Cullen Watson	f7b29d43a2	fix(indeed):sort relevance not date (#197 )	2024-09-18 18:42:25 -05:00
Cullen Watson	6f1490458c	fix key error (#186 )	2024-08-14 02:54:40 -05:00
Cullen Watson	6bb7d81ba8	change linkedin ep (#185 )	2024-08-14 02:39:43 -05:00
Cullen Watson	0e046432d1	fix:variable bug (#181 )	2024-08-05 12:47:55 -05:00
Cullen Watson	209e0e65b6	fix:malaysia indeed (#180 )	2024-08-03 22:48:53 -05:00
Cullen Watson	8570c0651e	fix:key error (#176 )	2024-07-21 13:05:18 -05:00
Cullen Watson	8678b0bbe4	enh: test on pr (#174 )	2024-07-19 14:25:25 -05:00
Cullen Watson	60d4d911c9	lock file (#173 )	2024-07-17 21:21:22 -05:00
Lluís Salord Quetglas	2a0cba8c7e	FEAT: Optional convertion to annual and know salary source (#170 )	2024-07-17 21:05:33 -05:00
Mason DePalma	de70189fa2	Update pyproject.toml (#172 ) Changed Numpy to the most recent version so the package can properly install	2024-07-17 20:54:08 -05:00
Cullen Watson	b55c0eb86d	docs:readme	2024-07-16 19:24:38 -05:00
Cullen Watson	88c95c4ad5	enh: estimated salary (#169 )	2024-07-16 19:20:34 -05:00
Cullen Watson	d8d33d602f	docs: readme	2024-07-15 21:30:11 -05:00
Cullen Watson	6330c14879	minor fix	2024-07-15 21:19:01 -05:00
Ali Bakhshi Ilani	48631ea271	Add company industry and job level to linkedin scraper (#166 )	2024-07-15 21:07:39 -05:00
Cullen Watson	edffe18e65	enh: listing source (#168 )	2024-07-15 20:30:04 -05:00
Lluís Salord Quetglas	0988230a24	FEAT: Add Glassdoor logo data if available (#167 )	2024-07-15 20:25:18 -05:00
Cullen Watson	d000a81eb3	Salary parse (#163 )	2024-06-09 17:45:38 -05:00
Cullen Watson	ccb0c17660	enh: ziprecruiter full description (#162 )	2024-06-09 16:21:01 -05:00
Cullen Watson	df339610fa	docs: readme	2024-05-29 19:32:32 -05:00
Cullen Watson	c501006bd8	docs: readme	2024-05-28 16:04:26 -05:00
Cullen Watson	89a3ee231c	enh(li): job function (#160 )	2024-05-28 16:01:29 -05:00
Cullen	6439f71433	chore: version	2024-05-28 15:39:24 -05:00
adamagassi	7f6271b2e0	LinkedIn scraper fixes: (#159 ) Correct initial page offset calculation Separate page variable from request counter Fix job offset starting value Increment offset by number of jobs returned instead of expected value	2024-05-28 15:38:13 -05:00
Cullen Watson	5cb7ffe5fd	enh: proxies (#157 ) * enh: proxies * enh: proxies	2024-05-25 14:04:09 -05:00
Cullen Watson	cd29f79796	docs: readme	2024-05-25 11:46:23 -05:00
Cullen Watson	65d2e5e707	Update pyproject.toml	2024-05-20 11:46:36 -05:00
fasih hussain	08d63a87a2	chore: id added for JobPost schema (#152 )	2024-05-20 11:45:52 -05:00
Cullen	1ffdb1756f	fix: dup line	2024-04-30 12:11:48 -05:00
Cullen Watson	1185693422	delete empty file	2024-04-30 12:06:20 -05:00
Lluís Salord Quetglas	dcd7144318	FIX: Allow Indeed search term with complex syntax (#139 )	2024-04-30 12:05:43 -05:00
Cullen Watson	bf73c061bd	enh: linkedin company logo (#141 )	2024-04-30 12:03:10 -05:00
Lluís Salord Quetglas	8dd08ed9fd	FEAT: Allow LinkedIn scraper to get external job apply url (#140 )	2024-04-30 11:36:01 -05:00
Cullen Watson	5d3df732e6	docs: readme	2024-03-12 20:46:25 -05:00
Kellen Mace	86f858e06d	Update scrape_jobs() parameters info in readme (#130 )	2024-03-12 20:45:13 -05:00
Cullen	1089d1f0a5	docs: readme	2024-03-11 21:30:57 -05:00
Cullen	3e93454738	fix(indeed): readd param	2024-03-11 21:23:20 -05:00
Cullen Watson	0d150d519f	docs: readme	2024-03-11 14:52:20 -05:00
Cullen Watson	cc3497f929	docs: readme	2024-03-11 14:45:17 -05:00
Cullen Watson	5986f75346	docs: readme	2024-03-11 14:41:12 -05:00
VitaminB16	4b7bdb9313	feat: Adjust log verbosity via verbose arg (#128 )	2024-03-11 14:38:44 -05:00