crawlee

Crawlee for Python

View on Pypi

Activity

Latest release: 3d ago
Total releases: 202
Cadence: ~daily
Last 12 months: 149

Details

License: Apache-2.0
First release: Jan 30, 2024

Releases

Version	Released
`1.11.1b2` pre	Oct 26, 2026
`1.11.1b1` pre	Oct 26, 2026
`1.11.0` minor	Oct 15, 2026
`1.10.5b5` pre	Oct 14, 2026
`1.10.5b4` pre	Oct 14, 2026
`1.10.5b3` pre	Oct 08, 2026
`1.10.5b2` pre	Oct 08, 2026
`1.10.5b1` pre	Oct 08, 2026
`1.10.4` patch	Oct 06, 2026
`1.10.4b4` pre	Oct 06, 2026
`1.10.4b3` pre	Oct 02, 2026
`1.10.4b2` pre	Oct 01, 2026
`1.10.4b1` pre	Oct 01, 2026
`1.10.3` patch	Sep 29, 2026
`1.10.3b6` pre	Sep 29, 2026
`1.10.3b5` pre	Sep 29, 2026
`1.10.3b4` pre	Sep 29, 2026
`1.10.3b3` pre	Sep 29, 2026
`1.10.3b2` pre	Sep 26, 2026
`1.10.3b1` pre	Sep 24, 2026
`1.10.2` patch	Sep 22, 2026
`1.10.2b1` pre	Sep 22, 2026
`1.10.1` patch	Sep 16, 2026
`1.10.1b5` pre	Sep 16, 2026
`1.10.1b4` pre	Sep 16, 2026
`1.10.1b3` pre	Sep 16, 2026
`1.10.1b2` pre	Sep 10, 2026
`1.10.1b1` pre	Sep 02, 2026
`1.10.0` minor	Aug 31, 2026
`1.9.4b5` pre	Aug 31, 2026
`1.9.4b4` pre	Aug 25, 2026
`1.9.4b3` pre	Aug 25, 2026
`1.9.4b2` pre	Aug 24, 2026
`1.9.4b1` pre	Aug 24, 2026
`1.9.3` patch	Aug 24, 2026
`1.9.3b6` pre	Aug 21, 2026
`1.9.3b5` pre	Aug 21, 2026
`1.9.3b4` pre	Aug 21, 2026
`1.9.3b3` pre	Aug 21, 2026
`1.9.3b2` pre	Aug 19, 2026
`1.9.3b1` pre	Aug 18, 2026
`1.9.2` patch	Aug 17, 2026
`1.9.2b4` pre	Aug 17, 2026
`1.9.2b3` pre	Aug 14, 2026
`1.9.2b2` pre	Aug 12, 2026
`1.9.2b1` pre	Aug 12, 2026
`1.9.1` patch	Aug 06, 2026
`1.9.1b5` pre	Aug 06, 2026
`1.9.1b4` pre	Aug 05, 2026
`1.9.1b3` pre	Aug 04, 2026

1–50 of 202

Dependencies (42)

Depends on

1.11.1b2

Used by

Supply-chain exposure

16 tracked projects depend on this, <1% of every package we track.

A known CVE here directly exposes every one of them.

Known CVEs (1)

CVE-2026-46497 PYSEC-2026-2430 GHSA-3r75-xc34-5f44 Jul 13, 2026

Crawlee for Python: SSRF via sitemap-derived URLs

Medium
Network Low None

Overview

Vulnerability type: Blind SSRF

Affected components: src/crawlee/_utils/sitemap.py, src/crawlee/_utils/robots.py, src/crawlee/request_loaders/_sitemap_request_loader.py, and all built-in HTTP clients.

Trigger: an attacker-controlled sitemap or robots.txt containing a URL that points to an internal host (layer 1) or uses a non-http scheme (layer 2).

Two-layer SSRF via sitemap-derived URLs:

1) Cross-host HTTP SSRF

Base case, affects every HTTP client.** Sitemap entries and robots.txt Sitemap: directives were accepted regardless of the host they pointed to. A sitemap on example.com could push http://internal.corp/admin into the crawler's queue, and the configured HTTP client would dispatch the request.

2) Non-HTTP scheme SSRF

Escalation, only CurlImpersonateHttpClient.** Nested-sitemap fetching dispatches the URL straight to the HTTP client, bypassing the Request construction step where Pydantic enforces http(s). Combined with the libcurl-backed CurlImpersonateHttpClient, this lets gopher://, file://, dict://, ftp://, etc., through.

Root cause

Crawlee already validates URL schemes through Pydantic's AnyHttpUrl (via validate_http_url in src/crawlee/_utils/urls.py) wherever a crawl target is materialised as a Request: the Request.url field is declared as Annotated[str, BeforeValidator(validate_http_url), Field(frozen=True)]. Anything that becomes a Request is therefore guaranteed to be http(s).

Two parts of the sitemap pipeline sidestepped this property in different ways:

1) Sitemap-derived URLs were enqueued without any host policy

SitemapRequestLoader took every <urlset><url><loc> entry, wrapped it in Request.from_url (which accepts any valid http(s) URL), and pushed the result into the request queue. RobotsTxtFile.get_sitemaps() returned every Sitemap: directive verbatim. Neither imposed any host check against the parent sitemap or robots.txt URL, so an attacker controlling that content could push internal-network HTTP URLs into the queue and have them crawled by whichever HTTP client was configured.

2) Nested sitemap fetching bypassed the Request chokepoint entirely

When _XmlSitemapParser encountered <sitemapindex><sitemap><loc>…</loc></sitemap></sitemapindex>, or when RobotsTxtFile.parse_sitemaps forwarded Sitemap: directives into the same pipeline, _fetch_and_process_sitemap dispatched the URL directly to the HTTP client:

async with http_client.stream( sitemap_url, method='GET', headers=SITEMAP_HEADERS, proxy_info=proxy_info, timeout=timeout, ) as response: ...

No Request was constructed, so the Pydantic validator never ran. Before the fix, the HTTP clients' own send_request() and stream() methods did not call validate_http_url either, so a non-http(s) scheme could pass straight through to the backend client.

The non-HTTP escalation in layer 2 is specific to CurlImpersonateHttpClient, which is backed by curl-cffi / libcurl and speaks gopher, file, dict, ftp, and other non-HTTP protocols. The other clients shipped with Crawlee (HttpxHttpClient, ImpitHttpClient, PlaywrightHttpClient) reject non-http(s) schemes at their own backend layer, regardless of what Crawlee passes in, so they were only affected by layer 1.

Vulnerable paths

Layer 1 — cross-host HTTP (all HTTP clients)

Source: an attacker-controlled sitemap that lists internal URLs under <urlset><url><loc> or <sitemapindex><sitemap><loc>, or an attacker-controlled robots.txt that lists internal URLs under Sitemap:.

Sink: the configured HTTP client issues GET requests against those URLs — either via client.request(url=request.url, …) inside crawl() for regular sitemap URLs, or via client.stream(url, …) inside the nested-sitemap fetch.

Layer 2 — non-HTTP schemes (CurlImpersonateHttpClient only)

Source: a nested <sitemap><loc> entry or a robots.txt Sitemap: directive pointing to a non-http(s) URL.

Sink: CurlImpersonateHttpClient.stream(...) hands the URL string verbatim to client.request(url=…, …), which dispatches via libcurl.

Hardening in 1.7.0 was added at both producer and consumer ends — see Remediation.

Exploitation preconditions

The crawler uses sitemap loading: any of SitemapRequestLoader, Sitemap.load / parse_sitemap, discover_valid_sitemaps, or RobotsTxtFile.parse_sitemaps.

The attacker controls the body of a sitemap or robots.txt that the crawler fetches — typically by being the target site, or by getting a target site to publish a malicious sitemap.

The crawler's network egress can reach the attacker-chosen destination (e.g., internal services on the same network).

The targeted endpoint accepts unauthenticated requests. Crawlee does not supply credentials to the forged destination, so authenticated services (IMDSv2 with token, password-protected Redis, protected admin panels) are not reachable through this path.

For layer 2 (non-HTTP), the configured HTTP client must additionally be CurlImpersonateHttpClient.

Impact

Layer 1 — cross-host HTTP (any client)

The crawler can be coerced into issuing GET requests against internal HTTP services on its own network: admin panels, unauthenticated internal APIs, cloud metadata endpoints, etc. Read-back is blind — Crawlee surfaces fetched content only through its local Dataset / KeyValueStore (push_data() etc.) and does not natively forward scraped bodies anywhere external — so direct impact is mostly existence/timing probing and occasional state changes via side-effecting GET endpoints. Read-side leakage of internal content is only exploitable end-to-end if the deployer's own application separately exposes scraped data (for example, a public summariser or aggregator built on top of Crawlee).

Layer 2 — non-HTTP escalation (only CurlImpersonateHttpClient)

Under the affected client, attackers gain the libcurl scheme set:

gopher:// is the canonical RESP-injection vector: pipeline FLUSHALL, CONFIG SET dir, CONFIG SET dbfilename, SAVE to an unauthenticated Redis on the crawler's network — enough to write attacker-controlled bytes to disk and, in the standard escalation, achieve remote code execution on the Redis host.

file:// allows the crawler to read local files (application secrets, configuration) on the crawler host.

dict:// and ftp:// permit fingerprinting and limited interaction with text-protocol services.

In both layers, the SSRF is blind in the default configuration. Write-side impact (gopher:// → Redis) and timing-based internal probing do not depend on read-back and remain viable regardless of whether the deployer surfaces scraped content.

Remediation

Both layers are fixed in crawlee==1.7.0. The fix is split across two PRs, applied at the two complementary boundaries of the affected pipeline:

Producer-side filtering — sitemap and robots.txt loaders (PR #1864). SitemapRequestLoader and RobotsTxtFile.get_sitemaps() now run every nested-sitemap entry, every regular sitemap URL, and every Sitemap: directive through crawlee._utils.urls.filter_url. This applies to an EnqueueStrategy (default 'same-hostname') against the parent sitemap / robots.txt URL — cross-host entries are dropped — and rejects non-http(s) schemes. The strategy is stamped onto the emitted Requests, so BasicCrawler._check_url_after_redirects continues policing the policy across redirects.

Consumer-side validation — HTTP-client boundary (PR #1862). validate_http_url(url) is now called at the top of send_request() and stream() in ImpitHttpClient, HttpxHttpClient, CurlImpersonateHttpClient, and PlaywrightHttpClient. Non-http(s) schemes raise pydantic.ValidationError before any backend call. crawl() was already covered, because Request.url is validated by Pydantic on construction.

After these changes, validation is enforced both where sitemap-derived HTTP requests are produced (sitemap and robots.txt loaders) and where they are consumed (HTTP clients). A regression at either layer is caught by the other.

Behaviour change for upgraders

SitemapRequestLoader and RobotsTxtFile.get_sitemaps() now default to enqueue_strategy='same-hostname'. Deployers that legitimately relied on cross-host sitemap entries (e.g., a sitemap index on sitemaps.example.com that points to content on www.example.com) must opt in explicitly with enqueue_strategy='same-domain' or enqueue_strategy='all'.

Finder credits

@r0otsu

@Yuremin (Zhengmin Yu)

@FORIMOC

@invoke1442 (Ethan Carter)

@Arturo0x90 (Arturo Melgarejo)

Affected versions

1.0.01.0.11.0.21.0.31.0.41.1.01.1.11.2.01.2.11.3.01.3.11.3.2

+ 34 more Show less

1.4.01.5.01.5.1b11.5.1b21.5.1b31.5.1b41.5.1b51.6.01.6.11.6.1b11.6.1b21.6.1b31.6.1b41.6.1b51.6.21.6.2b11.6.2b21.6.2b31.6.2b41.6.31.6.3b11.6.3b21.6.3b31.6.3b41.6.3b51.6.3b61.6.4b11.6.4b21.6.4b31.6.4b41.6.4b51.6.4b61.6.4b71.6.4b8

Fixed in

1.7.0

References

https://github.com/apify/crawlee-python/security/advisories/GHSA-3r75-xc34-5f44
https://nvd.nist.gov/vuln/detail/CVE-2026-46497
https://github.com/apify/crawlee-python
https://github.com/apify/crawlee-python/releases/tag/v1.7.0
https://pypi.org/project/crawlee
https://github.com/advisories/GHSA-3r75-xc34-5f44

Updated Jul 13, 2026 · Source: OSV.dev

Insights

Releases per month

last 12 months

Release mix

major 1
minor 17
patch 64
pre 119

202 releases

Dependencies over time

per release

Release calendar

2026

Jan

Feb

Mar

Apr

May

Jun

Jul

Aug

Sep

Oct

Nov

Dec

Less More

crawlee

Activity

Details

Overview

1) Cross-host HTTP SSRF

2) Non-HTTP scheme SSRF

Root cause

1) Sitemap-derived URLs were enqueued without any host policy

2) Nested sitemap fetching bypassed the `Request` chokepoint entirely

Vulnerable paths

Layer 1 — cross-host HTTP (all HTTP clients)

Layer 2 — non-HTTP schemes (`CurlImpersonateHttpClient` only)

Exploitation preconditions

Impact

Layer 1 — cross-host HTTP (any client)

Layer 2 — non-HTTP escalation (only `CurlImpersonateHttpClient`)

Remediation

Behaviour change for upgraders

Finder credits