Executive Summary

CVE-2026-46497 is a blind Server-Side Request Forgery (SSRF) vulnerability in the Crawlee Python library, affecting versions 1.0.0 to 1.6.3. It allows an attacker who controls sitemap or robots.txt files to manipulate the crawler into making HTTP requests to internal hosts or non-HTTP URLs by bypassing URL scheme validation.

The vulnerability has two layers: the first allows cross-host HTTP SSRF by enqueuing internal URLs from attacker-controlled sitemaps or robots.txt files; the second escalates to non-HTTP schemes (such as gopher, file, ftp) when using the CurlImpersonateHttpClient.

Exploitation requires the attacker to control the sitemap or robots.txt content, the crawler to fetch it, and network access to the target. The issue was fixed in version 1.7.0 by enforcing host-based filtering and stricter URL validation.

Useful 0 Not Useful 0

Impact Analysis

This vulnerability can allow an attacker to make the crawler send requests to internal services or non-HTTP resources that are normally inaccessible, potentially exposing sensitive internal network information.

Possible impacts include probing internal services, exploiting Redis servers via the gopher protocol, or accessing local files through the file protocol.

Because the SSRF is blind, the attacker may not see direct responses but can still infer information based on the crawler's behavior or side effects.

Useful 0 Not Useful 0

Detection Guidance

This vulnerability involves SSRF via sitemap-derived URLs in Crawlee versions 1.0.0 to 1.6.3, where an attacker can manipulate sitemap or robots.txt files to force HTTP requests to internal or non-HTTP URLs.

To detect exploitation attempts on your network or system, monitor HTTP requests initiated by Crawlee that target internal hosts or unusual URL schemes such as gopher://, file://, or ftp://.

You can use network monitoring tools or packet capture utilities (e.g., tcpdump, Wireshark) to filter outgoing requests from the Crawlee process.

• Use tcpdump to capture HTTP requests from the Crawlee host: tcpdump -i <interface> 'tcp port 80 or tcp port 443'
• Filter logs or network traffic for requests to internal IP ranges (e.g., 10.0.0.0/8, 192.168.0.0/16) or non-HTTP schemes.
• Inspect Crawlee logs for URLs enqueued from sitemaps or robots.txt that point to internal or non-HTTP URLs.

Useful 0 Not Useful 0

Mitigation Strategies

The primary mitigation is to upgrade Crawlee to version 1.7.0 or later, where the vulnerability is patched.

Version 1.7.0 enforces host-based filtering and stricter URL validation at both sitemap loading and HTTP client levels, preventing SSRF via manipulated sitemaps or robots.txt.

If upgrading immediately is not possible, consider implementing network-level restrictions to block Crawlee from making requests to internal IP ranges or non-HTTP URL schemes.

Additionally, review and restrict the sources of sitemaps and robots.txt files that Crawlee processes to trusted domains only.

Useful 0 Not Useful 0