Download Handlers

class kingfisher_scrapy.downloadhandlers.CurlImpersonateDownloadHandler(settings)

A download handler that uses curl_cffi to impersonate a browser’s TLS/JA3 fingerprint.

Some sites use an anti-bot service (like Cloudflare) that rejects Scrapy’s default Twisted client by its TLS/JA3 fingerprint. curl_cffi reproduces a real browser’s fingerprint.

To use it for a spider, override the https (and/or http) handler in the spider’s custom_settings:

custom_settings = {
    "DOWNLOAD_HANDLERS": {
        "https": "kingfisher_scrapy.downloadhandlers.CurlImpersonateDownloadHandler",
    },
}

And optionally:

• Set the CURL_IMPERSONATE setting to a browser profile. Choose a specific version (like "chrome146") for a consistent fingerprint across curl_cffi upgrades.

• Set the CURL_IP_VERSION setting to "4" or "6" for a consistent version across requests. If unset, curl_cffi chooses.

lazy = True

IP_RESOLVE = {'4': CurlIpResolve.V4, '6': CurlIpResolve.V6}

classmethod from_crawler(crawler)

async download_request(request)

async close()