Downloader Middlewares¶

class kingfisher_scrapy.downloadermiddlewares.BaseDownloaderMiddleware(crawler)[source]¶

Base class for downloader middlewares that need access to the spider instance.

classmethod from_crawler(crawler)[source]¶

class kingfisher_scrapy.downloadermiddlewares.ParaguayAuthMiddleware(crawler)[source]¶

Downloader middleware that manages API authentication for Paraguay scrapers.

Both DNCP (procurement authority) and Hacienda (finance ministry) use an authentication protocol based on OAuth 2.

This middleware helps us to manage the protocol, which consists on acquiring an access token every x minutes (usually 15) and sending the token on each request. The acquisition method of the token is delegated to the spider, since each publisher has their own credentials and requirements.

Apparently, a Downloader Middleware is the best place to set HTTP Request Headers (see https://docs.scrapy.org/en/latest/topics/architecture.html), but it’s not enough for this case :(. Tokens should be generated and assigned just before sending a request, but Scrapy does not provide any way to do this, which in turn means that sometimes we accidentally send expired tokens. For now, the issue seems to be avoided by setting the number of concurrent requests to 1, at cost of download speed.

class Paraguay:
    name = 'paraguay'

    # ParaguayAuthMiddleware
    access_token = None
    access_token_scheduled_at = None
    # The maximum age is less than the API's limit, since we don't precisely control Scrapy's scheduler.
    access_token_maximum_age = 14 * 60
    access_token_request_failed = False
    requests_backlog = []

    def build_access_token_request(self):
        self.access_token_scheduled_at = datetime.datetime.now()

        return scrapy.Request("https://example.com")

process_request(request)[source]¶

process_response(request, response)[source]¶

class kingfisher_scrapy.downloadermiddlewares.DelayedRequestMiddleware[source]¶

Downloader middleware that allows for delaying a request by a set ‘wait_time’ number of seconds.

A delayed request is useful when an API fails and works again after waiting a few minutes.

async process_request(request)[source]¶

class kingfisher_scrapy.downloadermiddlewares.CloudflareMiddleware(crawler)[source]¶

Downloader middleware that reuses a manually-solved Cloudflare Turnstile clearance and detects when it is stale.

It is active only for spiders that set cloudflare_protected = True, so it is safe to enable globally.

A human solves the “Verify you are human” challenge, and the spider reuses the resulting cf_clearance cookie (the CF_CLEARANCE setting). Cloudflare binds the cookie to the browser, operating system and IP that solved the challenge, and verifies all three on every later request. So, the challenge must be solved using the same browser, operating system and IP as future crawls:

The browser and operating system are impersonated using the CF_USER_AGENT (User-Agent header) and CURL_IMPERSONATE (TLS/JA3 fingerprint) settings.
The IP can not be impersonated, so the challenge must be solved using the same IP as future crawls, over the same IP version (the CURL_IP_VERSION setting).

An HTML content type is treated as a Cloudflare challenge. If so, the middleware posts a Slack message (to the SLACK_WEBHOOK_URL setting) and stops the crawl.