Big File Spider

class kingfisher_scrapy.base_spiders.big_file_spider.BigFileSpider(*args, **kwargs)

Collect data from a source that publishes very large packages.

Each package is split into smaller packages, each containing 100 releases or records. Users can then process the files without using an iterative parser and without having memory issues.

1. Inherit from BigFileSpider

2. Write a start() method to request the archive files

from kingfisher_scrapy.base_spiders import BigFileSpider
from kingfisher_scrapy.util import components

class MySpider(BigFileSpider):
    name = 'my_spider'

    async def start(self):
        yield self.build_request('https://example.com/api/package.json', formatter=components(-1)

Note

concatenated_json = True, line_delimited = True and root_path are not supported, because this spider yields items whose data field has package and data keys.

resize_package = True

classmethod from_crawler(crawler, *args, **kwargs)

parse(response)

Process response, i.e. extract data from it and generate new requests.

This is the default callback: Scrapy uses it for the response to any request that does not define a callback, such as the requests that start() yields by default.

Any cb_kwargs of the request are passed as keyword parameters.

Spiders must define this method, unless every request that they send defines a callback.

See Callback output about the supported return values.