Compressed File Spider

class kingfisher_scrapy.base_spiders.compressed_file_spider.CompressedFileSpider(*args, **kwargs)

Collect data from ZIP or RAR files.

It assumes all files have the same data type. Each compressed file is saved to disk. The archive file is not saved to disk.

1. Inherit from CompressedFileSpider

2. Set a data_type class attribute to the data type of the compressed files

3. Optionally, add a resize_package = True class attribute to split large packages (e.g. greater than 100MB)

4. Optionally, add a yield_non_archive_file = True class attribute if the spider requests both archive files and JSON files. Otherwise, the spider raises an UnknownArchiveFormatError exception.

5. Optionally, add a file_name_must_contain = 'text' class attribute to only decompress the files whose paths contain the given text.

6. Optionally, add a file_name_must_not_contain = 'text' class attribute to only decompress the files whose paths do not contain the given text.

7. Optionally, add a skip_empty_releases = True class attribute to skip files with empty releases arrays.

8. Write a start() method to request the archive files

from kingfisher_scrapy.base_spiders import CompressedFileSpider
from kingfisher_scrapy.util import components

class MySpider(CompressedFileSpider):
    name = 'my_spider'

    # CompressedFileSpider
    data_type = 'release_package'

    async def start(self):
        yield self.build_request('https://example.com/api/packages.zip', formatter=components(-1))

Note

concatenated_json = True, line_delimited = True, root_path, data_type = 'release' and data_type = 'record' are not supported if resize_package = True.

dont_truncate = True

yield_non_archive_file = False

file_name_must_contain = ''

file_name_must_not_contain = ''

skip_empty_releases = False

parse(response)

process_archive_file(response, archive_file_name, archive_data)