Simple Spider

class kingfisher_scrapy.base_spiders.simple_spider.SimpleSpider(*args, **kwargs)

Most spiders can inherit from this class.

It assumes all responses have the same data type.

1. Inherit from SimpleSpider

2. Set a data_type class attribute to the data type of the responses

3. Write a start() method (and any intermediate callbacks) to send requests

import scrapy

from kingfisher_scrapy.base_spiders import SimpleSpider

class MySpider(SimpleSpider):
    name = 'my_spider'

    # SimpleSpider
    data_type = 'release_package'

    async def start(self):
        yield scrapy.Request('https://example.com/api/package.json', meta={'file_name': 'all.json'})

parse(response)

Process response, i.e. extract data from it and generate new requests.

This is the default callback: Scrapy uses it for the response to any request that does not define a callback, such as the requests that start() yields by default.

Any cb_kwargs of the request are passed as keyword parameters.

Spiders must define this method, unless every request that they send defines a callback.

See Callback output about the supported return values.