Links Spider

class kingfisher_scrapy.base_spiders.links_spider.LinksSpider(*args, **kwargs)

Collect data from an API that implements the pagination pattern.

1. Inherit from LinksSpider

2. Set a data_type class attribute to the data type of the API responses

3. Set a formatter class attribute to set the file name like in build_request()

4. Set a next_link_formatter class attribute if pagination URLs differ from start URLs

5. Write a start() method to request the first page of API results

6. Optionally, set a next_pointer class attribute to the JSON Pointer for the next link (default “/links/next”)

If the API returns the number of total pages or results in the response, consider using IndexSpider instead.

import scrapy

from kingfisher_scrapy.base_spiders import LinksSpider

class MySpider(LinksSpider):
    name = 'my_spider'

    # SimpleSpider
    data_type = 'release_package'

    # LinksSpider
    formatter = staticmethod(parameters('page'))

    async def start(self):
        yield scrapy.Request('https://example.com/api/packages.json', meta={'file_name': 'page-1.json'})

next_pointer = '/links/next'

parse(response)

next_link(response, **kwargs)

If the JSON response has a links.next key, returns a scrapy.Request for the URL.