Periodic Spider

class kingfisher_scrapy.base_spiders.periodic_spider.PeriodicSpider(*args, **kwargs)

This class makes it easy to collect data from an API that accepts a year or a year and month as parameters.

1. Inherit from PeriodicSpider

2. Set a date_format class attribute to “year” or “year-month”

3. Set a pattern class attribute to a URL pattern, with placeholders. If the date_format is “year”, then a year is passed to the placeholder as an int. If the date_format is “year-month”, then the first day of the month is passed to the placeholder as a date, which you can format as, for example:

4. Set a formatter class attribute to set the file name like in build_request()

5. Set a default_from_date class attribute to a year (“YYYY”) or year-month (“YYYY-MM”)

6. If the source stopped publishing, set a default_until_date class attribute to a year or year-month

7. Optionally, set a start_requests_callback class attribute to a method’s name as a string - otherwise, it defaults to parse()

If sample is set, the data from the most recent year or month is retrieved.

date_required = True

start_requests_callback = 'parse'

start_requests()

build_urls(date)

Yields one or more URLs for the given date.