Periodic Spider

class kingfisher_scrapy.base_spiders.periodic_spider.PeriodicSpider(*args, **kwargs)

This class makes it easy to collect data from an API that accepts a year, year-month or date as a query string parameter or URL path component.

1. Inherit from PeriodicSpider

2. Set a date_format class attribute to “year”, “year-month” or “date”

3. Set a pattern class attribute to a URL pattern, with placeholders. If the date_format is “year”, then a year is passed to the placeholder as an int. If the date_format is “year-month”, then the first day of the month is passed to the placeholder as a date, which you can format as, for example:

4. Set a formatter class attribute to set the file name like in build_request()

5. Set a default_from_date class attribute to a year (“YYYY”) or year-month (“YYYY-MM”)

6. If the source stopped publishing, set a default_until_date class attribute to a year or year-month

7. Optionally, if the date_format is “date”, set a step class attribute to indicate the length of intervals, in days – otherwise, it defaults to 1

8. Optionally, set a start_requests_callback class attribute to a method’s name as a string - otherwise, it defaults to parse()

If sample is set, the data from the most recent year or month is retrieved.

date_required = True

step = 1

start_requests_callback = 'parse'

start_requests()

build_urls(from_date, until_date=None)

Yields one or more URLs for the given date.