Utilities

kingfisher_scrapy.util.pluck_filename(opts)

kingfisher_scrapy.util.replace_path_separator(string)

kingfisher_scrapy.util.components(start, stop=None)

Return a function that returns the selected non-empty path components, excluding the .json extension.

>>> components(-1)('http://example.com/api/planning.json')
'planning'

>>> components(-2, -1)('http://example.com/api/planning/package.json')
'planning'

kingfisher_scrapy.util.parameters(*keys, parser=None)

Return a function that returns the selected query string parameters.

>>> parameters('page')('http://example.com/api/packages.json?page=1')
'page-1'

>>> parameters('year', 'page')('http://example.com/api/packages.json?year=2000&page=1')
'year-2000-page-1'

kingfisher_scrapy.util.join(*functions, extension=None)

Return a function that joins the given functions’ outputs and sets the file extension, if provided.

>>> join(components(-1), parameters('page'))('http://example.com/api/planning.json?page=1')
'planning-page-1'

kingfisher_scrapy.util.date_range_by_interval(start, stop, step)

Yield date ranges from the start date to the stop date, in intervals of step days, in reverse chronological order.

kingfisher_scrapy.util.date_range_by_month(start, stop)

Yield the first day of the month as a date from the start to the stop dates, in reverse chronological order.

kingfisher_scrapy.util.date_range_by_year(start, stop)

Return the year as an int from the start to the stop years, in reverse chronological order.

kingfisher_scrapy.util.get_parameter_value(url, key)

Return the first value of the query string parameter.

kingfisher_scrapy.util.replace_parameters(url, **kwargs)

Return a URL after updating the query string parameters’ values.

kingfisher_scrapy.util.append_path_components(url, path)

Return a URL after appending path components to its path.

kingfisher_scrapy.util.add_query_string(method, params)

Return a function that yields the requests yielded by the wrapped method, after updating the query string parameter values in each request’s URL.

kingfisher_scrapy.util.add_path_components(method, path)

Return a function that yields the requests yielded by the wrapped method, after appending path components to each request’s URL.

kingfisher_scrapy.util.items_basecoro(target, prefix, map_type=None, skip_key=None)

Replicate the same function from ijson/common.py.

A skip_key argument is added. If the skip_key is in the current path, the current event is skipped. Otherwise, the method is identical.

kingfisher_scrapy.util.items(events, prefix, map_type=None, skip_key=None)

Replicate the same function from ijson/common.py.

A skip_key argument is added, which is passed as a keyword argument to items_basecoro(). Otherwise, the method is identical.

kingfisher_scrapy.util.default(obj)

Convert decimals to floats and iterables to lists.

class kingfisher_scrapy.util.TranscodeFile(file, encoding)

read(buf_size)

Re-encodes bytes read from the file to UTF-8.

kingfisher_scrapy.util.transcode_bytes(data, encoding)

Re-encodes bytes to UTF-8.

kingfisher_scrapy.util.transcode(spider, function, data, *args, **kwargs)

kingfisher_scrapy.util.grouper(iterable, n, fillvalue=None)

kingfisher_scrapy.util.get_file_name_and_extension(filename)

Given a filename, return its name and extension in two separate strings.

>>> get_file_name_and_extension('test.json')
('test', 'json')