diff options
Diffstat (limited to 'docs/plugins/writing/scrapers.rst')
| -rw-r--r-- | docs/plugins/writing/scrapers.rst | 48 |
1 files changed, 48 insertions, 0 deletions
diff --git a/docs/plugins/writing/scrapers.rst b/docs/plugins/writing/scrapers.rst new file mode 100644 index 0000000..258d3a8 --- /dev/null +++ b/docs/plugins/writing/scrapers.rst @@ -0,0 +1,48 @@ +Scrapers +======== + +A scraper extends the abstract :class:`~hircine.scraper.Scraper` class and +implements its :meth:`~hircine.scraper.Scraper.scrape` method. The latter is a +generator function yielding :ref:`scraped-data`. + +.. autoclass:: hircine.scraper.Scraper + :members: + :special-members: __init__ + +Exceptions +---------- + +A scraper may raise two kinds of exceptions: + +.. autoexception:: hircine.scraper.ScrapeWarning + +.. autoexception:: hircine.scraper.ScrapeError + +Utility functions +----------------- + +.. automodule:: hircine.scraper.utils + :members: + +Registering a scraper +--------------------- + +To register your class as a scraper, place it into the ``hircine.scraper`` +:ref:`entry point group <packaging:entry-points>`. For example, put the +following in a ``pyproject.toml`` file: + +.. code-block:: toml + + [project.entry-points.'hircine.scraper'] + my_scraper = 'myscraper.MyScraper' + +Example +------- + +.. literalinclude:: /_examples/example_scraper.py + :language: python + +The scraper above will scrape a JSON file with the following structure: + +.. literalinclude:: /_examples/example_scraper.json + :language: json |