path: root/docs/plugins/writing/scrapers.rst

Scrapers
========

A scraper extends the abstract :class:`~hircine.scraper.Scraper` class and
implements its :meth:`~hircine.scraper.Scraper.scrape` method. The latter is a
generator function yielding :ref:`scraped-data`.

.. autoclass:: hircine.scraper.Scraper
   :members:
   :special-members: __init__

Exceptions
----------

A scraper may raise two kinds of exceptions:

.. autoexception:: hircine.scraper.ScrapeWarning

.. autoexception:: hircine.scraper.ScrapeError

Utility functions
-----------------

.. automodule:: hircine.scraper.utils
   :members:

Registering a scraper
---------------------

To register your class as a scraper, place it into the ``hircine.scraper``
:ref:`entry point group <packaging:entry-points>`. For example, put the
following in a ``pyproject.toml`` file:

.. code-block:: toml

   [project.entry-points.'hircine.scraper']
   my_scraper = 'myscraper.MyScraper'

Example
-------

.. literalinclude:: /_examples/example_scraper.py
   :language: python

The scraper above will scrape a JSON file with the following structure:

.. literalinclude:: /_examples/example_scraper.json
   :language: json