doc: developer documentation explaining storage and pubsub cache

doc/developer.rst

@@ -0,0 +1,82 @@
+.. _developer:
+
+=======================
+Developer Documentation
+=======================
+
+This documentation is intended for people who wants to contribute or work with the
+internals of the project, it is not for end-users.
+
+Storage
+=======
+
+Since version 0.9, Libervia uses SQLAlchemy_ with its Object–Relational Mapping as a
+backend to store persistent data, and Alembic_ is used to handle schema and data
+migrations.
+
+SQLite_ is currently the only supported database, but it is planned to add support for
+other ones (notably PostgreSQL), probably during the development of 0.9 version.
+
+The mapping is done in ``sat.memory.sqla_mapping`` and working with database is done
+through high level methods found in ``sat.memory.sqla``.
+
+Before the move to SQLAlchemy, there was a strict separation between database
+implementation and the rest of the code. With 0.9, objects mapping to database can be used
+and manipulated directly outside of ``sat.memory.sqla`` to take profit of SQLAlchemy
+possibilities.
+
+Database state is detected when the backend starts, and the database will be created or
+migrated automatically if necessary.
+
+To create a new migration script, ``Alembic`` may be used directly. To do so, be sure to
+have an up-to-date database (and a backup in case of troubles), then activate the virtual
+environment where Libervia is installed (Alembic needs to access ORM mapping), go to
+``sat/memory/migration`` directory, and enter the following command::
+
+  alembic revision --autogenerate -m "some revision message"
+
+This will create a base migration file in ``versions`` directory. Adapt it to your needs,
+try to create both ``upgrade`` and ``downgrade`` method whenever possible, and be sure to
+test it in both directions (``alembic upgrade head`` and ``alembic downgrade
+<previous_revision>``). Please check Alembic documentation for more details.
+
+.. _SQLALchemy: https://www.sqlalchemy.org/
+.. _Alembic: https://alembic.sqlalchemy.org/
+.. _SQLite: https://sqlite.org
+
+Pubsub Cache
+============
+
+There is an internal cache for pubsub nodes and items, which is done in
+``plugin_pubsub_cache``. The ``PubsubNode`` and ``PubsubItem`` class are the one mapping
+the database.
+
+The cache is operated transparently to end-user, when a pubsub request is done, it uses a
+trigger to check if the requested node is or must be cached, and if possible returns
+result directly from database, otherwise it lets the normal workflow continue and query the
+pubsub service.
+
+To save resources, not all nodes are fully cached. When a node is checked, a series of
+analysers are checked, and the first one matching is used to determine if the node must be
+synchronised or not.
+
+Analysers can be registered by any plugins using ``registerAnalyser`` method:
+
+.. automethod:: sat.plugins.plugin_pubsub_cache.PubsubCache.registerAnalyser
+
+If no analyser is found, ``to_sync`` is false, or an error happens during the caching,
+the node won't be synchronised and the pubsub service will always be requested.
+
+Specifying an optional **parser** will store parsed data in addition to the raw XML of the
+items. This is more space consuming, but may be desired for the following reasons:
+
+* the parsing is resource consuming (network call or some CPU intensive operations are
+  done)
+* it is desirable to to queries on parsed data. Indeed the parsed data is stored in a
+  JSON_ field and its keys may be queried individually.
+
+The Raw XML is kept as the cache operates transparently, and a plugin may need raw data, or
+an user may be doing a low-level pubsub request.
+
+.. _JSON: https://docs.sqlalchemy.org/en/14/core/type_basics.html#sqlalchemy.types.JSON
+

doc/index.rst

@@ -26,6 +26,7 @@ You can follow this documentation to learn more on it, or join our official XMPP
    installation.rst
    overview.rst
    configuration.rst
+   developer.rst
    /libervia-cli/index.rst
    /libervia-tui/index.rst
    /contribuing/index.rst