82 lines
3.6 KiB
ReStructuredText
82 lines
3.6 KiB
ReStructuredText
.. _developer:
|
||
|
||
=======================
|
||
Developer Documentation
|
||
=======================
|
||
|
||
This documentation is intended for people who wants to contribute or work with the
|
||
internals of the project, it is not for end-users.
|
||
|
||
Storage
|
||
=======
|
||
|
||
Since version 0.9, Libervia uses SQLAlchemy_ with its Object–Relational Mapping as a
|
||
backend to store persistent data, and Alembic_ is used to handle schema and data
|
||
migrations.
|
||
|
||
SQLite_ is currently the only supported database, but it is planned to add support for
|
||
other ones (notably PostgreSQL), probably during the development of 0.9 version.
|
||
|
||
The mapping is done in ``libervia.backend.memory.sqla_mapping`` and working with database is done
|
||
through high level methods found in ``libervia.backend.memory.sqla``.
|
||
|
||
Before the move to SQLAlchemy, there was a strict separation between database
|
||
implementation and the rest of the code. With 0.9, objects mapping to database can be used
|
||
and manipulated directly outside of ``libervia.backend.memory.sqla`` to take profit of SQLAlchemy
|
||
possibilities.
|
||
|
||
Database state is detected when the backend starts, and the database will be created or
|
||
migrated automatically if necessary.
|
||
|
||
To create a new migration script, ``Alembic`` may be used directly. To do so, be sure to
|
||
have an up-to-date database (and a backup in case of troubles), then activate the virtual
|
||
environment where Libervia is installed (Alembic needs to access ORM mapping), go to
|
||
``libervia/backend/memory/migration`` directory, and enter the following command::
|
||
|
||
alembic revision --autogenerate -m "some revision message"
|
||
|
||
This will create a base migration file in ``versions`` directory. Adapt it to your needs,
|
||
try to create both ``upgrade`` and ``downgrade`` method whenever possible, and be sure to
|
||
test it in both directions (``alembic upgrade head`` and ``alembic downgrade
|
||
<previous_revision>``). Please check Alembic documentation for more details.
|
||
|
||
.. _SQLALchemy: https://www.sqlalchemy.org/
|
||
.. _Alembic: https://alembic.sqlalchemy.org/
|
||
.. _SQLite: https://sqlite.org
|
||
|
||
Pubsub Cache
|
||
============
|
||
|
||
There is an internal cache for pubsub nodes and items, which is done in
|
||
``plugin_pubsub_cache``. The ``PubsubNode`` and ``PubsubItem`` class are the one mapping
|
||
the database.
|
||
|
||
The cache is operated transparently to end-user, when a pubsub request is done, it uses a
|
||
trigger to check if the requested node is or must be cached, and if possible returns
|
||
result directly from database, otherwise it lets the normal workflow continue and query the
|
||
pubsub service.
|
||
|
||
To save resources, not all nodes are fully cached. When a node is checked, a series of
|
||
analysers are checked, and the first one matching is used to determine if the node must be
|
||
synchronised or not.
|
||
|
||
Analysers can be registered by any plugins using ``register_analyser`` method:
|
||
|
||
.. automethod:: libervia.backend.plugins.plugin_pubsub_cache.PubsubCache.register_analyser
|
||
|
||
If no analyser is found, ``to_sync`` is false, or an error happens during the caching,
|
||
the node won't be synchronised and the pubsub service will always be requested.
|
||
|
||
Specifying an optional **parser** will store parsed data in addition to the raw XML of the
|
||
items. This is more space consuming, but may be desired for the following reasons:
|
||
|
||
* the parsing is resource consuming (network call or some CPU intensive operations are
|
||
done)
|
||
* it is desirable to do queries on parsed data. Indeed the parsed data are stored in a
|
||
JSON_ field and its keys may be queried individually.
|
||
|
||
The Raw XML is kept as the cache operates transparently, and a plugin may need raw data, or
|
||
an user may be doing a low-level pubsub request.
|
||
|
||
.. _JSON: https://docs.sqlalchemy.org/en/14/core/type_basics.html#sqlalchemy.types.JSON
|
||
|