this new schema brings the new `public_id` column, which will be used to serve files via
HTTP, splits `mime_type` in 2 (`media_type` and `media_subtype`) to make indexing/media
filtering easier, and fixes indexes for `files` table.
`PyCrypto` is unmaintained for years but was used in SàT for password hashing. This patch
fixes that by replacing `PyCrypto` by the reference `cryptography` module which is well
maintained.
The behaviour stays the same (except that previously async `hash`, `encrypt` and `decrypt`
methods are now synchronous, as they are quick and using a deferToThread may actually be
more resource intensive than using blocking methods).
It is planed to improve `memory.crypto` by using more up-to-date cryptography/hashing
algorithms in the future.
PyCrypto is no more a dependency of SàT
A new Connection/autoconnect_backend param can be set for a profile or component to be
started automatically with backend. This is specially useful for components, but can be
useful for client profile too (e.g. on Android we need to start profile with backend to
get notifications, this part will come with following commits).
The new Sqlite.getIndParamValues method allows to retrieve the same parameters for all
profiles.
/!\ this is a huge commit
/!\ starting from this commit, SàT is needs Python 3.6+
/!\ SàT maybe be instable or some feature may not work anymore, this will improve with time
This patch port backend, bridge and frontends to Python 3.
Roughly this has been done this way:
- 2to3 tools has been applied (with python 3.7)
- all references to python2 have been replaced with python3 (notably shebangs)
- fixed files not handled by 2to3 (notably the shell script)
- several manual fixes
- fixed issues reported by Python 3 that where not handled in Python 2
- replaced "async" with "async_" when needed (it's a reserved word from Python 3.7)
- replaced zope's "implements" with @implementer decorator
- temporary hack to handle data pickled in database, as str or bytes may be returned,
to be checked later
- fixed hash comparison for password
- removed some code which is not needed anymore with Python 3
- deactivated some code which needs to be checked (notably certificate validation)
- tested with jp, fixed reported issues until some basic commands worked
- ported Primitivus (after porting dependencies like urwid satext)
- more manual fixes
the deferreds of the queries writing message to databases where not gathered and returned,
so the caller was not waiting for them to continue its workflow. This was resulting in
messages not always written when database was read just after the write (a case common
with MUC implementation), and message was appearing empty when sent to bridge.
fix 328
indexes were missing on foreign keys for "message", "subject" and "thread" tables, seriously impacting performances.
In addition to those indexes, two indexes have been added to speed ordering by timestamp on "history", and one on "files" table.
history.rowid is not used anymore as there is an index on (profile_id, received_timestamp) which will speed up the query.
Primary keys order has been changed to use automatic index in most common cases (filtering by profile_id then namespace).
- "info" messages deletion is now done by deactivating foreign keys. This seriously boost performances.
- "VACUUM" is done at the end to reduce database size.
Performance issue was due to bad ordering in table dropping, this has been fixed.
"infos" message are also deleted by this update as they are containing only presence data and take a lot of space for a barely useful data.
A config option may be available in the future to store presence data in logs.
/!\ Database schema change /!\
History was using a unique constraint on `profile_id, timestamp, source, dest, source_res, dest_res`, which can cause trouble because several messages send quickly by the same person can have a common timestamp (specially with delayed messages where precision is second), resulting in message loss.
The new constraint use `profile_id, stanza_id, source, dest` where `stanza_id` is XEP-0359 stanza_id, so it's unique by definition, and no message should be lost anymore.
Because sqlite doesn't support altering table with a constraint change, we have to create new tables and copy old data to new one, which can be pretty long.
Sqlite update mechanism with "specifics" has been fixed when several updates are applied (e.g. moving from v5 to v7) and a specific is in the workflow.
/!\ database schema change /!\
stanza_id is a new field in history added to prepare the implementation of MAM for messages.
A new "last_stanza_id" can be used in filters to retrieve last message with a know stanza id (useful for history synchronisation).
- moved directories from src and frontends/src to sat and sat_frontends, which is the recommanded naming convention
- move twisted directory to root
- removed all hacks from setup.py, and added missing dependencies, it is now clean
- use https URL for website in setup.py
- removed "Environment :: X11 Applications :: GTK", as wix is deprecated and removed
- renamed sat.sh to sat and fixed its installation
- added python_requires to specify Python version needed
- replaced glib2reactor which use deprecated code by gtk3reactor
sat can now be installed directly from virtualenv without using --system-site-packages anymore \o/
