- encryption is now allowed for group chats
- when an encryption is requested for a MUC, real jids or all occupants are used to
encrypt the message
- a cache for plain text message sent to MUC is used, because for security reason we can't
encrypt message for our own device with OMEMO (that would prevent ratchet and break the
prefect forward secrecy). Thus, message sent in MUC are cached for 5 min, and the
decrypted version is used when found. We don't send immediately the plain text message
to frontends and history because we want to keep the same MUC behaviour as for plain
text, and receiving a message means that it was received and sent back by MUC service
- <origin-id> is used to identify messages sent by our device
- a feedback_jid is now use to use correct entity for feedback message in case of problem:
with a room we have to send feedback message to the room and not the the emitter
- encryptMessage now only accepts list in the renamed "entity_bare_jids" argument
launch script was importing a method from memory (fixLocalDir), but memory is importing
twisted.internet.reactor which can result to a ReactorAlreadyInstalledError.
This patch fixes it by moving fixLocalDir to tools/config.
The launch script is now using /usr/bin/env.
/!\ this commit is huge. Jp is temporarily not working with `dbus` bridge /!\
This patch implements the port of jp to asyncio, so it is now correctly using the bridge
asynchronously, and it can be used with bridges like `pb`. This also simplify the code,
notably for things which were previously implemented with many callbacks (like pagination
with RSM).
During the process, some behaviours have been modified/fixed, in jp and backends, check
diff for details.
/!\ this is a huge commit
/!\ starting from this commit, SàT is needs Python 3.6+
/!\ SàT maybe be instable or some feature may not work anymore, this will improve with time
This patch port backend, bridge and frontends to Python 3.
Roughly this has been done this way:
- 2to3 tools has been applied (with python 3.7)
- all references to python2 have been replaced with python3 (notably shebangs)
- fixed files not handled by 2to3 (notably the shell script)
- several manual fixes
- fixed issues reported by Python 3 that where not handled in Python 2
- replaced "async" with "async_" when needed (it's a reserved word from Python 3.7)
- replaced zope's "implements" with @implementer decorator
- temporary hack to handle data pickled in database, as str or bytes may be returned,
to be checked later
- fixed hash comparison for password
- removed some code which is not needed anymore with Python 3
- deactivated some code which needs to be checked (notably certificate validation)
- tested with jp, fixed reported issues until some basic commands worked
- ported Primitivus (after porting dependencies like urwid satext)
- more manual fixes
the deferreds of the queries writing message to databases where not gathered and returned,
so the caller was not waiting for them to continue its workflow. This was resulting in
messages not always written when database was read just after the write (a case common
with MUC implementation), and message was appearing empty when sent to bridge.
fix 328
an old hardcoded value was used in several places to detect if an entity is a MUC, but this
value was not valid anymore. This has been fixed, and ENTITY_TYPE_MUC constant is now used
instead. This fixes message type autodetection for "groupchat" messages.
fixes 300
A "clear" method has been added to delete all data for a specific namespace (and profile).
Internal cache (self._cache) is not initialised with an empty dict instead of None to allow using methods like __setitem__ or clear without having to do a potentially expansive `load()` before. The only counterpart is that we won't have an exception anymore if developer forget to load() data before using them, which can make debugging a bit more difficult.
indexes were missing on foreign keys for "message", "subject" and "thread" tables, seriously impacting performances.
In addition to those indexes, two indexes have been added to speed ordering by timestamp on "history", and one on "files" table.
history.rowid is not used anymore as there is an index on (profile_id, received_timestamp) which will speed up the query.
Primary keys order has been changed to use automatic index in most common cases (filtering by profile_id then namespace).
so far, requestInfo failure where logged as a warning, and an empty disco was returned.
This prevented the requesting entity to know about the error, so now the error is logged and re-raised.
CancelError are converted to TimeoutError (requestInfo is only cancelled on timeout).
An empty disco is still cached in case of failure, to avoid re-waiting for the entity on next request.
- messageEncryptionStart and messageEncryptionStop are now async
- an encryption plugin can now have a startEncryption and endEncryption method, which are called automatically when suitable
- trust is now put in "trusted" key of message_data, and markAsTrusted has been added, it is used in OTR
- getTrustUI implemented for OTR, using legacy authenticate code
- catch potr.crypt.InvalidParameterError on decryption
- fixed some service OTR messages which were not correctly marked as not for storing, and thus were added to MAM archive/carbon copied
- other bug fixes
- encryptionNamespaceGet retrieves algorithm namespace from its short name
- encryptionTrustUIGet retrieves trust mangement XMLUI from encryption plugin
- new markAsUntrusted internal helper method, to add untrusted flag to message data
- "info" messages deletion is now done by deactivating foreign keys. This seriously boost performances.
- "VACUUM" is done at the end to reduce database size.
Performance issue was due to bad ordering in table dropping, this has been fixed.
"infos" message are also deleted by this update as they are containing only presence data and take a lot of space for a barely useful data.
A config option may be available in the future to store presence data in logs.
/!\ Database schema change /!\
History was using a unique constraint on `profile_id, timestamp, source, dest, source_res, dest_res`, which can cause trouble because several messages send quickly by the same person can have a common timestamp (specially with delayed messages where precision is second), resulting in message loss.
The new constraint use `profile_id, stanza_id, source, dest` where `stanza_id` is XEP-0359 stanza_id, so it's unique by definition, and no message should be lost anymore.
Because sqlite doesn't support altering table with a constraint change, we have to create new tables and copy old data to new one, which can be pretty long.
Sqlite update mechanism with "specifics" has been fixed when several updates are applied (e.g. moving from v5 to v7) and a specific is in the workflow.
/!\ database schema change /!\
stanza_id is a new field in history added to prepare the implementation of MAM for messages.
A new "last_stanza_id" can be used in filters to retrieve last message with a know stanza id (useful for history synchronisation).
XMPP server certificate is now checked, and connection is refused (by default) if it's not valid.
Certificate check can be disabled in the new parameter "Configuration/check_certificate".
If certificate checking is disabled, a warning note is sent on every new connection.
Twisted and Wokkel are temporarly monkey patched in sat.core.tls_patches module, until modifications are merged upstream.
Following methods are now available though bridge:
- messageEncryptionStop
- messageEncryptionGet: retrieve encryption data for a message session
- encryptionPluginsGet: retrieve all registered encryption plugin
Following methods are available for internal use:
- getPlugins: retrieve registerd plugins
- getNSFromName: retrieve namespace from plugin name
- getBridgeData: serialise session data (to be used with bridge)
- markAsEncrypted: mark message data as encrypted (to be set by encryption plugin in MessageReceived trigger)
Behaviours improvments:
- start and stop send messageEncryptionStarted and messageEncryptionStopped signals, and a message feedback
- new "replace" arguments in start allows to replace a plugin if one is already running (instead of raising a ConflictError)
- plugins declare themselves as "directed" (i.e. working with only one device at a time) or not.
This is checked while dealing with jids, an exception is raised when a full jid is received for a non directed encryption.
- use of new data_format (de)serialise
- "start" won't do anything if session is already encrypted with requested algorithm, and will raise a ConflictError if it's already encrypted but with an other algorithm
- implemented "stop", with an optional namespace to check we are stopping the expected algorithm
- "getSession" retrieve the current encryption session of a jid, if any
Plugin handling encryption can now register using host.registerEncryptionPlugin, and an encryption session can now be started using messageEncryptionStart bridge method.
This will make encryption handling more easy, as we now know if a session is clear or e2e encrypted, and which plugin handle it.
