Commit graph

4,470 commits

Author SHA1 Message Date
Goffi
d2f4623b53 doc: Add How-To for Email Gateway:
This How-To complete installation instruction with an easy-to follow document to
completely install the Email <=> XMPP gateway.

fix 466
2025-10-10 17:44:04 +02:00
Goffi
c1cb70ac1a doc (installation): Update instructions:
- Instructions have been checked and updated. OS packages dependencies have been added for
  Arch Linux.
- Added missing package.
- Reordered/refreshed instructions.
- Old/not relevant content has been removed.
- Tested on latest docker images of Debian/Arch/Unbutu.

rel 466
2025-10-10 17:44:04 +02:00
Goffi
8c35f015f1 tools (common/uri): fix missing f for f-string. 2025-10-10 17:44:04 +02:00
Goffi
fcb76a7dde cli (uri/build): fix call to uri.build_xmpp_uri. 2025-10-10 17:44:04 +02:00
Goffi
1b3593d95b plugin XEP-0050: Some more refactoring:
- Arguments are now grouped in a new `PageData` NamedTuple, making it simpler to use, and
  easier to extend.
- Page index is automatically calculated, and can be checked in callback to know which
  page to show.
- Form send by calling entity is automatically parsed, using original form to instantiate
  the right field types.
- A new `DataForm.get_field` method use Generic type as second argument to retrieve the
  right field when getting a `submit` form.
- Updated all classes registering Ad-Hoc commands to use the new system and simplify the
  code.
2025-10-10 17:44:04 +02:00
Goffi
e4e879d871 core (memory/sqla): Improve file_update by using raw value:
ORM was used, resulting in deserialisation/reserialisation of value checked for race
condition during update. In addition to the overhead, this could lead to modification even
if the value didn't change (e.g., because of JSON key order changed).

The raw value is not used to check if the row has not change between `select` and
`update`, making the method more robust.
2025-10-10 17:44:04 +02:00
Goffi
eaf6f1f00b plugin XEP-0050: Improve page data handling:
- Ad-Hoc callbacks now receive only a `client` and a `page_data`.
- `page_data` keep all important data useful to process current page in a namedtuple.
- Submitted form is automatically parsed, so callback just has to check the data from there.
- Page index is automatically handled, so the handling by callback is no longed needed.
- Session data is now splitted in 2 dict: private for `AdHocCommandHandler`, and public
  for the callback.
2025-10-10 17:44:04 +02:00
Goffi
697c7dbdbe component Emai Gateway: Add Ad-Hoc Commands:
- add Ad-Hoc commands to
    * delete an email
    * move an email to another mailbox
    * change flags on an email

- The email Message-ID is used to identify to email to manipulate, it is used as <message>
  ID on XMPP side.
- Add a `vebose` attribute on IMAPClient to see data exchanged with server.
- Refactor IDLE workflow, to fix synchronisation issues when command were emitted quickly.
- When in IDLE mode, and a command is emitted, there is now a 5s delay to restart IDLE
  mode, allowing for a series of command to be run without going to IDLE mode between each
  command.
- Fix pubsub nodes which were returned on each disco#items query.

fix 465
2025-10-10 17:44:00 +02:00
Goffi
19f4ce55de add type hints + minor reformatting. 2025-10-01 17:57:48 +02:00
Goffi
b07179a871 plugin XEP-0050: refactor to use Pydantic models:
Pydantic models are now used for Ad-Hoc commands callback data. The `AdHocCommand` model
can (de)serialise the callback by using its import string.
This change has several benefits:
- better data validation.
- the serialisation mechanism will make communication with frontends easier.
- open the door for automatic documentation.

rel 465
2025-10-01 17:57:47 +02:00
Goffi
286e931a7f tools (utils): Add maybe_async function:
`maybe_async` can be used to call a function which may be or not `async`. It doesn't use
`defer.ensureDeferred` contrary to `ad_defered`, so the call stack is not lost, and it's
easier to debug when an Exception is raised.
2025-10-01 17:57:47 +02:00
Goffi
bf93abb343 core (memory): Use better types for Sessions:
- `SessionData` is not a dedicated type instead of a simple `dict`.
- `SessionInfo` is now handled by a named tuple.
- Add type hints.

rel 465
2025-10-01 17:57:47 +02:00
Goffi
b882fb53ea plugin XEP-0004: New data form implementation using Pydantic:
This new model is easier to manipulate and serialize, and use the benefits from Pydantic's
data validation mechanisms. It has method to easily convert to/from Wokkel data form or
to/from a domish.Element.

rel 465
2025-10-01 17:57:47 +02:00
Goffi
a32b8bf9e8 plugin XEP-0050: use Enums and add type hints:
- Replaces namedtuple with enums which are more appropriate here.
- Add docstrings and type hints to `AdHocCommand`

rel 465
2025-10-01 17:57:47 +02:00
Goffi
a37101adc7 install [pyproject.toml]: update treq version. 2025-10-01 16:07:51 +02:00
Goffi
acd46d1bf8 install [pyproject.toml]: restrict Twisted maximum version. 2025-10-01 15:50:00 +02:00
Goffi
c4d00d63da cli: Fix exit traceback on Twisted based loop (e.g.; pb bridge). 2025-10-01 14:55:32 +02:00
Goffi
d9a5257438 bridge (pb): Fix broken call to init_bridge. 2025-10-01 14:55:27 +02:00
Goffi
96bf3af324 Global reformatting with black. 2025-09-13 10:46:10 +02:00
Goffi
5ec4fdd787 doc (components): Add message cleaning section to email gateway doc:
fix 464
2025-09-11 21:17:51 +02:00
Goffi
2b9b14ba1b tests (unit/components): Add tests for email gateway's cleaning module:
rel 464
2025-09-11 21:17:51 +02:00
Goffi
63edb309f8 component email message: Mailing list messages cleaning:
For mailing-list messages:
    - convert all text-only messages to XHTML for consistency.
    - detect and flag "noisy" content, i.e. content which are not useful and annoying in
      forum-like view (full-lenght quoted previous messages, mailing-list signature)

rel 464
2025-09-11 21:17:47 +02:00
Goffi
412db6a216 component email (imap): fetch all recent emails instead of only the last one. 2025-09-11 21:10:35 +02:00
Goffi
81f63ef93e plugin XEP-0277: add mentions to extra data:
rel 463
2025-09-06 16:31:24 +02:00
Goffi
5c9c57588c plugin XEP-0376: Fix type hint for node which is optional in subscriptions. 2025-09-05 19:32:47 +02:00
Goffi
0a42dd3da3 core (main): renamed hasFeature to has_feature to fix inconsistency. 2025-09-05 19:31:50 +02:00
Goffi
b816dc5e54 cli (pubsub): List support first draft + doc:
Basic support for listing hierarchy of nodes with pubsub extended discovery. Nodes are
displayed as a tree, depth can be specified.

rel 463
2025-08-31 12:34:37 +02:00
Goffi
57e71c326d component email gateway: implement pubsub relationships and pubsub extended discovery:
- Pubsub relationships are used to set node hierarchy (comments nodes are set as children of
corresponding pubsub nodes).
- XEP-0499 Extended Pubsub Discovery is used to return hierarchy of pubsub nodes and
  items.

rel 463
2025-08-31 12:34:37 +02:00
Goffi
70e76dd190 plugin XEP-0499 Pubsub Extended Discovery implementation:
- Implement Pubsub Extended Discovery, and Pydantic models to handle the data easily.
- `client.discoHandler` initialisation has been moved in `core.xmpp` at an earlier stage
  of the workflow as it has to be initialised so XEP-0499 could remove the default disco
  handler to handle disco itself.
- Removed `Disco handled` trigger point as it was not used.

rel 463
2025-08-31 12:34:30 +02:00
Goffi
5b18ba7313 plugin XEP-0060: Add a model to handle node metadata:
rel 463
2025-08-29 18:53:03 +02:00
Goffi
edb649eff7 memory (sqla): update set_pubsub_node and get_pubsub_nodes to handle pubsub relationships:
- `parent_node` and `linked_node` can now be specified in `set_pubsub_node`.
- `get_pubsub_nodes` has been rewritten to recursively traverse a tree and retrieve nodes
  matching criteria and their linking nodes/children/items.

rel 463
2025-08-29 18:52:58 +02:00
Goffi
e593f32f09 memory (sqla): add "parent_node_id" and "linked_mode_id" to PubsubNode:
This is to prepare for the implementation of XEP-0496 and XEP-0499.

rel 463
2025-08-29 17:59:12 +02:00
Goffi
f5a4b4b599 add .gitignore to .hgignore:
Some tools don't support Mercurial, and use `.gitignore`. Adding it here allow to have a
local `.gitignore` for dev local tools.
2025-08-29 11:44:43 +02:00
Goffi
009ef1aa4b doc (components): Document the handling of mailing lists in Email Gateway:
fix 462
2025-08-03 23:45:48 +02:00
Goffi
e1828b3672 component Email gateway: Convert mailing list to pubsub nodes:
- `Credentials` is now a Pydantic based model.
- Mailing list related emails are now detected hand saved as pubsub items, using XEP-0277
  blog items, and creating suitable nodes. The ID of the mailing list is used for root
  node.
- Mailing list items are currently restricted to recipient associated JID.
- Words in square bracket in title `[Like][That]` are removed and converted to blog
  categories.
- Method to convert between MbData and email (in both directions) have been created.

rel 462
2025-08-03 23:45:48 +02:00
Goffi
26002870a8 memory (sqla): Add columns and tables to handles permissions:
Add columns to handle `access_model` and `publish_model` and a table for `affiliations`.
With those new data, components handling pubsub can now manage permissions correctly.

rel 462
2025-08-03 23:45:45 +02:00
Goffi
3dbd990902 backend: various fixes, and type hints improvments. 2025-08-03 23:36:22 +02:00
Goffi
cab460152c plugin XEP-0277: Make MbData a Pydantic model:
- MbData is now a Pydantic based model.
- Bridge method has been updated to use `service` and `node` from MbData instead of using
  separate fields.
- Added `In-Reply-To` to MbData.
- Adapted code using XEP-0277 to work with the changes.

rel 462
2025-08-03 23:35:21 +02:00
Goffi
b1e0e336c2 plugin XEP-0297: Reworked forward method and add bridge method:
`Forward` method has been reworked and now includes a fallback. XEP-0297 ask to not use
fallback, but following a discussion on xsf@, we agreed that this is a legacy thing and a
fallback should nowadays be used, I'll propose a patch to the specification.

A `message_forward` has been added to bridge.

rel 461
2025-07-04 12:33:42 +02:00
Goffi
030be64c45 plugin XEP-0428: Add missing "for" attribute:
The "for" attribute was missing from <fallback> element.

rel 461
2025-07-04 12:30:20 +02:00
Goffi
16f9e07ce1 minor type update and unused import removal. 2025-07-04 12:28:40 +02:00
Goffi
0df0b0a5d0 core (types): Renamed libervia.backend.core.core_types.MessageData to MessageDataLegacy:
There are 2 concurrent models, this one is an historical legacy. The type to use the one
from `libervia.backend.models.core`, and it will progressively be used everywhere and
fully based on Pydantic.
2025-07-04 12:28:00 +02:00
Goffi
8302d09cf6 plugin data policy: Data Policy implementation:
This plugin implement data policy parsing and an algorithm to calculate a score based on
them.

rel 460
2025-06-26 17:02:33 +02:00
Goffi
6c95b3bec8 component email gateway: Add basic disco policy:
rel 460
2025-06-26 17:01:41 +02:00
Goffi
328e0ce775 memory (sqla): Add a filter for origin in history_get:
rel 459
2025-06-08 17:16:58 +02:00
Goffi
ceb2484a37 core (xmpp): add origin and origin_label to message data:
If a message comes from a gateway, its type is used as `origin` in `extra`, and it has a
name, it's used as `origin_label`.
`is-bot` is also set when the sending entity is advertised as a bot.

rel 459
2025-06-08 17:16:56 +02:00
Goffi
14f50afa64 tests (unit/sqla): Add test for keyword filtering:
rel 458
2025-06-06 10:45:54 +02:00
Goffi
660dd6d538 memory (sqla): Add filter for keyword:
A `keyword` filter can now be used to retrieve messages matching given keyword.

rel 458
2025-06-04 17:52:18 +02:00
Goffi
331e7dc5c4 plugin identity: add a timeout when getting identities. 2025-05-15 17:45:19 +02:00
Goffi
4a0d106b90 test (XEP-0461): some tests for XEP-0461:
rel 457
2025-05-06 00:34:01 +02:00