Matthias/conduit

Fork 0

Receiving out-of-line messages from before joining a room #143

New issue

Closed

opened 2021-09-02 08:50:27 +00:00 by Weasy · 8 comments

Weasy commented

2021-09-02 08:50:27 +00:00

(Migrated from gitlab.com)

Description

While running Conduit, i occasionally get out-of-line events. Most of those events are from a time before i joined a room.

Example:

Room join pdu:

{
  "auth_events": [
    "$fP9-VwJPUrB92dKnf0vcLK3Ntz1z6Xr-n87Fo388zdU",
    "$UfJ4H2SOR2X9yUQeJ6_Mm3Hl-lsn5nwTRQyFmc-iY3E",
    "$LqGoKWH6xTkllRooB4tjfqbViPCTEBuOfB_k8Dc-OJ8"
  ],
  "content": {
    "displayname": "weasy ⚡️",
    "membership": "join"
  },
  "depth": 21485,
  "event_id": "$_j3JxwRDeofxevo1GgjWIZmWMCq4iQzG0DhlFE23V8M",
  "hashes": {
    "sha256": "iHdcgr+EOw3S7pO/+6kQa7ITcvP27k+LH45FAFTY2CA"
  },
  "origin_server_ts": 1629142980912,
  "prev_events": [
    "$5NpgJErpIS2w7AL9g3ZByzPLMZ90yssHvEJekx8MIk0"
  ],
  "room_id": "!pWCROeqlZcGggueJLt:fachschaften.org",
  "sender": "@weasy:weasy-is-my.name",
  "signatures": {
    "weasy-is-my.name": {
      "ed25519:skvFbwnQ": "DJYc1Xq+axnnpsr8q1ATQY/3+YvDUgcknYmRekt2WEqewJ0/Qm5wQ6VeG3y0MQrWjb5bLdWxQbrKc5SLiMXxCw"
    }
  },
  "state_key": "@weasy:weasy-is-my.name",
  "type": "m.room.member",
  "unsigned": {}
}

Pdu of the 16. June 2021 message from Jonathan:

{
  "auth_events": [
    "$VMJIDcmFltqBX2XoUZ4UUj0UwX8hYeqpwdqWhvKrjpM",
    "$iuIkrkm3NTRoW7rf4qKjpBcjCazBSNyrX8QYSg4Ln-4",
    "$UfJ4H2SOR2X9yUQeJ6_Mm3Hl-lsn5nwTRQyFmc-iY3E"
  ],
  "content": {
    "body": "> <@ticho:cyberdi.sk> \"later\", wink wink :) I'm assuming dendrite developers will provide a way to migrate data at some point :)\n\nthe interest to do that will only last as long as people clamour for migration from synapse to dendrite",
    "format": "org.matrix.custom.html",
    "formatted_body": "<mx-reply><blockquote><a href=\"https://matrix.to/#/!pWCROeqlZcGggueJLt:fachschaften.org/$3TbJ9EGPHo9LAS__BNz7nlXgkXWL4xSuKF2R4BsHNQg?via=nordgedanken.dev&via=matrix.org&via=conduit.rs\">In reply to</a> <a href=\"https://matrix.to/#/@ticho:cyberdi.sk\">@ticho:cyberdi.sk</a><br>&quot;later&quot;, wink wink :) I&#39;m assuming dendrite developers will provide a way to migrate data at some point :)</blockquote></mx-reply>the interest to do that will only last as long as people clamour for migration from synapse to dendrite",
    "m.relates_to": {
      "m.in_reply_to": {
        "event_id": "$3TbJ9EGPHo9LAS__BNz7nlXgkXWL4xSuKF2R4BsHNQg"
      }
    },
    "msgtype": "m.text"
  },
  "depth": 9942,
  "event_id": "$M2SGirLWaI2kpOYRFpmKQVw5NStTR6D_2cNhLoKzlDE",
  "hashes": {
    "sha256": "BC98ZSPPxTiD7V6PBnlTPmzF3iACA4vPmu46pFMHkAg"
  },
  "origin": "jboi.nl",
  "origin_server_ts": 1623845960828,
  "prev_events": [
    "$IhcS7RAv8BjpaDsOEQYj7E_PYEOjNUXQzZpoSeSvJ88"
  ],
  "prev_state": [],
  "room_id": "!pWCROeqlZcGggueJLt:fachschaften.org",
  "sender": "@jboi:jboi.nl",
  "signatures": {
    "jboi.nl": {
      "ed25519:a_fZmz": "/ft1fRcs2UNQUyUTomb/luzKvSSPuHFDYsPPSKY4sancypZdfw/axiA2+en8Itqn7UA/c/lb57PHSKDmnQVXDA"
    }
  },
  "type": "m.room.message",
  "unsigned": {
    "age": 6686938829
  }
}

System Configuration

Conduit Version: 0.2.0 (commit: 102ff574)
Database backend (default is sqlite): sqlite

### Description While running Conduit, i occasionally get out-of-line events. Most of those events are from a time before i joined a room. Example: ![image](/uploads/ceffd80fa587c6c5d150d4fabbb0f289/image.png) Room join pdu: ```json { "auth_events": [ "$fP9-VwJPUrB92dKnf0vcLK3Ntz1z6Xr-n87Fo388zdU", "$UfJ4H2SOR2X9yUQeJ6_Mm3Hl-lsn5nwTRQyFmc-iY3E", "$LqGoKWH6xTkllRooB4tjfqbViPCTEBuOfB_k8Dc-OJ8" ], "content": { "displayname": "weasy ⚡️", "membership": "join" }, "depth": 21485, "event_id": "$_j3JxwRDeofxevo1GgjWIZmWMCq4iQzG0DhlFE23V8M", "hashes": { "sha256": "iHdcgr+EOw3S7pO/+6kQa7ITcvP27k+LH45FAFTY2CA" }, "origin_server_ts": 1629142980912, "prev_events": [ "$5NpgJErpIS2w7AL9g3ZByzPLMZ90yssHvEJekx8MIk0" ], "room_id": "!pWCROeqlZcGggueJLt:fachschaften.org", "sender": "@weasy:weasy-is-my.name", "signatures": { "weasy-is-my.name": { "ed25519:skvFbwnQ": "DJYc1Xq+axnnpsr8q1ATQY/3+YvDUgcknYmRekt2WEqewJ0/Qm5wQ6VeG3y0MQrWjb5bLdWxQbrKc5SLiMXxCw" } }, "state_key": "@weasy:weasy-is-my.name", "type": "m.room.member", "unsigned": {} } ``` Pdu of the 16. June 2021 message from Jonathan: ```json { "auth_events": [ "$VMJIDcmFltqBX2XoUZ4UUj0UwX8hYeqpwdqWhvKrjpM", "$iuIkrkm3NTRoW7rf4qKjpBcjCazBSNyrX8QYSg4Ln-4", "$UfJ4H2SOR2X9yUQeJ6_Mm3Hl-lsn5nwTRQyFmc-iY3E" ], "content": { "body": "> <@ticho:cyberdi.sk> \"later\", wink wink :) I'm assuming dendrite developers will provide a way to migrate data at some point :)\n\nthe interest to do that will only last as long as people clamour for migration from synapse to dendrite", "format": "org.matrix.custom.html", "formatted_body": "<mx-reply><blockquote><a href=\"https://matrix.to/#/!pWCROeqlZcGggueJLt:fachschaften.org/$3TbJ9EGPHo9LAS__BNz7nlXgkXWL4xSuKF2R4BsHNQg?via=nordgedanken.dev&via=matrix.org&via=conduit.rs\">In reply to</a> <a href=\"https://matrix.to/#/@ticho:cyberdi.sk\">@ticho:cyberdi.sk</a><br>"later", wink wink :) I'm assuming dendrite developers will provide a way to migrate data at some point :)</blockquote></mx-reply>the interest to do that will only last as long as people clamour for migration from synapse to dendrite", "m.relates_to": { "m.in_reply_to": { "event_id": "$3TbJ9EGPHo9LAS__BNz7nlXgkXWL4xSuKF2R4BsHNQg" } }, "msgtype": "m.text" }, "depth": 9942, "event_id": "$M2SGirLWaI2kpOYRFpmKQVw5NStTR6D_2cNhLoKzlDE", "hashes": { "sha256": "BC98ZSPPxTiD7V6PBnlTPmzF3iACA4vPmu46pFMHkAg" }, "origin": "jboi.nl", "origin_server_ts": 1623845960828, "prev_events": [ "$IhcS7RAv8BjpaDsOEQYj7E_PYEOjNUXQzZpoSeSvJ88" ], "prev_state": [], "room_id": "!pWCROeqlZcGggueJLt:fachschaften.org", "sender": "@jboi:jboi.nl", "signatures": { "jboi.nl": { "ed25519:a_fZmz": "/ft1fRcs2UNQUyUTomb/luzKvSSPuHFDYsPPSKY4sancypZdfw/axiA2+en8Itqn7UA/c/lb57PHSKDmnQVXDA" } }, "type": "m.room.message", "unsigned": { "age": 6686938829 } } ``` ### System Configuration Conduit Version: 0.2.0 (commit: 102ff574) Database backend (default is sqlite): sqlite

shsorbom commented

2021-09-02 15:44:59 +00:00

(Migrated from gitlab.com)

So far, this only seems to happen in federated rooms for me.

timokoesters commented

2021-09-02 18:23:05 +00:00

(Migrated from gitlab.com)

If you have no sensitive data you DM me the db.

Weasy commented

2021-09-02 20:34:59 +00:00

(Migrated from gitlab.com)

Will do that. I don't think i will find the time to take a closer look before sunday.

timokoesters commented

2021-09-03 09:28:29 +00:00

(Migrated from gitlab.com)

assigned to @timokoesters

timokoesters commented

2021-09-03 09:42:56 +00:00

(Migrated from gitlab.com)

mentioned in merge request !188

sspaeth commented

2021-09-03 09:56:52 +00:00

(Migrated from gitlab.com)

Sending you my db. Here are 2 events where a June 29th even follows a Sep 2 event:
(
Room ID: !pWCROeqlZcGggueJLt:fachschaften.org)

{
  "content": {
    "displayname": "ish",
    "membership": "join"
  },
  "event_id": "$5cBDx8_SJ7CpPLliBjNyP8LL5qaYq-SFHBY1SNdfUQs",
  "origin_server_ts": 1630618086303,
  "room_id": "!pWCROeqlZcGggueJLt:fachschaften.org",
  "sender": "@jasonish:matrix.org",
  "state_key": "@jasonish:matrix.org",
  "type": "m.room.member",
  "unsigned": {
    "age_ts": 1630618086303
  }
}


{
  "content": {
    "body": "> <@tglman:matrix.org> that will increase the data itself, and yes definitely a real use case, so it does make sense, one thing i was interested more was more messages and more rooms, in general mor data\n\nThe romeo and juliet benchmark is easy to tweak",
    "format": "org.matrix.custom.html",
    "formatted_body": "<mx-reply><blockquote><a href=\"https://matrix.to/#/!pWCROeqlZcGggueJLt:fachschaften.org/$jqB_8s1MONfex2pzDVv-b6vTMrU0n60icmSrrFCmz_8?via=nordgedanken.dev&via=matrix.org&via=conduit.rs\">In reply to</a> <a href=\"https://matrix.to/#/@tglman:matrix.org\">@tglman:matrix.org</a><br>that will increase the data itself, and yes definitely a real use case, so it does make sense, one thing i was interested more was more messages and more rooms, in general mor data</blockquote></mx-reply>The romeo and juliet benchmark is easy to tweak",
    "m.relates_to": {
      "m.in_reply_to": {
        "event_id": "$jqB_8s1MONfex2pzDVv-b6vTMrU0n60icmSrrFCmz_8"
      }
    },
    "msgtype": "m.text"
  },
  "event_id": "$Bm4b0htsxt-PqdnQUfpADayFe9DS_4Mieli8AFe5II8",
  "origin_server_ts": 1624984645208,
  "room_id": "!pWCROeqlZcGggueJLt:fachschaften.org",
  "sender": "@timo:conduit.rs",
  "type": "m.room.message",
  "unsigned": {}
}

Plenty messages from July 2 and 3 follow, before it resumes the Sep3 conversation.

Sending you my db. Here are 2 events where a June 29th even follows a Sep 2 event: ( Room ID: !pWCROeqlZcGggueJLt:fachschaften.org) ``` { "content": { "displayname": "ish", "membership": "join" }, "event_id": "$5cBDx8_SJ7CpPLliBjNyP8LL5qaYq-SFHBY1SNdfUQs", "origin_server_ts": 1630618086303, "room_id": "!pWCROeqlZcGggueJLt:fachschaften.org", "sender": "@jasonish:matrix.org", "state_key": "@jasonish:matrix.org", "type": "m.room.member", "unsigned": { "age_ts": 1630618086303 } } { "content": { "body": "> <@tglman:matrix.org> that will increase the data itself, and yes definitely a real use case, so it does make sense, one thing i was interested more was more messages and more rooms, in general mor data\n\nThe romeo and juliet benchmark is easy to tweak", "format": "org.matrix.custom.html", "formatted_body": "<mx-reply><blockquote><a href=\"https://matrix.to/#/!pWCROeqlZcGggueJLt:fachschaften.org/$jqB_8s1MONfex2pzDVv-b6vTMrU0n60icmSrrFCmz_8?via=nordgedanken.dev&via=matrix.org&via=conduit.rs\">In reply to</a> <a href=\"https://matrix.to/#/@tglman:matrix.org\">@tglman:matrix.org</a><br>that will increase the data itself, and yes definitely a real use case, so it does make sense, one thing i was interested more was more messages and more rooms, in general mor data</blockquote></mx-reply>The romeo and juliet benchmark is easy to tweak", "m.relates_to": { "m.in_reply_to": { "event_id": "$jqB_8s1MONfex2pzDVv-b6vTMrU0n60icmSrrFCmz_8" } }, "msgtype": "m.text" }, "event_id": "$Bm4b0htsxt-PqdnQUfpADayFe9DS_4Mieli8AFe5II8", "origin_server_ts": 1624984645208, "room_id": "!pWCROeqlZcGggueJLt:fachschaften.org", "sender": "@timo:conduit.rs", "type": "m.room.message", "unsigned": {} } ``` Plenty messages from July 2 and 3 follow, before it resumes the Sep3 conversation.

Weasy commented

2021-09-08 10:03:06 +00:00

(Migrated from gitlab.com)

Looks like there still is a bug in message ordering.

and

My server was down from the afternoon to early morning the next day, because of an optic fiber cable defect outside the datacenter. It looks like Conduit then fetched most (some images are blank) of the missing messages/PDUs, but ordered them wrong.

Looks like there still is a bug in message ordering. >![image](/uploads/3620a888ba4e38ab273ee6c2bc237a2d/image.png) and >![image](/uploads/cd175a3bc7cadfde1c2db61c05a1efe0/image.png) My server was down from the afternoon to early morning the next day, because of an optic fiber cable defect outside the datacenter. It looks like Conduit then fetched most (some images are blank) of the missing messages/PDUs, but ordered them wrong.

timokoesters commented

2021-09-08 12:56:31 +00:00

(Migrated from gitlab.com)

Yeah when the server was offline this can sometimes happen. We won't fix this anytime soon.