3 ms·

The issue of broken rooms was reportedly the database having gotten corrupted indexes, which affected a bunch of rooms (2/55 of ones I was in): https://github.c

by dzaima 1y ago

The issue of broken rooms was reportedly the database having gotten corrupted indexes, which affected a bunch of rooms (2/55 of ones I was in): https://github.com/element-hq/synapse/issues/18606#issuecomment-3038332995 https://github.com/element-hq/synapse/issues/18606#issuecomm.... Not much a protocol/implementation can do about that. It did take over 2 weeks for it to be resolved though, which is a rather long period of time. Also on the status page - https://status.matrix.org/incidents/8gljb3gtlv11 https://status.matrix.org/incidents/8gljb3gtlv11 (shows start at Jul 2, but I noticed it on Jun 28) After that was resolved, messages that were sent and received by/on other homeservers during that time never ended up on matrix.org, so much for federation :/

udev4096 1y ago
Wait, did anyone investigate on why messages were not stored by other homeservers? That's rather odd, given the selling point of matrix is federation
- dzaima 1y ago
  I've heard that messages not getting through federation in some cases is just a thing that can happen. How that's compatible with calling matrix as having good federation, I don't know.
  - Arathorn 1y ago
    I’m not aware of any scenarios, including this one, where messages don’t federate (unless the destination server has been unavailable and marked down, obviously).
  - udev4096 1y ago
    No, you are making things up. That cannot happen unless the homeservers are not configured properly. If you don't have anything to back that story up, I suggest you slow down on making matrix look bad
    - dzaima 1y ago
      Unfortunately I cannot find where I had read that, might've just misinterpreted discussion about delays as about dropping messages / interpreting messages as answers to questions I didn't actually post... Sorry for the unnecessary negativity!
      - Arathorn 1y ago
        in dzaima’s defence they did hit an instance here where messages were effectively dropped - but only because the matrix.org db was so corrupted it incorrectly quarantined them (and even then, they can be flushed through). corruot db is effectively an extreme form of misconfiguration, i guess.
Arathorn 1y ago
The corrupt room bug which pushed the OP over the edge was due to a corrupted DB index on matrix.org which caused millions of rows to be incorrectly deleted and was an absolute nightmare to debug and mitigate. You can see some of the story at https://marc.info/?t=175161814600006&r=1&w=2 https://marc.info/?t=175161814600006&r=1&w=2. We need to write up the incident report; it’s not clear whether the root cause was a bug in postgres (potentially years ago, and only caused problems when we started using the corrupt piece of the index by coincidence) or due to potential corruption from a HW failure years ago. The rooms affected should now work again after reconstructing the lost data. No data should have been lost over federation though; the room history should have synced up as normal. Also, only room state (not msgs) was effected. Do you have a bug report anywhere for the missing msgs?
- dzaima 1y ago
  Thanks for the link to the postgres discussion! An incident report would definitely be interesting. I haven't reported the missing messages thing anywhere as I assumed that was just expected (IIRC (and this is a big IIRC) it was mentioned a long time ago (i.e. entirely unrelated to this) by TravisR on Discord bridge problems as a possible cause for missing messages). Here is one event in an affected room, made by a t2bot.io account on July 6, to which t2bot.io accounts can reply to, but which gets an "Event not found." error when requested by me, @dzaima:matrix.org: (and for reference t2bot.io functioned properly on other rooms during that time, so overall t2bot.io↔matrix.org federation should've been fine; and for reference there's a third homeserver that federated with t2bot.io fine in that room just fine too; and to be clear the room works fine now) https://matrix.org/_matrix/client/v3/rooms/!WpdazzauuDxyGNAiCr:matrix.org/context/$BM9s8e3vUhk_iHfdg7D_s7CsQ5COFSPdlJdygz7TKvU
  - Arathorn 1y ago
    Huge thanks for giving details we can investigate; i've taken a look: * That event is in the matrix.org DB... * ...but it failed authentication checks and got rejected from being added to matrix.org's copy of the room, presumably because matrix.org didn't think the sender was in the room at the point that the message was sent * This was almost certainly because the state groups that tracked that the sender was in the room had been deleted due to the where clause on DELETEs matching unrelated rows thanks to the corrupted index So the sending server probably thought it had successfully sent the message, but matrix.org quarantined it due to the db corruption. Therefore we need to manually flush the rejected messages to get them re-authorised in the impacted rooms; we hadn't spotted the problem so forgot to do so. I've nudged the folks who run the server to take a look on Monday. Sorry you got hit by this, and thanks again for the actionable report.

by dzaima 1y ago

udev4096 1y ago

Wait, did anyone investigate on why messages were not stored by other homeservers? That's rather odd, given the selling point of matrix is federation

dzaima 1y ago
I've heard that messages not getting through federation in some cases is just a thing that can happen. How that's compatible with calling matrix as having good federation, I don't know.
- Arathorn 1y ago
  I’m not aware of any scenarios, including this one, where messages don’t federate (unless the destination server has been unavailable and marked down, obviously).
- udev4096 1y ago
  No, you are making things up. That cannot happen unless the homeservers are not configured properly. If you don't have anything to back that story up, I suggest you slow down on making matrix look bad
  - dzaima 1y ago
    Unfortunately I cannot find where I had read that, might've just misinterpreted discussion about delays as about dropping messages / interpreting messages as answers to questions I didn't actually post... Sorry for the unnecessary negativity!
    - Arathorn 1y ago
      in dzaima’s defence they did hit an instance here where messages were effectively dropped - but only because the matrix.org db was so corrupted it incorrectly quarantined them (and even then, they can be flushed through). corruot db is effectively an extreme form of misconfiguration, i guess.

Arathorn 1y ago

The corrupt room bug which pushed the OP over the edge was due to a corrupted DB index on matrix.org which caused millions of rows to be incorrectly deleted and was an absolute nightmare to debug and mitigate. You can see some of the story at https://marc.info/?t=175161814600006&r=1&w=2 https://marc.info/?t=175161814600006&r=1&w=2. We need to write up the incident report; it’s not clear whether the root cause was a bug in postgres (potentially years ago, and only caused problems when we started using the corrupt piece of the index by coincidence) or due to potential corruption from a HW failure years ago. The rooms affected should now work again after reconstructing the lost data. No data should have been lost over federation though; the room history should have synced up as normal. Also, only room state (not msgs) was effected. Do you have a bug report anywhere for the missing msgs?

dzaima 1y ago
Thanks for the link to the postgres discussion! An incident report would definitely be interesting. I haven't reported the missing messages thing anywhere as I assumed that was just expected (IIRC (and this is a big IIRC) it was mentioned a long time ago (i.e. entirely unrelated to this) by TravisR on Discord bridge problems as a possible cause for missing messages). Here is one event in an affected room, made by a t2bot.io account on July 6, to which t2bot.io accounts can reply to, but which gets an "Event not found." error when requested by me, @dzaima:matrix.org: (and for reference t2bot.io functioned properly on other rooms during that time, so overall t2bot.io↔matrix.org federation should've been fine; and for reference there's a third homeserver that federated with t2bot.io fine in that room just fine too; and to be clear the room works fine now) https://matrix.org/_matrix/client/v3/rooms/!WpdazzauuDxyGNAiCr:matrix.org/context/$BM9s8e3vUhk_iHfdg7D_s7CsQ5COFSPdlJdygz7TKvU
- Arathorn 1y ago
  Huge thanks for giving details we can investigate; i've taken a look: * That event is in the matrix.org DB... * ...but it failed authentication checks and got rejected from being added to matrix.org's copy of the room, presumably because matrix.org didn't think the sender was in the room at the point that the message was sent * This was almost certainly because the state groups that tracked that the sender was in the room had been deleted due to the where clause on DELETEs matching unrelated rows thanks to the corrupted index So the sending server probably thought it had successfully sent the message, but matrix.org quarantined it due to the db corruption. Therefore we need to manually flush the rejected messages to get them re-authorised in the impacted rooms; we hadn't spotted the problem so forgot to do so. I've nudged the folks who run the server to take a look on Monday. Sorry you got hit by this, and thanks again for the actionable report.