4 ms·

The corrupt room bug which pushed the OP over the edge was due to a corrupted DB index on matrix.org which caused millions of rows to be incorrectly deleted and

The corrupt room bug which pushed the OP over the edge was due to a corrupted DB index on matrix.org which caused millions of rows to be incorrectly deleted and was an absolute nightmare to debug and mitigate. You can see some of the story at https://marc.info/?t=175161814600006&r=1&w=2 https://marc.info/?t=175161814600006&r=1&w=2. We need to write up the incident report; it’s not clear whether the root cause was a bug in postgres (potentially years ago, and only caused problems when we started using the corrupt piece of the index by coincidence) or due to potential corruption from a HW failure years ago. The rooms affected should now work again after reconstructing the lost data. No data should have been lost over federation though; the room history should have synced up as normal. Also, only room state (not msgs) was effected. Do you have a bug report anywhere for the missing msgs?

dzaima 1y ago
Thanks for the link to the postgres discussion! An incident report would definitely be interesting. I haven't reported the missing messages thing anywhere as I assumed that was just expected (IIRC (and this is a big IIRC) it was mentioned a long time ago (i.e. entirely unrelated to this) by TravisR on Discord bridge problems as a possible cause for missing messages). Here is one event in an affected room, made by a t2bot.io account on July 6, to which t2bot.io accounts can reply to, but which gets an "Event not found." error when requested by me, @dzaima:matrix.org: (and for reference t2bot.io functioned properly on other rooms during that time, so overall t2bot.io↔matrix.org federation should've been fine; and for reference there's a third homeserver that federated with t2bot.io fine in that room just fine too; and to be clear the room works fine now) https://matrix.org/_matrix/client/v3/rooms/!WpdazzauuDxyGNAiCr:matrix.org/context/$BM9s8e3vUhk_iHfdg7D_s7CsQ5COFSPdlJdygz7TKvU
- Arathorn 1y ago
  Huge thanks for giving details we can investigate; i've taken a look: * That event is in the matrix.org DB... * ...but it failed authentication checks and got rejected from being added to matrix.org's copy of the room, presumably because matrix.org didn't think the sender was in the room at the point that the message was sent * This was almost certainly because the state groups that tracked that the sender was in the room had been deleted due to the where clause on DELETEs matching unrelated rows thanks to the corrupted index So the sending server probably thought it had successfully sent the message, but matrix.org quarantined it due to the db corruption. Therefore we need to manually flush the rejected messages to get them re-authorised in the impacted rooms; we hadn't spotted the problem so forgot to do so. I've nudged the folks who run the server to take a look on Monday. Sorry you got hit by this, and thanks again for the actionable report.
  - dzaima 1y ago
    FWIW, my specific room didn't lose much due to the missed syncing, I manually copied over the lost messages (..and we're used to doing so due to the flaky discord bridge). I just noted it as it seemed like a generally relevant thing (and of course perhaps more important for other affected rooms). Thanks for looking into it though, good to know that's at least not how it's supposed to function!

dzaima 1y ago

Thanks for the link to the postgres discussion! An incident report would definitely be interesting. I haven't reported the missing messages thing anywhere as I assumed that was just expected (IIRC (and this is a big IIRC) it was mentioned a long time ago (i.e. entirely unrelated to this) by TravisR on Discord bridge problems as a possible cause for missing messages). Here is one event in an affected room, made by a t2bot.io account on July 6, to which t2bot.io accounts can reply to, but which gets an "Event not found." error when requested by me, @dzaima:matrix.org: (and for reference t2bot.io functioned properly on other rooms during that time, so overall t2bot.io↔matrix.org federation should've been fine; and for reference there's a third homeserver that federated with t2bot.io fine in that room just fine too; and to be clear the room works fine now) https://matrix.org/_matrix/client/v3/rooms/!WpdazzauuDxyGNAiCr:matrix.org/context/$BM9s8e3vUhk_iHfdg7D_s7CsQ5COFSPdlJdygz7TKvU

Arathorn 1y ago
Huge thanks for giving details we can investigate; i've taken a look: * That event is in the matrix.org DB... * ...but it failed authentication checks and got rejected from being added to matrix.org's copy of the room, presumably because matrix.org didn't think the sender was in the room at the point that the message was sent * This was almost certainly because the state groups that tracked that the sender was in the room had been deleted due to the where clause on DELETEs matching unrelated rows thanks to the corrupted index So the sending server probably thought it had successfully sent the message, but matrix.org quarantined it due to the db corruption. Therefore we need to manually flush the rejected messages to get them re-authorised in the impacted rooms; we hadn't spotted the problem so forgot to do so. I've nudged the folks who run the server to take a look on Monday. Sorry you got hit by this, and thanks again for the actionable report.
- dzaima 1y ago
  FWIW, my specific room didn't lose much due to the missed syncing, I manually copied over the lost messages (..and we're used to doing so due to the flaky discord bridge). I just noted it as it seemed like a generally relevant thing (and of course perhaps more important for other affected rooms). Thanks for looking into it though, good to know that's at least not how it's supposed to function!