3 ms·

Thanks for the link to the postgres discussion! An incident report would definitely be interesting. I haven't reported the missing messages thing anywhere as I

by dzaima 1y ago

Thanks for the link to the postgres discussion! An incident report would definitely be interesting. I haven't reported the missing messages thing anywhere as I assumed that was just expected (IIRC (and this is a big IIRC) it was mentioned a long time ago (i.e. entirely unrelated to this) by TravisR on Discord bridge problems as a possible cause for missing messages). Here is one event in an affected room, made by a t2bot.io account on July 6, to which t2bot.io accounts can reply to, but which gets an "Event not found." error when requested by me, @dzaima:matrix.org: (and for reference t2bot.io functioned properly on other rooms during that time, so overall t2bot.io↔matrix.org federation should've been fine; and for reference there's a third homeserver that federated with t2bot.io fine in that room just fine too; and to be clear the room works fine now) https://matrix.org/_matrix/client/v3/rooms/!WpdazzauuDxyGNAiCr:matrix.org/context/$BM9s8e3vUhk_iHfdg7D_s7CsQ5COFSPdlJdygz7TKvU

Arathorn 1y ago
Huge thanks for giving details we can investigate; i've taken a look: * That event is in the matrix.org DB... * ...but it failed authentication checks and got rejected from being added to matrix.org's copy of the room, presumably because matrix.org didn't think the sender was in the room at the point that the message was sent * This was almost certainly because the state groups that tracked that the sender was in the room had been deleted due to the where clause on DELETEs matching unrelated rows thanks to the corrupted index So the sending server probably thought it had successfully sent the message, but matrix.org quarantined it due to the db corruption. Therefore we need to manually flush the rejected messages to get them re-authorised in the impacted rooms; we hadn't spotted the problem so forgot to do so. I've nudged the folks who run the server to take a look on Monday. Sorry you got hit by this, and thanks again for the actionable report.
- dzaima 1y ago
  FWIW, my specific room didn't lose much due to the missed syncing, I manually copied over the lost messages (..and we're used to doing so due to the flaky discord bridge). I just noted it as it seemed like a generally relevant thing (and of course perhaps more important for other affected rooms). Thanks for looking into it though, good to know that's at least not how it's supposed to function!

by dzaima 1y ago

Arathorn 1y ago

Huge thanks for giving details we can investigate; i've taken a look: * That event is in the matrix.org DB... * ...but it failed authentication checks and got rejected from being added to matrix.org's copy of the room, presumably because matrix.org didn't think the sender was in the room at the point that the message was sent * This was almost certainly because the state groups that tracked that the sender was in the room had been deleted due to the where clause on DELETEs matching unrelated rows thanks to the corrupted index So the sending server probably thought it had successfully sent the message, but matrix.org quarantined it due to the db corruption. Therefore we need to manually flush the rejected messages to get them re-authorised in the impacted rooms; we hadn't spotted the problem so forgot to do so. I've nudged the folks who run the server to take a look on Monday. Sorry you got hit by this, and thanks again for the actionable report.

dzaima 1y ago
FWIW, my specific room didn't lose much due to the missed syncing, I manually copied over the lost messages (..and we're used to doing so due to the flaky discord bridge). I just noted it as it seemed like a generally relevant thing (and of course perhaps more important for other affected rooms). Thanks for looking into it though, good to know that's at least not how it's supposed to function!