3 ms·

> State resolution is just a total mess. Not since https://matrix.org/blog/2025/08/project-hydra-improving-state-res/ https://matrix.org/blog/2025/08/project-h

by Arathorn 10mo ago

> State resolution is just a total mess. Not since https://matrix.org/blog/2025/08/project-hydra-improving-state-res/ https://matrix.org/blog/2025/08/project-hydra-improving-stat... > I spoke to a homeserver admin once who found that the DB storage space of just the member list can easily reach multiple gigabytes for larger rooms. This is nothing to do with state resolution; it's due to Synapse's implementation deliberately cutting corners on storage efficiency while trading off for speed. I showed how it could be fixed a few months ago here: https://youtu.be/D5zAgVYBuGk?t=1853 https://youtu.be/D5zAgVYBuGk?t=1853, but we prioritised fixing state resets instead. > Also years later, we still don't have custom emojis, user statuses, user bios, invite links etc There are MSCs for all of these now, and implementations are starting to filter through. The reality is that the project has been in a funding crunch since 2023 and we've had to focus on survival by prioritising stuff people pay for (i.e. big servers for govtech deployments) rather than custom emoji.

juliangoldsmith 10mo ago
>trading off for speed If speed is a concern, why did you all stick with Synapse (essentially single-threaded due to the GIL) over moving to Dendrite? As far as I can tell, Dendrite is, for all intents and purposes, abandoned.
- Arathorn 10mo ago
  Because we didn't have enough people or cash to do a good job of simultaneously writing two servers, and as Synapse had gone into production across *.gouv.fr and other critical deployments, we instead frantically backported Dendrite's main novelties to Synapse - adding instead worker processes to Synapse so it could easily scale beyond the GIL: https://matrix.org/blog/2020/11/03/how-we-fixed-synapse-s-scalability/ https://matrix.org/blog/2020/11/03/how-we-fixed-synapse-s-sc... The hope was always that we would then get back to Dendrite and be able to invest in it and migrate over, but the cash situation got worse in 2022 due to Matrix being more and more successful (causing the available $ in the industry to be flow to integrators rather than the upstream project), and instead we had to essentially park Dendrite dev in 2023 other than for critical fixes. Meanwhile, to try to fix the $ situation, we added Rust workers to Synapse as "Synapse Pro" to give customers a reason to actually route money to us as the upstream project, and nowadays Element is actually on a more economically sustainable path. However, at this point the likelihood is that rather than progressing Dendrite we'll instead look to add more Rust to Synapse and fix its resource usage. That said, others are of course very welcome to continue progressing Dendrite forwards, and I personally find it super depressing that we failed to progress both servers at the same time.
  - The_President 10mo ago
    Matrix team is doing a solid job of running - Keep it up and keep eating the Slack/Teams marketshare up with competitive features and pricing. Additional business considerations like HQ location costs, tax liabilities, and talent pool availability on paper also affect what you have to work with. London tax, talent, and labor pay versus Austin for example. Also I got your name wrong last time - I apologize for that.
  - majoe 10mo ago
    > nowadays Element is actually on a more economically sustainable path Good to hear. Keep up the good work.
  - juliangoldsmith 10mo ago
    It sounds like you were stuck between a rock and a hard place there. Hope the Rust integration goes well.
  - xethos 10mo ago
    > the likelihood is that rather than progressing Dendrite we'll instead look to add more Rust to Synapse I thought the goal of Dendrite was decentralization done right? Namely, the ability to run a homeserver from the very phone one is using the client on?
    - Arathorn 10mo ago
      Dendrite did subsequently switch to powering the P2P Matrix work… which also got paused in 2023. We’re currently resurrecting it, but it’s not clear whether Dendrite will be the clientside server impl.
  - DANmode 10mo ago
    Been loving your well-placed comments for 5 years. Keep up the great work - your team should be giving you a raise; you’re a great reflection on them.
  - catgirlinspace 10mo ago
    What’s the best way to fund development of this stuff? I’m aware of donating to the matrix.org foundation, but as far as I can tell none of that goes towards funding server and client implementations since those are Element instead of the foundation.
dfajgljsldkjag 10mo ago
> This is nothing to do with state resolution; it's due to Synapse's implementation deliberately cutting corners on storage efficiency while trading off for speed. It's specifically to increase the speed if *state resolution*. If it weren't for the convoluted state resolution system, there wouldn't be a need to store gigabytes worth of state groups in the database. * https://element-hq.github.io/synapse/latest/usage/administration/state_groups.html https://element-hq.github.io/synapse/latest/usage/administra... * https://github.com/matrix-org/rust-synapse-compress-state https://github.com/matrix-org/rust-synapse-compress-state Maybe there's a way to calculate state without state groups, but I sure don't see one that I can use if I were to run a matrix server. > Not since https://matrix.org/blog/2025/08/project-hydra-improving-stat https://matrix.org/blog/2025/08/project-hydra-improving-stat... Simply fixes some of the many ways that rooms can explode or be bricked. Zero confidence that room brickings are totally fixed once and for all. > There are MSCs for all of these now, and implementations are starting to filter through. The reality is that the project has been in a funding crunch since 2023 and we've had to focus on survival by prioritising stuff people pay for (i.e. big servers for govtech deployments) rather than custom emoji. A funding crunch since 2023 yet those features have been necessary for many years before 2023.
- tcfhgj 10mo ago
  The room state is cached to not need to recompute the current room state from the beginning of time. You probably would do that even if there was no state resolution at all > Simply fixes some of the many ways that rooms can explode or be bricked. How many other ways are there? Afaik none is known
  - dfajgljsldkjag 10mo ago
    > Afaik none is known Before project hydra people didn't know about the room exploit either. They just knew that rooms exploded somehow every once in a while.
    - Arathorn 10mo ago
      Most people actually working on Matrix have been aware of state resets for quite a while. Hydra is just the name of the project which addresses them. There are 3 phases, of which the 1st covers the most serious ones; the 2nd and 3rd phases should drop next year. As an analogy, it's not dissimilar to how Git has added various different merge resolution approaches over the years in order to come up with more predictable and more "do what i mean" algorithms (resolve, recursive, ORT, octopus, etc). It's slightly different in that a bad merge in Matrix feels very unexpected and problematic, whereas manually unpicking collisions in a VCS is just part of the territory.
2Gkashmiri 10mo ago
https://kashmirlife.net/14-messaging-apps-blocked-in-jk-316305/ https://kashmirlife.net/14-messaging-apps-blocked-in-jk-3163... You guys gave up on the national security threat and caved. Dont want authorities knocking my door down for using an app
DANmode 10mo ago
Custom emoji also stood out to me as “who cares”, when compared with the other concerns throughout the thread.

juliangoldsmith 10mo ago

>trading off for speed If speed is a concern, why did you all stick with Synapse (essentially single-threaded due to the GIL) over moving to Dendrite? As far as I can tell, Dendrite is, for all intents and purposes, abandoned.

Arathorn 10mo ago
Because we didn't have enough people or cash to do a good job of simultaneously writing two servers, and as Synapse had gone into production across *.gouv.fr and other critical deployments, we instead frantically backported Dendrite's main novelties to Synapse - adding instead worker processes to Synapse so it could easily scale beyond the GIL: https://matrix.org/blog/2020/11/03/how-we-fixed-synapse-s-scalability/ https://matrix.org/blog/2020/11/03/how-we-fixed-synapse-s-sc... The hope was always that we would then get back to Dendrite and be able to invest in it and migrate over, but the cash situation got worse in 2022 due to Matrix being more and more successful (causing the available $ in the industry to be flow to integrators rather than the upstream project), and instead we had to essentially park Dendrite dev in 2023 other than for critical fixes. Meanwhile, to try to fix the $ situation, we added Rust workers to Synapse as "Synapse Pro" to give customers a reason to actually route money to us as the upstream project, and nowadays Element is actually on a more economically sustainable path. However, at this point the likelihood is that rather than progressing Dendrite we'll instead look to add more Rust to Synapse and fix its resource usage. That said, others are of course very welcome to continue progressing Dendrite forwards, and I personally find it super depressing that we failed to progress both servers at the same time.
- The_President 10mo ago
  Matrix team is doing a solid job of running - Keep it up and keep eating the Slack/Teams marketshare up with competitive features and pricing. Additional business considerations like HQ location costs, tax liabilities, and talent pool availability on paper also affect what you have to work with. London tax, talent, and labor pay versus Austin for example. Also I got your name wrong last time - I apologize for that.
- majoe 10mo ago
  > nowadays Element is actually on a more economically sustainable path Good to hear. Keep up the good work.
- juliangoldsmith 10mo ago
  It sounds like you were stuck between a rock and a hard place there. Hope the Rust integration goes well.
- xethos 10mo ago
  > the likelihood is that rather than progressing Dendrite we'll instead look to add more Rust to Synapse I thought the goal of Dendrite was decentralization done right? Namely, the ability to run a homeserver from the very phone one is using the client on?
  - Arathorn 10mo ago
    Dendrite did subsequently switch to powering the P2P Matrix work… which also got paused in 2023. We’re currently resurrecting it, but it’s not clear whether Dendrite will be the clientside server impl.
- DANmode 10mo ago
  Been loving your well-placed comments for 5 years. Keep up the great work - your team should be giving you a raise; you’re a great reflection on them.
- catgirlinspace 10mo ago
  What’s the best way to fund development of this stuff? I’m aware of donating to the matrix.org foundation, but as far as I can tell none of that goes towards funding server and client implementations since those are Element instead of the foundation.

dfajgljsldkjag 10mo ago

> This is nothing to do with state resolution; it's due to Synapse's implementation deliberately cutting corners on storage efficiency while trading off for speed. It's specifically to increase the speed if *state resolution*. If it weren't for the convoluted state resolution system, there wouldn't be a need to store gigabytes worth of state groups in the database. * https://element-hq.github.io/synapse/latest/usage/administration/state_groups.html https://element-hq.github.io/synapse/latest/usage/administra... * https://github.com/matrix-org/rust-synapse-compress-state https://github.com/matrix-org/rust-synapse-compress-state Maybe there's a way to calculate state without state groups, but I sure don't see one that I can use if I were to run a matrix server. > Not since https://matrix.org/blog/2025/08/project-hydra-improving-stat https://matrix.org/blog/2025/08/project-hydra-improving-stat... Simply fixes some of the many ways that rooms can explode or be bricked. Zero confidence that room brickings are totally fixed once and for all. > There are MSCs for all of these now, and implementations are starting to filter through. The reality is that the project has been in a funding crunch since 2023 and we've had to focus on survival by prioritising stuff people pay for (i.e. big servers for govtech deployments) rather than custom emoji. A funding crunch since 2023 yet those features have been necessary for many years before 2023.

tcfhgj 10mo ago
The room state is cached to not need to recompute the current room state from the beginning of time. You probably would do that even if there was no state resolution at all > Simply fixes some of the many ways that rooms can explode or be bricked. How many other ways are there? Afaik none is known
- dfajgljsldkjag 10mo ago
  > Afaik none is known Before project hydra people didn't know about the room exploit either. They just knew that rooms exploded somehow every once in a while.
  - Arathorn 10mo ago
    Most people actually working on Matrix have been aware of state resets for quite a while. Hydra is just the name of the project which addresses them. There are 3 phases, of which the 1st covers the most serious ones; the 2nd and 3rd phases should drop next year. As an analogy, it's not dissimilar to how Git has added various different merge resolution approaches over the years in order to come up with more predictable and more "do what i mean" algorithms (resolve, recursive, ORT, octopus, etc). It's slightly different in that a bad merge in Matrix feels very unexpected and problematic, whereas manually unpicking collisions in a VCS is just part of the territory.

2Gkashmiri 10mo ago

https://kashmirlife.net/14-messaging-apps-blocked-in-jk-316305/ https://kashmirlife.net/14-messaging-apps-blocked-in-jk-3163... You guys gave up on the national security threat and caved. Dont want authorities knocking my door down for using an app

DANmode 10mo ago

Custom emoji also stood out to me as “who cares”, when compared with the other concerns throughout the thread.