3 ms·

For the key-value storage, would it make sense to use it as a "large" KV database? Eg, for sake of simplicity, imagine you're storing Git in Matrix (similar to

For the key-value storage, would it make sense to use it as a "large" KV database? Eg, for sake of simplicity, imagine you're storing Git in Matrix (similar to what i'm doing, if i use Matrix). I've got two primary concepts: 1. A mutable set of pointers. 2. A series of key values, where the keys are content addresses, and the values are a byte array. My initial assumption would be to use the messages to post all newly created blocks and changes of pointers. Eg, Send a message with a new content address and the bytes. Send a message with a new address for a pointer. For a Git-like to work in that model, it just needs to slurp up any messages in the datastream. Making sure to also slurp in old messages properly that maybe are out of order (ie, reconnecting after being offline for a while). But being a Git-like, it's a fairly simple model. With that said, perhaps the metadata storage in Matrix would better suite some of this? I just imagine Matrix isn't intended to host TiBs of `/state` metadata. Thoughts?

Arathorn 6y ago
Yeah, Matrix isn't a good fit yet for freeform large KV storage. For instance, there isn't a good way to paginate state yet: you can query values by a given key, or a given type, but not paginate. Also, each KV pair gets fed into the merge resolution algorithm, which can start to get sluggish with more than ~100K events (although it's got way better in Synapse since https://github.com/matrix-org/synapse/blob/master/docs/auth_chain_difference_algorithm.md https://github.com/matrix-org/synapse/blob/master/docs/auth_... landed a few weeks ago). In the longer term we'd (well, I'd) love to make it work well with freeform KV data (or object graphs! you gotta store your VR scene graphs in Matrix, after all) and act like some kind of crazy global zero-trust CouchDB or Cassandra. But right now the KV state is meant to be for relatively simple freeform metadata about a room.
- southerntofu 6y ago
  > In the longer term we'd (well, I'd) love to make it work well with freeform KV data That would be great for a decentralized forge based on matrix. This is an area where i think matrix decentralized rooms would shine, as they would effectively prevent DMCA takedowns for software projects.

Arathorn 6y ago

Yeah, Matrix isn't a good fit yet for freeform large KV storage. For instance, there isn't a good way to paginate state yet: you can query values by a given key, or a given type, but not paginate. Also, each KV pair gets fed into the merge resolution algorithm, which can start to get sluggish with more than ~100K events (although it's got way better in Synapse since https://github.com/matrix-org/synapse/blob/master/docs/auth_chain_difference_algorithm.md https://github.com/matrix-org/synapse/blob/master/docs/auth_... landed a few weeks ago). In the longer term we'd (well, I'd) love to make it work well with freeform KV data (or object graphs! you gotta store your VR scene graphs in Matrix, after all) and act like some kind of crazy global zero-trust CouchDB or Cassandra. But right now the KV state is meant to be for relatively simple freeform metadata about a room.

southerntofu 6y ago
> In the longer term we'd (well, I'd) love to make it work well with freeform KV data That would be great for a decentralized forge based on matrix. This is an area where i think matrix decentralized rooms would shine, as they would effectively prevent DMCA takedowns for software projects.