9 ms·

Show HN: Matrix-CRDT – real-time collaborative apps using Matrix as backend

320 points by YousefED 5y ago

Hi HN! Matrix-CRDT connects the worlds of Yjs [1] (a proven, high performance CRDT) with that of Matrix.org [2]. It started as an experiment, asking myself; can we store "state updates of a datastore" in Matrix instead of chat messages? Now, I'm convinced it's actually a really powerful combination to develop real-time, collaborative software. I'm using it for a new project and so far didn't have to write a backend yet. Matrix takes care of a lot of stuff: Authentication, E2EE, federation, hosting, etc. - so I can focus on the client. I love the ideas of Local First [3] software. Personally, I'm convinced the technologies in this ecosystem (CRDTs, etc) are really powerful and can do more to decentralize software than many web3 technologies that currently receive much of the hype. However, it's still early days, so I decided to open source parts that I think will be useful to others (I recently also shared SyncedStore [4]) [1] https://github.com/yjs/yjs [2] https://www.matrix.org [3] https://www.inkandswitch.com/local-first/ [4] https://www.syncedstore.org

hammyhavoc 5y ago
This is a whole other level of cool. Jesus Christ.
outside1234 5y ago
Lots of Matrix ignorance here - so sorry if this is a dumb question. When you auth with Matrix, is that a centralized action (one company / org behind it) or is that a decentralized action (its more a protocol for auth amongst the community of users)? ie. Is using Matrix for auth decentralized or a different centralization?
- YousefED 5y ago
  You authenticate against a Matrix server (the “homeserver”). This can be a server that you host yourself, or one that is readily available (matrix.org is the most popular one). This design is fairly similar to email where you can be @gmail, @hotmail, or @whateveryoulike.com
  - outside1234 5y ago
    Great - got it - thanks!
deleted 5y ago
[deleted]
samwillis 5y ago
Awesome project! I have said it before and I will say it again, I think this year is going to be the year of Yjs. The ecosystem around it is brilliant and I believe it will become the leading CRDT framework (it probably already is). Happy to see you using TipTap too for the demo, a brilliant rich text editor framework (with support for Vue.js, React and Svelte) built on top ProseMirror with first class Yjs support. https://tiptap.dev https://tiptap.dev The TipTap guys (Philipp and Hans) along with Kevin Jahns of yJS have stated the y-collective inorder to centralise funding and organising of the Yjs ecosystem: https://opencollective.com/y-collective https://opencollective.com/y-collective Personally I'm excited about the us of Yjs and CRDTs outside of just rich text editing, I think there is great potential to build a distributed offline enabled database targeting mobile and PWAs with it. Something like PouchDB but with automatic conflict resolution. Finally there is a Rust port in development to improve the (already very good) performance and make it cross platform with other languages. https://github.com/y-crdt/y-crdt https://github.com/y-crdt/y-crdt
- YousefED 5y ago
  Thanks! And definitely agree, can 100% recommend TipTap!
- hanspagel 5y ago
  Thanks for the mention, Sam! Great to have you and Yousef in the community. BTW, we are already tinkering on some interesting stuff with the Rust port, too. :-)
  - samwillis 5y ago
    Thanks! Sadly haven’t had the time in the last 6 months to continue working on what I was experimenting with. Hopefully will soon.
- thruflo 5y ago
  > I think there is great potential to build a distributed offline enabled database targeting mobile and PWAs with it. Something like PouchDB but with automatic conflict resolution. Take a look at https://concordant.io https://concordant.io
infogulch 5y ago
I recall that matrix itself has some CRDT-like data structure to transmit messages. If that's accurate, could you map some of the functionality of yjs directly into native matrix ops?
- YousefED 5y ago
  Indeed, I think Matrix state resolution has some CRDT-like properties. Afaik, the goal of this is mainly to manage room state across servers when using federation - but I'd have to dive deeper into this. However, not all CRDTs are the same (actually, I think the definition is still somewhat vague - most things "eventually consistent" could be called a CRDT). Yjs is specifically designed for high-performance operations and has great support for rich-text collaboration. It also works great while offline - and you can connect different providers as you like (for example, store updates both locally in IndexedDB, and transmit messages over websocket / webrtc / and now, also Matrix). Definitely wanted to keep this flexibility.
- Arathorn 5y ago
  On the Matrix core team, we're working on 'native' collaborative editing collaboration via the Matrix DAG - while shamelessly learning from CRDT-over-Matrix projects like this one :)
  - kevinjahns 5y ago
    What I dislike about these attempts is that you will just end up with yet another CRDT implementation that is incompatible with the existing ecosystem (editors, drawing apps, state management, ...). Instead, I want to encourage you to build an API that others can use to efficiently store shared data. Feel free to ping me if you need input. - Kevin (Author of Yjs)
    - infogulch 5y ago
      I'm happy to see representatives from both the matrix and yjs communities interact more directly. Can you expand on what you mean by this? > an API that others can use to efficiently store shared data What would you expect this API look like in a bit more detail? Would it be able to abstract any of the underlying CRDT logic? Would it just be a raw stream of authenticated messages with partial ordering? Something in-between?
noman-land 5y ago
This is so cool. Come on, everyone! Let's make matrix the new thing, already.
toyg 5y ago
I wish this was available for Python. Does anybody know what the Py-equivalent would be (if there is one)?
- samwillis 5y ago
  The Rust port of Yjs is aiming to have Python bindings. https://github.com/y-crdt/y-crdt https://github.com/y-crdt/y-crdt https://github.com/y-crdt/y-crdt/tree/main/y-py https://github.com/y-crdt/y-crdt/tree/main/y-py
ath92 5y ago
Amazing work. I think there's a whole host of apps that just store some small amount of data for each user that could work entirely on something like this. Some nice things: - Anyone can host their own Matrix homeserver, which can then sync its data within a federation, allowing users pretty fine-grained control over what data they share and with whom. - Add end-to-end encryption, and it becomes even better: data stored somewhere in the cloud (i.e. the user doesn't have to manage a server), but the server knows almost nothing about what you've stored aside from some metadata.
- YousefED 5y ago
  Exactly. One of the reasons to build on top of Matrix was that it opens the path to end-to-end-encryption. Matrix E2EE (Olm) is heavily based on Signal, and wouldn't be trivial to build from the ground up. I have a prototype running that uses Matrix-CRDT and Matrix E2EE, and it worked great. It's still a bit of a hassle to set up though (mainly, configuring the matrix sdk correctly), I hope to make that easier later this year.
maelito 5y ago
I've used yjs for a project that aims at computing individual climate footprints as a group in a conference. It works beautifully with almost no code. I used Yjs P2P, but P2P is blocked in numerous networks. If you want to try, with an hackernews room, but sorry it's in french : https://nosgestesclimat.fr/conférence/hackernews https://nosgestesclimat.fr/conférence/hackernews So using Matrix as a backend is a great idea. The only caveat I found to Yjs is that if you want to persist data (and not lose it when the server crashes), nothing's really plug-and-play, so I went with Supabase.
kukabynd 5y ago
Awesome, thanks for sharing! I’ve been building using yjs and it’s indeed something much more exciting than a bunch of bs we see with web3.
- noman-land 5y ago
  Matrix is for sure part of web3 since it's an open protocol.
  - tinco 5y ago
    Open protocols and decentralization are web1 not web3. It's how the web was originally envisioned. Web3 is specifically about replacing centralized authority with cryptographic authority. But whether that's actually useful enough for anyone for it to be considered a revolution on the web like web2 was remains to be seen.
    - noman-land 5y ago
      Unlike web1 and web2, web3 is an ethos more than a set of technologies. web1 and web2 are descriptions of the past, whereas web3 is a vision for the future.
      - tinco 5y ago
        Alright, but if you can use that ethos to argue that e-mail, Bittorrent and IRC are the future I'm going to look at you funny.
        opan 5y ago
        I'd like to think email, irc, and bittorrent will still be around far into the future, at least.
      - heartbeats 5y ago
        Unlike web1 and web2, web3 is a grift. In contrast, this appears to be a genuinely innovative technology.
jeroenhd 5y ago
This is an interesting use of the Matrix protocol. I like the many creative ways Matrix is being used outside the chat space, where its predominant use lies. However, I have several questions about this approach. - What happens with E2EE when there are key exchange problems? (i.e. a server goes down and the account is moved) - What happens when someone redacts a message? I assume you'd want to disable redactions for these rooms or you'll end up with broken update trees! - How do you communicate clearly that any data you add is in the event chain forever? In Word and GDocs you can remove old revisions of a document, but I don't think you can in this system? That could be a feature, of course! - What's the performance impact of such a system on a server editing reasonably large documents? If twenty people each edit ten to twenty documents you end up with quite a large set of state, especially over time!
- kevinjahns 5y ago
  I can only answer to the last question. Yjs uses several performance optimizations to produce small documents (both in memory and in the encoded state). Since humans type relatively slow (<60 actions per minute), it is impossible for humans to create a document that has performance problems. I showed this in [1]. Relm [2] even models a 3d world using Yjs. I don't necessarily recommend doing this as 3d applications usually produce a lot more actions per minute than text applications. This required some workarounds and deep knowledge of how Yjs' optimizations work. But it's definitely possible. [1]: https://blog.kevinjahns.de/are-crdts-suitable-for-shared-editing/ https://blog.kevinjahns.de/are-crdts-suitable-for-shared-edi... [2]: https://www.relm.us/ https://www.relm.us/
- YousefED 5y ago
  Thanks for the questions! - If you'd enable E2EE for the Matrix client that you pass to Matrix-CRDT, key management is covered in the same way that Matrix does it normally. If key sharing with a particular user is broken, then you won't see updates from that user anymore and v.v. Basically, the transport between you and that user is broken. As CRDTs such as Yjs are designed specifically to work Local-first, at the core there is no assumption that all clients should always be connected to each other. Once the clients are able to communicate again, potential conflicts would be "resolved" according to the CRDT design. - Redacting: yes, basically you need to trust clients not to fiddle with messages (I think this is fair as you trust them to work with you on the same data already) - UX / communication: good question! Technically would be possible to purge old (deleted data), but I think this is still something we need to explore together while we start to see more mature software built on these technologies. - Kevin already answered your performance questions. Matrix-CRDT makes an additional optimization so that not the entire history of the Matrix room needs to be retrieved (see "Snapshots" in the readme)
JanisIO 5y ago
Thank you for your work! First SyncedStore, now this. Yjs keeps getting better and is already my favourite technology to work with. Going to include this in my app :) https://app.lity.cc https://app.lity.cc
- sunbum 5y ago
  Kinda a dick move that it hijacks the back button :(
octopoc 5y ago
(dreaming big here) You could make this a basis for a generic conversational UI[1] for web data. It wouldn't be a perfect UI for most applications, but it provide a couple of important features: 1. It would be easy to manage data sharing. Post updates to data in channels that the relevant people have access to. 2. It would be easy to debug data changes. I could see it becoming the equivalent of a terminal app but for the web. [1] https://en.wikipedia.org/wiki/Conversational_user_interface https://en.wikipedia.org/wiki/Conversational_user_interface
Cthulhu_ 5y ago
Yjs looks interesting; I'm currently building a management UI that for now is a very straightforward set of REST APIs and a database, but there's a lot of concerns about people editing the same configuration simultaneously; I wonder if something like yjs can be used to create a collaborative editing environment. I'd probably have to re-architect the whole front- and back-end though. Although on the other hand, there will be some code editors (currently using Monaco) where this could be usable.
- er4hn 5y ago
  My understanding of CRDT is that it shines the most when you have collaborative documents. For a mgmt UI along the lines of "listen on port X, log at level Y" where does CRDT become handy? Wouldn't you just want "last writer wins" for that sort of use case?
robto 5y ago
See, this is what I keep going on about to my friends - Matrix is actually a graph syncing protocol, not an instant messaging protocol, and is a great way to build federated applications. I get that Element needs to focus on one use case in order to pay the bills, but there's a lot of room for applications that have barely been explored - calendars, contacts, games, social media, heck - you could build a federated gitlab-like collaboration site with truly federated data. You get multi-device sync, e2e encryption, and identity management out of the box. Thanks for putting this together, I hope there will be a gold rush as people realize how much low-hanging fruit there is!
- mxuribe 5y ago
  I think the very beginning the matrix project folks have used language similar to matrix being data synching...etc...But, that, the first use case is chat...but won;t end there...etc. Obviously i'm paraphrasing heavily...but once i read that, i've used similar language when i introduce matrix to friends...basically, chat/instant messenger is the first but not last/not only use case/scenario for this very cool technology.
- all2 5y ago
  > calendars, contacts I'm interested in this just to keep my devices sync'd up. I know Samsung/Apple/MS etc all have syncing services, but I really really want to get all my data out of their ecosystems. Just a distributed calendar app would be pretty amazing. No more iCal subscription/export/whatevers, 'just' federate the calendar between devices, and arbitrarily share it with anyone else. I have no clue where to start with building an app on these abstractions. Does anyone have any pointers? Documentation to read? High level overviews? Etc.?
  - robto 5y ago
    I'd start with reading the Matrix spec, you'll want to know how the syncing protocol works, and this will give you enough familiarity that you can start playing around with curl to actually see it in action. I'd highly recommend understanding how conflict resolution works, the "Analysis of the Matrix Event GraphReplicated Data Type"[0] lays it out pretty well. Matrix-CRDT posted here should abstract most of the nitty gritty stuff, but there are no shortcuts to designing with CRDTs so familiarizing yourself with those is important. Other than that, just start playing and asking questions! The #matrix:matrix.org room is full of friendly and helpful people and I'm sure you'd be able to get answers. [0]https://arxiv.org/pdf/2011.06488.pdf https://arxiv.org/pdf/2011.06488.pdf
panick21_ 5y ago
Having an Datomic like store backed by something like this. https://github.com/replikativ/datahike https://github.com/replikativ/datahike Is an Open Source variant of Datomic. Lambdaforge wants to eventually have this work with CRDTs. Using the Matrix ecosystem for this is quite interesting as it solves many problems for you already.
- dunham 5y ago
  I'd also love to see an offline-first version of the datomic model. I've thought about datomic + crdts in the past, and it seems like ":db/unique :db.unique/identity" properties would be an issue. Whether or not the transaction fails depends on the current content of the database. I don't know if there is a way around this (while still being a CRDT) or how necessary this feature is.
pixel_tracing 5y ago
Awesome project! Quick question I notice this uses WebRTC, why integrate / bake this in directly rather than abstract it out?
- YousefED 5y ago
  Good spot. The WebRTC part is currently experimental and might be superceded by a different solution. If that's not the case I'll probably abstract it out indeed!
outside1234 5y ago
Just to understand, is the concept here that you send all of the updates for a particular data item to a Matrix "room" and then converge on the current state of that data item by each client replaying that data?
- YousefED 5y ago
  Yep, thats pretty much correct. With the addition of sending occasional “snapshots” so clients don't need to read the entire room history initially (more about this in the bottom of the readme).
outside1234 5y ago
Is there any community where projects like this have a center of gravity? (eg. conferences, Discord, Matrix, ...) for people interested in getting involved in the community?
- YousefED 5y ago
  I just set up a Discord @ https://discord.gg/9GxSXCeQyN https://discord.gg/9GxSXCeQyN. Please join! It’s called TypeCell as that’s the (yet to be announced) project both Matrix-CRDT and SyncedStore are building blocks for. If you’re interested in tech like this and maybe also “future of programming” (those bret victor things ;) ), knowledge management software (notion, roam, obsidian) - i’m sure you’ll like it!
  - DenseComet 5y ago
    Heh. I get why you're using Discord, but it's unfortunate that a project building on Matrix does not use it for chat.
    - YousefED 5y ago
      Fair point haha. I’ll also set up a matrix room and bridge it to Discord.
      - trenchgun 5y ago
        Excellent. Discord is not just proprietary, but it also has a messy UI and bad UX.
  - brunoqc 5y ago
    oh! I dream of a CRDT note-taking app.
lgas 5y ago
This sounds like (at least a big step towards) distributed event sourcing which is something that I have been anticipating for a while now but haven't heard any news of.
klabb3 5y ago
> Personally, I'm convinced the technologies in this ecosystem (CRDTs, etc) are really powerful and can do more to decentralize software than many web3 technologies that currently receive much of the hype. 100% agree! Web3 is really only tackling monetary instruments, which is interesting but very narrow. The majority of apps people need don't require global consensus, and it's too slow and expensive anyway. Yet, there's so much interesting stuff that can be decentralized/federated. > Matrix takes care of a lot of stuff: Authentication, E2EE, federation, hosting, etc. - so I can focus on the client. This is great to hear! What else besides CRDTs do you think is needed in order for matrix to be a platform for general purpose apps like this?
- YousefED 5y ago
  Thanks! Personally, I’m looking forward to stronger / cleaner (TypeScript) sdks with for example easier auth and e2ee key management. I heard this is on their roadmap for this year, so that’s pretty exciting!

320 points by YousefED 5y ago

hammyhavoc 5y ago

This is a whole other level of cool. Jesus Christ.

outside1234 5y ago

Lots of Matrix ignorance here - so sorry if this is a dumb question. When you auth with Matrix, is that a centralized action (one company / org behind it) or is that a decentralized action (its more a protocol for auth amongst the community of users)? ie. Is using Matrix for auth decentralized or a different centralization?

YousefED 5y ago
You authenticate against a Matrix server (the “homeserver”). This can be a server that you host yourself, or one that is readily available (matrix.org is the most popular one). This design is fairly similar to email where you can be @gmail, @hotmail, or @whateveryoulike.com
- outside1234 5y ago
  Great - got it - thanks!

deleted 5y ago

[deleted]

samwillis 5y ago

Awesome project! I have said it before and I will say it again, I think this year is going to be the year of Yjs. The ecosystem around it is brilliant and I believe it will become the leading CRDT framework (it probably already is). Happy to see you using TipTap too for the demo, a brilliant rich text editor framework (with support for Vue.js, React and Svelte) built on top ProseMirror with first class Yjs support. https://tiptap.dev https://tiptap.dev The TipTap guys (Philipp and Hans) along with Kevin Jahns of yJS have stated the y-collective inorder to centralise funding and organising of the Yjs ecosystem: https://opencollective.com/y-collective https://opencollective.com/y-collective Personally I'm excited about the us of Yjs and CRDTs outside of just rich text editing, I think there is great potential to build a distributed offline enabled database targeting mobile and PWAs with it. Something like PouchDB but with automatic conflict resolution. Finally there is a Rust port in development to improve the (already very good) performance and make it cross platform with other languages. https://github.com/y-crdt/y-crdt https://github.com/y-crdt/y-crdt

YousefED 5y ago
Thanks! And definitely agree, can 100% recommend TipTap!
hanspagel 5y ago
Thanks for the mention, Sam! Great to have you and Yousef in the community. BTW, we are already tinkering on some interesting stuff with the Rust port, too. :-)
- samwillis 5y ago
  Thanks! Sadly haven’t had the time in the last 6 months to continue working on what I was experimenting with. Hopefully will soon.
thruflo 5y ago
> I think there is great potential to build a distributed offline enabled database targeting mobile and PWAs with it. Something like PouchDB but with automatic conflict resolution. Take a look at https://concordant.io https://concordant.io

infogulch 5y ago

I recall that matrix itself has some CRDT-like data structure to transmit messages. If that's accurate, could you map some of the functionality of yjs directly into native matrix ops?

YousefED 5y ago
Indeed, I think Matrix state resolution has some CRDT-like properties. Afaik, the goal of this is mainly to manage room state across servers when using federation - but I'd have to dive deeper into this. However, not all CRDTs are the same (actually, I think the definition is still somewhat vague - most things "eventually consistent" could be called a CRDT). Yjs is specifically designed for high-performance operations and has great support for rich-text collaboration. It also works great while offline - and you can connect different providers as you like (for example, store updates both locally in IndexedDB, and transmit messages over websocket / webrtc / and now, also Matrix). Definitely wanted to keep this flexibility.
Arathorn 5y ago
On the Matrix core team, we're working on 'native' collaborative editing collaboration via the Matrix DAG - while shamelessly learning from CRDT-over-Matrix projects like this one :)
- kevinjahns 5y ago
  What I dislike about these attempts is that you will just end up with yet another CRDT implementation that is incompatible with the existing ecosystem (editors, drawing apps, state management, ...). Instead, I want to encourage you to build an API that others can use to efficiently store shared data. Feel free to ping me if you need input. - Kevin (Author of Yjs)
  - infogulch 5y ago
    I'm happy to see representatives from both the matrix and yjs communities interact more directly. Can you expand on what you mean by this? > an API that others can use to efficiently store shared data What would you expect this API look like in a bit more detail? Would it be able to abstract any of the underlying CRDT logic? Would it just be a raw stream of authenticated messages with partial ordering? Something in-between?

noman-land 5y ago

This is so cool. Come on, everyone! Let's make matrix the new thing, already.

toyg 5y ago

I wish this was available for Python. Does anybody know what the Py-equivalent would be (if there is one)?

samwillis 5y ago
The Rust port of Yjs is aiming to have Python bindings. https://github.com/y-crdt/y-crdt https://github.com/y-crdt/y-crdt https://github.com/y-crdt/y-crdt/tree/main/y-py https://github.com/y-crdt/y-crdt/tree/main/y-py

ath92 5y ago

Amazing work. I think there's a whole host of apps that just store some small amount of data for each user that could work entirely on something like this. Some nice things: - Anyone can host their own Matrix homeserver, which can then sync its data within a federation, allowing users pretty fine-grained control over what data they share and with whom. - Add end-to-end encryption, and it becomes even better: data stored somewhere in the cloud (i.e. the user doesn't have to manage a server), but the server knows almost nothing about what you've stored aside from some metadata.

YousefED 5y ago
Exactly. One of the reasons to build on top of Matrix was that it opens the path to end-to-end-encryption. Matrix E2EE (Olm) is heavily based on Signal, and wouldn't be trivial to build from the ground up. I have a prototype running that uses Matrix-CRDT and Matrix E2EE, and it worked great. It's still a bit of a hassle to set up though (mainly, configuring the matrix sdk correctly), I hope to make that easier later this year.

maelito 5y ago

I've used yjs for a project that aims at computing individual climate footprints as a group in a conference. It works beautifully with almost no code. I used Yjs P2P, but P2P is blocked in numerous networks. If you want to try, with an hackernews room, but sorry it's in french : https://nosgestesclimat.fr/conférence/hackernews https://nosgestesclimat.fr/conférence/hackernews So using Matrix as a backend is a great idea. The only caveat I found to Yjs is that if you want to persist data (and not lose it when the server crashes), nothing's really plug-and-play, so I went with Supabase.

kukabynd 5y ago

Awesome, thanks for sharing! I’ve been building using yjs and it’s indeed something much more exciting than a bunch of bs we see with web3.

noman-land 5y ago
Matrix is for sure part of web3 since it's an open protocol.
- tinco 5y ago
  Open protocols and decentralization are web1 not web3. It's how the web was originally envisioned. Web3 is specifically about replacing centralized authority with cryptographic authority. But whether that's actually useful enough for anyone for it to be considered a revolution on the web like web2 was remains to be seen.
  - noman-land 5y ago
    Unlike web1 and web2, web3 is an ethos more than a set of technologies. web1 and web2 are descriptions of the past, whereas web3 is a vision for the future.
    - tinco 5y ago
      Alright, but if you can use that ethos to argue that e-mail, Bittorrent and IRC are the future I'm going to look at you funny.
      - opan 5y ago
        I'd like to think email, irc, and bittorrent will still be around far into the future, at least.
    - heartbeats 5y ago
      Unlike web1 and web2, web3 is a grift. In contrast, this appears to be a genuinely innovative technology.

jeroenhd 5y ago

This is an interesting use of the Matrix protocol. I like the many creative ways Matrix is being used outside the chat space, where its predominant use lies. However, I have several questions about this approach. - What happens with E2EE when there are key exchange problems? (i.e. a server goes down and the account is moved) - What happens when someone redacts a message? I assume you'd want to disable redactions for these rooms or you'll end up with broken update trees! - How do you communicate clearly that any data you add is in the event chain forever? In Word and GDocs you can remove old revisions of a document, but I don't think you can in this system? That could be a feature, of course! - What's the performance impact of such a system on a server editing reasonably large documents? If twenty people each edit ten to twenty documents you end up with quite a large set of state, especially over time!

kevinjahns 5y ago
I can only answer to the last question. Yjs uses several performance optimizations to produce small documents (both in memory and in the encoded state). Since humans type relatively slow (<60 actions per minute), it is impossible for humans to create a document that has performance problems. I showed this in [1]. Relm [2] even models a 3d world using Yjs. I don't necessarily recommend doing this as 3d applications usually produce a lot more actions per minute than text applications. This required some workarounds and deep knowledge of how Yjs' optimizations work. But it's definitely possible. [1]: https://blog.kevinjahns.de/are-crdts-suitable-for-shared-editing/ https://blog.kevinjahns.de/are-crdts-suitable-for-shared-edi... [2]: https://www.relm.us/ https://www.relm.us/
YousefED 5y ago
Thanks for the questions! - If you'd enable E2EE for the Matrix client that you pass to Matrix-CRDT, key management is covered in the same way that Matrix does it normally. If key sharing with a particular user is broken, then you won't see updates from that user anymore and v.v. Basically, the transport between you and that user is broken. As CRDTs such as Yjs are designed specifically to work Local-first, at the core there is no assumption that all clients should always be connected to each other. Once the clients are able to communicate again, potential conflicts would be "resolved" according to the CRDT design. - Redacting: yes, basically you need to trust clients not to fiddle with messages (I think this is fair as you trust them to work with you on the same data already) - UX / communication: good question! Technically would be possible to purge old (deleted data), but I think this is still something we need to explore together while we start to see more mature software built on these technologies. - Kevin already answered your performance questions. Matrix-CRDT makes an additional optimization so that not the entire history of the Matrix room needs to be retrieved (see "Snapshots" in the readme)

JanisIO 5y ago

Thank you for your work! First SyncedStore, now this. Yjs keeps getting better and is already my favourite technology to work with. Going to include this in my app :) https://app.lity.cc https://app.lity.cc

sunbum 5y ago
Kinda a dick move that it hijacks the back button :(

octopoc 5y ago

(dreaming big here) You could make this a basis for a generic conversational UI[1] for web data. It wouldn't be a perfect UI for most applications, but it provide a couple of important features: 1. It would be easy to manage data sharing. Post updates to data in channels that the relevant people have access to. 2. It would be easy to debug data changes. I could see it becoming the equivalent of a terminal app but for the web. [1] https://en.wikipedia.org/wiki/Conversational_user_interface https://en.wikipedia.org/wiki/Conversational_user_interface

Cthulhu_ 5y ago

Yjs looks interesting; I'm currently building a management UI that for now is a very straightforward set of REST APIs and a database, but there's a lot of concerns about people editing the same configuration simultaneously; I wonder if something like yjs can be used to create a collaborative editing environment. I'd probably have to re-architect the whole front- and back-end though. Although on the other hand, there will be some code editors (currently using Monaco) where this could be usable.

er4hn 5y ago
My understanding of CRDT is that it shines the most when you have collaborative documents. For a mgmt UI along the lines of "listen on port X, log at level Y" where does CRDT become handy? Wouldn't you just want "last writer wins" for that sort of use case?

robto 5y ago

See, this is what I keep going on about to my friends - Matrix is actually a graph syncing protocol, not an instant messaging protocol, and is a great way to build federated applications. I get that Element needs to focus on one use case in order to pay the bills, but there's a lot of room for applications that have barely been explored - calendars, contacts, games, social media, heck - you could build a federated gitlab-like collaboration site with truly federated data. You get multi-device sync, e2e encryption, and identity management out of the box. Thanks for putting this together, I hope there will be a gold rush as people realize how much low-hanging fruit there is!

mxuribe 5y ago
I think the very beginning the matrix project folks have used language similar to matrix being data synching...etc...But, that, the first use case is chat...but won;t end there...etc. Obviously i'm paraphrasing heavily...but once i read that, i've used similar language when i introduce matrix to friends...basically, chat/instant messenger is the first but not last/not only use case/scenario for this very cool technology.
all2 5y ago
> calendars, contacts I'm interested in this just to keep my devices sync'd up. I know Samsung/Apple/MS etc all have syncing services, but I really really want to get all my data out of their ecosystems. Just a distributed calendar app would be pretty amazing. No more iCal subscription/export/whatevers, 'just' federate the calendar between devices, and arbitrarily share it with anyone else. I have no clue where to start with building an app on these abstractions. Does anyone have any pointers? Documentation to read? High level overviews? Etc.?
- robto 5y ago
  I'd start with reading the Matrix spec, you'll want to know how the syncing protocol works, and this will give you enough familiarity that you can start playing around with curl to actually see it in action. I'd highly recommend understanding how conflict resolution works, the "Analysis of the Matrix Event GraphReplicated Data Type"[0] lays it out pretty well. Matrix-CRDT posted here should abstract most of the nitty gritty stuff, but there are no shortcuts to designing with CRDTs so familiarizing yourself with those is important. Other than that, just start playing and asking questions! The #matrix:matrix.org room is full of friendly and helpful people and I'm sure you'd be able to get answers. [0]https://arxiv.org/pdf/2011.06488.pdf https://arxiv.org/pdf/2011.06488.pdf

panick21_ 5y ago

Having an Datomic like store backed by something like this. https://github.com/replikativ/datahike https://github.com/replikativ/datahike Is an Open Source variant of Datomic. Lambdaforge wants to eventually have this work with CRDTs. Using the Matrix ecosystem for this is quite interesting as it solves many problems for you already.

dunham 5y ago
I'd also love to see an offline-first version of the datomic model. I've thought about datomic + crdts in the past, and it seems like ":db/unique :db.unique/identity" properties would be an issue. Whether or not the transaction fails depends on the current content of the database. I don't know if there is a way around this (while still being a CRDT) or how necessary this feature is.

pixel_tracing 5y ago

Awesome project! Quick question I notice this uses WebRTC, why integrate / bake this in directly rather than abstract it out?

YousefED 5y ago
Good spot. The WebRTC part is currently experimental and might be superceded by a different solution. If that's not the case I'll probably abstract it out indeed!

outside1234 5y ago

Just to understand, is the concept here that you send all of the updates for a particular data item to a Matrix "room" and then converge on the current state of that data item by each client replaying that data?

YousefED 5y ago
Yep, thats pretty much correct. With the addition of sending occasional “snapshots” so clients don't need to read the entire room history initially (more about this in the bottom of the readme).

outside1234 5y ago

Is there any community where projects like this have a center of gravity? (eg. conferences, Discord, Matrix, ...) for people interested in getting involved in the community?

YousefED 5y ago
I just set up a Discord @ https://discord.gg/9GxSXCeQyN https://discord.gg/9GxSXCeQyN. Please join! It’s called TypeCell as that’s the (yet to be announced) project both Matrix-CRDT and SyncedStore are building blocks for. If you’re interested in tech like this and maybe also “future of programming” (those bret victor things ;) ), knowledge management software (notion, roam, obsidian) - i’m sure you’ll like it!
- DenseComet 5y ago
  Heh. I get why you're using Discord, but it's unfortunate that a project building on Matrix does not use it for chat.
  - YousefED 5y ago
    Fair point haha. I’ll also set up a matrix room and bridge it to Discord.
    - trenchgun 5y ago
      Excellent. Discord is not just proprietary, but it also has a messy UI and bad UX.
- brunoqc 5y ago
  oh! I dream of a CRDT note-taking app.

lgas 5y ago

This sounds like (at least a big step towards) distributed event sourcing which is something that I have been anticipating for a while now but haven't heard any news of.

klabb3 5y ago

> Personally, I'm convinced the technologies in this ecosystem (CRDTs, etc) are really powerful and can do more to decentralize software than many web3 technologies that currently receive much of the hype. 100% agree! Web3 is really only tackling monetary instruments, which is interesting but very narrow. The majority of apps people need don't require global consensus, and it's too slow and expensive anyway. Yet, there's so much interesting stuff that can be decentralized/federated. > Matrix takes care of a lot of stuff: Authentication, E2EE, federation, hosting, etc. - so I can focus on the client. This is great to hear! What else besides CRDTs do you think is needed in order for matrix to be a platform for general purpose apps like this?

YousefED 5y ago
Thanks! Personally, I’m looking forward to stronger / cleaner (TypeScript) sdks with for example easier auth and e2ee key management. I heard this is on their roadmap for this year, so that’s pretty exciting!