3 ms·
You can move CRDT data over whatever data layer you have on hand, and insert them into whatever data stores you want. Care for the special semantics of CRDT are
by jitl 5y ago
You can move CRDT data over whatever data layer you have on hand, and insert them into whatever data stores you want. Care for the special semantics of CRDT are only needed when merging CRDTs, storage and transportation is like any other encoded format, like moving a JPEG encoded image over HTTP, or storing a JPEG as a byte stream in a database. Neither the HTTP client or the database need to know how to turn the JPEG bytes into screen pixels.
Here’s a possible design for a CRDT editor system:
Browser client composes a CRDT locally in-memory, and saves the CRDT data into IndexedDB periodically as a Blob of bytes.
When the client is online, it periodically performs a sync with the server using HTTP, with this protocol:
1. HTTP GET /doc/:id/stateVector - returns the latest state vector on the central server. A little analogous for checking the refs of a git remote.
2. HTTP POST /doc/:id/push - send the difference between the state vector returned from (1) and the client’s local state to the server. This is kinda like `git push`.
3. The server responds to (2) with the missing state on the client. It can compute this from the data the client sent.
Inside the server, writes are handled like this:
1. Start a RW transaction on a Postgres row containing the CRDT data.
2. Read the data from the DB and merge it with the incoming data in the client post.
3. Compute the update delta for the client
4. Write the merged data back into the DB
5. Commit.
6. Respond with the delta computed in (3).
This scheme is not optimally efficient, but shows that you can put together a CRDT system with a normal database and a bit of HTTP.
See https://docs.yjs.dev/api/document-updates#syncing-clients https://docs.yjs.dev/api/document-updates#syncing-clients for more info on how Yjs (the most production ready library) handles these concepts.
- josephg 5y agoYep. I've done something very similar on top of Diamond Types for a little personal wiki. This page[1] is synced between all users who have the page open. Its a remarkably small piece of code, outside of the CRDT library itself (which is in rust via wasm). The way it works is: - On page load, the server sends the whole CRDT document to the browser, and the server streams changes from that point onwards. - When a change happens in the browser, it makes that change locally then and sends anything the server doesn't know about upstream. - Whenever the server finds out about a new change, it re-broadcasts that change to any subscribed browser streams. I'm using the braid HTTP protocol for changes - but we could easily switch to a SSE or websocket solution. It doesn't really matter. At the moment I'm just using flat files for storage, but there's nothing stopping you using a database instead, except that its a bit awkward to use efficient CRDT packing techniques in a database. [1] https://wiki.seph.codes/hn https://wiki.seph.codes/hn Code is here, if anyone is interested. The whole thing is a few hundred lines all up: https://github.com/josephg/diamond-types/tree/0cb5d7ecf49364a0a8c0e52c5b325d5dafcb883e/wiki https://github.com/josephg/diamond-types/tree/0cb5d7ecf49364...