3 ms·

I’ve built a successful business from a TTRPG campaign manager (LegendKeeper) using CRDTs, specifically the Yjs kind. It’s been great, and the UX of CRDT-powere

I’ve built a successful business from a TTRPG campaign manager (LegendKeeper) using CRDTs, specifically the Yjs kind. It’s been great, and the UX of CRDT-powered stuff is excellent. Zero latency, eventually consistent; overall users love the performance and offline capability. That said, there are a lot of trade offs. Some things that are easy in a traditional server-client model become difficult in the local-first context that CRDTs provide. Role based authorization is hard, data model changes must be done additive (never mutative), and knowing what state a client is in when debugging is tough too, without a lot of “full-surveillance”-level tooling. Also with the automatic , bidirectional syncing a lot of CRDT architectures afford you, a bug in production that corrupts data can virally propagate and cause a huge headache. Investor-funded services like Liveblocks are starting to pop up that promise to make this stuff easier, but as an indie I find them expensive; I’m sure they’re a great value for big corps or funded teams though. Rolling my own infrastructure for Yjs has been taxing, but I’ve learned a lot, and have been able to tailor it exactly to my needs.

paulgb 3y ago
> knowing for sure what state a user’s client is in when debugging is tough too My team has built an open-source debugger for Yjs that might interest you (docs: https://y-sweet.cloud/advanced/debugger https://y-sweet.cloud/advanced/debugger) You mention the investor-funded services that pop up to make this stuff easier -- our goal with Y-Sweet is to build the same type of DX you’d get from those services, but build it on a fully open-source (MIT) platform with Yjs at the core: https://github.com/drifting-in-space/y-sweet https://github.com/drifting-in-space/y-sweet
- braden-lk 3y ago
  Ooh hell yeah; excited to check this out.
- janesconference 3y ago
  A couple of questions about y-sweet, based on the experiences I had with CRDTs: 1) Does the server keeps in memory the "active" documents? In other words, does the server need to open a document and keep it in RAM while clients are connected to it (I assume there's a websocket connection somewhere in the client that keeps it hot)? Or is the server stateless - just connects to the store when needed? I found the latter very hard to do. 2) Does the client persist entries using indexeddb? If yes, does opening many tabs cause redundant writes as they all sync with the server? If not, does the client need to fully re-sync with the server anytime it wakes up? 3) Is it possible to observe updates on the client as they come? One of the major use cases of CRDTs is to index data on the client - then you can have a dumb server that just syncs data between clients and a smart client that does search, graphs, visualizations etc. on the data it receives. To do that, the client needs to observe updates one by one and process them to create secondary indexes. Is it possible to do with y-sweet without forking its source code? I remember getting updates yn Y.js being quite inefficient as you need to replay them all or something similar, but that was a couple of years ago.
  - paulgb 3y ago
    Good questions! > 1 The server keeps the documents in memory when they are open, but it is horizontally scalable by hosting using CloudFlare Durable Objects. (We also plan to support Plane.dev but that's not built out yet). > 2 The client is based on Yjs, so it's compatible with Yjs' y-indexeddb provider to store in IndexedDB. Tabs synchronize state between each other using a local broadcast channel. The client only synchronizes unsynced state with the server, so if one tab has already pushed the local offline edits to the server, the other tabs can discover that and avoid pushing them. That said, I'm not 100% sure if Yjs deals with the race condition where two tabs wake up at the same time so the server has not yet received offline edits from either, I'd have to check on that. >3 Yes, Yjs types have an `observe` method that takes a callback, which receives an event with details of each edit. Here’s an example for observing events of a Y.Map: https://docs.yjs.dev/api/shared-types/y.map#observing-changes-y.mapevent https://docs.yjs.dev/api/shared-types/y.map#observing-change...
    - janesconference 3y ago
      Thanks! Re: 1, the docs are not really deep on that, but from your answer it seems it possible to self-host y-sweet on Cloudflare workers (I guess) with Durable Objects as storage? Also, if you guys are going to have a paid plan, how do you see the prices going? Comparable to, say, Supabase per user, less or more?
      - paulgb 3y ago
        Sorry about that, I’ll clarify the docs. You can self-host on Cloudflare, but the storage is R2/S3/S3-compatible blob storage. Our tentative pricing is $25/month + $10/10k minutes of “open connection” time (per-document, not per-connection, so multiple users with the same doc open are not double-counted). Storage is free if you bring your own R2/S3 bucket, or a nominal fee if you use ours. Unlike supabase we don’t do any of the relational stuff, but for Figma-like apps where a lot of documents are never touched, I think our hot/cold storage model can be significantly cheaper at scale than a hosted postgres database like supabase.
janesconference 3y ago
How does it scale? What happens if you have many users or a bunch of users with huge history on their data? If you haven't hit these limitations, what do you plan to do when (if) you hit them?
- braden-lk 3y ago
  I don't know how well the original y-websocket provider scales, as it holds ydocs in memory. I imagine lots of folks are using it just fine in production, though. I wrote a less-stateful version of y-websocket that uses the Yjs Differential Updates API to save and serve updates without loading the docs into memory. As long as you have garbage collection turned on for your Ydocs, they stay pretty small, especially if you are avoiding using YMaps. (The strings that serve as YMap keys can't be GC'd, from what I understand. YMaps are great for bounded domain objects, but not so great for storing collections, dictionary-style. Y-KeyValue solves this problem.) I eventually added a X MB document size limit on the backend, but only after doing a statistical analysis on existing documents. I found a size threshold that was a strong indicator of abnormal/buggy behavior, and set a limit under that. Without the limit, occasionally I had huge Ydocs, usually created by a bug or weird user behavior, clogging up database resources. Now I block those ydocs on the backend and send a messsage to the user with some mitigation/recovery tips. I plan to add automatic document repair, but just haven't gotten to it yet. As LK matures and I get better with Yjs, these bugs become much rarer.
Karrot_Kream 3y ago
Is LegendKeeper built on top of the Websocket Yjs provider? If so, do you run the Websocket server yourself? If not, do you use WebRTC and have you had any STUN/TURN issues with that? LegendKeeper looks really awesome btw, I might bring this up for my own campaign use. I've been thinking of using Yjs to build some character sheet builders myself which is why I'm asking.
- braden-lk 3y ago
  I built a custom solution based on y-websocket that handles multiplexing and syncing multitudes of Ydocs at once; a single YDoc is not really enough for complex apps. I run the server myself; originally on GKE because I was learning it for work, but then once I went FT on LegendKeeper, k8s became super-overkill minus the learning context. Finally switching to Render after tiring of fighting weird K8s internal DNS issues.
doctorpangloss 3y ago
> data model changes must be done additive (never mutative) Sounds painful. It means your mutative data model changes, which exist, live somewhere else.
- braden-lk 3y ago
  Yep, though coming from mobile development this was somewhat familiar. A lot of mobile apps are local-first, but I don't think it was called that back when I was doing mobile. Most mobile platforms expect this and provide tools to ease migrations, like Room on Android. Since CRDT approaches are still fresh, I imagine most people are rolling their own adhoc migration strategies. Ink & Switch is working on this: https://www.inkandswitch.com/cambria/ https://www.inkandswitch.com/cambria/

paulgb 3y ago

> knowing for sure what state a user’s client is in when debugging is tough too My team has built an open-source debugger for Yjs that might interest you (docs: https://y-sweet.cloud/advanced/debugger https://y-sweet.cloud/advanced/debugger) You mention the investor-funded services that pop up to make this stuff easier -- our goal with Y-Sweet is to build the same type of DX you’d get from those services, but build it on a fully open-source (MIT) platform with Yjs at the core: https://github.com/drifting-in-space/y-sweet https://github.com/drifting-in-space/y-sweet

braden-lk 3y ago
Ooh hell yeah; excited to check this out.
janesconference 3y ago
A couple of questions about y-sweet, based on the experiences I had with CRDTs: 1) Does the server keeps in memory the "active" documents? In other words, does the server need to open a document and keep it in RAM while clients are connected to it (I assume there's a websocket connection somewhere in the client that keeps it hot)? Or is the server stateless - just connects to the store when needed? I found the latter very hard to do. 2) Does the client persist entries using indexeddb? If yes, does opening many tabs cause redundant writes as they all sync with the server? If not, does the client need to fully re-sync with the server anytime it wakes up? 3) Is it possible to observe updates on the client as they come? One of the major use cases of CRDTs is to index data on the client - then you can have a dumb server that just syncs data between clients and a smart client that does search, graphs, visualizations etc. on the data it receives. To do that, the client needs to observe updates one by one and process them to create secondary indexes. Is it possible to do with y-sweet without forking its source code? I remember getting updates yn Y.js being quite inefficient as you need to replay them all or something similar, but that was a couple of years ago.
- paulgb 3y ago
  Good questions! > 1 The server keeps the documents in memory when they are open, but it is horizontally scalable by hosting using CloudFlare Durable Objects. (We also plan to support Plane.dev but that's not built out yet). > 2 The client is based on Yjs, so it's compatible with Yjs' y-indexeddb provider to store in IndexedDB. Tabs synchronize state between each other using a local broadcast channel. The client only synchronizes unsynced state with the server, so if one tab has already pushed the local offline edits to the server, the other tabs can discover that and avoid pushing them. That said, I'm not 100% sure if Yjs deals with the race condition where two tabs wake up at the same time so the server has not yet received offline edits from either, I'd have to check on that. >3 Yes, Yjs types have an `observe` method that takes a callback, which receives an event with details of each edit. Here’s an example for observing events of a Y.Map: https://docs.yjs.dev/api/shared-types/y.map#observing-changes-y.mapevent https://docs.yjs.dev/api/shared-types/y.map#observing-change...
  - janesconference 3y ago
    Thanks! Re: 1, the docs are not really deep on that, but from your answer it seems it possible to self-host y-sweet on Cloudflare workers (I guess) with Durable Objects as storage? Also, if you guys are going to have a paid plan, how do you see the prices going? Comparable to, say, Supabase per user, less or more?
    - paulgb 3y ago
      Sorry about that, I’ll clarify the docs. You can self-host on Cloudflare, but the storage is R2/S3/S3-compatible blob storage. Our tentative pricing is $25/month + $10/10k minutes of “open connection” time (per-document, not per-connection, so multiple users with the same doc open are not double-counted). Storage is free if you bring your own R2/S3 bucket, or a nominal fee if you use ours. Unlike supabase we don’t do any of the relational stuff, but for Figma-like apps where a lot of documents are never touched, I think our hot/cold storage model can be significantly cheaper at scale than a hosted postgres database like supabase.

janesconference 3y ago

How does it scale? What happens if you have many users or a bunch of users with huge history on their data? If you haven't hit these limitations, what do you plan to do when (if) you hit them?

braden-lk 3y ago
I don't know how well the original y-websocket provider scales, as it holds ydocs in memory. I imagine lots of folks are using it just fine in production, though. I wrote a less-stateful version of y-websocket that uses the Yjs Differential Updates API to save and serve updates without loading the docs into memory. As long as you have garbage collection turned on for your Ydocs, they stay pretty small, especially if you are avoiding using YMaps. (The strings that serve as YMap keys can't be GC'd, from what I understand. YMaps are great for bounded domain objects, but not so great for storing collections, dictionary-style. Y-KeyValue solves this problem.) I eventually added a X MB document size limit on the backend, but only after doing a statistical analysis on existing documents. I found a size threshold that was a strong indicator of abnormal/buggy behavior, and set a limit under that. Without the limit, occasionally I had huge Ydocs, usually created by a bug or weird user behavior, clogging up database resources. Now I block those ydocs on the backend and send a messsage to the user with some mitigation/recovery tips. I plan to add automatic document repair, but just haven't gotten to it yet. As LK matures and I get better with Yjs, these bugs become much rarer.

Karrot_Kream 3y ago

Is LegendKeeper built on top of the Websocket Yjs provider? If so, do you run the Websocket server yourself? If not, do you use WebRTC and have you had any STUN/TURN issues with that? LegendKeeper looks really awesome btw, I might bring this up for my own campaign use. I've been thinking of using Yjs to build some character sheet builders myself which is why I'm asking.

braden-lk 3y ago
I built a custom solution based on y-websocket that handles multiplexing and syncing multitudes of Ydocs at once; a single YDoc is not really enough for complex apps. I run the server myself; originally on GKE because I was learning it for work, but then once I went FT on LegendKeeper, k8s became super-overkill minus the learning context. Finally switching to Render after tiring of fighting weird K8s internal DNS issues.

doctorpangloss 3y ago

> data model changes must be done additive (never mutative) Sounds painful. It means your mutative data model changes, which exist, live somewhere else.

braden-lk 3y ago
Yep, though coming from mobile development this was somewhat familiar. A lot of mobile apps are local-first, but I don't think it was called that back when I was doing mobile. Most mobile platforms expect this and provide tools to ease migrations, like Room on Android. Since CRDT approaches are still fresh, I imagine most people are rolling their own adhoc migration strategies. Ink & Switch is working on this: https://www.inkandswitch.com/cambria/ https://www.inkandswitch.com/cambria/