8 ms·

I use CRDTs in production at Jackbox for audience functionality and honestly I don't know why the only thing that people talk about when it comes to CRDTs is co

by zemo 6y ago

I use CRDTs in production at Jackbox for audience functionality and honestly I don't know why the only thing that people talk about when it comes to CRDTs is collaborative text editing. Like, sure, cool, but that's literally a single problem domain and like, Google Docs already exists and works well for the majority of users so how many developers actually need to create a collaborative text editor? CRDTs are an incredibly abstract and versatile concept; collaborative text editing is a tiny problem domain. I would really like to see more writing about CRDTs that is NOT about collaborative text editing.

rhencke 6y ago
Would you consider being the writer of one of those articles?
- zemo 6y ago
  well I really walked into that question huh
  - gen220 6y ago
    I'd read this article. Or, if you'd rather not, can you share some examples of problem domains, that we at home can search for?
    - zemo 6y ago
      a lot of other use cases are interested in using CRDTs for complex mutations with a relatively small number of users. Most collaborative editing sessions will be a small handful of users. Our use cases are things more like "three thousand people are presented with a multiple-choice question simultaneously and answer it simultaneously and we have to show the results on an xbox and the entire thing has to happen in under 15 seconds". We mostly just use the standard g-counter and pn-counter types. We have a thing that's like ... a high score list for a trivia game that doesn't have a well known type (it's kinda like ... having a zset where only the top N items in the zset are replicated) and a CRDT implementation of a ring buffer of strings for audience comments in ... Quiplash 2, iirc. I'm a bit tied up since we have a new pack coming out October 15th, but I might have time to write some stuff up after that.
  - davidhowlett 6y ago
    If you write an article and post it here I will read it.
  - djm_ 6y ago
    Followed you in case you ever do! I'm a big fan of Jackbox and I'd love to read about where CRDTs fit in your stack.
- joekim 6y ago
  +1 Nicely done.
wenc 6y ago
Here’s one written by an employee of Redis Labs about CRDTs for georeplication of databases for strong eventual consistency. [1] There's also 2-year old list of CRDTs used in prod [2]. I know Azure CosmosDB (NoSQL) advertises it's own use of CRDTs for global replication. [1] https://www.infoworld.com/article/3305321/when-to-use-a-crdt-based-database.html https://www.infoworld.com/article/3305321/when-to-use-a-crdt... [2] https://github.com/ipfs/notes/issues/404 https://github.com/ipfs/notes/issues/404
sagichmal 6y ago
1000% this.
PaulDavisThe1st 6y ago
We want something like that for distributed collaboration in Ardour, a cross-platform DAW. The relevant state is serialized as an XML file, and used natively as a complex object tree. Users want to be able to edit (in the DAW) locally, then share (and merge) their results with collaborators.
- jka 6y ago
  I've plugged this collaboration project a few times recently, and have no relationship to it other than discovering it (via YJS' "who is using" list[1]) and finding it fascinating: http://cattaz.io/ http://cattaz.io/ What I find most interesting about it is that it has reduced the state of multiple 'smart' user-facing widgets/apps into a common, lowest-common-denominator format (a text document) that lends itself more easily and intuitively to collaborative editing and CRDT operations. I don't know for sure whether this is the path forward for CRDT-based applications in general, but I think there are valuable ideas there. It does raise the possibility of the widgets/applications occasionally being in 'invalid' states; but rarely in a way that the human participants wouldn't notice or be able to fix themselves. Whether that scales to the complexity of the state management for a multi-track audio editing session, I don't know; but it could be instructional to compare. [1] - https://github.com/yjs/yjs#who-is-using-yjs https://github.com/yjs/yjs#who-is-using-yjs
- zemo 6y ago
  in real-time? Well, I have thoughts, but I'm not super familiar with Ardour itself, so I'm not sure if you're trying to merge during a live performance or if you're talking more of a distributed studio recording session type situation. I have working knowledge of Reason and Logic and ChucK (which I use with JACK and do some networked OSC stuff with, although I haven't touched it in a few years). The approach we use at Jackbox for making the state of an xbox game mutable to thousands of live viewers on twitch is to have lots of little CRDT values, mostly just counters and sets of strings, and you merge the little values independently of one another, which is very different from the situation of editing a text document, which is typically structured as one big value. I wonder if, for a DAW, you could merge at the track or control level instead of the workspace level. E.g., communicate as an independent value the state of an individual fader, and communicate either states or operations on that fader and have each client merge them. In this example, the fader's state would be encoded as a PN-counter with a range condition, and you'd replicate increment and decrement options, like it was a networked rotary encoder. So every mutable thing in the DAW would be a value having operations that can be merged, instead of having a single big value representing the entire state of the DAW. My use-case is also funky because I have potentially thousands of writers writing to the same key, but only a single reader, and the reader doesn't need an update at every change, so I use state-based CRDTs, but I think most other people using CRDTs use operation-based CRDTs. Also not sure how you would mutate two separate values transactionally or if that's a thing you even need.
uryga 6y ago
just wanted to say, i really like Jackbox's games and its really fun to see you here! i've been exposed to your games via youtube vids and it feels weird (but cool!) to have that intersect with HN. would love to read more about how you use CRDTs to make it all work! (i'm pretty clueless re: multiplayer games so even a high-level explanation would be interesting)
rad_gruchalski 6y ago
Why not writing an article about how you use them?
josephg 6y ago
Author of the blog post here. I’ll let you in on a dirty secret: I agree with you. I see text editing as the beachhead for this tech. Text editing is hard enough that other systems don’t work well, so you’re kind of forced to use OT or CRDTs to make it work. And text documents are lists of characters - so once you’ve made it work there you have an implementation that can also sync arbitrary lists. At that point, editing maps (js objects) and adding support for arbitrary moves and reparenting will allow us to add real-time editing to a lot more software. I think there’s a lot of interesting software architectures that can be enabled by making everything in a system a CRDT document. But text is still the beachhead. And a good litmus test for performance. And an important application in its own right.
- zemo 6y ago
  on the one hand, the generality of the text editing solutions is really powerful, and I see what you mean in terms of that solution generalizing to other domains. But on the other hand, I always think back to how popular Memcache or Redis were even early on when they had very very few features. Just having a fast, in memory cache of strings empowered a lot of interesting new product development. I really wonder how much the average developer on a random project could get out of an appliance that lets you create and mutate an arbitrarily large number of values of the well-known, simple CRDT types like g-counters, pn-counters, 2P-sets, etc. Most of the literature is focused on "how do we merge the most complex data type", and not questions like "how do we manage the entire lifecycle of CRDT stores", "how does a CRDT store fit into most people's stack", "should CRDT types be added to existing stores or should developers expect dedicated CRDT-only stores", or "do people generally need to define their own CRDT types or do most people just want a box full of common ones". I hand-rolled my own CRDT setup just to get the most simple CRDT types because I didn't see anything out there that makes directly consuming CRDT types by application developers accessible. E.g., you make a g-counter and literally all a client can do with it is increment it or read its value. That's it. We have that, and it's totally useful! We also do entirely state-based replication because expressing the operations on the data would be so much larger than the data itself. But our situation is just so off-base for many people because clients are only ever interested in the current state (and never care about the past state), and we can safely just ignore the problem of when to delete the data; we just keep it around until you're finished playing the game, and then delete it when the game is over.
spockz 6y ago
We use CRDTs for having a scalable, availability first, service discovery implementation. I’m sure there are more uses out there. We use Akka Distributed Data and there are many users of that.
an_opabinia 6y ago
I love the Jackbox games. However in my experience most games's shared world state cannot be reconciled if one player's action is inserted in time prior to another player action. Or future actions reveal hidden information. Even turn based games have turn timers. Including Jackbox games! Among public databases there may only be two correct CRDT implementations in the world - Cassandra's and etcd's. Compare to how many database products make CRDT or CRDT-like promises, and how few probably actually work. Lots of software promises CRDTs-like-over-actors, e.g. Erlang, the new Cloudflare product. Not really qualified to talk about them. Most applications normal people use boil down to chat conversations - chat apps, Gmail, Facebook, collaborative text like you said, all sorts of log-like things - or serial, recommendation-driven consumption - YouTube, Netflix, which use incremental / batched cells in a matrix to update their recs. This stuff is insensitive to order and time. Then there's e-commerce, which is inherently linear, centralized, single-actor, all-or-nothing transactions, CRDTs provide no advantages here. It's tricky. On the one hand you'd get this interesting discussion of CRDT applications. On the other it may wither under intense scrutiny.
nwsm 6y ago
Thanks for your work on Jackbox! Never had an issue in the games and they are perfect for parties. I always thought it would be fun to make a Jackbox-type game system.
nunez 6y ago
Holy shit CRDTs totally make sense for Jackbox. (Btw thank you for the awesome games!)

by zemo 6y ago

rhencke 6y ago

Would you consider being the writer of one of those articles?

zemo 6y ago
well I really walked into that question huh
- gen220 6y ago
  I'd read this article. Or, if you'd rather not, can you share some examples of problem domains, that we at home can search for?
  - zemo 6y ago
    a lot of other use cases are interested in using CRDTs for complex mutations with a relatively small number of users. Most collaborative editing sessions will be a small handful of users. Our use cases are things more like "three thousand people are presented with a multiple-choice question simultaneously and answer it simultaneously and we have to show the results on an xbox and the entire thing has to happen in under 15 seconds". We mostly just use the standard g-counter and pn-counter types. We have a thing that's like ... a high score list for a trivia game that doesn't have a well known type (it's kinda like ... having a zset where only the top N items in the zset are replicated) and a CRDT implementation of a ring buffer of strings for audience comments in ... Quiplash 2, iirc. I'm a bit tied up since we have a new pack coming out October 15th, but I might have time to write some stuff up after that.
- davidhowlett 6y ago
  If you write an article and post it here I will read it.
- djm_ 6y ago
  Followed you in case you ever do! I'm a big fan of Jackbox and I'd love to read about where CRDTs fit in your stack.
joekim 6y ago
+1 Nicely done.

wenc 6y ago

Here’s one written by an employee of Redis Labs about CRDTs for georeplication of databases for strong eventual consistency. [1] There's also 2-year old list of CRDTs used in prod [2]. I know Azure CosmosDB (NoSQL) advertises it's own use of CRDTs for global replication. [1] https://www.infoworld.com/article/3305321/when-to-use-a-crdt-based-database.html https://www.infoworld.com/article/3305321/when-to-use-a-crdt... [2] https://github.com/ipfs/notes/issues/404 https://github.com/ipfs/notes/issues/404

sagichmal 6y ago

1000% this.

PaulDavisThe1st 6y ago

We want something like that for distributed collaboration in Ardour, a cross-platform DAW. The relevant state is serialized as an XML file, and used natively as a complex object tree. Users want to be able to edit (in the DAW) locally, then share (and merge) their results with collaborators.

jka 6y ago
I've plugged this collaboration project a few times recently, and have no relationship to it other than discovering it (via YJS' "who is using" list[1]) and finding it fascinating: http://cattaz.io/ http://cattaz.io/ What I find most interesting about it is that it has reduced the state of multiple 'smart' user-facing widgets/apps into a common, lowest-common-denominator format (a text document) that lends itself more easily and intuitively to collaborative editing and CRDT operations. I don't know for sure whether this is the path forward for CRDT-based applications in general, but I think there are valuable ideas there. It does raise the possibility of the widgets/applications occasionally being in 'invalid' states; but rarely in a way that the human participants wouldn't notice or be able to fix themselves. Whether that scales to the complexity of the state management for a multi-track audio editing session, I don't know; but it could be instructional to compare. [1] - https://github.com/yjs/yjs#who-is-using-yjs https://github.com/yjs/yjs#who-is-using-yjs
zemo 6y ago
in real-time? Well, I have thoughts, but I'm not super familiar with Ardour itself, so I'm not sure if you're trying to merge during a live performance or if you're talking more of a distributed studio recording session type situation. I have working knowledge of Reason and Logic and ChucK (which I use with JACK and do some networked OSC stuff with, although I haven't touched it in a few years). The approach we use at Jackbox for making the state of an xbox game mutable to thousands of live viewers on twitch is to have lots of little CRDT values, mostly just counters and sets of strings, and you merge the little values independently of one another, which is very different from the situation of editing a text document, which is typically structured as one big value. I wonder if, for a DAW, you could merge at the track or control level instead of the workspace level. E.g., communicate as an independent value the state of an individual fader, and communicate either states or operations on that fader and have each client merge them. In this example, the fader's state would be encoded as a PN-counter with a range condition, and you'd replicate increment and decrement options, like it was a networked rotary encoder. So every mutable thing in the DAW would be a value having operations that can be merged, instead of having a single big value representing the entire state of the DAW. My use-case is also funky because I have potentially thousands of writers writing to the same key, but only a single reader, and the reader doesn't need an update at every change, so I use state-based CRDTs, but I think most other people using CRDTs use operation-based CRDTs. Also not sure how you would mutate two separate values transactionally or if that's a thing you even need.

uryga 6y ago

just wanted to say, i really like Jackbox's games and its really fun to see you here! i've been exposed to your games via youtube vids and it feels weird (but cool!) to have that intersect with HN. would love to read more about how you use CRDTs to make it all work! (i'm pretty clueless re: multiplayer games so even a high-level explanation would be interesting)

rad_gruchalski 6y ago

Why not writing an article about how you use them?

josephg 6y ago

Author of the blog post here. I’ll let you in on a dirty secret: I agree with you. I see text editing as the beachhead for this tech. Text editing is hard enough that other systems don’t work well, so you’re kind of forced to use OT or CRDTs to make it work. And text documents are lists of characters - so once you’ve made it work there you have an implementation that can also sync arbitrary lists. At that point, editing maps (js objects) and adding support for arbitrary moves and reparenting will allow us to add real-time editing to a lot more software. I think there’s a lot of interesting software architectures that can be enabled by making everything in a system a CRDT document. But text is still the beachhead. And a good litmus test for performance. And an important application in its own right.

zemo 6y ago
on the one hand, the generality of the text editing solutions is really powerful, and I see what you mean in terms of that solution generalizing to other domains. But on the other hand, I always think back to how popular Memcache or Redis were even early on when they had very very few features. Just having a fast, in memory cache of strings empowered a lot of interesting new product development. I really wonder how much the average developer on a random project could get out of an appliance that lets you create and mutate an arbitrarily large number of values of the well-known, simple CRDT types like g-counters, pn-counters, 2P-sets, etc. Most of the literature is focused on "how do we merge the most complex data type", and not questions like "how do we manage the entire lifecycle of CRDT stores", "how does a CRDT store fit into most people's stack", "should CRDT types be added to existing stores or should developers expect dedicated CRDT-only stores", or "do people generally need to define their own CRDT types or do most people just want a box full of common ones". I hand-rolled my own CRDT setup just to get the most simple CRDT types because I didn't see anything out there that makes directly consuming CRDT types by application developers accessible. E.g., you make a g-counter and literally all a client can do with it is increment it or read its value. That's it. We have that, and it's totally useful! We also do entirely state-based replication because expressing the operations on the data would be so much larger than the data itself. But our situation is just so off-base for many people because clients are only ever interested in the current state (and never care about the past state), and we can safely just ignore the problem of when to delete the data; we just keep it around until you're finished playing the game, and then delete it when the game is over.

spockz 6y ago

We use CRDTs for having a scalable, availability first, service discovery implementation. I’m sure there are more uses out there. We use Akka Distributed Data and there are many users of that.

an_opabinia 6y ago

I love the Jackbox games. However in my experience most games's shared world state cannot be reconciled if one player's action is inserted in time prior to another player action. Or future actions reveal hidden information. Even turn based games have turn timers. Including Jackbox games! Among public databases there may only be two correct CRDT implementations in the world - Cassandra's and etcd's. Compare to how many database products make CRDT or CRDT-like promises, and how few probably actually work. Lots of software promises CRDTs-like-over-actors, e.g. Erlang, the new Cloudflare product. Not really qualified to talk about them. Most applications normal people use boil down to chat conversations - chat apps, Gmail, Facebook, collaborative text like you said, all sorts of log-like things - or serial, recommendation-driven consumption - YouTube, Netflix, which use incremental / batched cells in a matrix to update their recs. This stuff is insensitive to order and time. Then there's e-commerce, which is inherently linear, centralized, single-actor, all-or-nothing transactions, CRDTs provide no advantages here. It's tricky. On the one hand you'd get this interesting discussion of CRDT applications. On the other it may wither under intense scrutiny.

nwsm 6y ago

Thanks for your work on Jackbox! Never had an issue in the games and they are perfect for parties. I always thought it would be fun to make a Jackbox-type game system.

nunez 6y ago

Holy shit CRDTs totally make sense for Jackbox. (Btw thank you for the awesome games!)