6 ms·

CRDT resources

119 points by stichers 5y ago

nuerow 5y ago
The content is interesting but the page is totally unusable in mobile.
- nikivi 5y ago
  Author of the wiki here, I think GitBook is quite usable on mobile for me. What is the issue?
  - mkl 5y ago
    (Not OP) For me on Android Firefox the topic menu takes up the whole screen, doesn't scroll, and doesn't hide. On Chrome it seems to work.
    - teodorlu 5y ago
      Happens to me too =/ Firefox on Android.
  - stichers 5y ago
    OP here — works for me on mobile Safari
    - skeeter2020 5y ago
      2021 and developers are still responding to support issues with "works on my machine!"
      - deleted 5y ago
        [deleted]
      - stichers 5y ago
        Ha ha! I'm not a dev any more, but old habits die hard, I guess. My comment was, in fact, to confirm for anyone who wanted to access with mobile Safari but who hadn't yet, that it indeed would work fine.
conaclos 5y ago
Interested readers may also find interest in https://crdt.tech https://crdt.tech
globular-toast 5y ago
At least just write out the name in full before using the acronym. "CRDT" could mean literally anything and it doesn't help when half the links in the list also don't bother to name it before using the acronym. Resist using acronyms before defining them. Don't just do it because everyone else does it. You are only creating barriers for people who might be interested in what you are talking about. I know people do this intentionally too. Don't be so insecure. You don't need to invent special language to remain relevant. Another user commented on a similar thing earlier today: https://news.ycombinator.com/item?id=28997945 https://news.ycombinator.com/item?id=28997945
- zffr 5y ago
  Since no one else defined it, CRDT = Conflict Free Replicated Data Types. There are 2 variants which confusingly have a very similar acronym: - CmRDT: Commutative Replicated Data Types (also known as operation-based CRDTs). These CRDTs replicate state by transmitting update operations to peers. Peers are always able to apply these operations in any order and without conflicts. - CvRDT: Convergent Replicated Data Types (also known as state-based CRDTs). These CRDTs replicate state by transmitting the entire object every time an update is made to a local replica. Peers are able to merge the state they receive with their local copy without conflicts.
  - dboreham 5y ago
    It's ok that they have confusingly similar acronyms, because they're basically the same anyway.
tomsonj 5y ago
I've observed that implementing a CRDT system requires blogging about it afterward
- hecturchi 5y ago
  Hehe, or even writing a paper! https://arxiv.org/abs/2004.00107# https://arxiv.org/abs/2004.00107# The link takes you to our Merkle-CRDTs paper, which includes a nice intro to CRDTs in general, so no prior knowledge needed!
- nosianu 5y ago
  What method did you use to determine which/if anyone implementing CRDTs did not write a blog post? Or did you observe that 100% of the people who blogged about CRDTs implemented CRDTs and then wrote a blog post?
- sunny--tech 5y ago
  I have implemented a CRDT system and have also blogged about it afterwards. So I can confirm this. Joking aside though, CRDTs are still a pretty esoteric space and all the various blog posts came in handy when I was researching how to build my own CRDT. Most of the resources I've seen on CRDTs are from whitepapers and those can be difficult to read if you don't have a math background. I gave a talk at a Papers We Love [1] explicitly because I found the academic papers a big turn off from many people interested in the space. [1]: https://www.youtube.com/watch?v=1Bs3Fj9rvks&t=2169s&ab_channel=PapersWeLove https://www.youtube.com/watch?v=1Bs3Fj9rvks&t=2169s&ab_chann...
- dang 5y ago
  "Don't be snarky." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
keewee7 5y ago
What are some interesting use cases for CRDTs beyond collaborative (text) editing?
- lijogdfljk 5y ago
  I mean.. data, right? I'm trying to learn about CRDTs for complex, nested data structures. The use cases are for.. well, nested data, anything you'd use BTree's and etc for over distributed systems. A big thing i'm currently learning with them is to write a content addressable system with a more forgiving merge policy between parties. Yea i often see people nitpick CRDT about user intention, and where it should be `ADBC` or `ABCD`, but in my case i'm focusing on multi-device, not multi-user - and even in multi-user it's still best-in-breed when you are designing away from centralization. I'm still struggling to learn the more complex approaches to CRDT. So many resources focus on the low hanging fruit of CRDT. Grow counters, basic text editing, etc. I need to build the full suite of data structures; maps, sets, lists, etc.
- sunny--tech 5y ago
  Quite a few data stores use CRDTs under the hood. Redis uses CRDTs for active-active architectures [1] and for some of their native data structures [2]. Riak also uses them in their data store [3]. And looks like PayPal might use them for consensus purposes (I found this while looking up the Riak talk so I haven't actually watched it) [4] 1: https://redis.com/blog/diving-into-crdts/ https://redis.com/blog/diving-into-crdts/ 2: https://redis.com/videos/active-active-geo-distribution-redis-enterprise/ https://redis.com/videos/active-active-geo-distribution-redi... 3: https://www.youtube.com/watch?v=f20882ZSdkU&ab_channel=ErlangSolutions https://www.youtube.com/watch?v=f20882ZSdkU&ab_channel=Erlan... 4: https://www.infoq.com/presentations/crdt-production/ https://www.infoq.com/presentations/crdt-production/
- deleted 5y ago
  [deleted]
mox111 5y ago
Most of the CRDT examples I've seen appear to be Electron apps e.g. https://github.com/automerge/pushpin https://github.com/automerge/pushpin. My understanding is that CRDT's rely on having a safe place to store data on the user's machine (otherwise it's a bit like doing a `git clone` to receive new data, rather than a `git pull`). Is this not a major limitation for people hoping to use it for web apps?
- sunny--tech 5y ago
  I know that YJS is a common library that people use for using CRDTS in the browser and web-apps. https://github.com/yjs/yjs https://github.com/yjs/yjs
- nisa 5y ago
  The Distributed Mutable Containers spec looks super exciting: https://dream.public.cat/ https://dream.public.cat/ https://inqlab.net/projects/dmc/ https://inqlab.net/projects/dmc/ - https://inqlab.net/projects/eris/ https://inqlab.net/projects/eris/ - it's in https://openengiadina.net/ https://openengiadina.net/ - https://gitlab.com/openengiadina https://gitlab.com/openengiadina
  - mro_name 5y ago
    Thanks for sharing, nice to see djb (netstrings in c-s-exps) shine through from far below. I wonder about the energy consumption aspect of content adressing. Redundancy eats resources.
cyber_kinetist 5y ago
> “Distributed state is so fundamentally complex that I think we actually need CRDTs (or something like them) to reason about it effectively.” Gamedevs working on multiplayer FPSs and MMOs (which requires resolving incredibly complex state synchronizations at millisecond-scales) have done this for decades, and they haven’t been using any fancy CRDTs. Maybe they might have some ideas on how to achieve fast document synchronization as well? If you forget about P2P and only think about server/client type connections (since P2P doesn’t give you that much advantages in a Google-Docs type service), I think there’s a lot of overlap between multiplayer games and collaborative document editing, and maybe some cross-domain pollination might be needed to solve this problem.
- nivenkos 5y ago
  If you drop the P2P requirement, you can use Operational Transformations - https://en.wikipedia.org/wiki/Operational_transformation https://en.wikipedia.org/wiki/Operational_transformation
- dboreham 5y ago
  > they haven’t been using any fancy CRDTs CRDT is actually "the tricks we have always used + math to prove whether they work or not". Read the papers. You'll find old school stuff like Lamport Clocks.
- sunny--tech 5y ago
  This is only tangentially related to your point, but Riot uses CRDTs for their chat system. https://technology.riotgames.com/news/chat-service-architecture-persistence https://technology.riotgames.com/news/chat-service-architect...
  - cyber_kinetist 5y ago
    Maybe CRDTs are useful for implementing chat systems since the operational semantics for it are simple enough to make a CRDT system for it? The blog post I've read in the first link seems to say that CRDTs gets very complex when applying it to a domain where there are all kinds of different rich operations. I guess CRDTs aren't fundamentally the right solution for the collaborative document editing problem then?
pfraze 5y ago
Off topic - probably the most successful piece of "content" I've ever made is the CRDT notes [1] item that's nestled in there. I saw this submission and thought, "I wonder if my repo made it in there" and indeed it did. Why I find that funny is that I made that repo on a whim while I was doing my own reading, and then did nothing with it. Maybe I tweeted it? But somehow it SEOed well with Google for a stretch and I've been getting a steady stream of stars on that repo ever since. I assume it was because I created it when CRDTs were still early and so it got the clicks. I'm sure a lot of folks here know what it's like to try to put projects out there and go looking for traction. It's always made me chuckle that one of my biggest successes was the unintentional one. 1. https://github.com/pfrazee/crdt_notes https://github.com/pfrazee/crdt_notes
dang 5y ago
Lists don't make good HN submissions—they're too generic. The only thing to discuss is the lowest common denominator of the items on the list, and that's usually some very general topic about which there's nothing particularly new to say. Also, HN is itself a list. A pointer to a pointer to a pointer is too much indirection! It's better to post the most interesting item on the list. That increases the chance that there's something specific to talk about. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sort=byDate&type=comment&query=denominator%20list%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...
- dqpb 5y ago
  As a counterpoint, I upvoted and favorited this post before seeing your comment.
  - deleted 5y ago
    [deleted]
  - dang 5y ago
    Well, lists do make good bookmarks, and people tend to favorite things they want to look at later, so your comment and mine are quite compatible. My concern is thread quality. Lists of "resources about X" don't have anything really to discuss beyond "X in general", and maybe supplying links that didn't make the list. That's not, in the general case, enough to support a curious conversation—when it comes to forum threads, "generic" implies "shallow" and "repetitive". In this case the thread wasn't so bad, probably because CRDTs aren't in the most-discussed-topic set [1]. So as a generic submission it's better than, say, a list of Rust resources or something [2]. Still, we have to derive moderation principles for the general case, and this principle (about lists) is surprisingly reliable and solid. [1] https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=comments%3E3%20CRDT&sort=byDate&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... [2] https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=comments%3E3%20rust&sort=byDate&type=story&storyText=none https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

CRDT resources

119 points by stichers 5y ago

nuerow 5y ago

The content is interesting but the page is totally unusable in mobile.

nikivi 5y ago
Author of the wiki here, I think GitBook is quite usable on mobile for me. What is the issue?
- mkl 5y ago
  (Not OP) For me on Android Firefox the topic menu takes up the whole screen, doesn't scroll, and doesn't hide. On Chrome it seems to work.
  - teodorlu 5y ago
    Happens to me too =/ Firefox on Android.
- stichers 5y ago
  OP here — works for me on mobile Safari
  - skeeter2020 5y ago
    2021 and developers are still responding to support issues with "works on my machine!"
    - deleted 5y ago
      [deleted]
    - stichers 5y ago
      Ha ha! I'm not a dev any more, but old habits die hard, I guess. My comment was, in fact, to confirm for anyone who wanted to access with mobile Safari but who hadn't yet, that it indeed would work fine.

conaclos 5y ago

Interested readers may also find interest in https://crdt.tech https://crdt.tech

globular-toast 5y ago

At least just write out the name in full before using the acronym. "CRDT" could mean literally anything and it doesn't help when half the links in the list also don't bother to name it before using the acronym. Resist using acronyms before defining them. Don't just do it because everyone else does it. You are only creating barriers for people who might be interested in what you are talking about. I know people do this intentionally too. Don't be so insecure. You don't need to invent special language to remain relevant. Another user commented on a similar thing earlier today: https://news.ycombinator.com/item?id=28997945 https://news.ycombinator.com/item?id=28997945

zffr 5y ago
Since no one else defined it, CRDT = Conflict Free Replicated Data Types. There are 2 variants which confusingly have a very similar acronym: - CmRDT: Commutative Replicated Data Types (also known as operation-based CRDTs). These CRDTs replicate state by transmitting update operations to peers. Peers are always able to apply these operations in any order and without conflicts. - CvRDT: Convergent Replicated Data Types (also known as state-based CRDTs). These CRDTs replicate state by transmitting the entire object every time an update is made to a local replica. Peers are able to merge the state they receive with their local copy without conflicts.
- dboreham 5y ago
  It's ok that they have confusingly similar acronyms, because they're basically the same anyway.

tomsonj 5y ago

I've observed that implementing a CRDT system requires blogging about it afterward

hecturchi 5y ago
Hehe, or even writing a paper! https://arxiv.org/abs/2004.00107# https://arxiv.org/abs/2004.00107# The link takes you to our Merkle-CRDTs paper, which includes a nice intro to CRDTs in general, so no prior knowledge needed!
nosianu 5y ago
What method did you use to determine which/if anyone implementing CRDTs did not write a blog post? Or did you observe that 100% of the people who blogged about CRDTs implemented CRDTs and then wrote a blog post?
sunny--tech 5y ago
I have implemented a CRDT system and have also blogged about it afterwards. So I can confirm this. Joking aside though, CRDTs are still a pretty esoteric space and all the various blog posts came in handy when I was researching how to build my own CRDT. Most of the resources I've seen on CRDTs are from whitepapers and those can be difficult to read if you don't have a math background. I gave a talk at a Papers We Love [1] explicitly because I found the academic papers a big turn off from many people interested in the space. [1]: https://www.youtube.com/watch?v=1Bs3Fj9rvks&t=2169s&ab_channel=PapersWeLove https://www.youtube.com/watch?v=1Bs3Fj9rvks&t=2169s&ab_chann...
dang 5y ago
"Don't be snarky." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

keewee7 5y ago

What are some interesting use cases for CRDTs beyond collaborative (text) editing?

lijogdfljk 5y ago
I mean.. data, right? I'm trying to learn about CRDTs for complex, nested data structures. The use cases are for.. well, nested data, anything you'd use BTree's and etc for over distributed systems. A big thing i'm currently learning with them is to write a content addressable system with a more forgiving merge policy between parties. Yea i often see people nitpick CRDT about user intention, and where it should be `ADBC` or `ABCD`, but in my case i'm focusing on multi-device, not multi-user - and even in multi-user it's still best-in-breed when you are designing away from centralization. I'm still struggling to learn the more complex approaches to CRDT. So many resources focus on the low hanging fruit of CRDT. Grow counters, basic text editing, etc. I need to build the full suite of data structures; maps, sets, lists, etc.
sunny--tech 5y ago
Quite a few data stores use CRDTs under the hood. Redis uses CRDTs for active-active architectures [1] and for some of their native data structures [2]. Riak also uses them in their data store [3]. And looks like PayPal might use them for consensus purposes (I found this while looking up the Riak talk so I haven't actually watched it) [4] 1: https://redis.com/blog/diving-into-crdts/ https://redis.com/blog/diving-into-crdts/ 2: https://redis.com/videos/active-active-geo-distribution-redis-enterprise/ https://redis.com/videos/active-active-geo-distribution-redi... 3: https://www.youtube.com/watch?v=f20882ZSdkU&ab_channel=ErlangSolutions https://www.youtube.com/watch?v=f20882ZSdkU&ab_channel=Erlan... 4: https://www.infoq.com/presentations/crdt-production/ https://www.infoq.com/presentations/crdt-production/
deleted 5y ago
[deleted]

mox111 5y ago

Most of the CRDT examples I've seen appear to be Electron apps e.g. https://github.com/automerge/pushpin https://github.com/automerge/pushpin. My understanding is that CRDT's rely on having a safe place to store data on the user's machine (otherwise it's a bit like doing a `git clone` to receive new data, rather than a `git pull`). Is this not a major limitation for people hoping to use it for web apps?

sunny--tech 5y ago
I know that YJS is a common library that people use for using CRDTS in the browser and web-apps. https://github.com/yjs/yjs https://github.com/yjs/yjs
nisa 5y ago
The Distributed Mutable Containers spec looks super exciting: https://dream.public.cat/ https://dream.public.cat/ https://inqlab.net/projects/dmc/ https://inqlab.net/projects/dmc/ - https://inqlab.net/projects/eris/ https://inqlab.net/projects/eris/ - it's in https://openengiadina.net/ https://openengiadina.net/ - https://gitlab.com/openengiadina https://gitlab.com/openengiadina
- mro_name 5y ago
  Thanks for sharing, nice to see djb (netstrings in c-s-exps) shine through from far below. I wonder about the energy consumption aspect of content adressing. Redundancy eats resources.

cyber_kinetist 5y ago

> “Distributed state is so fundamentally complex that I think we actually need CRDTs (or something like them) to reason about it effectively.” Gamedevs working on multiplayer FPSs and MMOs (which requires resolving incredibly complex state synchronizations at millisecond-scales) have done this for decades, and they haven’t been using any fancy CRDTs. Maybe they might have some ideas on how to achieve fast document synchronization as well? If you forget about P2P and only think about server/client type connections (since P2P doesn’t give you that much advantages in a Google-Docs type service), I think there’s a lot of overlap between multiplayer games and collaborative document editing, and maybe some cross-domain pollination might be needed to solve this problem.

nivenkos 5y ago
If you drop the P2P requirement, you can use Operational Transformations - https://en.wikipedia.org/wiki/Operational_transformation https://en.wikipedia.org/wiki/Operational_transformation
dboreham 5y ago
> they haven’t been using any fancy CRDTs CRDT is actually "the tricks we have always used + math to prove whether they work or not". Read the papers. You'll find old school stuff like Lamport Clocks.
sunny--tech 5y ago
This is only tangentially related to your point, but Riot uses CRDTs for their chat system. https://technology.riotgames.com/news/chat-service-architecture-persistence https://technology.riotgames.com/news/chat-service-architect...
- cyber_kinetist 5y ago
  Maybe CRDTs are useful for implementing chat systems since the operational semantics for it are simple enough to make a CRDT system for it? The blog post I've read in the first link seems to say that CRDTs gets very complex when applying it to a domain where there are all kinds of different rich operations. I guess CRDTs aren't fundamentally the right solution for the collaborative document editing problem then?

pfraze 5y ago

Off topic - probably the most successful piece of "content" I've ever made is the CRDT notes [1] item that's nestled in there. I saw this submission and thought, "I wonder if my repo made it in there" and indeed it did. Why I find that funny is that I made that repo on a whim while I was doing my own reading, and then did nothing with it. Maybe I tweeted it? But somehow it SEOed well with Google for a stretch and I've been getting a steady stream of stars on that repo ever since. I assume it was because I created it when CRDTs were still early and so it got the clicks. I'm sure a lot of folks here know what it's like to try to put projects out there and go looking for traction. It's always made me chuckle that one of my biggest successes was the unintentional one. 1. https://github.com/pfrazee/crdt_notes https://github.com/pfrazee/crdt_notes

dang 5y ago

Lists don't make good HN submissions—they're too generic. The only thing to discuss is the lowest common denominator of the items on the list, and that's usually some very general topic about which there's nothing particularly new to say. Also, HN is itself a list. A pointer to a pointer to a pointer is too much indirection! It's better to post the most interesting item on the list. That increases the chance that there's something specific to talk about. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sort=byDate&type=comment&query=denominator%20list%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...

dqpb 5y ago
As a counterpoint, I upvoted and favorited this post before seeing your comment.
- deleted 5y ago
  [deleted]
- dang 5y ago
  Well, lists do make good bookmarks, and people tend to favorite things they want to look at later, so your comment and mine are quite compatible. My concern is thread quality. Lists of "resources about X" don't have anything really to discuss beyond "X in general", and maybe supplying links that didn't make the list. That's not, in the general case, enough to support a curious conversation—when it comes to forum threads, "generic" implies "shallow" and "repetitive". In this case the thread wasn't so bad, probably because CRDTs aren't in the most-discussed-topic set [1]. So as a generic submission it's better than, say, a list of Rust resources or something [2]. Still, we have to derive moderation principles for the general case, and this principle (about lists) is surprisingly reliable and solid. [1] https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=comments%3E3%20CRDT&sort=byDate&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... [2] https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=comments%3E3%20rust&sort=byDate&type=story&storyText=none https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...