4 ms·

> I've been kicking around the idea of writing a CRDT-based editor using this model. I got around to creating a data layer (p2p, browser-based, CRDT-backed) fo

> I've been kicking around the idea of writing a CRDT-based editor using this model. I got around to creating a data layer (p2p, browser-based, CRDT-backed) for something like this: https://github.com/hyperhyperspace/hyperhyperspace-core https://github.com/hyperhyperspace/hyperhyperspace-core I'd be interested in collaborating on your editor

FractalHQ 5y ago
Excited to check this out!! I’ve got experience with this stuff, count me in! Check out svelt-yjs[0]. It makes complex CRDT apps trivial. Websockets or P2P WebRTC in a few lines. I’ve got a live collab Svelte REPL using Monaco almost complete. It uses Postgres via Supabase for auth / row level security / saving and sharing Apps you make. Heck, plug yjs into this Svelte based notion style block editor[1] and you’re more than halfway done in an evening. [0] https://github.com/relm-us/svelt-yjs https://github.com/relm-us/svelt-yjs [1] https://github.com/djyde/plastic-editor https://github.com/djyde/plastic-editor
- deleted 5y ago
  [deleted]
- sbazerque 5y ago
  That's cool!
gumby 5y ago
This looks quite interesting.
isaacimagine 5y ago
I've also been toying with this idea for a while (p2p, CRDT backed, wasm-based)! It's a great concept, and I'd be interested in collaborating as well. One thing I've been working on is rich structural editors for the scripting language used to build these local-first applications. So the environment becomes something like squeak or a lisp machine, where all application data is automatically synchronized over the network.
- jitl 5y ago
  This is a very interesting idea. In general, it's interesting to think about synchronizing computation or memory state of a WASM module. Could you build an operational transform model for WASM memory state, to enable OTP-style process migration for more general WASM programs?
  - isaacimagine 5y ago
    This is quite the question, and I'll do my best to address it. First off, let's discuss more general Wasm programs. Currently, there are three primary modes of execution for Wasm modules - browser, node, or native runtime. The issue here is the dependence on JS - If you want to do any high-level interop between the language you're compiling to Wasm and some external library, you're most likely going to have to go through JS, pulling in a tool like emscripten to enforce the 'ABI' (so to speak) at the boundary between Wasm and JS. There are proposals, such as interface types, which may alleviate this in the future, but for now it's either go through JS or roll your own. I'm not a huge fan of having to pull in JS for something like this, which is why I've instead been working on an alternative 'ABI' for Wasm, which supports algebraic datatypes. The basic idea is pretty simple: we hand the Wasm module a large hunk of Wasm linear memory that is encoded in accordance with the ABI. Wasm modules must expose an `update` function that takes this memory and updates it in-place. The runtime then 'deserializes' the shared memory, extracts useful data, and calls out to external libraries, modifying the shared memory (once again in-place) with any results. We toss this `update` function in a loop, run it in a background thread, and watch it to ensure it doesn't crash, write out bad data, or hang. That's the rough idea, but the actual format itself is pretty straightforward (similar to messagepack); all we need to implement is a library to serialize/deserialize this shared memory representation for each language that compiles to Wasm. All programs operate on these algebraic datatypes, so to save program state, we can just serialize the shared memory and write it directly to disk. By making a module stateless, we can easily synchronize computation and memory. I'm currently using a hybrid operational transform / CRDT model that operates on the algebraic datatype model expressed by the ABI. Tombstones for text are largely inefficient, so we use a diff-based patch model for text inside blocks, but CRDTs for individual block-level transformations. (Wasm programs also have the ability to choose how to resolve conflicts.) Blocks are synchronized with a protocol similar to DAT, but, as stated, the protocol has built-in support for multiple writers and conflict resolution. So, with respect to Erlang/OTP-style process migration - it's certainly a possibility. If we exposed an actor-style library to Wasm programs through the aforementioned ABI, implementing message passing would be as simple as sharing an append-only log between two computers on a network - the data synchronization would happen automatically. Spawning a new process is as simple as creating a new block in the algebraic datatype tree, and giving a Wasm module control over it. It's all pretty interesting to think about, and I'm just skimming here, but that's the basic idea. To render out the user interface, we do something pretty simple - in addition to providing an `update` function, modules may provide multiple `translate` functions that take an ADT and translate it into another. Each node in the ADT is like a notion-style block, so all we need to do is define a `translate` function that takes the program-specific shared linear memory, and translates it to an ADT that represents a DOM (not a browser DOM, just a DOM). There are a number of optimizations that can be easily made here. TL;DR: It's easy to synchronize Wasm state if modules are stateless, program state can be encoded as a block-based ADT, we can synchronize this state through a mixture of Operational Transforms and CRDTs, GUIs are a translation of program data. (Woah, that's a lot of text...)
jitl 5y ago
How do you model text in your CRDT?
- sbazerque 5y ago
  It's not really designed to model collaboratively edited text (yet?) but as the backend for multi user apps (social, workspace, commerce, etc)
1_player 5y ago
Any relation to the hypercore project? Sounds similar in tech and in name, yet I can't find any reference to it.
- sbazerque 5y ago
  No relation, IIRC HyperCore was called DAT when I chose the Hyper Hyper Space name \o/

FractalHQ 5y ago

Excited to check this out!! I’ve got experience with this stuff, count me in! Check out svelt-yjs[0]. It makes complex CRDT apps trivial. Websockets or P2P WebRTC in a few lines. I’ve got a live collab Svelte REPL using Monaco almost complete. It uses Postgres via Supabase for auth / row level security / saving and sharing Apps you make. Heck, plug yjs into this Svelte based notion style block editor[1] and you’re more than halfway done in an evening. [0] https://github.com/relm-us/svelt-yjs https://github.com/relm-us/svelt-yjs [1] https://github.com/djyde/plastic-editor https://github.com/djyde/plastic-editor

deleted 5y ago
[deleted]
sbazerque 5y ago
That's cool!

gumby 5y ago

This looks quite interesting.

isaacimagine 5y ago

I've also been toying with this idea for a while (p2p, CRDT backed, wasm-based)! It's a great concept, and I'd be interested in collaborating as well. One thing I've been working on is rich structural editors for the scripting language used to build these local-first applications. So the environment becomes something like squeak or a lisp machine, where all application data is automatically synchronized over the network.

jitl 5y ago
This is a very interesting idea. In general, it's interesting to think about synchronizing computation or memory state of a WASM module. Could you build an operational transform model for WASM memory state, to enable OTP-style process migration for more general WASM programs?
- isaacimagine 5y ago
  This is quite the question, and I'll do my best to address it. First off, let's discuss more general Wasm programs. Currently, there are three primary modes of execution for Wasm modules - browser, node, or native runtime. The issue here is the dependence on JS - If you want to do any high-level interop between the language you're compiling to Wasm and some external library, you're most likely going to have to go through JS, pulling in a tool like emscripten to enforce the 'ABI' (so to speak) at the boundary between Wasm and JS. There are proposals, such as interface types, which may alleviate this in the future, but for now it's either go through JS or roll your own. I'm not a huge fan of having to pull in JS for something like this, which is why I've instead been working on an alternative 'ABI' for Wasm, which supports algebraic datatypes. The basic idea is pretty simple: we hand the Wasm module a large hunk of Wasm linear memory that is encoded in accordance with the ABI. Wasm modules must expose an `update` function that takes this memory and updates it in-place. The runtime then 'deserializes' the shared memory, extracts useful data, and calls out to external libraries, modifying the shared memory (once again in-place) with any results. We toss this `update` function in a loop, run it in a background thread, and watch it to ensure it doesn't crash, write out bad data, or hang. That's the rough idea, but the actual format itself is pretty straightforward (similar to messagepack); all we need to implement is a library to serialize/deserialize this shared memory representation for each language that compiles to Wasm. All programs operate on these algebraic datatypes, so to save program state, we can just serialize the shared memory and write it directly to disk. By making a module stateless, we can easily synchronize computation and memory. I'm currently using a hybrid operational transform / CRDT model that operates on the algebraic datatype model expressed by the ABI. Tombstones for text are largely inefficient, so we use a diff-based patch model for text inside blocks, but CRDTs for individual block-level transformations. (Wasm programs also have the ability to choose how to resolve conflicts.) Blocks are synchronized with a protocol similar to DAT, but, as stated, the protocol has built-in support for multiple writers and conflict resolution. So, with respect to Erlang/OTP-style process migration - it's certainly a possibility. If we exposed an actor-style library to Wasm programs through the aforementioned ABI, implementing message passing would be as simple as sharing an append-only log between two computers on a network - the data synchronization would happen automatically. Spawning a new process is as simple as creating a new block in the algebraic datatype tree, and giving a Wasm module control over it. It's all pretty interesting to think about, and I'm just skimming here, but that's the basic idea. To render out the user interface, we do something pretty simple - in addition to providing an `update` function, modules may provide multiple `translate` functions that take an ADT and translate it into another. Each node in the ADT is like a notion-style block, so all we need to do is define a `translate` function that takes the program-specific shared linear memory, and translates it to an ADT that represents a DOM (not a browser DOM, just a DOM). There are a number of optimizations that can be easily made here. TL;DR: It's easy to synchronize Wasm state if modules are stateless, program state can be encoded as a block-based ADT, we can synchronize this state through a mixture of Operational Transforms and CRDTs, GUIs are a translation of program data. (Woah, that's a lot of text...)

jitl 5y ago

How do you model text in your CRDT?

sbazerque 5y ago
It's not really designed to model collaboratively edited text (yet?) but as the backend for multi user apps (social, workspace, commerce, etc)

1_player 5y ago

Any relation to the hypercore project? Sounds similar in tech and in name, yet I can't find any reference to it.

sbazerque 5y ago
No relation, IIRC HyperCore was called DAT when I chose the Hyper Hyper Space name \o/