5 ms·

Celld: Self-hosted, distributed Durable Objects

286 points by calvinfo 2mo ago

jauntywundrkind 2mo ago
Well that couldn't have been any more timely. Did they know? Were they keeping this in a back pocket for the time? CloudFlare OS: https://blog.cloudflare.com/cloudflare-os/ https://blog.cloudflare.com/cloudflare-os/ https://news.ycombinator.com/item?id=49182996 https://news.ycombinator.com/item?id=49182996 And yes, celld is isolates! The very lightweight within v8 isolation boundary. Hence the "very low idle cost". The deno team rolled their own new runtime! This one without deno_core! Shout out to this excellent 2022 post: https://deno.com/blog/roll-your-own-javascript-runtime https://deno.com/blog/roll-your-own-javascript-runtime https://news.ycombinator.com/item?id=35819990 https://news.ycombinator.com/item?id=35819990 https://news.ycombinator.com/item?id=35819990 https://news.ycombinator.com/item?id=35819990 It's super super exciting having an age where many pieces of software can all run with low profiles. Very timely, again, for the unbelievably order of magnitude (and thats 10 to the not 2 to the, for you fellow computer nerds) ish ram crunch we are in. Next up, at some point, ideally we get some v8-like runtimes where we can share libraries across multiple isolates! Separating the shared code from the shared data, so we have multiple instances of the library, seems much harder. (It feels like wasm has some/much promise here and I eagerly await clearer signals that code-sharing while sandboxing is indeed possible)
- reactordev 2mo ago
  Now you’re getting into the realm of old SOA… where data exchanges are just as stochastic as services on the pipe.
- wartywhoa23 2mo ago
  The order of magnitude of this is simply mindblowing.
- tdrz 2mo ago
  > Next up, at some point, ideally we get some v8-like runtimes where we can share libraries across multiple isolates! I'm probably missing your point, but aren't wasm memories achieving exactly this? Just an example from wasmtime: https://docs.wasmtime.dev/examples-multimemory.html https://docs.wasmtime.dev/examples-multimemory.html
vmg12 2mo ago
All we need is a way to run these on spot instances
- wmf 2mo ago
  I don't see why not. Somebody once wrote a simple Lambda that spawns a new instance when your spot instance dies so there's always at least one running.
q3k 2mo ago
> addressed by name and replicated to an S3-compatible bucket you own; nodes coordinate through that bucket alone, with no control plane or consensus ... but that S3 _is_ the control plane and consensus layer, no? You're just pushing this down the stack to whoever runs that S3 clone.
- jitl 2mo ago
  sure, but s3 is far easier to buy and also far more reliable than typical distributed systems coordinator stuff like etcd/zookeeper/consul. every major cloud offers it, and there’s several decent open source options.
  - srvraw 2mo ago
    I haven't deep-dived into celld. Does it requires s3's strong consistency to be correct? or is any s3 api compatible object storage sufficient even if they lack strong consistency guarantee?
    - jitl 2mo ago
      it’s relying on consistency w conditional writes
zuzululu 2mo ago
i donno...i have some doubts about this cloudflare gives you that instant geo-sync across the world that is hard to beat with Celld do i need to buy bare metal in major continents
- vmg12 2mo ago
  Geo-sync? I'm pretty sure durable objects are tied to the datacenter they are created in.
  - zuzululu 2mo ago
    nothing beats cloudflare's reach they have nodes everywhere thats hard to beat
- jauntywundrkind 2mo ago
  on the one hand, hosting is hard, you should pay people to host, especially distributed systems. on the other hand, deno land folks stole from (used open source from) the best. the coordination layer/control-plane is all S3 CAS of dumb json files, which is a fantastic common-mode infra requirement for most orgs anyways, & perhaps a durable control-plane substrate you'd feel comfortable having someone else run (such as aws or others), while you run the data-plane (workers) yourself. and then for the durable objects themselves, they used litestream, which is a pretty top pick, excellent way to get radical distribution (but actual topology not included, some assembly required)! https://hn.algolia.com/?q=litestream https://hn.algolia.com/?q=litestream there's no reason this wouldn't run fine on most hosting, you definitely don't need bare metal. the virtues of v8 sandboxing / isolates! no need for vm's at all, no nested vm difficulties if you are trying to host on a shared host! but if you're asking questions like this, i want to again point you back to my top point.
- wmf 2mo ago
  With apologies to Obama, if you like Cloudflare you can keep your Cloudflare. It looks like celld is not intended to be an exact clone of Cloudflare but is for someone who wants to run a bunch of workers in a single region.
- 2mo ago
khalidx 2mo ago
Finally! So happy to see support for running durable objects outside of one provider. Upvoted. The "durable object" concept has been repeatably demonstrated to be a valuable abstraction. "Each object is its own SQLite database, addressed by name and replicated to an S3-compatible bucket you own" -- this concept can take you a long way, both in its power and simplicity.
- semicognitive 2mo ago
  Check out https://rivet.dev/actors/ https://rivet.dev/actors/
  - crabmusket 2mo ago
    I believe Restate has something which can work similarly: https://docs.restate.dev/foundations/services https://docs.restate.dev/foundations/services There was also Jamsocket and Plane, though it looks like they've shut down after acquisition: https://github.com/jamsocket/plane https://github.com/jamsocket/plane This seems to be the first one which is providing drop-in compatibility for Cloudflare's JS-side APIs, including JSRPC etc.
- jeremycarter 2mo ago
  This concept has been around for a long time. https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/Orleans-MSR-TR-2014-41.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/...
hhthrowaway1230 2mo ago
I like it! Would be great if I could run locally from the start without configuring an s3 for easy playing/prototyping
- wmf 2mo ago
  Maybe Wrangler is a better way to do that.
deleted 2mo ago
[deleted]
jitl 2mo ago
what is the difference between celld and Cloudflare Workers open source version workerd? https://github.com/cloudflare/workerd https://github.com/cloudflare/workerd
- rough-sea 2mo ago
  workerd is missing the scheduler - the thing that allows it to operate across nodes. Durable Object storage are just local sqlite databases. celld is the full distributed system (albeit single tenant). It distributes DOs (cells) across any number of VMs. The databases for each DO are in object storage with RPO=0 guarantee.
  - kentonv 2mo ago
    You beat us to it! Been meaning to land this but got distracted. https://github.com/cloudflare/workerd/pull/6780 https://github.com/cloudflare/workerd/pull/6780 But honestly I love that there are multiple implementations now.
    - jitl 2mo ago
      bummer this expects NFS; this is all a brand new implementation vs what Cloudflare runs internally in production for DOs? I can’t imagine all your workerds share a single global NFS… What would a more realistic workerd with DO support look like? ah well, we have celld!
      - kentonv 2mo ago
        Honestly, you wouldn't want to run our internal implementation -- it's far too complicated. Unless you have 100+ datacenters around the world -- then maybe. The goal of this new design is to scale to a cluster while being operationally very easy to set up. Ideal for self-hosting. NFSv4 is an easy first step, convenient because it's broadly understood, has many implementations, and requires no client libraries. I could imagine a follow-up to support LiteFS instead of NFS could make a lot of sense, though it'll get more complicated. But yes, celld is definitely ahead of us here. No doubt about that.
myshapeprotocol 2mo ago
[flagged]
jumploops 2mo ago
I recently spun up a simple app for our annual mango tasting event[0] using Cloudflare Workers and Durable Objects. It worked really well! Excited to see more options outside of Cloudflare. [0]https://github.com/jumploops/mangotango https://github.com/jumploops/mangotango
sakesun 2mo ago
Wonder if this will become common practice from now on ? > Pull requests are disabled. Coding agents make it too easy to send a large, > low-context change that costs maintainers more time than it saves. > Thoughtful contributions are welcome; please understand the code, > keep the patch focused, and respect the review time you are asking for. > > Send a git format-patch attachment to ...
- dbmikus 2mo ago
  time to bring back the dev email distribution list
  - 7bit 2mo ago
    Deargodplsnooo... Meme
- szmarczak 2mo ago
  I wonder if the code was written by AI?
  - crabmusket 2mo ago
    The website surely was, so...
- aitchnyu 2mo ago
  Could be done by an Openclaw agent named Jarvis.
- ryanrasti 2mo ago
  Funny, the same thing stuck out to me from Cloudflare OS's README (https://github.com/cloudflare/cloudflare-os#contributing https://github.com/cloudflare/cloudflare-os#contributing): > At this time, we are not seeking outside contribution. > AI has made writing code easy. The hard part, today, is not writing the code, but reviewing it, making sure quality stays high, and keeping the product coherent. In that light, unfortunately, external code contributions are "donating" the easy part of the job, while creating more of the hard work. Feels very weird, but is logically sound: owners know exactly what they want and so they can work with Claude et al to iterate on features faster than with most drive-by contributors. It reads like an "end of an era" but I imagine the steady state will be somewhere in the middle: high trust, high context contributors will still be able to contribute meaningful work.
jiveturkey 2mo ago
claim is that very many things are not needed, but they are. they are just provided by the layer beneath (S3). so isn't correctness highly dependent on s3 correctness? was s3 built for that, vs just plain durability? for example, would it be just as reliable with minio? i would prefer a thing that was more self-contained, not dependent on a black box service layer underneath. much apologies if i just have a poor understanding.
- mrkeen 2mo ago
  I think you might be asking too much when it comes to correctness. It's like fixing all the problems with democracy by putting everyone in charge of their own 1-person election.
rtpg 2mo ago
Could someone explain to me in practice how one would use these to do… a thing? Like a straightforward use case? Having trouble figuring out what these are for
- pelagicAustral 2mo ago
  Multiplayer games, or apps like document editing for teams... I don't know, highly concurrent stuff
- wett 2mo ago
  Simple example 1:35 -7:40 https://youtu.be/qF2PuYnBahw https://youtu.be/qF2PuYnBahw More practical example (serverless WebSockets) https://youtu.be/FgWVoryZ8PU https://youtu.be/FgWVoryZ8PU
  - thefreeman 2mo ago
    This was a great video. Thanks!
7bit 2mo ago
Hey, is this something that compares to Temporal or Restate or Azure Durable Functions? If so, what are the differences and when would I use Cells instead of the aforementioned?
- crabmusket 2mo ago
  There's some overlap and some similar use cases. I've seen a few people being confused between Durable Objects and Durable Execution. Short writeup: https://crabmusket.net/2024/durable-execution-versus-session-backends/ https://crabmusket.net/2024/durable-execution-versus-session... Summary: I think the way to decide is, do you want to program with "objects" or "processes"? I'd use Durable Objects (or what I called Session Backends in my article, after Jamsocket) if I wanted to model an "entity" that lasts indefinitely (e.g. a Figma doc, a user, a concert/event). I'd use a temporal/restate function for something that has a linear sequence of events and eventually comes to an end. This isn't a hard rule. You can get the same outcome out of both technologies. (For example, Cloudflare built Workflows on top of Durable Objects; Rivet did the reverse.) You can use either one to implement the other, but you're going to be going out of your way based on what APIs are provided.
dzonga 2mo ago
Ryan & the Deno team thanks for this. not that I'm going to use it anytime soon - but yeah having durable objects outside Cloudflare and also self hosted ones means a lot.
meowtimemania 2mo ago
Wish there were more performance related benchmarks that compare this to cloudflare durable objects.
ironqcold 2mo ago
I don't think AI is killing open source. It's just making it way easier to spam maintainers with code. Good contributors are probably worth even more now.
somerandom1 2mo ago
Seeing something like this reminds me of Kleppmann's blog post: https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html https://martin.kleppmann.com/2016/02/08/how-to-do-distribute.... celld's README states "Object-storage compare-and-swap ensures that exactly one node owns a cell at a time," but I'm skeptical this actually holds at the point where data is written to storage - I had an AI skim through the code with me, and the actual segment writes looked like plain, unconditional PUTs with no epoch check. In many cases, I think using Cloudflare's Durable Objects is probably the right call instead.

286 points by calvinfo 2mo ago

Well that couldn't have been any more timely. Did they know? Were they keeping this in a back pocket for the time? CloudFlare OS: https://blog.cloudflare.com/cloudflare-os/ https://blog.cloudflare.com/cloudflare-os/ https://news.ycombinator.com/item?id=49182996 https://news.ycombinator.com/item?id=49182996 And yes, celld is isolates! The very lightweight within v8 isolation boundary. Hence the "very low idle cost". The deno team rolled their own new runtime! This one without deno_core! Shout out to this excellent 2022 post: https://deno.com/blog/roll-your-own-javascript-runtime https://deno.com/blog/roll-your-own-javascript-runtime https://news.ycombinator.com/item?id=35819990 https://news.ycombinator.com/item?id=35819990 https://news.ycombinator.com/item?id=35819990 https://news.ycombinator.com/item?id=35819990 It's super super exciting having an age where many pieces of software can all run with low profiles. Very timely, again, for the unbelievably order of magnitude (and thats 10 to the not 2 to the, for you fellow computer nerds) ish ram crunch we are in. Next up, at some point, ideally we get some v8-like runtimes where we can share libraries across multiple isolates! Separating the shared code from the shared data, so we have multiple instances of the library, seems much harder. (It feels like wasm has some/much promise here and I eagerly await clearer signals that code-sharing while sandboxing is indeed possible)

reactordev 2mo ago
Now you’re getting into the realm of old SOA… where data exchanges are just as stochastic as services on the pipe.
wartywhoa23 2mo ago
The order of magnitude of this is simply mindblowing.
tdrz 2mo ago
> Next up, at some point, ideally we get some v8-like runtimes where we can share libraries across multiple isolates! I'm probably missing your point, but aren't wasm memories achieving exactly this? Just an example from wasmtime: https://docs.wasmtime.dev/examples-multimemory.html https://docs.wasmtime.dev/examples-multimemory.html

vmg12 2mo ago

All we need is a way to run these on spot instances

wmf 2mo ago
I don't see why not. Somebody once wrote a simple Lambda that spawns a new instance when your spot instance dies so there's always at least one running.

q3k 2mo ago

> addressed by name and replicated to an S3-compatible bucket you own; nodes coordinate through that bucket alone, with no control plane or consensus ... but that S3 _is_ the control plane and consensus layer, no? You're just pushing this down the stack to whoever runs that S3 clone.

jitl 2mo ago
sure, but s3 is far easier to buy and also far more reliable than typical distributed systems coordinator stuff like etcd/zookeeper/consul. every major cloud offers it, and there’s several decent open source options.
- srvraw 2mo ago
  I haven't deep-dived into celld. Does it requires s3's strong consistency to be correct? or is any s3 api compatible object storage sufficient even if they lack strong consistency guarantee?
  - jitl 2mo ago
    it’s relying on consistency w conditional writes

zuzululu 2mo ago

i donno...i have some doubts about this cloudflare gives you that instant geo-sync across the world that is hard to beat with Celld do i need to buy bare metal in major continents

vmg12 2mo ago
Geo-sync? I'm pretty sure durable objects are tied to the datacenter they are created in.
- zuzululu 2mo ago
  nothing beats cloudflare's reach they have nodes everywhere thats hard to beat
jauntywundrkind 2mo ago
on the one hand, hosting is hard, you should pay people to host, especially distributed systems. on the other hand, deno land folks stole from (used open source from) the best. the coordination layer/control-plane is all S3 CAS of dumb json files, which is a fantastic common-mode infra requirement for most orgs anyways, & perhaps a durable control-plane substrate you'd feel comfortable having someone else run (such as aws or others), while you run the data-plane (workers) yourself. and then for the durable objects themselves, they used litestream, which is a pretty top pick, excellent way to get radical distribution (but actual topology not included, some assembly required)! https://hn.algolia.com/?q=litestream https://hn.algolia.com/?q=litestream there's no reason this wouldn't run fine on most hosting, you definitely don't need bare metal. the virtues of v8 sandboxing / isolates! no need for vm's at all, no nested vm difficulties if you are trying to host on a shared host! but if you're asking questions like this, i want to again point you back to my top point.
wmf 2mo ago
With apologies to Obama, if you like Cloudflare you can keep your Cloudflare. It looks like celld is not intended to be an exact clone of Cloudflare but is for someone who wants to run a bunch of workers in a single region.
2mo ago

khalidx 2mo ago

Finally! So happy to see support for running durable objects outside of one provider. Upvoted. The "durable object" concept has been repeatably demonstrated to be a valuable abstraction. "Each object is its own SQLite database, addressed by name and replicated to an S3-compatible bucket you own" -- this concept can take you a long way, both in its power and simplicity.

semicognitive 2mo ago
Check out https://rivet.dev/actors/ https://rivet.dev/actors/
- crabmusket 2mo ago
  I believe Restate has something which can work similarly: https://docs.restate.dev/foundations/services https://docs.restate.dev/foundations/services There was also Jamsocket and Plane, though it looks like they've shut down after acquisition: https://github.com/jamsocket/plane https://github.com/jamsocket/plane This seems to be the first one which is providing drop-in compatibility for Cloudflare's JS-side APIs, including JSRPC etc.
jeremycarter 2mo ago
This concept has been around for a long time. https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/Orleans-MSR-TR-2014-41.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/...

hhthrowaway1230 2mo ago

I like it! Would be great if I could run locally from the start without configuring an s3 for easy playing/prototyping

wmf 2mo ago
Maybe Wrangler is a better way to do that.

deleted 2mo ago

[deleted]

jitl 2mo ago

what is the difference between celld and Cloudflare Workers open source version workerd? https://github.com/cloudflare/workerd https://github.com/cloudflare/workerd

rough-sea 2mo ago
workerd is missing the scheduler - the thing that allows it to operate across nodes. Durable Object storage are just local sqlite databases. celld is the full distributed system (albeit single tenant). It distributes DOs (cells) across any number of VMs. The databases for each DO are in object storage with RPO=0 guarantee.
- kentonv 2mo ago
  You beat us to it! Been meaning to land this but got distracted. https://github.com/cloudflare/workerd/pull/6780 https://github.com/cloudflare/workerd/pull/6780 But honestly I love that there are multiple implementations now.
  - jitl 2mo ago
    bummer this expects NFS; this is all a brand new implementation vs what Cloudflare runs internally in production for DOs? I can’t imagine all your workerds share a single global NFS… What would a more realistic workerd with DO support look like? ah well, we have celld!
    - kentonv 2mo ago
      Honestly, you wouldn't want to run our internal implementation -- it's far too complicated. Unless you have 100+ datacenters around the world -- then maybe. The goal of this new design is to scale to a cluster while being operationally very easy to set up. Ideal for self-hosting. NFSv4 is an easy first step, convenient because it's broadly understood, has many implementations, and requires no client libraries. I could imagine a follow-up to support LiteFS instead of NFS could make a lot of sense, though it'll get more complicated. But yes, celld is definitely ahead of us here. No doubt about that.

myshapeprotocol 2mo ago

[flagged]

jumploops 2mo ago

I recently spun up a simple app for our annual mango tasting event[0] using Cloudflare Workers and Durable Objects. It worked really well! Excited to see more options outside of Cloudflare. [0]https://github.com/jumploops/mangotango https://github.com/jumploops/mangotango

sakesun 2mo ago

Wonder if this will become common practice from now on ? > Pull requests are disabled. Coding agents make it too easy to send a large, > low-context change that costs maintainers more time than it saves. > Thoughtful contributions are welcome; please understand the code, > keep the patch focused, and respect the review time you are asking for. > > Send a git format-patch attachment to ...

dbmikus 2mo ago
time to bring back the dev email distribution list
- 7bit 2mo ago
  Deargodplsnooo... Meme
szmarczak 2mo ago
I wonder if the code was written by AI?
- crabmusket 2mo ago
  The website surely was, so...
aitchnyu 2mo ago
Could be done by an Openclaw agent named Jarvis.
ryanrasti 2mo ago
Funny, the same thing stuck out to me from Cloudflare OS's README (https://github.com/cloudflare/cloudflare-os#contributing https://github.com/cloudflare/cloudflare-os#contributing): > At this time, we are not seeking outside contribution. > AI has made writing code easy. The hard part, today, is not writing the code, but reviewing it, making sure quality stays high, and keeping the product coherent. In that light, unfortunately, external code contributions are "donating" the easy part of the job, while creating more of the hard work. Feels very weird, but is logically sound: owners know exactly what they want and so they can work with Claude et al to iterate on features faster than with most drive-by contributors. It reads like an "end of an era" but I imagine the steady state will be somewhere in the middle: high trust, high context contributors will still be able to contribute meaningful work.

jiveturkey 2mo ago

claim is that very many things are not needed, but they are. they are just provided by the layer beneath (S3). so isn't correctness highly dependent on s3 correctness? was s3 built for that, vs just plain durability? for example, would it be just as reliable with minio? i would prefer a thing that was more self-contained, not dependent on a black box service layer underneath. much apologies if i just have a poor understanding.

mrkeen 2mo ago
I think you might be asking too much when it comes to correctness. It's like fixing all the problems with democracy by putting everyone in charge of their own 1-person election.

rtpg 2mo ago

Could someone explain to me in practice how one would use these to do… a thing? Like a straightforward use case? Having trouble figuring out what these are for

pelagicAustral 2mo ago
Multiplayer games, or apps like document editing for teams... I don't know, highly concurrent stuff
wett 2mo ago
Simple example 1:35 -7:40 https://youtu.be/qF2PuYnBahw https://youtu.be/qF2PuYnBahw More practical example (serverless WebSockets) https://youtu.be/FgWVoryZ8PU https://youtu.be/FgWVoryZ8PU
- thefreeman 2mo ago
  This was a great video. Thanks!

7bit 2mo ago

Hey, is this something that compares to Temporal or Restate or Azure Durable Functions? If so, what are the differences and when would I use Cells instead of the aforementioned?

crabmusket 2mo ago
There's some overlap and some similar use cases. I've seen a few people being confused between Durable Objects and Durable Execution. Short writeup: https://crabmusket.net/2024/durable-execution-versus-session-backends/ https://crabmusket.net/2024/durable-execution-versus-session... Summary: I think the way to decide is, do you want to program with "objects" or "processes"? I'd use Durable Objects (or what I called Session Backends in my article, after Jamsocket) if I wanted to model an "entity" that lasts indefinitely (e.g. a Figma doc, a user, a concert/event). I'd use a temporal/restate function for something that has a linear sequence of events and eventually comes to an end. This isn't a hard rule. You can get the same outcome out of both technologies. (For example, Cloudflare built Workflows on top of Durable Objects; Rivet did the reverse.) You can use either one to implement the other, but you're going to be going out of your way based on what APIs are provided.

dzonga 2mo ago

Ryan & the Deno team thanks for this. not that I'm going to use it anytime soon - but yeah having durable objects outside Cloudflare and also self hosted ones means a lot.

meowtimemania 2mo ago

Wish there were more performance related benchmarks that compare this to cloudflare durable objects.

ironqcold 2mo ago

I don't think AI is killing open source. It's just making it way easier to spam maintainers with code. Good contributors are probably worth even more now.

somerandom1 2mo ago

Seeing something like this reminds me of Kleppmann's blog post: https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html https://martin.kleppmann.com/2016/02/08/how-to-do-distribute.... celld's README states "Object-storage compare-and-swap ensures that exactly one node owns a cell at a time," but I'm skeptical this actually holds at the point where data is written to storage - I had an AI skim through the code with me, and the actual segment writes looked like plain, unconditional PUTs with no epoch check. In many cases, I think using Cloudflare's Durable Objects is probably the right call instead.