3 ms·

Zero-latency SQLite storage in every Durable Object

23 points by jgrahamc 2y ago

xyzzy_plugh 2y ago
This is pretty cool. What I don't like is that they're basically doing the equivalent to disabling fsync on quorum writes. As long as a quorum acks the writes, it's considered committed, even if storage hasn't flushed yet. I agree that the likelihood of this is rare, especially if each committer is in a different zone, but stranger things have happened. You're gambling with your operations credit. I hope you have enough! This seems like the typical CAP "actually all three" footgun most folks fall into. I'd love to see the failure scenarios that result in data loss enumerated. If you're going to make this choice on behalf of your users, I think it's only fair. I get that some point you're just hedging against failure scenarios by stacking redundancy, and that's probably what's happening here, but I can't see inside their implementation to know. I'm immediately dubious of folks who try to sell systems where fsync is disabled for commits. We don't all drink this kool-aid, the onus is in you. Unfortunately this always seems to degenerate into discussion of whether or not fsync does or can even work in practice. Even if I agree that it doesn't work, it becomes a moot point as we then can't productively discuss durability in the face of power loss.
- kentonv 2y ago
  As described in the blog post, each write is replicated to machines in five different datacenters, and not confirmed until a quorum of three acknowledge it. So you'd need a power loss in three geographically-separated datacenters simultaneously to lose data. Write confirmations block the "output gate", which means the app can keep running "speculatively" while waiting for write confirmation, but if the write fails then no one in the rest of the world will ever observe that any code after the write executed.
tarasglek 2y ago
I like this, can we have cdc notifications for table writes and DO reacting to them behind that output gate too? That's cool that you guys are looking beyond s3 for distributed primitives
- kentonv 2y ago
  I've thought about this... what if output gates were a sort of contagious thing across Cloudflare's network, so they don't block communications to other workers (including DOs), but instead the output gate extends around the other worker? Haven't done this yet but in principle it makes sense.

23 points by jgrahamc 2y ago

This is pretty cool. What I don't like is that they're basically doing the equivalent to disabling fsync on quorum writes. As long as a quorum acks the writes, it's considered committed, even if storage hasn't flushed yet. I agree that the likelihood of this is rare, especially if each committer is in a different zone, but stranger things have happened. You're gambling with your operations credit. I hope you have enough! This seems like the typical CAP "actually all three" footgun most folks fall into. I'd love to see the failure scenarios that result in data loss enumerated. If you're going to make this choice on behalf of your users, I think it's only fair. I get that some point you're just hedging against failure scenarios by stacking redundancy, and that's probably what's happening here, but I can't see inside their implementation to know. I'm immediately dubious of folks who try to sell systems where fsync is disabled for commits. We don't all drink this kool-aid, the onus is in you. Unfortunately this always seems to degenerate into discussion of whether or not fsync does or can even work in practice. Even if I agree that it doesn't work, it becomes a moot point as we then can't productively discuss durability in the face of power loss.

kentonv 2y ago
As described in the blog post, each write is replicated to machines in five different datacenters, and not confirmed until a quorum of three acknowledge it. So you'd need a power loss in three geographically-separated datacenters simultaneously to lose data. Write confirmations block the "output gate", which means the app can keep running "speculatively" while waiting for write confirmation, but if the write fails then no one in the rest of the world will ever observe that any code after the write executed.

tarasglek 2y ago

I like this, can we have cdc notifications for table writes and DO reacting to them behind that output gate too? That's cool that you guys are looking beyond s3 for distributed primitives

kentonv 2y ago
I've thought about this... what if output gates were a sort of contagious thing across Cloudflare's network, so they don't block communications to other workers (including DOs), but instead the output gate extends around the other worker? Haven't done this yet but in principle it makes sense.