4 ms·

sewen (et al) This is lovely and I'm itching to try it. One question: We have a use case where a location gets cut off completely from the internet at large.

by zellyn 2y ago

sewen (et al) This is lovely and I'm itching to try it. One question: We have a use case where a location gets cut off completely from the internet at large. In that case, it makes sense for the local hardware (typically Android and/or iOS tablets or equivalent) to take over as a log owner: even though you're cut off, if you're willing to swallow the risk (and hence cost) of offline payments, you should be able to create orders, fulfill them, pay for them, close them out, send tickets to the kitchen to cook the food or to the warehouse to fetch the tractor, etc. Does restate include something that covers that use-case? In the noodling/daydreaming a colleague and I have done, we ended up with something very close to restate (I imagined just using Kafka), except that additionally many operations would have a CRDT nature: eg. you should _always_ be allowed to add a payment to an order, because presumably a real-life payment happened. I've also noodled with the idea of logs whose canonical ownership can be transferred. That covers cases where you start offline and then reconnect, but doesn't work so well for transactions that start out connected (and thus owned in the datacenter) and need to continue offline. One could also imagine ensuring that > n/2 consensus members are always located inside the restaurant/hardware store/etc., so if you go offline, you can still proceed. It might even be possible to recognize disconnection and then take one full vote to further subdivide that pool of consensus members so if one dies it doesn't halt progress. This feels like it would be getting very tricksy…

withinboredom 2y ago
I'm actually working on a database implementation for this exact use-case... It's a distributed edge database and still quite a long ways to go -- https://github.com/bottledcode/atlas-db https://github.com/bottledcode/atlas-db if you want to give it a star. It's mostly based on wpaxos (wide-area consensus), spaxos, fpaxos and pretty neat. The repo above is a productionization of several proof of concepts to get there. > One could also imagine ensuring that > n/2 consensus members are always located inside the restaurant/hardware store/etc., so if you go offline, you can still proceed. This is what annoys me to no end about RAFT. It's a great protocol, don't get me wrong, but its too simple for these types of problems. RAFT fails when it doesn't have consensus and because the consensus is non-deterministic, it must have an odd number of nodes. PAXOS, while far more complex than RAFT in terms of "grok", is deterministic so you don't need an odd number of nodes. If you throw in some flexible quorums, you can do some really neat stuff, like how Atlas handles a "region" (ie, areas connected via the internet instead of the same network) becoming disconnected; but I'm not ready yet. There's still a long way to go!
mrkeen 2y ago
> One could also imagine ensuring that > n/2 consensus members are always located inside the restaurant/hardware store/etc. If the simple-majority is inside the restaurant, then the restaurant is online and the rest of the world is offline.
WilcoKruijer 2y ago
Interesting use case, I've also thought about this in the past. I wonder why you want to designate the local device as "owner". The devices could simply keep the (closed) payments in a local outbox, and send them to the server when it comes back online. If there is no way for syncing the local changes to the server to fail, you've essentially created a CRDT (at least the "conflict-free" part). Of course that means you cannot have any server side validation at all (maybe this is what you mean by "owner"?). Because that would mean closed out payments would be rejected by the server and you'd end up in an inconsistent state. A consequence of this is that there could be no balance checking (as an example). Payments in a ledger are inherently a CRDT! The consensus part is interesting. Would you really need it? Wouldn't one device be sufficient to validate a transaction? And would consensus really even mean something within a small building? If the restaurant catches on fire the data would be lost anyway, so if you don't achieve "high-availability", why bother?

by zellyn 2y ago

withinboredom 2y ago

I'm actually working on a database implementation for this exact use-case... It's a distributed edge database and still quite a long ways to go -- https://github.com/bottledcode/atlas-db https://github.com/bottledcode/atlas-db if you want to give it a star. It's mostly based on wpaxos (wide-area consensus), spaxos, fpaxos and pretty neat. The repo above is a productionization of several proof of concepts to get there. > One could also imagine ensuring that > n/2 consensus members are always located inside the restaurant/hardware store/etc., so if you go offline, you can still proceed. This is what annoys me to no end about RAFT. It's a great protocol, don't get me wrong, but its too simple for these types of problems. RAFT fails when it doesn't have consensus and because the consensus is non-deterministic, it must have an odd number of nodes. PAXOS, while far more complex than RAFT in terms of "grok", is deterministic so you don't need an odd number of nodes. If you throw in some flexible quorums, you can do some really neat stuff, like how Atlas handles a "region" (ie, areas connected via the internet instead of the same network) becoming disconnected; but I'm not ready yet. There's still a long way to go!

mrkeen 2y ago

> One could also imagine ensuring that > n/2 consensus members are always located inside the restaurant/hardware store/etc. If the simple-majority is inside the restaurant, then the restaurant is online and the rest of the world is offline.

WilcoKruijer 2y ago

Interesting use case, I've also thought about this in the past. I wonder why you want to designate the local device as "owner". The devices could simply keep the (closed) payments in a local outbox, and send them to the server when it comes back online. If there is no way for syncing the local changes to the server to fail, you've essentially created a CRDT (at least the "conflict-free" part). Of course that means you cannot have any server side validation at all (maybe this is what you mean by "owner"?). Because that would mean closed out payments would be rejected by the server and you'd end up in an inconsistent state. A consequence of this is that there could be no balance checking (as an example). Payments in a ledger are inherently a CRDT! The consensus part is interesting. Would you really need it? Wouldn't one device be sufficient to validate a transaction? And would consensus really even mean something within a small building? If the restaurant catches on fire the data would be lost anyway, so if you don't achieve "high-availability", why bother?