3 ms·
Won't the proposed implementation risk doing duplicate work? e.g. another machine starts on your work, but then you ask for it before it's done. So you'd proba
by ColonelPhantom 3y ago
Won't the proposed implementation risk doing duplicate work? e.g. another machine starts on your work, but then you ask for it before it's done.
So you'd probably need the "not available, but already 'stolen' case". For example, the current machine could look for work to steal itself. But this probably will increase latency a lot, so it would only make sense if the batches are huge.
- lunarcave 3y agoIt's really hard to have exactly-once delivery[1]. But exactly once processing is viable. With Differential, the implementation does allow for idempotency in cases where you need exactly-once processing. And it does handle the "not available, but already 'stolen' case". Disclaimer: I'm an author. [1] https://news.ycombinator.com/item?id=34986691 https://news.ycombinator.com/item?id=34986691
- JonChesterfield 3y agoThe sketch above makes duplicated work and inefficiency very likely. It's explicitly choosing at-least-once execution. The problem with the "already scheduled" pattern is when the machine holding the task falls over. Not a good choice for bank transactions. A good option for something like a distributed build farm - it doesn't matter hugely if you compile some piece of program repeatedly, but it matters a lot if you send the only copy of some source to a machine that immediately goes offline. Mainly bringing it up as a case in which RPC works beautifully - remote machine failures can be completely hidden by a willingness to do the work locally instead.