3 ms·
This looks potentially fantastic. If I could beg one wish from the developers of this (and almost every other project anywhere near this space), though, it wou
by heavenlyhash 10y ago
This looks potentially fantastic. If I could beg one wish from the developers of this (and almost every other project anywhere near this space), though, it would be one tiny piece of documentation:
What's your unique ID scheme?
Let's say I'm willing to believe[1] that you've got Durable and Consistent down, once messages make it committed in to the system. What's the story for messages on their way in? My application logs are buffered to the local disk, now I'm streaming them into central storage, and halfway through a TCP connection that's shuffled 2mb of thousands of messages into storage, the connection terminates -- unexpectedly, midmessage. Could the service have committed more messages than it acknowledged? Or many less than I've sent? Both could be true from the network standpoint.[2]
So, what I need to know, and what should be very easy to answer, front-and-center in your docs, pretty please:
1) Where should my log uploader resume?
2) Is there any danger of repeatedly entering some lines?
3) If I have log lines that are legitimately duplicates, will they be stored at the correct count?
These are questions that may have a different answer than the durability after data makes it fully into the system. It also may provide useful information about how complexity the code in a submitting client is, because good answers tend to require some kind of ID sequence being assigned on submitting clients, afaict. And it's really just plain critical to sanity.
----
[1] well, no, I'm not, "trust by verify" in all things etc etc; but let's suppose that's more believable and something I have to mechanically verify anyway, and doesn't have an obviously observable boolean at the protocol level as to whether it's going to work well or not, and system internals simply don't have such a sordid history of being over-simplified until they're broken like client interfaces so often are, so...! We'll handwave that to a later and more involved step of quality investigation.
[2] https://en.wikipedia.org/wiki/Two_Generals'_Problem https://en.wikipedia.org/wiki/Two_Generals'_Problem
- chillacy 10y agoI stumbled upon this page while reading the docs: https://twitter.github.io/distributedlog/html/design/main.html#consistency https://twitter.github.io/distributedlog/html/design/main.ht... It looks like they use fencing and a two-phase commit to prevent duplicate writes. Whether that covers all failure scenarios I'm not sure.
- sandspit 10y ago> Could the service have committed more messages than it acknowledged? Yes of course. It may have committed records but not had time to to ACK success. > Or many less than I've sent? Yes again. However it will NEVER ACK success for writes which it hasn't committed. So the persistence story is pretty straightforward (I'll see if we can update docs if this is missing). If you want to avoid duplication on the write side, you would have to retrieve the last log record-- with id x-- and start adding records again from id x+1. Fencing + locking combine to provide efficient exclusive access and the fat client (for now) guarantees write ordering.