5 ms·

If this is the plugging subthread: gazette is another project in this space which plays particularly well with batch workflows (https://gazette.dev https://gaze

If this is the plugging subthread: gazette is another project in this space which plays particularly well with batch workflows (https://gazette.dev https://gazette.dev). It’s streaming consumer framework is Go only today, however.

ryanworl 7y ago
It is interesting that you made the decision to commit to all replicas of a journal before acknowledging a write and rely on etcd for explicit failure monitoring and re-configuration. FoundationDB also does this, relying on an internal metadata store called the "coordinators", which store some small configuration state. Have you ever found the 20s failure detection window to be too long? FoundationDB's failure monitor pings hosts approximately once a second, so failures are detected very quickly. If you were willing to dedicate a process in the cluster to failure monitoring, you might be able to shrink that window.
- jgraettinger1 7y ago
  What you describe is how Etcd leases work, just inverted: clients instead check in with the Etcd leader at prescribed intervals or face lease revocation. The leader is doing some add'l clever things like scaling check-in intervals to not DoS itself. 20s is configurable per-client, and gazette lifts this default from etcd. This one number is trying to summarize the tension and trade-off inherent in the (famously hard) problem of distributed failure detection. Network flakes or partitions of a few seconds are all too common and you don't necessarily want to assume failure if one occurs (particularly given the compounding effects of resend timers, etc). In practice, no, it's not been a problem since clients typically buffer writes anyway, in order to batch many small writes into few larger ones.
  - ryanworl 7y ago
    I didn’t mean to imply they were different approaches (they aren’t really), only that you could offload the job from etcd to somewhere else to rely on it less, which would hopefully help scalability or allow you to lower the ping time. And yes, FDB separates the act of detecting a failure from reacting to a detected failure for that reason. In FDB’s case, a 20s network hiccup on a specific process could mean it is better to initiate a recovery anyway. I think for your use case that trade off makes sense as there are lots of journals, whereas FDB only runs a single database for clients to access.
pm90 7y ago
Interesting. How does the system recover from corrupt "journals"? Do you have to rely on e.g. the blob stores' data recovery function to do a restore... is that even possible? (since the journal sounds like its immutable)
- jgraettinger1 7y ago
  By "corruption" do you mean "whoopsie, I wrote bad data"? Same way as always - there's some kind of delimiter in the message serialization, and you can "re-sync" by finding that delimiter and hoping to continue reading valid messages. For example, a bad CSV stream can be recovered by finding the next newline and hoping you read correct CSV again. The problem is just moved to the client library which is layering a notion of messages atop journals, and is not a direct broker concern.
  - pm90 7y ago
    Yep, thanks. Reading more about it, the "immutability" of the append log seems like a convention rather than something that is enforced programmatically (i.e. you can always write over the persisted fragments since they're objects in blob storage), which is what I was concerned about.

ryanworl 7y ago

It is interesting that you made the decision to commit to all replicas of a journal before acknowledging a write and rely on etcd for explicit failure monitoring and re-configuration. FoundationDB also does this, relying on an internal metadata store called the "coordinators", which store some small configuration state. Have you ever found the 20s failure detection window to be too long? FoundationDB's failure monitor pings hosts approximately once a second, so failures are detected very quickly. If you were willing to dedicate a process in the cluster to failure monitoring, you might be able to shrink that window.

jgraettinger1 7y ago
What you describe is how Etcd leases work, just inverted: clients instead check in with the Etcd leader at prescribed intervals or face lease revocation. The leader is doing some add'l clever things like scaling check-in intervals to not DoS itself. 20s is configurable per-client, and gazette lifts this default from etcd. This one number is trying to summarize the tension and trade-off inherent in the (famously hard) problem of distributed failure detection. Network flakes or partitions of a few seconds are all too common and you don't necessarily want to assume failure if one occurs (particularly given the compounding effects of resend timers, etc). In practice, no, it's not been a problem since clients typically buffer writes anyway, in order to batch many small writes into few larger ones.
- ryanworl 7y ago
  I didn’t mean to imply they were different approaches (they aren’t really), only that you could offload the job from etcd to somewhere else to rely on it less, which would hopefully help scalability or allow you to lower the ping time. And yes, FDB separates the act of detecting a failure from reacting to a detected failure for that reason. In FDB’s case, a 20s network hiccup on a specific process could mean it is better to initiate a recovery anyway. I think for your use case that trade off makes sense as there are lots of journals, whereas FDB only runs a single database for clients to access.

pm90 7y ago

Interesting. How does the system recover from corrupt "journals"? Do you have to rely on e.g. the blob stores' data recovery function to do a restore... is that even possible? (since the journal sounds like its immutable)

jgraettinger1 7y ago
By "corruption" do you mean "whoopsie, I wrote bad data"? Same way as always - there's some kind of delimiter in the message serialization, and you can "re-sync" by finding that delimiter and hoping to continue reading valid messages. For example, a bad CSV stream can be recovered by finding the next newline and hoping you read correct CSV again. The problem is just moved to the client library which is layering a notion of messages atop journals, and is not a direct broker concern.
- pm90 7y ago
  Yep, thanks. Reading more about it, the "immutability" of the append log seems like a convention rather than something that is enforced programmatically (i.e. you can always write over the persisted fragments since they're objects in blob storage), which is what I was concerned about.