6 ms·
How does LogDevice differ from Kafka?
by ashu 8y ago
How does LogDevice differ from Kafka?
- mdasen 8y agoI’m wondering this as well. In the description it says that it ensures total ordering while Kafka only ensures partition ordering. I haven’t read enough to say more.
- manigandham 8y agoKafka brokers handle both the computation (partition/topic management, sequencing, assignments, etc) and storage together. This coupling creates scaling and operational challenges which LogDevice removes by separating the layers. Storage nodes can be as simple as object stores (but optimized for appending files) and use multiple non-deterministic locations for a given piece of data to randomize placement. They read, write and recover data very quickly by working together in a mesh. Meanwhile the compute layer becomes very lightweight and almost stateless, which is easy to scale. In LogDevice, the Sequencers are potential bottlenecks but generating a series of incrementing numbers is about the fastest thing you can do so it'll outpace any actual data ingest to a single log, while giving you a total order of all entries within that log. The numbers (LSNs) follow the Hi/Lo sequence pattern so if a Sequencer fails, another one takes its place with a greater "High" number, so it's guaranteed that all of its LSNs will be greater than the previous Sequencer as a result. This also provides a built-in buffer to still accept messages and assign the permanent LSNs to them after recovery in case a Sequencer fails. Apache Pulsar is similar to LogDevice but goes further where brokers manage connections, routing and message acknowledgements while data is sent to a separate layer of Apache Bookkeeper nodes which store the data in append-optimized log files.
- Rapzid 8y agoInteresting. Microsoft's Tango paper had some interesting things to say about sequencers/sequences as well.
- StreamBright 8y agoAmazing! We had lots of operational issues because of the coupling you mentioned. One question though: will Presto support querying from LogDevice directly? :)
- pdpi 8y agoScribe is the Facebook-internal Kafka equivalent. LogDevice is the storage layer used by Scribe. Scribe isn’t the only place where LogDevice is used though — Facebook has documented using it for TAO as well (as part of the secondary indices)
- saryant 8y agoI don't believe Scribe and Kafka are equivalent. Isn't Scribe at-most-once? Unless we're talking about two different projects named Scribe, which is certainly possible.
- lclarkmichalek 8y agoScribe doesn't really make decisions about that, it doesn't store checkpoints for readers. Readers are commonly more-than-once.
- pdpi 8y ago> Scribe doesn't really make decisions about that It kind of does, on the write pipeline. Tailers vary, but scribed controls the semantics of how your message gets delivered to LogDevice.
- pdpi 8y agoAs lclarkmichalek said, there’s more than one way to skin a cat. At any rate, I used “equivalent” here to mean that, while different trade-offs have been made, it has the sort of users building the same sort of applications on the same sort of abstractions — it plays the same role, for all intents and purposes.
- martincmartin 8y agoI worked on LogDevice at FB until about 6 months ago. I'm not that familiar with Kafka, but in general LogDevice emphasizes write availability over read availability. There are many applications where data is being generated all the time, and if you don't write it, it will be lost. However, if reading is delayed, it just means readers are a little behind and will need to catch up. So, when a sequencer node dies and we need to figure out what happened to the records that were in flight -- which ones ended up on disk & can be replicated, what the last record was -- LogDevice still accepts new writes. However, to ensure ordering, these new writes aren't visible to readers until the earlier writes are sorted out.
- ivankelly 8y agoWhat happens with a sequencer which appears to fail but hasn't really, and then comes back up after the process to figure out inflight records has completed? If that sequencer receives a record for a log, will it be able to write it to the storage nodes? I.e. is there any fencing mechanism to tell the storage nodes that the epoch has been bumped, so don't access writes for that epoch anymore?
- sh00s 8y agoyes, in LogDevice it's called "sealing". However, as it stands, a newly activated sequencer won't wait for sealing on the old epoch to complete before taking new writes - in the tradeoff between write availability and consistency LogDevice picks higher availability. Blocking new writes until sealing is complete, however, should be fairly easy to integrate into LogDevice as an option.