3 ms·

Reading the blog its not clear to me how they deal with gaps in the LSN sequence. The scalability & performance properties derive from 1) Using a separate seque

by jbit84 9y ago

Reading the blog its not clear to me how they deal with gaps in the LSN sequence. The scalability & performance properties derive from 1) Using a separate sequencer that issues increasing sequence numbers, 2) Uncoordinated distributed writes of actual record value to storage nodes, 3) reconstitution of ordered log at consumer side. How does a consumer that have retrieved N and N+2 know if N+1 is not yet written, or if it failed and will never be written? Perhaps they write-through the sequencer with subsequent writes waiting on acknowledgements, so 'gaps' only occur on epoch changes?

martincmartin 9y ago
Hi, I work on the LogDevice team at Facebook. Within an epoch, the sequencer is a single process on a single machine and gives out LSNs sequentially. When a sequencer dies, and a new one is started, its first job is to fix up the end of the last epoch. If it can't find any copies of a given record, it inserts a "hole plug," to store the fact that the record is lost. So, except for hole plugs (which should be very rare), the only gaps are between epochs as you say. Suppose you have 10 LogDevice servers, and you store 3 copies of every record. Then the client will have connections to all 10 machines, and each machine will push whatever records it has to the client. Crucially, the servers always push records in order. So if a client gets record N from machine 7, then gets record N+2, it can be sure that machine 7 doesn't have a copy of record N+1. Once you've got record N+2 or higher from at least 8 machines, without getting record N+1 from any of them, you can be sure that at least one copy of the record has been lost. If those 8 servers have complete information, you can report to the user that the record has been lost.
- jbit84 9y ago
  Hi, thank you for the answer. Can you comment on how writes to the LogDevice servers are performed - particularly if the writes have to go through the sequencer, or if the individual producers can write directly after obtaining a LSN? Since a given LogDevice server will only receive a non-deterministic sub-sequence of records, I would think it has to receive its set of writes in order? (To have enough information to push to clients in order). Unless there is actually some mechanism by which it can determine the elements it should receive. Trying to understand if your per-log throughput will be capped by the max traffic that can be pushed through a single sequencer, or the round-trip latency for waiting for acknowledged writes from LogDevice servers.
  - martincmartin 9y ago
    Trying to understand if your per-log throughput will be capped by the max traffic that can be pushed through a single sequencer, Yes. You can think of an individual log like a database shard. We have plans to allow the sequencer to just give out sequence numbers, and allow the clients to send the data directly to the storage nodes. But its not a high priority for us, since it's just a constant factor improvement. (Although a large one!) Users would still need to shard their data, although not as much.

by jbit84 9y ago

martincmartin 9y ago

Hi, I work on the LogDevice team at Facebook. Within an epoch, the sequencer is a single process on a single machine and gives out LSNs sequentially. When a sequencer dies, and a new one is started, its first job is to fix up the end of the last epoch. If it can't find any copies of a given record, it inserts a "hole plug," to store the fact that the record is lost. So, except for hole plugs (which should be very rare), the only gaps are between epochs as you say. Suppose you have 10 LogDevice servers, and you store 3 copies of every record. Then the client will have connections to all 10 machines, and each machine will push whatever records it has to the client. Crucially, the servers always push records in order. So if a client gets record N from machine 7, then gets record N+2, it can be sure that machine 7 doesn't have a copy of record N+1. Once you've got record N+2 or higher from at least 8 machines, without getting record N+1 from any of them, you can be sure that at least one copy of the record has been lost. If those 8 servers have complete information, you can report to the user that the record has been lost.

jbit84 9y ago
Hi, thank you for the answer. Can you comment on how writes to the LogDevice servers are performed - particularly if the writes have to go through the sequencer, or if the individual producers can write directly after obtaining a LSN? Since a given LogDevice server will only receive a non-deterministic sub-sequence of records, I would think it has to receive its set of writes in order? (To have enough information to push to clients in order). Unless there is actually some mechanism by which it can determine the elements it should receive. Trying to understand if your per-log throughput will be capped by the max traffic that can be pushed through a single sequencer, or the round-trip latency for waiting for acknowledged writes from LogDevice servers.
- martincmartin 9y ago
  Trying to understand if your per-log throughput will be capped by the max traffic that can be pushed through a single sequencer, Yes. You can think of an individual log like a database shard. We have plans to allow the sequencer to just give out sequence numbers, and allow the clients to send the data directly to the storage nodes. But its not a high priority for us, since it's just a constant factor improvement. (Although a large one!) Users would still need to shard their data, although not as much.