3 ms·

One big global write lock in LMDB maps pretty well to one single stream of replicated log entries in Raft, IMO. And logical address space is still far in exces

by jbooth 12y ago

One big global write lock in LMDB maps pretty well to one single stream of replicated log entries in Raft, IMO. And logical address space is still far in excess of what most disks or arrays can fit, right? 40 bits or so on linux? EDIT: 47 bits, for 128TB -- http://stackoverflow.com/questions/2159456/whats-the-max-file-mapping-size-in-64bits-machine http://stackoverflow.com/questions/2159456/whats-the-max-fil...

peterwwillis 12y ago
I don't see how? One global write lock means a single instance can't update multiple ranges at a time, so determining consensus and writing from multiple peers would just take a long time for no reason. The whole point of an SSI MVCC is to get around difficult locks.... If Moore's law holds, a single SSD will outgrow the address space in around 7 years. In four years, an array of eight disks would outgrow the address space. This is just for a single server. If you want a linearly-scaling, robust solution for future requirements (like multi-petabyte and exabyte distributed datastores), there's no reason to lock yourself into technology that'll be obsolete in half a decade. (edit: SanDisk says it may release 8TB SSDs next year, also adding "We see reaching the 4TB mark as really just the beginning and expect to continue doubling the capacity every year or two, far outpacing the growth for traditional HDDs")
- jbooth 12y ago
  I'm not on the inside of cockroachDB's raft implementation, but typically you've got a single thread processing AppendEntries requests in a defined order, exactly one at a time, to guarantee the same order of execution on every node. There might be some small savings from doing a couple of updates concurrently here and there but your overall flow should be single threaded. As far as the address space and big SSDs thing.. I'd be willing to gamble on linux supporting mmap up to the biggest devices on the market, one way or another. Heck, there's only 16 more bits after that 47 before every FS under VFS has to be rewritten, right?
- t1m 12y ago
  IIRC, current x86-64 chips are limited to 48bits virtual address to simplify the address translation logic (cheaper to manufacture). This makes sense for the current generation of storage sub-systems, though it would be misleading to say using memory map technology will be "obsolete in half a decade". The 48 bit limit is arbitrary. Manufacturers have 56 bit designs on the table right now, and there is nothing stopping them from implementing full 64 bit virtual address support.

by jbooth 12y ago

peterwwillis 12y ago

I don't see how? One global write lock means a single instance can't update multiple ranges at a time, so determining consensus and writing from multiple peers would just take a long time for no reason. The whole point of an SSI MVCC is to get around difficult locks.... If Moore's law holds, a single SSD will outgrow the address space in around 7 years. In four years, an array of eight disks would outgrow the address space. This is just for a single server. If you want a linearly-scaling, robust solution for future requirements (like multi-petabyte and exabyte distributed datastores), there's no reason to lock yourself into technology that'll be obsolete in half a decade. (edit: SanDisk says it may release 8TB SSDs next year, also adding "We see reaching the 4TB mark as really just the beginning and expect to continue doubling the capacity every year or two, far outpacing the growth for traditional HDDs")

jbooth 12y ago
I'm not on the inside of cockroachDB's raft implementation, but typically you've got a single thread processing AppendEntries requests in a defined order, exactly one at a time, to guarantee the same order of execution on every node. There might be some small savings from doing a couple of updates concurrently here and there but your overall flow should be single threaded. As far as the address space and big SSDs thing.. I'd be willing to gamble on linux supporting mmap up to the biggest devices on the market, one way or another. Heck, there's only 16 more bits after that 47 before every FS under VFS has to be rewritten, right?
t1m 12y ago
IIRC, current x86-64 chips are limited to 48bits virtual address to simplify the address translation logic (cheaper to manufacture). This makes sense for the current generation of storage sub-systems, though it would be misleading to say using memory map technology will be "obsolete in half a decade". The 48 bit limit is arbitrary. Manufacturers have 56 bit designs on the table right now, and there is nothing stopping them from implementing full 64 bit virtual address support.