4 ms·

I'm only familiar with other key-value storage engines, not FoundationDB, but it seems like the goals are: "distributed key-value database, read latencies below

I'm only familiar with other key-value storage engines, not FoundationDB, but it seems like the goals are: "distributed key-value database, read latencies below 500 microseconds, ACID, scalability". I remember evaluating a few low latency key-value storage solutions, and one of these was Stanford's RAMCloud, which is supposed to give 4-5 microseconds reads, 15 microseconds writes, scale up to 10,000 boxes and provide data durability. https://ramcloud.atlassian.net/wiki/display/RAM/RAMCloud https://ramcloud.atlassian.net/wiki/display/RAM/RAMCloud Seems like, that would be "Databases at 2000Mhz". I've actually studied the code that was handling the network and it had been written pretty nicely, and as far as I know, it should work both over 10Gbe and Infiniband with similar latencies. And I'm not at all surprised, they could get pretty clean looking 4-5us latency distribution, with the code like that. How does it compare with FoundationDB? Is it completely different technology?

jrallison 12y ago
Don't know much about RAMCloud, but from the description: "Many more issues remain, such as whether we can provide higher-level features such as secondary indexes and multiple-object transactions without sacrificing the latency or scalability of the system. We are currently exploring several of these issues." Sounds it doesn't provide multi-key ACID transactions at the very least.
- dchichkov 12y ago
  Yes, It looks like it doesn't support multi-key ACID transactions. As per: https://ramcloud.atlassian.net/wiki/display/RAM/Deciding+Whether+to+Use+RAMCloud https://ramcloud.atlassian.net/wiki/display/RAM/Deciding+Whe... > If your application requires the higher-level data model features of a relational database, such as secondary indexes and transactions spanning multiple objects, then it may be difficult to get that application running on RAMCloud. The current RAMCloud data model is a key-value store with a few extension (such as conditional writes, table enumeration, and multi-read/multi-write operations); it does not yet support secondary indexes and multi-object transactions. Most of these operations can be implemented on top of the existing RAMCloud features, but it will take extra work and these operations will not run as fast as the built-in RAMCloud operations. We are currently working on adding higher-level features to RAMCloud. Still. Aside from that, it is free, and open source and gives hundred times better latency?
  - misframer 12y ago
    > Still. Aside from that, it is free, and open source and gives hundred times better latency? It doesn't have the same guarantees. If you need those guarantees, then there's no comparison and it doesn't make sense to compare latencies.
    - dchichkov 12y ago
      I'm not so sure. Have the latencies claimed by FoundationDB been measured on multi-key transactions? And if you need these guaranties, it seems like "Most of these operations can be implemented on top of the existing RAMCloud features". I would expect it to be possible to implement that and stay well withing the latency budget, provided that a single-key durable write is 100 times faster.
      - nlavezzo 12y ago
        All of the transactions in the test noted in the blog are multi-key transactions. (20 key updates)
        dchichkov 12y ago
        When claiming the 14.4MHz number, is that per 20-key update? Or a single key update in the 20-key update? And what's the latency on the 20-key update?
        itp 12y ago
        The writes we are measuring are individual keys, modified atomically in transactions of 20 keys at a time. So this test is doing 14 million writes per second as 720 thousand transactions per second.
      - jrallison 12y ago
        Multi-key transactions means coordination. Coordination is not cheap. You'll be paying a hefty cost somewhere.
imaginenore 12y ago
But it doesn't store the data on disk. It should be compared to Memcached, not to a real storage engine.
- dchichkov 12y ago
  Unlike Memcached or Redis, RAMCloud provides durability: * Automatic replication, crash recovery, and fail-over (no loss of availability if a server fails) * Durability guarantee: data is always replicated and durable before operations return, without significant performance hit (subject to the requirement for persistent buffers on backups).
  - imaginenore 12y ago
    So? Power goes out in your data center, all the data is lost. Nothing stops you from replicating memcached.
  - grogers 12y ago
    You can't make a durable write to disk in 15us. SSDs are around 2 orders of magnitude slower than that. 15us is about right for in memory on two nodes in a single data center though. Most people wouldn't consider that durable though.

jrallison 12y ago

Don't know much about RAMCloud, but from the description: "Many more issues remain, such as whether we can provide higher-level features such as secondary indexes and multiple-object transactions without sacrificing the latency or scalability of the system. We are currently exploring several of these issues." Sounds it doesn't provide multi-key ACID transactions at the very least.

dchichkov 12y ago
Yes, It looks like it doesn't support multi-key ACID transactions. As per: https://ramcloud.atlassian.net/wiki/display/RAM/Deciding+Whether+to+Use+RAMCloud https://ramcloud.atlassian.net/wiki/display/RAM/Deciding+Whe... > If your application requires the higher-level data model features of a relational database, such as secondary indexes and transactions spanning multiple objects, then it may be difficult to get that application running on RAMCloud. The current RAMCloud data model is a key-value store with a few extension (such as conditional writes, table enumeration, and multi-read/multi-write operations); it does not yet support secondary indexes and multi-object transactions. Most of these operations can be implemented on top of the existing RAMCloud features, but it will take extra work and these operations will not run as fast as the built-in RAMCloud operations. We are currently working on adding higher-level features to RAMCloud. Still. Aside from that, it is free, and open source and gives hundred times better latency?
- misframer 12y ago
  > Still. Aside from that, it is free, and open source and gives hundred times better latency? It doesn't have the same guarantees. If you need those guarantees, then there's no comparison and it doesn't make sense to compare latencies.
  - dchichkov 12y ago
    I'm not so sure. Have the latencies claimed by FoundationDB been measured on multi-key transactions? And if you need these guaranties, it seems like "Most of these operations can be implemented on top of the existing RAMCloud features". I would expect it to be possible to implement that and stay well withing the latency budget, provided that a single-key durable write is 100 times faster.
    - nlavezzo 12y ago
      All of the transactions in the test noted in the blog are multi-key transactions. (20 key updates)
      - dchichkov 12y ago
        When claiming the 14.4MHz number, is that per 20-key update? Or a single key update in the 20-key update? And what's the latency on the 20-key update?
        itp 12y ago
        The writes we are measuring are individual keys, modified atomically in transactions of 20 keys at a time. So this test is doing 14 million writes per second as 720 thousand transactions per second.
    - jrallison 12y ago
      Multi-key transactions means coordination. Coordination is not cheap. You'll be paying a hefty cost somewhere.

imaginenore 12y ago

But it doesn't store the data on disk. It should be compared to Memcached, not to a real storage engine.

dchichkov 12y ago
Unlike Memcached or Redis, RAMCloud provides durability: * Automatic replication, crash recovery, and fail-over (no loss of availability if a server fails) * Durability guarantee: data is always replicated and durable before operations return, without significant performance hit (subject to the requirement for persistent buffers on backups).
- imaginenore 12y ago
  So? Power goes out in your data center, all the data is lost. Nothing stops you from replicating memcached.
- grogers 12y ago
  You can't make a durable write to disk in 15us. SSDs are around 2 orders of magnitude slower than that. 15us is about right for in memory on two nodes in a single data center though. Most people wouldn't consider that durable though.