4 ms·

I'm really curious as to why RocksDB was so slow at loading data from the Optane. An uneducated guess would be that there's some weird interactions with (f)sync

I'm really curious as to why RocksDB was so slow at loading data from the Optane. An uneducated guess would be that there's some weird interactions with (f)sync () but I'd love to hear anyone else's idea as to why that might be.

hyc_symas 8y ago
I'd really love to know too, but so far we haven't found any explanations. The loader is actually using all asynchronous writes, so there are no fsync() calls during the loading phase at all. The CPU usage is about the same as for the load on flash, so the delay can only be attributable to the actual I/O operations. Perhaps this data set just magically happens to cause a write access pattern that's pathologically bad for Optane SSDs. We re-tested multiple times though and always had the same result - runs fine on flash, slows down on optane. Even LMDB on the raw block device, which immediately did a physical write upon every logical write, wasn't that slow. It's just bizarre.
- mehrdadn 8y ago
  I have zero ideas how any of these work underneath, but out of a wild guess, is there any chance that alignment (or similar) is the issue? e.g. maybe RocksDB tries to align its writes to some boundary in a way that's awful for Optane's technology?
  - wtallis 8y ago
    Alignment sounds like a plausible factor, but in my testing I've found that the penalty for unaligned writes on the Optane P4800X is less than a factor of two.
    - hyc_symas 8y ago
      Hadn't really thought of that. I suppose we could have retried by making a single partition taking up the entire space, and letting fdisk take care of aligning the start of the partition. But I assumed that alignment wouldn't be an issue when using the unpartitioned block device.
- antirez 8y ago
  That's definitely a problem because the same yet to understand pattern may happen later with lmdb if there are the conditions (given that they are not understood) and create a random performance issue.
- hashmash 8y ago
  The LMDB test is a bit contrived. By choosing a key of 16 bytes and a value of 4000 bytes, it's nearly optimal. The underlying block device wants 4KiB aligned writes for best performance, and that's pretty much what this test is doing. RocksDB is best at performing writes into very small records, where it can coalesce them and reduce write amplification. LMDB can't do this. As the size of the records increases, the RocksDB performance advantage disappears, and then it eventually loses out. In my own tests, I've found that the crossover point is about 250 bytes. A better test comparing LMDB to RocksDB would show what this crossover is, in a more controlled fashion than what I have done.
wtallis 8y ago
Write latency can be lower on flash-based SSDs than Optane because the former tend to have DRAM caches, but Optane SSDs have little to no caching. The difference isn't enough to produce that large of a discrepancy, so I suspect that the data being written to the Optane SSD was either not being batched into blocks of the same size as for the flash SSDs, or that fewer operations were being queued simultaneously for the Optane SSDs. I wonder if there were some mkfs or mount options that defaulted to a different value for the Optane SSD. I'm also curious whether the NVMe driver was used in polling mode.
- hyc_symas 8y ago
  Since the loader was performing asynchronous writes, actual batching would be whatever the OS is doing when it eventually flushes dirty pages to the underlying device. I can't imagine the OS using different queue depths during the RocksDB run than the LMDB run. mkfs/mount options were identical for both. We didn't touch the NVMe driver at all, so whatever its default behavior is, was used on all tests. By the way - I think you can still get free access to run your own tests. https://twitter.com/IntelStorage/status/1010284314121129985 https://twitter.com/IntelStorage/status/1010284314121129985 Let me know if you want to try to rerun these, and doublecheck the results. Also, we found that 4.x kernels performed significantly better than 3.x kernels on these tests.
  - wtallis 8y ago
    I already have the hardware needed to run my own tests (albeit only one P4800X), but I've been almost exclusively focusing on consumer SSDs recently. I'll have some time later this month to try to replicate your results. I don't have experience setting up RocksDB or LMDB. Is the process of getting your benchmark running pretty self-explanatory?
    - hyc_symas 8y ago
      All the shell scripts I used are in the data.tgz file linked at the bottom of the page. You should be able to run after editing a few pathnames. Compiling the dbbench drivers should be simple enough; I may be able to send you binaries if not.

hyc_symas 8y ago

I'd really love to know too, but so far we haven't found any explanations. The loader is actually using all asynchronous writes, so there are no fsync() calls during the loading phase at all. The CPU usage is about the same as for the load on flash, so the delay can only be attributable to the actual I/O operations. Perhaps this data set just magically happens to cause a write access pattern that's pathologically bad for Optane SSDs. We re-tested multiple times though and always had the same result - runs fine on flash, slows down on optane. Even LMDB on the raw block device, which immediately did a physical write upon every logical write, wasn't that slow. It's just bizarre.

mehrdadn 8y ago
I have zero ideas how any of these work underneath, but out of a wild guess, is there any chance that alignment (or similar) is the issue? e.g. maybe RocksDB tries to align its writes to some boundary in a way that's awful for Optane's technology?
- wtallis 8y ago
  Alignment sounds like a plausible factor, but in my testing I've found that the penalty for unaligned writes on the Optane P4800X is less than a factor of two.
  - hyc_symas 8y ago
    Hadn't really thought of that. I suppose we could have retried by making a single partition taking up the entire space, and letting fdisk take care of aligning the start of the partition. But I assumed that alignment wouldn't be an issue when using the unpartitioned block device.
antirez 8y ago
That's definitely a problem because the same yet to understand pattern may happen later with lmdb if there are the conditions (given that they are not understood) and create a random performance issue.
hashmash 8y ago
The LMDB test is a bit contrived. By choosing a key of 16 bytes and a value of 4000 bytes, it's nearly optimal. The underlying block device wants 4KiB aligned writes for best performance, and that's pretty much what this test is doing. RocksDB is best at performing writes into very small records, where it can coalesce them and reduce write amplification. LMDB can't do this. As the size of the records increases, the RocksDB performance advantage disappears, and then it eventually loses out. In my own tests, I've found that the crossover point is about 250 bytes. A better test comparing LMDB to RocksDB would show what this crossover is, in a more controlled fashion than what I have done.

wtallis 8y ago

Write latency can be lower on flash-based SSDs than Optane because the former tend to have DRAM caches, but Optane SSDs have little to no caching. The difference isn't enough to produce that large of a discrepancy, so I suspect that the data being written to the Optane SSD was either not being batched into blocks of the same size as for the flash SSDs, or that fewer operations were being queued simultaneously for the Optane SSDs. I wonder if there were some mkfs or mount options that defaulted to a different value for the Optane SSD. I'm also curious whether the NVMe driver was used in polling mode.

hyc_symas 8y ago
Since the loader was performing asynchronous writes, actual batching would be whatever the OS is doing when it eventually flushes dirty pages to the underlying device. I can't imagine the OS using different queue depths during the RocksDB run than the LMDB run. mkfs/mount options were identical for both. We didn't touch the NVMe driver at all, so whatever its default behavior is, was used on all tests. By the way - I think you can still get free access to run your own tests. https://twitter.com/IntelStorage/status/1010284314121129985 https://twitter.com/IntelStorage/status/1010284314121129985 Let me know if you want to try to rerun these, and doublecheck the results. Also, we found that 4.x kernels performed significantly better than 3.x kernels on these tests.
- wtallis 8y ago
  I already have the hardware needed to run my own tests (albeit only one P4800X), but I've been almost exclusively focusing on consumer SSDs recently. I'll have some time later this month to try to replicate your results. I don't have experience setting up RocksDB or LMDB. Is the process of getting your benchmark running pretty self-explanatory?
  - hyc_symas 8y ago
    All the shell scripts I used are in the data.tgz file linked at the bottom of the page. You should be able to run after editing a few pathnames. Compiling the dbbench drivers should be simple enough; I may be able to send you binaries if not.