7 ms·
Things I have learned about SSDs: If you want to go fast & save NAND lifetime, use append-only log structures. If you want to go even faster & save even more
by bob1029 5y ago
Things I have learned about SSDs:
If you want to go fast & save NAND lifetime, use append-only log structures.
If you want to go even faster & save even more NAND lifetime, batch your writes in software (i.e. some ring buffer with natural back-pressure mechanism) and then serialize them with a single writer into an append-only log structure. Many newer devices have something like this at the hardware level, but your block size is still a constraint when working in hardware. If you batch in software, you can hypothetically write multiple logical business transactions per block I/O. When you physical block size is 4k and your logical transactions are averaging 512b of data, you would be leaving a lot of throughput on the table.
Going down 1 level of abstraction seems important if you want to extract the most performance from an SSD. Unsurprisingly, the above ideas also make ordinary magnetic disk drives more performant & potentially last longer.
- pclmulqdq 5y agoI used to think the same thing, but now that I work on SSD-based storage systems, I'm not sure this holds up in today's storage stacks. Log structuring really helped with HDDs since it meant fewer seeks. In particular, the filesystem tends to undo a lot of the benefits you get from log-structuring unless you are using a filesystem designed to keep your files log-structured. Using huge writes definitely still helps, though. A paper that I really like goes deeper into this: http://pages.cs.wisc.edu/~jhe/eurosys17-he.pdf http://pages.cs.wisc.edu/~jhe/eurosys17-he.pdf Edit: I had originally said "designed for flash" instead of "designed to keep your files log-structured." F2FS is designed for flash, but in my testing does relatively poorly with log-structured files because of how it works internally. Edit 2: de-googled the link. Thank you for pointing that out.
- trulyme 5y agoDegoogled link: http://pages.cs.wisc.edu/~jhe/eurosys17-he.pdf http://pages.cs.wisc.edu/~jhe/eurosys17-he.pdf
- 10000truths 5y agoAchieving cutting-edge storage performance tends to require bypassing the filesystem anyways. Traditionally, that meant using SPDK. Nowadays, opening /dev/nvme* with O_DIRECT and operating on it with io_uring will get you most of the way there. In either case, the advice given in the article and by the OP is filesystem agnostic.
- quotemstr 5y agoWhy would you want to bypass the filesystem by talking to the block device directly? Doesn't O_DIRECT on a preallocated regular file accomplish the same thing with less management complexity and special OS permissions? Granted, the file extents might be fragmented a bit, but that can be fixed.
- lazide 5y agoIf you understand your workload and the hardware well enough to understand how doing direct I/O on a file will help - then you’re going to generally do better against a direct block device because there are fewer intermediate layers doing the wrong optimizations or otherwise messing you up. From a pure performance perspective anyway. Extents are one part of the issue, flushes to disk (and how/when they happen), caching, etc. Doesn’t mean it isn’t easier to deal with as a file from an administration perspective (and you can do snapshots, or whatever!), but Lvm can do that too for a block device, and many other things.
- quotemstr 5y agoWith O_DIRECT though you're opting out of the filesystem's caching (well, VFS's), forced flushes, and most FS level optimizations, so I'd expect it to perform on par with direct partition access. Do you have numbers showing an advantage of going directly to the block device? Personally, I'd consider the management advantages of a filesystem compelling absent specific performance numbers showing the benefit of direct partition access.
- lazide 5y ago
- remram 5y agoShouldn't the OS or libc take care of that? If I write and don't immediately flush()?
- KMag 5y agoI don't think most libc implementations take care to buffer to filesystem block/cluster boundaries.
- scns 5y agoLike this? https://en.wikipedia.org/wiki/NILFS?wprov=sfla1 https://en.wikipedia.org/wiki/NILFS?wprov=sfla1
- AtlasBarfed 5y agoThis is basically the purpose of rocksdb, and to a lesser extent Cassandra
- gravypod 5y agoThis is the "secret sauce" behind LevelDB: https://github.com/google/leveldb#performance https://github.com/google/leveldb#performance
- bob1029 5y agoThis looks to be a similar technique. In my testing of these ideas, I've been able to push over 2 million transactions per second (~1Kb per transaction) to a Samsung 960 Pro. For reference, its rated for 2.1GB/s sequential writes, so I've got it pretty much 100% saturated. The implementation for something like this is actually really underwhelming when you figure out how to put all the pieces together. I assembled this prototype (also a key-value store) using .NET5, LMAX Disruptor, and a splay tree implementation i copied from google somewhere. The hardest part was figuring out how to wait for write completion on the caller side (multiple calling threads are ultimately serialized into a single worker thread via the Disruptor). Turns out, busy wait for a few thousand cycles followed by a yield to the OS is a pretty good trick. You just do a while(true) over a completion flag on the transaction object which is set en masse by the handling thread after the write goes to disk. Batch sizes are determined dynamically based on how long the previous batch took to write. In practice, I never observed a batch that took longer than 2-3 milliseconds on my 960 pro. Max batch size is 4096, and it is permanently full when 100% loaded. A full batch = a nice big IO to disk.
- ww520 5y agoLMDB has similar write characteristics where its b-tree is append-only. This gives LMDB amazing performance and very robust ACID transaction support as immutability is baked in.
- fulafel 5y agoThis is quite common in traditional DBs too. Eg PostgreSQL has its write-ahead log. Both LMDB and PostgreSQL then occasionally need to do do some kind of compaction, checkpoint or garbage collection, whatever it's called in various systems, the write-only log is reset and any live data in it improted into the main db data.
- ww520 5y agoI only have a cursory knowledge on LMDB (listening to a podcast while biking). Anyway, LMDB has no transaction log nor write ahead log. There's no overwrite during update. Data page update is copy-on-write and b+tree index update is append only. The update on the b+tree pages is performed from the bottom of the tree to the root, linking newly appended pages to higher level pages. The transaction is committed when the new root page is appended. When there's a crash, the incomplete appended index pages have not been linked up to the root page yet and are not reachable from the previous valid root page. They can be just thrown away. Recovery just means searching for the last valid root index page. There's no need for a WAL and undo/redo of the transaction log. Deleted pages and obsolete pages are actively put back into a free list (tracked by another b+tree), which will be reused for new page allocation. This avoids the long garbage collection phase to walk all the live pages for compaction (no vacuum is needed).
- fulafel 5y agoThanks for the explanation. Clever stuff, LMDB is taking advantage of not having to support multiple writers here.
- hyc_symas 5y agoNo. LMDB is copy-on-write, with double buffering/shadow pages for the root page updates. No searching for the last valid root page. Looks like you have the other details right.
- hypertele-Xii 5y agoBuffering writes is fine if you're ok with losing your data. For some applications that's acceptable, but when I'm writing to disk, it's because I want persistence. "It'll get flushed to disk at some point as long as power doesn't go out" is hardly that.
- senderista 5y agoAlso: parallelize your writes. This is the biggest difference between SSDs and HDDs: internal parallelism. You’ll have a hard tine saturating I/O bandwidth even with huge sequential writes if you don’t introduce some parallelism. Fortunately, io_uring makes this easy from a single thread.