8 ms·

When are SSDs slow?

133 points by eatonphil 2y ago

vermaden 2y ago
Sometimes its the LEAST expected thing (that one this article does not cover). Some time ago I had some really cheap SATA 2.5 SSD 256 GB in my laptop. If generally worked well - system (FreeBSD) was responsive but ... when I deleted large file (like 1GB+ size) then the system almost FROZE ... until file is deleted. After some investigation with Allan Jude (thanks again) we came to the conclusion that a single TRIM operation (for delete) take about 500ms (half a second) ... By default ZFS on FreeBSD uses up to 64 TRIM operations PER SECOND (the vfs.zfs.vdev.trim_max_active setting) - so the SSD was literally DoS with TRIM operation and it crippled it to its knees. After switching this option from 64 to 1 it solved all the problems and everything was smooth and fast again. Details below. % grep -A 1 -i trim /etc/sysctl.conf # ZFS DELETE FUCKUP TRIM # vfs.zfs.vdev.trim_max_active: 64 -> 1 vfs.zfs.vdev.trim_max_active=1 Hope that will help someone. Regards, vermaden
wmf 2y ago
This "non-production mode" has a bad side effect where benchmarks on laptops will be artificially fast and people will "forget" about the data loss disclaimer, just like the bad old days of MongoDB.
- jauntywundrkind 2y ago
  The article also mentions that enterprise SSD often have supercaps & can be basically free because there's enough runtime (even if power goes out) to persist any in-flight data. So it seems like it wouldnt be an issue. But it would be running your dev & prod in two different modes.
jakedata 2y ago
In Postgres, synchronous_commits = off can seem like magic during benchmarks but in real-life applications the anticipated benefits can be underwhelming. Ultimately, a reliable non-volatile write cache, sized for your workload is the answer. It evens out all the peaks and valleys of flash writes and hands control back to the database engine so it can get on with its work.
- lvogel 2y ago
  > Ultimately, a reliable non-volatile write cache, sized for your workload is the answer. Author here, I agree! It's quite sad that we need such an involved solution to offset the inherent complexity of the flash medium (latency spikes, erase blocks, ...). We nearly had the perfect solution with Optane[1]: 100ns latency, instantly persisted writes and all that good stuff. I'm still not over Intel killing it while I did my PhD on it. [1] https://en.m.wikipedia.org/wiki/3D_XPoint https://en.m.wikipedia.org/wiki/3D_XPoint
  - throwaway81523 2y ago
    There used to be battery backed ramdisks with SATA interfaces. Are they history now? These days it would be NVMe or whatever, of course.
    - jakedata 2y ago
      Radian makes ultracapacitor backed RAM cards with flash backup that appear to be NVMe devices. They do a nice job for things like Write Ahead Logs and ZFS ARC/ZIL. They offer effectively unlimited write endurance with a tradeoff of cost and capacity.
  - hi-v-rocknroll 2y ago
    More industry experience in real world would've made it clear that they're totally useless for datacenters where there are rarely power outages with many 9's of electrical uptime measured in months or years. UPSes and BBWC evolved to bring reliability to production gear running in non-DC environments when mainline servers used spinning rust without backup power. Today, it's largely a vendor up-charge. Write barriers cause far too much latency in practice on servers in tier IV datacenters, so they're almost always turned off except for a tiny fraction of systems. There has never been a "perfect" or a universal solution, only a risk budget and suitability for a specific use-case.
    - jakedata 2y ago
      Write barriers aren't just for durability - they also can even out major latency spikes when bloated buffers ultimately must be flushed. Database, filesystem, RAID, device, flash management and individual chips all become saturated at some point. Managing performance at the database engine layer gives you visibility into the issue at a high level as opposed to rolling merrily along until commits start taking 2000 ms. As an example, ZFS is terrifyingly unpredictable at high loads.
sam_bristow 2y ago
Related: "What Modern NVMe Storage Can Do, and How to Exploit it"[1] by Gabriel Haas and Viktor Leis. [1]https://dl.acm.org/doi/10.14778/3598581.3598584 https://dl.acm.org/doi/10.14778/3598581.3598584
soonnow 2y ago
We used a small embedded db in our desktop product only for durability and SQL semantics. After we got multiple database corruptions we switched to a small file based store written in Java with simple DB semantics. Looking carefully at what happens when data is lost we found that it's basically all recoverable. This lead to an amazing performance, as data is flushed only when necessary. We flush only every 10 seconds now or when the write through cache is full. My lesson learned from that is to carefully think about the actual requirements when it comes to transactions and having loose constraints may be the quickest way to speed up performance. For many applications even a second or two of lost data may not be a big issue.
hi-v-rocknroll 2y ago
How far we've come from spinning rust on dev boxes and mainline warm servers back in 2011.[0] 0. https://web.archive.org/web/20111112045055/http://buyafuckingssd.com/ https://web.archive.org/web/20111112045055/http://buyafuckin...
- sroussey 2y ago
  People don’t warm servers before entering production anymore?
  - pixl97 2y ago
    "Smaller" servers, not really much any more. With high speed networking/SSD even 64GB-12GB servers warm up really fast. Back in the disk days that could take a very long time to read from disk especially if it was random IO.
Havoc 2y ago
Small optanes (50-100 gigs) are quite accessible cost wise and solve most of the queue depth problem
- ssl-3 2y ago
  Optane is neat tech, but it's also dead tech.
  - lvogel 2y ago
    Tell me about it... Was just too expensive compared to flash. But the tech was definitely awesome and I hope it'll come back in one way or another. When designing CedarDB, we recently had multiple instances where we thought: "If we had just a few KiB of Optane here ... "
    - ssl-3 2y ago
      Oh, it's amazing. I use a small chunk of it at home for ZIL, even though my synch writes are few and far between. It's just approximately the Right Thing to use for some things, and I had room for it, and it was rather inexpensive when I bought it. :) 10/10, would recommend and buy again. (Whoops!)
      - ComputerGuru 2y ago
        I have various generations of optane I stocked up on for my own small business and home lab use, and fully agree. I don’t know if you’re aware but Samsung introduced their own version of optane just shortly before Intel killed their product line. It was called Samsung Z and it was prohibitively expensive for some reason, but I didn’t hear that it was officially killed off (though it likely has been).
        ComputerGuru 2y ago
        I found it deep in my Amazon “saved for later” but it’s no longer available: https://amzn.to/3z7QeGu https://amzn.to/3z7QeGu
justinclift 2y ago
This seems doesn't seem right: Unfortunately, my laptop, and probably many other developers’ laptops, do not have an enterprise-grade SSD, or even the option to install one. Enterprise grade SATA SSDs are fairly easy to obtain (Samsung PM893, Kingston DC600M). Enterprise M.2 SSDs are also a thing (Kingston DC1000B). If they're using a mac and are limited to external connectivity, then they could throw the above into a thunderbolt enclosure.
- wtallis 2y ago
  Even when there are enterprise SSDs that are mechanically and electrically compatible with the bays/slots in a laptop, they're often untenable due to the lack of idle power management features on enterprise SSDs (because sleeping when idle kills the latency metrics). Consider that the 480GB DC1000B is officially rated for 1.9W at idle and in practice I wasn't able to get it below 1.2W. Consumer SSDs with working power management would idle under 10mW.
  - justinclift 2y ago
    Interesting, haven't seen mention of that aspect of enterprise ssds at all.
    - timschmidt 2y ago
      I usually have to tweak my home rackmount equipment over IPMI to set the fan settings to idle when temps are low instead of 24/7/365 jet engine mode which the local university datacenter seems to prefer. I imagine running the fans full blast all the time results in slightly more reliable performance and reliability, but I don't have the luxury of a soundproof HVAC controlled room to lock everything away in, so the lower noise at lower utilization matters to me. This trade-off definitely happens all over the place in enterprise gear.
      - justinclift 2y ago
        You've been through all the stuff like setting max device power for the NVMe drives and similar yeah? https://nvmexpress.org/resource/technology-power-features/ https://nvmexpress.org/resource/technology-power-features/
jeffbee 2y ago
The problem of commit latency bottlenecking global throughput seems to be a matter of treating the device as a black box that must be quiesced as a unit. If the NVMe device presented multiple namespaces to the operating system, and if other unnecessary abstractions like the filesystem are discarded, then the database can make progress on one namespace while blocked on another, even within a single host.
- senderista 2y ago
  Isn't this basically what FUA does on SCSI? Presumably O_DIRECT+O_SYNC/O_DSYNC will invoke this path when possible?
  - jeffbee 2y ago
    Yeah but a hard disk drive has no internal parallelism, while an SSD has immense internal parallelism.
- lmz 2y ago
  There is a feature known as NVMe namespaces but not sure how well it's supported outside of expensive enterprise gear https://unix.stackexchange.com/questions/520231/what-are-nvme-namespaces-how-do-they-work https://unix.stackexchange.com/questions/520231/what-are-nvm...
  - jeffbee 2y ago
    To a close approximation, NVMe devices that come in U.2 or U.3 form factors support multiple namespaces, and those that come in M.2 support only 1, and 4KiB LBA format support follows the same pattern with a few exceptions. But you don't have to spend an insane amount to get these features. The Micro 7x00 series costs ~$100/TB.
    - mgerdts 2y ago
      Generally datacenter drives like Samsung PM9A3, Micron 7xxx, etc., that are available in M.2 and U.2 have the same features between the form factors. The maximum size and performance may differ between them, but this is likely a side effect of physical size for NAND chips and power envelope.
louwrentius 2y ago
Cool! They used my bench-fio and fio-plot tool for the 3D graph :-) [0] What the 3D graph (also) shows is how poorly SSDs actually perform in single-threaded, low queue depth situations. Many enterprise or datacenter grade SSDs often perform much better and consistently on this front than the average consumer SSD. [0] https://github.com/louwrentius/fio-plot https://github.com/louwrentius/fio-plot
refset 2y ago
Slightly off-topic but CedarDB is extremely exciting. It's the commercialization of the widely cited Umbra research DBMS [0] that has been in the works for several years, which benchmarks faster than DuckDB for OLAP [1] whilst simultaneously being really strong for transactional workloads. Also discussed recently here [2]. [0] https://umbra-db.com/ https://umbra-db.com/ [1] https://cedardb.com/blog/ode_to_postgres/ https://cedardb.com/blog/ode_to_postgres/ [2] https://news.ycombinator.com/item?id=40241150 https://news.ycombinator.com/item?id=40241150
- riku_iki 2y ago
  > which benchmarks faster than DuckDB for OLAP [1] that link doesn't do any performance comparison. They claim that CedarDB executes less "code branches" than duckdb, which may or may not translate to faster performance.
  - ChrisWint 2y ago
    Author of that blogpost here. > less "code branches" than duckdb, which may or may not translate to faster performance. In that case it was about 2.5x faster than DuckDB end to end, so a bit less than the difference in branches. If you want to see some independent benchmarks on Umbra, our underlying technology, its currently first place on Clickbench [1]. You can compare against duckdb there as well. [1] https://benchmark.clickhouse.com/ https://benchmark.clickhouse.com/
    - riku_iki 2y ago
      clickbench is a toy benchmark: small dataset, very specific queries. Benchmarking full tcp-h (not just one query like in your post) on sizable dataset (few TBs) would be very good close to real world scenario, but vendors usually avoid this.
      - pfent 2y ago
        There are TPC-H numbers in another post: https://cedardb.com/blog/simple_efficient_hash_tables/ https://cedardb.com/blog/simple_efficient_hash_tables/
JosephRedfern 2y ago
Maybe slight off topic, but does anyone know what the deal is with the NVMe K/V command set spec? There was a fair bit of noise made about it at the time, but I’ve not seen any drives that support it (despite a few enquiries)
- wmf 2y ago
  I think the plan is for ZNS to never be available at retail and I imagine key-value will be the same.
- wtallis 2y ago
  Those new command sets that require a re-write of your entire software storage stack exist solely to satisfy the hyperscalers for whom re-writing the entire software storage stack can be a worthwhile optimization effort. If you don't command enough purchasing power to already be getting custom SKUs from your drive vendors, they don't have enough incentive to add those features to the models they offer you.
  - riku_iki 2y ago
    I imagine it could be cool for some DB vendors too, so they can optimize base on direct access and skip all OS/FS overhead.
floating-io 2y ago
The seemingly critical parallel job performance issue sounds like it can and should be handled transparently by the OS when multiple pages are involved, and should thus be an area of research if it isn't already. Frankly, the idea that it isn't already a solved problem makes me wonder if that could possibly be the case... The writes thing... Outside of very rare circumstances, I would think that properly batching inserts/updates within transactions at the app level would resolve half of that, and multi-user concurrency would buy back the rest. What am I missing? P.S. I'm mad at the author for enlightening me to the real-world performance impact of enterprise SSD. i'm about to buy, and that might just cost me...
- wtallis 2y ago
  > The seemingly critical parallel job performance issue sounds like it can and should be handled transparently by the OS when multiple pages are involved, A thread can only fault on one page at a time, and only sequential access patterns can benefit from the OS bringing in multiple pages per fault. Having all of your CPU cores busy handling page faults and context switches to threads that are about to trigger another page fault is only going to generate enough read traffic to keep a few SSDs busy, not a full shelf of 24+ drives. mmap() and synchronous IO don't scale well enough for today's SSDs.
  - floating-io 2y ago
    That sounds like it would be an issue for anything. If the kernel knows that N pages need to be paged in, it should be able to optimize that behind the scenes in whatever way is needed. I'll agree that mmap() would probably have issues with prediction there (other than basic readahead), but I would think that issuing a simple bulk read() call (or more likely an equivalent zero copy call; vread() if memory serves? Been too long) would give the kernel enough information to optimize bulk transfers. Small async reads will always be slow if done serially, but there's already "standard" scatter/gather calls that should let the kernel optimize that as well. In short, all but a truly naive application should be limited by the SSD or PCIe bandwidth before they ever hit issues like that if the kernel is optimizing correctly, and assuming that the application can predict its own needs.
prewett 2y ago
A little off-topic, but since the article mentioned macOS not necessary actually doing a flush even on fsync(), I've been wondering how long it is before macOS actually writes to the drive. A non-trivial number of my writes are a compile followed by, oh yeah, quick fix, recompile. It'd be nice if it would hold off for a while. Given that I'm on a laptop, the risk of losing data is limited to the very rare times I actually run out of power or a kernel panic.
the8472 2y ago
> If you want your data to be stored persistent for real, you need to issue a sync command to make the operating system block and only return control after the data has been persisted. Well on linux the work that has to be done during a blocking sync() can be minimized by triggering async writeback via sync_file_range in advance.
cogman10 2y ago
> Instead of writing every single change to the SSD immediately, we can instead queue many such writes and persist the whole queue in one go. Instead of having one latency-sensitive round trip per commit, we now have one round trip per queue flush, let’s say every 100 commits. We just have to be careful not to tell the user that their data has been committed before the queue containing their data has been flushed. I'm a little bit unclear, is the suggested method here issuing 100 `write` commands and then a `sync` (or something similar) and only informing the downstream of success when the `sync` is finish? Or is the DB actually maintaining a queue of commits to be written that get fsynced all at once?
- Dalewyn 2y ago
  CPU: "Hey SSD, I need to write 100MB of data." SSD Controller: "Okay, go for it." CPU: <Intense Writing Noises /> SSD Controller: "(Hey OS, can you dump what the CPU's throwing at me in your RAM instead? Don't tell him.)" OS: "Okay." <100MBs later /> CPU: "All done! You got it, SSD?" SSD Controller: "Uhh, yeah, one sec." SSD Controller: "(Hey OS, throw me all that data before the CPU gets wind of this.)" OS: "Okay." OS: <Intense Writing Noise /> SSD Controller: "Alright, come on finish writing ya damn NAND." NAND: "Hold on, man." CPU: "Sitrep, SSD?" SSD Controller: "Almost done!" SSD Controller: "All done!" CPU: "Great!" --- Long story short, big bursty writes are faster than small intermittent writes and SSD controllers are bloody liars.
WarOnPrivacy 2y ago
> If you really want durable writes with sub-millisecond latency, you might want to pay the ~20% premium that vendors charge for such enterprise SSDs. A 20% premium for an enterprise SSD? No. A 120% premium is closer to it. ref: https://www.newegg.com/Enterprise-SSDs/SubCategory/ID-2021 https://www.newegg.com/Enterprise-SSDs/SubCategory/ID-2021
ggm 2y ago
Slow is relative. Slower than rated SSD speed I could believe. Slower than comparable HDD? Slower enough that it impacts performance visibly for the same load?
- TristanBall 2y ago
  It's been a while since I was testing but sustained FUA ( Force Unit Access or something like that), writes, which are used by some DB's in stead of cache write + sync degrade consumer SSD'S to HDD levels of performance. Perhaps not as slow as HDDs themselves would degrade under the same workload, but certainly 10s of milliseconds. So yes, visible, extremely so. On drives commonly known as excellent consumer drives and that I myself happily use on a general desktop/gaming PC. From the limited testing available from my personal budget it would suggest that newer drives are still impacted, although not as badly, and larger drives are ( perhaps obviously ) impacted much less than than smaller ones ( 2tb vs 256gb ). I've seen 2 level drops too.. first when cache/slc writes are overrun, and then a second, when I think the drive is forced into some kind of garbage collection/reorg at the same time. But all that stuff is very controller specific so it's a little hard to generalize. But equally, I now buy old enterprise drives, because they work so much better under these workloads at equivalent sizes, even if consumers drives would appear faster according to spec sheets or "drive friendly" benchmarks. I personally use old intel ( 37nn and 46nn series ) but that's mostly just what I know, there are similarly performing models from most enterprise ssd vendors
userbinator 2y ago
...when they run out of SLC cache. Fortunately it looks like enterprise SLC SSDs have come back into popularity somewhat, and for those who don't want to overpay, some cheap TLC and QLC drives can be forced to run in full-SLC mode too: https://news.ycombinator.com/item?id=40405578 https://news.ycombinator.com/item?id=40405578

133 points by eatonphil 2y ago

Sometimes its the LEAST expected thing (that one this article does not cover). Some time ago I had some really cheap SATA 2.5 SSD 256 GB in my laptop. If generally worked well - system (FreeBSD) was responsive but ... when I deleted large file (like 1GB+ size) then the system almost FROZE ... until file is deleted. After some investigation with Allan Jude (thanks again) we came to the conclusion that a single TRIM operation (for delete) take about 500ms (half a second) ... By default ZFS on FreeBSD uses up to 64 TRIM operations PER SECOND (the vfs.zfs.vdev.trim_max_active setting) - so the SSD was literally DoS with TRIM operation and it crippled it to its knees. After switching this option from 64 to 1 it solved all the problems and everything was smooth and fast again. Details below. % grep -A 1 -i trim /etc/sysctl.conf # ZFS DELETE FUCKUP TRIM # vfs.zfs.vdev.trim_max_active: 64 -> 1 vfs.zfs.vdev.trim_max_active=1 Hope that will help someone. Regards, vermaden

wmf 2y ago

This "non-production mode" has a bad side effect where benchmarks on laptops will be artificially fast and people will "forget" about the data loss disclaimer, just like the bad old days of MongoDB.

jauntywundrkind 2y ago
The article also mentions that enterprise SSD often have supercaps & can be basically free because there's enough runtime (even if power goes out) to persist any in-flight data. So it seems like it wouldnt be an issue. But it would be running your dev & prod in two different modes.

jakedata 2y ago

In Postgres, synchronous_commits = off can seem like magic during benchmarks but in real-life applications the anticipated benefits can be underwhelming. Ultimately, a reliable non-volatile write cache, sized for your workload is the answer. It evens out all the peaks and valleys of flash writes and hands control back to the database engine so it can get on with its work.

lvogel 2y ago
> Ultimately, a reliable non-volatile write cache, sized for your workload is the answer. Author here, I agree! It's quite sad that we need such an involved solution to offset the inherent complexity of the flash medium (latency spikes, erase blocks, ...). We nearly had the perfect solution with Optane[1]: 100ns latency, instantly persisted writes and all that good stuff. I'm still not over Intel killing it while I did my PhD on it. [1] https://en.m.wikipedia.org/wiki/3D_XPoint https://en.m.wikipedia.org/wiki/3D_XPoint
- throwaway81523 2y ago
  There used to be battery backed ramdisks with SATA interfaces. Are they history now? These days it would be NVMe or whatever, of course.
  - jakedata 2y ago
    Radian makes ultracapacitor backed RAM cards with flash backup that appear to be NVMe devices. They do a nice job for things like Write Ahead Logs and ZFS ARC/ZIL. They offer effectively unlimited write endurance with a tradeoff of cost and capacity.
- hi-v-rocknroll 2y ago
  More industry experience in real world would've made it clear that they're totally useless for datacenters where there are rarely power outages with many 9's of electrical uptime measured in months or years. UPSes and BBWC evolved to bring reliability to production gear running in non-DC environments when mainline servers used spinning rust without backup power. Today, it's largely a vendor up-charge. Write barriers cause far too much latency in practice on servers in tier IV datacenters, so they're almost always turned off except for a tiny fraction of systems. There has never been a "perfect" or a universal solution, only a risk budget and suitability for a specific use-case.
  - jakedata 2y ago
    Write barriers aren't just for durability - they also can even out major latency spikes when bloated buffers ultimately must be flushed. Database, filesystem, RAID, device, flash management and individual chips all become saturated at some point. Managing performance at the database engine layer gives you visibility into the issue at a high level as opposed to rolling merrily along until commits start taking 2000 ms. As an example, ZFS is terrifyingly unpredictable at high loads.

sam_bristow 2y ago

Related: "What Modern NVMe Storage Can Do, and How to Exploit it"[1] by Gabriel Haas and Viktor Leis. [1]https://dl.acm.org/doi/10.14778/3598581.3598584 https://dl.acm.org/doi/10.14778/3598581.3598584

soonnow 2y ago

We used a small embedded db in our desktop product only for durability and SQL semantics. After we got multiple database corruptions we switched to a small file based store written in Java with simple DB semantics. Looking carefully at what happens when data is lost we found that it's basically all recoverable. This lead to an amazing performance, as data is flushed only when necessary. We flush only every 10 seconds now or when the write through cache is full. My lesson learned from that is to carefully think about the actual requirements when it comes to transactions and having loose constraints may be the quickest way to speed up performance. For many applications even a second or two of lost data may not be a big issue.

hi-v-rocknroll 2y ago

How far we've come from spinning rust on dev boxes and mainline warm servers back in 2011.[0] 0. https://web.archive.org/web/20111112045055/http://buyafuckingssd.com/ https://web.archive.org/web/20111112045055/http://buyafuckin...

sroussey 2y ago
People don’t warm servers before entering production anymore?
- pixl97 2y ago
  "Smaller" servers, not really much any more. With high speed networking/SSD even 64GB-12GB servers warm up really fast. Back in the disk days that could take a very long time to read from disk especially if it was random IO.

Havoc 2y ago

Small optanes (50-100 gigs) are quite accessible cost wise and solve most of the queue depth problem

ssl-3 2y ago
Optane is neat tech, but it's also dead tech.
- lvogel 2y ago
  Tell me about it... Was just too expensive compared to flash. But the tech was definitely awesome and I hope it'll come back in one way or another. When designing CedarDB, we recently had multiple instances where we thought: "If we had just a few KiB of Optane here ... "
  - ssl-3 2y ago
    Oh, it's amazing. I use a small chunk of it at home for ZIL, even though my synch writes are few and far between. It's just approximately the Right Thing to use for some things, and I had room for it, and it was rather inexpensive when I bought it. :) 10/10, would recommend and buy again. (Whoops!)
    - ComputerGuru 2y ago
      I have various generations of optane I stocked up on for my own small business and home lab use, and fully agree. I don’t know if you’re aware but Samsung introduced their own version of optane just shortly before Intel killed their product line. It was called Samsung Z and it was prohibitively expensive for some reason, but I didn’t hear that it was officially killed off (though it likely has been).
      - ComputerGuru 2y ago
        I found it deep in my Amazon “saved for later” but it’s no longer available: https://amzn.to/3z7QeGu https://amzn.to/3z7QeGu

justinclift 2y ago

This seems doesn't seem right: Unfortunately, my laptop, and probably many other developers’ laptops, do not have an enterprise-grade SSD, or even the option to install one. Enterprise grade SATA SSDs are fairly easy to obtain (Samsung PM893, Kingston DC600M). Enterprise M.2 SSDs are also a thing (Kingston DC1000B). If they're using a mac and are limited to external connectivity, then they could throw the above into a thunderbolt enclosure.

wtallis 2y ago
Even when there are enterprise SSDs that are mechanically and electrically compatible with the bays/slots in a laptop, they're often untenable due to the lack of idle power management features on enterprise SSDs (because sleeping when idle kills the latency metrics). Consider that the 480GB DC1000B is officially rated for 1.9W at idle and in practice I wasn't able to get it below 1.2W. Consumer SSDs with working power management would idle under 10mW.
- justinclift 2y ago
  Interesting, haven't seen mention of that aspect of enterprise ssds at all.
  - timschmidt 2y ago
    I usually have to tweak my home rackmount equipment over IPMI to set the fan settings to idle when temps are low instead of 24/7/365 jet engine mode which the local university datacenter seems to prefer. I imagine running the fans full blast all the time results in slightly more reliable performance and reliability, but I don't have the luxury of a soundproof HVAC controlled room to lock everything away in, so the lower noise at lower utilization matters to me. This trade-off definitely happens all over the place in enterprise gear.
    - justinclift 2y ago
      You've been through all the stuff like setting max device power for the NVMe drives and similar yeah? https://nvmexpress.org/resource/technology-power-features/ https://nvmexpress.org/resource/technology-power-features/

jeffbee 2y ago

The problem of commit latency bottlenecking global throughput seems to be a matter of treating the device as a black box that must be quiesced as a unit. If the NVMe device presented multiple namespaces to the operating system, and if other unnecessary abstractions like the filesystem are discarded, then the database can make progress on one namespace while blocked on another, even within a single host.

senderista 2y ago
Isn't this basically what FUA does on SCSI? Presumably O_DIRECT+O_SYNC/O_DSYNC will invoke this path when possible?
- jeffbee 2y ago
  Yeah but a hard disk drive has no internal parallelism, while an SSD has immense internal parallelism.
lmz 2y ago
There is a feature known as NVMe namespaces but not sure how well it's supported outside of expensive enterprise gear https://unix.stackexchange.com/questions/520231/what-are-nvme-namespaces-how-do-they-work https://unix.stackexchange.com/questions/520231/what-are-nvm...
- jeffbee 2y ago
  To a close approximation, NVMe devices that come in U.2 or U.3 form factors support multiple namespaces, and those that come in M.2 support only 1, and 4KiB LBA format support follows the same pattern with a few exceptions. But you don't have to spend an insane amount to get these features. The Micro 7x00 series costs ~$100/TB.
  - mgerdts 2y ago
    Generally datacenter drives like Samsung PM9A3, Micron 7xxx, etc., that are available in M.2 and U.2 have the same features between the form factors. The maximum size and performance may differ between them, but this is likely a side effect of physical size for NAND chips and power envelope.

louwrentius 2y ago

Cool! They used my bench-fio and fio-plot tool for the 3D graph :-) [0] What the 3D graph (also) shows is how poorly SSDs actually perform in single-threaded, low queue depth situations. Many enterprise or datacenter grade SSDs often perform much better and consistently on this front than the average consumer SSD. [0] https://github.com/louwrentius/fio-plot https://github.com/louwrentius/fio-plot

refset 2y ago

Slightly off-topic but CedarDB is extremely exciting. It's the commercialization of the widely cited Umbra research DBMS [0] that has been in the works for several years, which benchmarks faster than DuckDB for OLAP [1] whilst simultaneously being really strong for transactional workloads. Also discussed recently here [2]. [0] https://umbra-db.com/ https://umbra-db.com/ [1] https://cedardb.com/blog/ode_to_postgres/ https://cedardb.com/blog/ode_to_postgres/ [2] https://news.ycombinator.com/item?id=40241150 https://news.ycombinator.com/item?id=40241150

riku_iki 2y ago
> which benchmarks faster than DuckDB for OLAP [1] that link doesn't do any performance comparison. They claim that CedarDB executes less "code branches" than duckdb, which may or may not translate to faster performance.
- ChrisWint 2y ago
  Author of that blogpost here. > less "code branches" than duckdb, which may or may not translate to faster performance. In that case it was about 2.5x faster than DuckDB end to end, so a bit less than the difference in branches. If you want to see some independent benchmarks on Umbra, our underlying technology, its currently first place on Clickbench [1]. You can compare against duckdb there as well. [1] https://benchmark.clickhouse.com/ https://benchmark.clickhouse.com/
  - riku_iki 2y ago
    clickbench is a toy benchmark: small dataset, very specific queries. Benchmarking full tcp-h (not just one query like in your post) on sizable dataset (few TBs) would be very good close to real world scenario, but vendors usually avoid this.
    - pfent 2y ago
      There are TPC-H numbers in another post: https://cedardb.com/blog/simple_efficient_hash_tables/ https://cedardb.com/blog/simple_efficient_hash_tables/

JosephRedfern 2y ago

Maybe slight off topic, but does anyone know what the deal is with the NVMe K/V command set spec? There was a fair bit of noise made about it at the time, but I’ve not seen any drives that support it (despite a few enquiries)

wmf 2y ago
I think the plan is for ZNS to never be available at retail and I imagine key-value will be the same.
wtallis 2y ago
Those new command sets that require a re-write of your entire software storage stack exist solely to satisfy the hyperscalers for whom re-writing the entire software storage stack can be a worthwhile optimization effort. If you don't command enough purchasing power to already be getting custom SKUs from your drive vendors, they don't have enough incentive to add those features to the models they offer you.
- riku_iki 2y ago
  I imagine it could be cool for some DB vendors too, so they can optimize base on direct access and skip all OS/FS overhead.

floating-io 2y ago

The seemingly critical parallel job performance issue sounds like it can and should be handled transparently by the OS when multiple pages are involved, and should thus be an area of research if it isn't already. Frankly, the idea that it isn't already a solved problem makes me wonder if that could possibly be the case... The writes thing... Outside of very rare circumstances, I would think that properly batching inserts/updates within transactions at the app level would resolve half of that, and multi-user concurrency would buy back the rest. What am I missing? P.S. I'm mad at the author for enlightening me to the real-world performance impact of enterprise SSD. i'm about to buy, and that might just cost me...

wtallis 2y ago
> The seemingly critical parallel job performance issue sounds like it can and should be handled transparently by the OS when multiple pages are involved, A thread can only fault on one page at a time, and only sequential access patterns can benefit from the OS bringing in multiple pages per fault. Having all of your CPU cores busy handling page faults and context switches to threads that are about to trigger another page fault is only going to generate enough read traffic to keep a few SSDs busy, not a full shelf of 24+ drives. mmap() and synchronous IO don't scale well enough for today's SSDs.
- floating-io 2y ago
  That sounds like it would be an issue for anything. If the kernel knows that N pages need to be paged in, it should be able to optimize that behind the scenes in whatever way is needed. I'll agree that mmap() would probably have issues with prediction there (other than basic readahead), but I would think that issuing a simple bulk read() call (or more likely an equivalent zero copy call; vread() if memory serves? Been too long) would give the kernel enough information to optimize bulk transfers. Small async reads will always be slow if done serially, but there's already "standard" scatter/gather calls that should let the kernel optimize that as well. In short, all but a truly naive application should be limited by the SSD or PCIe bandwidth before they ever hit issues like that if the kernel is optimizing correctly, and assuming that the application can predict its own needs.

prewett 2y ago

A little off-topic, but since the article mentioned macOS not necessary actually doing a flush even on fsync(), I've been wondering how long it is before macOS actually writes to the drive. A non-trivial number of my writes are a compile followed by, oh yeah, quick fix, recompile. It'd be nice if it would hold off for a while. Given that I'm on a laptop, the risk of losing data is limited to the very rare times I actually run out of power or a kernel panic.

the8472 2y ago

> If you want your data to be stored persistent for real, you need to issue a sync command to make the operating system block and only return control after the data has been persisted. Well on linux the work that has to be done during a blocking sync() can be minimized by triggering async writeback via sync_file_range in advance.

cogman10 2y ago

> Instead of writing every single change to the SSD immediately, we can instead queue many such writes and persist the whole queue in one go. Instead of having one latency-sensitive round trip per commit, we now have one round trip per queue flush, let’s say every 100 commits. We just have to be careful not to tell the user that their data has been committed before the queue containing their data has been flushed. I'm a little bit unclear, is the suggested method here issuing 100 `write` commands and then a `sync` (or something similar) and only informing the downstream of success when the `sync` is finish? Or is the DB actually maintaining a queue of commits to be written that get fsynced all at once?

Dalewyn 2y ago
CPU: "Hey SSD, I need to write 100MB of data." SSD Controller: "Okay, go for it." CPU: <Intense Writing Noises /> SSD Controller: "(Hey OS, can you dump what the CPU's throwing at me in your RAM instead? Don't tell him.)" OS: "Okay." <100MBs later /> CPU: "All done! You got it, SSD?" SSD Controller: "Uhh, yeah, one sec." SSD Controller: "(Hey OS, throw me all that data before the CPU gets wind of this.)" OS: "Okay." OS: <Intense Writing Noise /> SSD Controller: "Alright, come on finish writing ya damn NAND." NAND: "Hold on, man." CPU: "Sitrep, SSD?" SSD Controller: "Almost done!" SSD Controller: "All done!" CPU: "Great!" --- Long story short, big bursty writes are faster than small intermittent writes and SSD controllers are bloody liars.

WarOnPrivacy 2y ago

> If you really want durable writes with sub-millisecond latency, you might want to pay the ~20% premium that vendors charge for such enterprise SSDs. A 20% premium for an enterprise SSD? No. A 120% premium is closer to it. ref: https://www.newegg.com/Enterprise-SSDs/SubCategory/ID-2021 https://www.newegg.com/Enterprise-SSDs/SubCategory/ID-2021

ggm 2y ago

Slow is relative. Slower than rated SSD speed I could believe. Slower than comparable HDD? Slower enough that it impacts performance visibly for the same load?

TristanBall 2y ago
It's been a while since I was testing but sustained FUA ( Force Unit Access or something like that), writes, which are used by some DB's in stead of cache write + sync degrade consumer SSD'S to HDD levels of performance. Perhaps not as slow as HDDs themselves would degrade under the same workload, but certainly 10s of milliseconds. So yes, visible, extremely so. On drives commonly known as excellent consumer drives and that I myself happily use on a general desktop/gaming PC. From the limited testing available from my personal budget it would suggest that newer drives are still impacted, although not as badly, and larger drives are ( perhaps obviously ) impacted much less than than smaller ones ( 2tb vs 256gb ). I've seen 2 level drops too.. first when cache/slc writes are overrun, and then a second, when I think the drive is forced into some kind of garbage collection/reorg at the same time. But all that stuff is very controller specific so it's a little hard to generalize. But equally, I now buy old enterprise drives, because they work so much better under these workloads at equivalent sizes, even if consumers drives would appear faster according to spec sheets or "drive friendly" benchmarks. I personally use old intel ( 37nn and 46nn series ) but that's mostly just what I know, there are similarly performing models from most enterprise ssd vendors

userbinator 2y ago

...when they run out of SLC cache. Fortunately it looks like enterprise SLC SSDs have come back into popularity somewhat, and for those who don't want to overpay, some cheap TLC and QLC drives can be forced to run in full-SLC mode too: https://news.ycombinator.com/item?id=40405578 https://news.ycombinator.com/item?id=40405578