4 ms·

I wish SSDs gave you CPU performance style metrics about their activity

92 points by ingve 1y ago

jauntywundrkind 1y ago
One of the big innovations of NVMe over SATA was giving us a bunch of separate command queues. It'd be lovely to get some per queue information. I feel like maybe some of this info is already available we just don't commonly look at it: knowing how deep the queue is, how many commands are outstanding at any given moment is probably a decent start. I haven't spent time digging into blk-mq to see what's available, to understand the hardware dispatch queue (how the kernel represents the many hardware queues available) info. https://www.kernel.org/doc/html/v5.16/block/blk-mq.html https://www.kernel.org/doc/html/v5.16/block/blk-mq.html
- adgjlsfhk1 1y ago
  I think a lot of the problem is that the ssd is fast enough and far enough away that by the time you get a response back it's fairly out of date
  - dleary 1y ago
    That’s a separate concern. Every command that you issue to the ssd returns a response. It would be nice to have a bunch of performance counters that tell us where the time went with each of the commands we give it. GPUs have this already.
  - wmf 1y ago
    This was solved in networking by having the device collect a histogram of queue occupancy.
- fh973 1y ago
  Queues tend to be always full or always empty (see queueing theory). There is no steady state with a half full queue. For NVMe in particular you will have a hard time filling their queues. Your perceived performance is mostly latency, as there is hardly an application that can submit enough concurrent requests.
andai 1y ago
This reminds me of not too long ago when you could hear the sound of the spinny disk in action, and you'd know if there was an issue (e.g. low on RAM and swapping a lot, or the dreaded Windows search indexer). You get many of the same problems these days, but they're a bit harder to diagnose. You have to go looking at system monitors to see what's going on. Whereas, if the computer just communicated to you what it was doing, in an ambient way, this stuff would be immediately obvious. I've heard stories like this where people worked on older computers that were loud, and then you could actually hear what it was doing. If it got stuck in an infinite loop, you'd literally hear it. That seems like very much a feature to me.
- buildbot 1y ago
  You can hear some modern GPU/CPUs (well really their power electronics) when they get heavily loaded! With training runs it makes a little beat and you can tell when it checkpoints because there’s a little skip. Or a GPU drops off the bus…
  - huflungdung 1y ago
    [dead]
  - dataflow 1y ago
    > You can hear some modern GPU/CPUs (well really their power electronics) when they get heavily loaded! I'd hope you hear their fans too...
    - tomsmeding 1y ago
      Sure, but that has a resolution of seconds at best. The coil whine in the power electronics is milliseconds-accurate.
      - wpm 1y ago
        Yeah I get a nice reminder to limit frame rates when I hear the atrocious coil whine from my 4090 as it renders 1500fps of a static loading screen. First world problems.
zokier 1y ago
Doesn't nvme have lot of log capabilities, like telemetry log etc?
- wmf 1y ago
  Consumer NVMe probably doesn't have that. It has SMART which is hard to interpret.
cjensen 1y ago
This is asking too much. The management of trim, reallocation, wear leveling, and so much more is very complex. It's a full software stack hiding behind the abstraction of NVMe. Every manufacturer is running a different stack with different features and tradeoffs. The "stats" the author is asking for would be entirely different between manufacturers, and I doubt there is that much to be gained from peering behind the curtain.
- zekrioca 1y ago
  It has been done previously for CPUs, which are much more complex than SSDs. Why couldn’t each manufacturer expose whatever performance metrics there are, in whichever way they want (as the post argues, eg., through SMART), and then let system engineers exploit this information to optimize their use-cases?
  - jeffbee 1y ago
    Seems like a poor example since CPU performance metrics differ not only between ISAs, and between vendors of one ISA (AMD vs. Intel, for example) but also between items from a single vendor. There's a 1000-page PDF that tries to explain what all the Intel PMU counters mean on different CPUs and it's full of errors and omissions as well.
    - zekrioca 1y ago
      Yes, but these differences don’t really matter. There are multiple techniques that system engineers can use to perform both variable selection and regularization (to help with differences across multiple architectures) to help them select counters that matter for their specific use cases. But then saying “it is too much to ask” is just another way to limit what user can do with the specific resources they paid for.
- jeffbee 1y ago
  The abstraction is the problem. Get rid of the translation layer, manage flash directly in the operating system, and suddenly the ambiguity dissolves. You would get meaningful, uniform statistics with semantics necessarily matching those used by your operating system.
srean 1y ago
I have had a long held, far far simpler wish that has remained out of my reach since ever -- can SMART implementation by vendors be not so half assed. I have had the wish since the days of spinning disks.
tanelpoder 1y ago
Indeed, would be nice if there were a standardized API/naming for internal NVMe events, so you'd not have to look up the vendor-specific RAW counters and their offsets. Somewhat like the libpfm/PerfMon2 library for standardized naming for common CPU counters/events across architectures. The `nvme id-ctrl -H` (human readable) option does parse and explain some configuration settings and hardware capabilities in a more standardized human readable fashion, but availability of internal activity counters, events vary greatly across vendors, products, firmware versions (and even your currently installed nvme & smartctl software package versions). Regarding eBPF (for OS level view), the `biolatency` tool supports -F option to additionally break down I/Os by the IORQ flags. I have added the iorq_flags to my eBPF `xcapture` tool as well, so I can break down IOs (and latencies) by submitter PID, user, program, etc and see IORQ flags like "WRITE|SYNC|FUA" that help to understand why some write operations are slower than others (especially on commodity SSDs without power-loss-protected write cache). An example output of viewing IORQ flags in general is below: https://tanelpoder.com/posts/xcapture-xtop-beta/#disk-io-waits-and-filenameblockdevice-latency-drilldown https://tanelpoder.com/posts/xcapture-xtop-beta/#disk-io-wai...
- Avamander 1y ago
  It's not only NVMe/SSD that could use such standardization. If you want detailed Ryzen stats you have to use ryzen_monitor. If you want detailed Seagate HDD stats you have to use OpenSeaChest. If you want detailed NIC queue stats there's ethq. I'm sure there are other examples as well. Most hardware metrics are still really difficult to collect, understand and monitor.
LebanonJon 1y ago
Most recent NVMe SSDs support OCP NVMe spec which has c0 log page. Can just get from standard NVMe cli OCP plugin. https://www.opencompute.org/documents/datacenter-nvme-ssd-specification-v2-6-2-pdf https://www.opencompute.org/documents/datacenter-nvme-ssd-sp...

92 points by ingve 1y ago

One of the big innovations of NVMe over SATA was giving us a bunch of separate command queues. It'd be lovely to get some per queue information. I feel like maybe some of this info is already available we just don't commonly look at it: knowing how deep the queue is, how many commands are outstanding at any given moment is probably a decent start. I haven't spent time digging into blk-mq to see what's available, to understand the hardware dispatch queue (how the kernel represents the many hardware queues available) info. https://www.kernel.org/doc/html/v5.16/block/blk-mq.html https://www.kernel.org/doc/html/v5.16/block/blk-mq.html

adgjlsfhk1 1y ago
I think a lot of the problem is that the ssd is fast enough and far enough away that by the time you get a response back it's fairly out of date
- dleary 1y ago
  That’s a separate concern. Every command that you issue to the ssd returns a response. It would be nice to have a bunch of performance counters that tell us where the time went with each of the commands we give it. GPUs have this already.
- wmf 1y ago
  This was solved in networking by having the device collect a histogram of queue occupancy.
fh973 1y ago
Queues tend to be always full or always empty (see queueing theory). There is no steady state with a half full queue. For NVMe in particular you will have a hard time filling their queues. Your perceived performance is mostly latency, as there is hardly an application that can submit enough concurrent requests.

andai 1y ago

This reminds me of not too long ago when you could hear the sound of the spinny disk in action, and you'd know if there was an issue (e.g. low on RAM and swapping a lot, or the dreaded Windows search indexer). You get many of the same problems these days, but they're a bit harder to diagnose. You have to go looking at system monitors to see what's going on. Whereas, if the computer just communicated to you what it was doing, in an ambient way, this stuff would be immediately obvious. I've heard stories like this where people worked on older computers that were loud, and then you could actually hear what it was doing. If it got stuck in an infinite loop, you'd literally hear it. That seems like very much a feature to me.

buildbot 1y ago
You can hear some modern GPU/CPUs (well really their power electronics) when they get heavily loaded! With training runs it makes a little beat and you can tell when it checkpoints because there’s a little skip. Or a GPU drops off the bus…
- huflungdung 1y ago
  [dead]
- dataflow 1y ago
  > You can hear some modern GPU/CPUs (well really their power electronics) when they get heavily loaded! I'd hope you hear their fans too...
  - tomsmeding 1y ago
    Sure, but that has a resolution of seconds at best. The coil whine in the power electronics is milliseconds-accurate.
    - wpm 1y ago
      Yeah I get a nice reminder to limit frame rates when I hear the atrocious coil whine from my 4090 as it renders 1500fps of a static loading screen. First world problems.

zokier 1y ago

Doesn't nvme have lot of log capabilities, like telemetry log etc?

wmf 1y ago
Consumer NVMe probably doesn't have that. It has SMART which is hard to interpret.

cjensen 1y ago

This is asking too much. The management of trim, reallocation, wear leveling, and so much more is very complex. It's a full software stack hiding behind the abstraction of NVMe. Every manufacturer is running a different stack with different features and tradeoffs. The "stats" the author is asking for would be entirely different between manufacturers, and I doubt there is that much to be gained from peering behind the curtain.

zekrioca 1y ago
It has been done previously for CPUs, which are much more complex than SSDs. Why couldn’t each manufacturer expose whatever performance metrics there are, in whichever way they want (as the post argues, eg., through SMART), and then let system engineers exploit this information to optimize their use-cases?
- jeffbee 1y ago
  Seems like a poor example since CPU performance metrics differ not only between ISAs, and between vendors of one ISA (AMD vs. Intel, for example) but also between items from a single vendor. There's a 1000-page PDF that tries to explain what all the Intel PMU counters mean on different CPUs and it's full of errors and omissions as well.
  - zekrioca 1y ago
    Yes, but these differences don’t really matter. There are multiple techniques that system engineers can use to perform both variable selection and regularization (to help with differences across multiple architectures) to help them select counters that matter for their specific use cases. But then saying “it is too much to ask” is just another way to limit what user can do with the specific resources they paid for.
jeffbee 1y ago
The abstraction is the problem. Get rid of the translation layer, manage flash directly in the operating system, and suddenly the ambiguity dissolves. You would get meaningful, uniform statistics with semantics necessarily matching those used by your operating system.

srean 1y ago

I have had a long held, far far simpler wish that has remained out of my reach since ever -- can SMART implementation by vendors be not so half assed. I have had the wish since the days of spinning disks.

tanelpoder 1y ago

Indeed, would be nice if there were a standardized API/naming for internal NVMe events, so you'd not have to look up the vendor-specific RAW counters and their offsets. Somewhat like the libpfm/PerfMon2 library for standardized naming for common CPU counters/events across architectures. The `nvme id-ctrl -H` (human readable) option does parse and explain some configuration settings and hardware capabilities in a more standardized human readable fashion, but availability of internal activity counters, events vary greatly across vendors, products, firmware versions (and even your currently installed nvme & smartctl software package versions). Regarding eBPF (for OS level view), the `biolatency` tool supports -F option to additionally break down I/Os by the IORQ flags. I have added the iorq_flags to my eBPF `xcapture` tool as well, so I can break down IOs (and latencies) by submitter PID, user, program, etc and see IORQ flags like "WRITE|SYNC|FUA" that help to understand why some write operations are slower than others (especially on commodity SSDs without power-loss-protected write cache). An example output of viewing IORQ flags in general is below: https://tanelpoder.com/posts/xcapture-xtop-beta/#disk-io-waits-and-filenameblockdevice-latency-drilldown https://tanelpoder.com/posts/xcapture-xtop-beta/#disk-io-wai...

Avamander 1y ago
It's not only NVMe/SSD that could use such standardization. If you want detailed Ryzen stats you have to use ryzen_monitor. If you want detailed Seagate HDD stats you have to use OpenSeaChest. If you want detailed NIC queue stats there's ethq. I'm sure there are other examples as well. Most hardware metrics are still really difficult to collect, understand and monitor.

LebanonJon 1y ago

Most recent NVMe SSDs support OCP NVMe spec which has c0 log page. Can just get from standard NVMe cli OCP plugin. https://www.opencompute.org/documents/datacenter-nvme-ssd-specification-v2-6-2-pdf https://www.opencompute.org/documents/datacenter-nvme-ssd-sp...