3 ms·

Exploiting Byte-Accessibility of SSDs Within a Unified Memory-Storage Hierarchy [pdf]

55 points by lainon 7y ago

twotwotwo 7y ago
I think the sources of the claimed benefits here are 1) the OS can stay out of some reads entirely, because PCIe devices directly answering requests to read/write memory is a thing, and 2) for small reads [or writes, as arcticbull notes below] you don't have to send the whole page between the SSD and the host system, freeing up bandwidth. The gains measured in the paper are using emulation, so, like, they didn't really hack SSD firmware. There's got to be some cost on the controller side to service these new requests. And they seem to be accounting for this, but of course taking away some overheads doesn't take away the underlying limits of the storage medium or controller. A couple things make something like this more interesting in the future: PCIe 4 doubles bandwidth and is supposed to be coming in some AMD chipsets soon (and the PCIe 5 spec has been finalized and, once it's really shipping, provides another doubling), and of course Intel/Micron's new storage medium has zippier read/writes than Flash. I wonder whether arrangements like this over PCIe end up a middle ground between using swap/mmap'ing vs. way pricier setups with nonvolatile DIMMs.
- arcticbull 7y ago
  Some of the benefits claimed here are that in the paging model, even if you want to write a couple of bytes here or there (the FS metadata issue) you have to bring an entire page into memory, update it, then write the entire block back out immediately. They're suggesting directly making byte-level mutations to the PCIe device which (battery-backed to ensure consistency) tracks those changes and writes them back out when reasonable. In terms of SSD durability, by tracking mutations at the controller you can avoid unnecessary re-writes so where reading may have been (page-in-block, interpret, page-out-block), reading can now be direct and limited to the range of interest. At least that was my first-glance take.
  - infogulch 7y ago
    It basically eliminates the "paging" abstraction, and lets the storage controller, with specific knowledge of the hardware implementation, to intelligently do the minimum necessary reads/writes with the actual hardware to express the desired changes. It makes me wonder how locking/race conditions/consistency is implemented since that's one of the features of the page abstraction that's not obviously addressed with this tech.
- joehandzik 7y ago
  Some variation of this gain is implemented in various HPC systems as well. For PCIe response vs routing through the CPU/OS, you see all sorts of offload engines and CPU bypass technology being used in networking stacks. For small IO being stored efficiently to minimize the lookup->retrieval process, data-on-metadata in Lustre (I'm sure other scale-out storage systems have similar implementations). The march is of course to have the devices on the most high performance bus that makes sense from a cost efficiency standpoint, hence their mention of GenZ and similar technologies. We're definitely getting closer in the industry, I was just using an FPGA connected to a GenZ bus that exposed an NVMe device this week! Early prototype, but it worked in Ubuntu 18.04.

55 points by lainon 7y ago

I think the sources of the claimed benefits here are 1) the OS can stay out of some reads entirely, because PCIe devices directly answering requests to read/write memory is a thing, and 2) for small reads [or writes, as arcticbull notes below] you don't have to send the whole page between the SSD and the host system, freeing up bandwidth. The gains measured in the paper are using emulation, so, like, they didn't really hack SSD firmware. There's got to be some cost on the controller side to service these new requests. And they seem to be accounting for this, but of course taking away some overheads doesn't take away the underlying limits of the storage medium or controller. A couple things make something like this more interesting in the future: PCIe 4 doubles bandwidth and is supposed to be coming in some AMD chipsets soon (and the PCIe 5 spec has been finalized and, once it's really shipping, provides another doubling), and of course Intel/Micron's new storage medium has zippier read/writes than Flash. I wonder whether arrangements like this over PCIe end up a middle ground between using swap/mmap'ing vs. way pricier setups with nonvolatile DIMMs.

arcticbull 7y ago
Some of the benefits claimed here are that in the paging model, even if you want to write a couple of bytes here or there (the FS metadata issue) you have to bring an entire page into memory, update it, then write the entire block back out immediately. They're suggesting directly making byte-level mutations to the PCIe device which (battery-backed to ensure consistency) tracks those changes and writes them back out when reasonable. In terms of SSD durability, by tracking mutations at the controller you can avoid unnecessary re-writes so where reading may have been (page-in-block, interpret, page-out-block), reading can now be direct and limited to the range of interest. At least that was my first-glance take.
- infogulch 7y ago
  It basically eliminates the "paging" abstraction, and lets the storage controller, with specific knowledge of the hardware implementation, to intelligently do the minimum necessary reads/writes with the actual hardware to express the desired changes. It makes me wonder how locking/race conditions/consistency is implemented since that's one of the features of the page abstraction that's not obviously addressed with this tech.
joehandzik 7y ago
Some variation of this gain is implemented in various HPC systems as well. For PCIe response vs routing through the CPU/OS, you see all sorts of offload engines and CPU bypass technology being used in networking stacks. For small IO being stored efficiently to minimize the lookup->retrieval process, data-on-metadata in Lustre (I'm sure other scale-out storage systems have similar implementations). The march is of course to have the devices on the most high performance bus that makes sense from a cost efficiency standpoint, hence their mention of GenZ and similar technologies. We're definitely getting closer in the industry, I was just using an FPGA connected to a GenZ bus that exposed an NVMe device this week! Early prototype, but it worked in Ubuntu 18.04.