9 ms·

NVMe over Fabrics Explained

93 points by nickysielicki 5y ago

jbrot 5y ago
From reading the article, it sounds like NVMe-oF exposes a drive directly to a network, but I don’t really get how that works. Does the drive still get connected to a host? How does this compare to other technologies for having networked storage, like NFS?
- helge9210 5y ago
  > it sounds like NVMe-oF exposes a drive directly to a network The drive is connected to a storage device. Storage is exposed by the device to network. Exposed storage and the drive do not correspond 1to1 (redundancy, etc.)
- vbernat 5y ago
  It still needs to be connected to a host. This is an alternative to iSCSI. Instead of using the SCSI protocol, it uses the NVMe protocol.
  - karatinversion 5y ago
    And the use case is when you have a sufficiently high-performance storage controller that the SCSI protocol (that you are tunneling over IP or Fibre channel) becomes a bottle neck.
    - trasz 5y ago
      SCSI still does have some tricks (eg VAAI) which I don't think have NVMe counterparts yet? Still, even just being able to get rid of the huge complexity and historical baggage of SCSI is a good thing. One of my favourite examples is how the spec defines the LUN number to not be a number.
      - wtallis 5y ago
        I think NVMe is pretty close to feature parity with SCSI, or at least the subset of SCSI that is actually relevant. NVMe includes thin provisioning, some copy offload, atomic operations and of course deallocate/trim—which I think covers all the VAAI features.
fouc 5y ago
This is not related to cloth or some sort of smart sheets like I thought it was going to be
ggm 5y ago
There is a wheel of life story here. Everything winds up being used as an IP protocol carrier. Working backwards from USB-C I can't think of a modern-day plug which hasn't wound up having a MAC address, bootstrap-to-IP path added to it. NVMe to ethernet is simply making the SSD the endpoint with a functional NFS/SMB and IP stack. Soon it will move out of the backplane into a stand-alone cage, and be a NAS...
- karatinversion 5y ago
  > NVMe to ethernet is simply making the SSD the endpoint Confused by this comment. NVMeoF is about tunneling the NVMe command set over a network, instead of tunneling SCSI over IP or Fibre Channel, when communicating with a NAS-like device; not about plugging an ethernet cable to the SSD.
  - wtallis 5y ago
    There are also native Ethernet SSDs that speak NVMeoF. Marvell is a big supporter of that idea since they make Ethernet switch chips but not PCIe switch chips. They make a NVMe/PCIe to Ethernet converter chip that can be used on small interposer boards between drives and the backplane or integrated into the drive itself.
bullen 5y ago
It seems to me you might have a hard time adding 12 NVMe drives to a single CPU then? SATA might be slower, but I much prefer a slower interface that can break but supports 12x drives than a single NVMe drive!!! Also I need to see some CPU overhead graphs on those 12x SATA drives vs. a single NVMe?! My suspicion is that the overhead will be lower if the SATA control chip is buffering data to avoid the CPU waiting.
- deleted 5y ago
  [deleted]
- fh973 5y ago
  You can connect many NVMEs to a single CPU, but the PCI bandwidth becomes a bottleneck much earlier of course. Just as with SATA. Think of the SATA HBA as an adapter that bridges between PCI and SATA. For NVMe the counterpart is a PCI switch, with PCI on both sides.
  - puzzlingcaptcha 5y ago
    Supermicro sells single-CPU servers with 24 NVMe bays. Not a big problem if your CPU has 128 PCIe lanes. You could get 12 quite easily even on consumer hardware. AMD Threadripper gives you 64 PCIe 4.0 lanes (so up to 72 with the TRX40 chipset), so you just use three PCIe x16 expansion cards each providing four NVMe 4x slots.
- deleted 5y ago
  [deleted]
- MisterTea 5y ago
  > It seems to me you might have a hard time adding 12 NVMe drives to a single CPU then? Attachment is a solved issue as all you do is buy a PCIe switch and hang more nvme off your cpu. The issue I see is no clear standard for nvme form factors or connectors. m.2 is a consumer standard that is more mobile friendly than desktop or server yet it appears to be the more common standard in all three. U.2 nvme in the classic 2.5" spinning rust form factor but I don't see them used in desktops. Then there are numerous "ruler" form factors designed to plug into 1U servers. Connectors are also a mixed bag with m.2, U.2, a 36pin quad channel sff motherboard connector, sata express connector, and OCulink. And further this whole article is more old-is-new again blogspam as all it talks about yet another SAN technology.
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
nmaludy 5y ago
I'm curious how NVMe-oF compares to iSER? From a quick search it seems that NVMe-oF is still early days with limited implementations, where iSER is more established. Both rely on RDMA. I could not find any performance comparisons. Any insight here?
- benlwalker 5y ago
  I don't have any formal comparisons handy but there's lots of NVMe-oF benchmarks here: https://spdk.io/doc/performance_reports.html https://spdk.io/doc/performance_reports.html NVMe-oF can do both TCP (akin to iSCSI) and RDMA (akin to iSER). There are likely several reasons NVMe-oF is faster, but one big one is that each connection within a session shares state in iSCSI, so you either process all connections for a session on a single thread or you take locks. In NVMe-oF, it's possible to keep every connection entirely independent, so the NVMe-oF implementations scale better.
flemhans 5y ago
Does anyone know of a good NVMe-of storage chassis where the pricing is listed on their web site?
- alberth 5y ago
  https://www.siliconmechanics.com/systems/storage/storform https://www.siliconmechanics.com/systems/storage/storform They mainly use SuperMicro. They have been around for decades and have top notch service and pricing. Also note, check out the Server section as well since there’s a few more in that section that have high NVMe capacity. Like below https://www.siliconmechanics.com/systems/servers/rackform#feature:nvme https://www.siliconmechanics.com/systems/servers/rackform#fe...
  - buildbot 5y ago
    Oh hey, Silicon Mechanics in the wild! My first software dev job was there. They really do try to have the best prices/test their stuff better than Supermicro does.
    - alberth 5y ago
      If you're one of the people who created their web price configurator - then THANK YOU a million times over. It's so easy to use and understand ... as well as being informative as well.
      - buildbot 5y ago
        No that was a different group of people - the backend system that runs that is a CRAZY complex PHP app ;) It's pretty impressive how many configurations/checks it can handle, supermicro probably sells as many different SKUs as Dell in total? I worked on a new stress test suite/baremetal OS deployment automation. No idea if they still use it.
        alberth 5y ago
        Does Silicon Mechanics license out their web configurator? I ask because ThinkMate.com web configurator is identical to SiliconMechanics.com EDIT: I'm going to answer my own question. It appears Silicon Mechanics was acquired by Source Code Corporation, who also owns ThinkMate.
baybal2 5y ago
Ask HN: Do you people also see high performance SSDs vanishing from the shelves, and going up in price at a rapid pace these days?
- ajoseps 5y ago
  Not sure if it’s the cause but Chia is a new-ish cryptocurrency which uses Proof-of-Space: https://en.wikipedia.org/wiki/Chia_(cryptocurrency)?wprov=sfti1 https://en.wikipedia.org/wiki/Chia_(cryptocurrency)?wprov=sf... Might be related
  - lgats 5y ago
    Chia uses high-performance SSD / NVMe drives to 'mine' for plots. This is a very read/write intensive process. Once a plot is generated, it can be stored on a slower HDD for the proof-of-space to take place.
- buildbot 5y ago
  Yes,I have a script that searches eBay for the lowest price per gb, currently prices are _equal_ to buying new for many higher performance drives. There is also far fewer enterprise drives listed than their used to be, but I don’t have a hard number for that. Just no more large fusion io drives for example.
olavgg 5y ago
NVMe-oF is already supported in Linux with target(host) and initiator(client). I've already configured and use this in my homelab with Mellanox network cards with RDMA. Performance is awesome and I have not had any problems. It is a bit messy to configure on the target. But is very easy to configure on the initiator. There is also a free NVMe-oF client driver for Windows, https://www.starwindsoftware.com/starwind-nvme-of-initiator https://www.starwindsoftware.com/starwind-nvme-of-initiator but I have not tested it yet. I use NVMe-oF both for block devices(ZFS volumes from traditional hard drives) and Intel Optane. It also works with SATA/SAS SSD's. It is not as fast as native, you get some additional latency but still much much better than the alternatives. What is really cool is that Optane is so fast that I can easily create 4+ partitions and share the Optane device to 4+ different systems.

93 points by nickysielicki 5y ago

From reading the article, it sounds like NVMe-oF exposes a drive directly to a network, but I don’t really get how that works. Does the drive still get connected to a host? How does this compare to other technologies for having networked storage, like NFS?

helge9210 5y ago
> it sounds like NVMe-oF exposes a drive directly to a network The drive is connected to a storage device. Storage is exposed by the device to network. Exposed storage and the drive do not correspond 1to1 (redundancy, etc.)
vbernat 5y ago
It still needs to be connected to a host. This is an alternative to iSCSI. Instead of using the SCSI protocol, it uses the NVMe protocol.
- karatinversion 5y ago
  And the use case is when you have a sufficiently high-performance storage controller that the SCSI protocol (that you are tunneling over IP or Fibre channel) becomes a bottle neck.
  - trasz 5y ago
    SCSI still does have some tricks (eg VAAI) which I don't think have NVMe counterparts yet? Still, even just being able to get rid of the huge complexity and historical baggage of SCSI is a good thing. One of my favourite examples is how the spec defines the LUN number to not be a number.
    - wtallis 5y ago
      I think NVMe is pretty close to feature parity with SCSI, or at least the subset of SCSI that is actually relevant. NVMe includes thin provisioning, some copy offload, atomic operations and of course deallocate/trim—which I think covers all the VAAI features.

fouc 5y ago

This is not related to cloth or some sort of smart sheets like I thought it was going to be

ggm 5y ago

There is a wheel of life story here. Everything winds up being used as an IP protocol carrier. Working backwards from USB-C I can't think of a modern-day plug which hasn't wound up having a MAC address, bootstrap-to-IP path added to it. NVMe to ethernet is simply making the SSD the endpoint with a functional NFS/SMB and IP stack. Soon it will move out of the backplane into a stand-alone cage, and be a NAS...

karatinversion 5y ago
> NVMe to ethernet is simply making the SSD the endpoint Confused by this comment. NVMeoF is about tunneling the NVMe command set over a network, instead of tunneling SCSI over IP or Fibre Channel, when communicating with a NAS-like device; not about plugging an ethernet cable to the SSD.
- wtallis 5y ago
  There are also native Ethernet SSDs that speak NVMeoF. Marvell is a big supporter of that idea since they make Ethernet switch chips but not PCIe switch chips. They make a NVMe/PCIe to Ethernet converter chip that can be used on small interposer boards between drives and the backplane or integrated into the drive itself.

bullen 5y ago

It seems to me you might have a hard time adding 12 NVMe drives to a single CPU then? SATA might be slower, but I much prefer a slower interface that can break but supports 12x drives than a single NVMe drive!!! Also I need to see some CPU overhead graphs on those 12x SATA drives vs. a single NVMe?! My suspicion is that the overhead will be lower if the SATA control chip is buffering data to avoid the CPU waiting.

deleted 5y ago
[deleted]
fh973 5y ago
You can connect many NVMEs to a single CPU, but the PCI bandwidth becomes a bottleneck much earlier of course. Just as with SATA. Think of the SATA HBA as an adapter that bridges between PCI and SATA. For NVMe the counterpart is a PCI switch, with PCI on both sides.
- puzzlingcaptcha 5y ago
  Supermicro sells single-CPU servers with 24 NVMe bays. Not a big problem if your CPU has 128 PCIe lanes. You could get 12 quite easily even on consumer hardware. AMD Threadripper gives you 64 PCIe 4.0 lanes (so up to 72 with the TRX40 chipset), so you just use three PCIe x16 expansion cards each providing four NVMe 4x slots.
deleted 5y ago
[deleted]
MisterTea 5y ago
> It seems to me you might have a hard time adding 12 NVMe drives to a single CPU then? Attachment is a solved issue as all you do is buy a PCIe switch and hang more nvme off your cpu. The issue I see is no clear standard for nvme form factors or connectors. m.2 is a consumer standard that is more mobile friendly than desktop or server yet it appears to be the more common standard in all three. U.2 nvme in the classic 2.5" spinning rust form factor but I don't see them used in desktops. Then there are numerous "ruler" form factors designed to plug into 1U servers. Connectors are also a mixed bag with m.2, U.2, a 36pin quad channel sff motherboard connector, sata express connector, and OCulink. And further this whole article is more old-is-new again blogspam as all it talks about yet another SAN technology.

deleted 5y ago

[deleted]

deleted 5y ago

[deleted]

nmaludy 5y ago

I'm curious how NVMe-oF compares to iSER? From a quick search it seems that NVMe-oF is still early days with limited implementations, where iSER is more established. Both rely on RDMA. I could not find any performance comparisons. Any insight here?

benlwalker 5y ago
I don't have any formal comparisons handy but there's lots of NVMe-oF benchmarks here: https://spdk.io/doc/performance_reports.html https://spdk.io/doc/performance_reports.html NVMe-oF can do both TCP (akin to iSCSI) and RDMA (akin to iSER). There are likely several reasons NVMe-oF is faster, but one big one is that each connection within a session shares state in iSCSI, so you either process all connections for a session on a single thread or you take locks. In NVMe-oF, it's possible to keep every connection entirely independent, so the NVMe-oF implementations scale better.

flemhans 5y ago

Does anyone know of a good NVMe-of storage chassis where the pricing is listed on their web site?

alberth 5y ago
https://www.siliconmechanics.com/systems/storage/storform https://www.siliconmechanics.com/systems/storage/storform They mainly use SuperMicro. They have been around for decades and have top notch service and pricing. Also note, check out the Server section as well since there’s a few more in that section that have high NVMe capacity. Like below https://www.siliconmechanics.com/systems/servers/rackform#feature:nvme https://www.siliconmechanics.com/systems/servers/rackform#fe...
- buildbot 5y ago
  Oh hey, Silicon Mechanics in the wild! My first software dev job was there. They really do try to have the best prices/test their stuff better than Supermicro does.
  - alberth 5y ago
    If you're one of the people who created their web price configurator - then THANK YOU a million times over. It's so easy to use and understand ... as well as being informative as well.
    - buildbot 5y ago
      No that was a different group of people - the backend system that runs that is a CRAZY complex PHP app ;) It's pretty impressive how many configurations/checks it can handle, supermicro probably sells as many different SKUs as Dell in total? I worked on a new stress test suite/baremetal OS deployment automation. No idea if they still use it.
      - alberth 5y ago
        Does Silicon Mechanics license out their web configurator? I ask because ThinkMate.com web configurator is identical to SiliconMechanics.com EDIT: I'm going to answer my own question. It appears Silicon Mechanics was acquired by Source Code Corporation, who also owns ThinkMate.

baybal2 5y ago

Ask HN: Do you people also see high performance SSDs vanishing from the shelves, and going up in price at a rapid pace these days?

ajoseps 5y ago
Not sure if it’s the cause but Chia is a new-ish cryptocurrency which uses Proof-of-Space: https://en.wikipedia.org/wiki/Chia_(cryptocurrency)?wprov=sfti1 https://en.wikipedia.org/wiki/Chia_(cryptocurrency)?wprov=sf... Might be related
- lgats 5y ago
  Chia uses high-performance SSD / NVMe drives to 'mine' for plots. This is a very read/write intensive process. Once a plot is generated, it can be stored on a slower HDD for the proof-of-space to take place.
buildbot 5y ago
Yes,I have a script that searches eBay for the lowest price per gb, currently prices are _equal_ to buying new for many higher performance drives. There is also far fewer enterprise drives listed than their used to be, but I don’t have a hard number for that. Just no more large fusion io drives for example.

olavgg 5y ago

NVMe-oF is already supported in Linux with target(host) and initiator(client). I've already configured and use this in my homelab with Mellanox network cards with RDMA. Performance is awesome and I have not had any problems. It is a bit messy to configure on the target. But is very easy to configure on the initiator. There is also a free NVMe-oF client driver for Windows, https://www.starwindsoftware.com/starwind-nvme-of-initiator https://www.starwindsoftware.com/starwind-nvme-of-initiator but I have not tested it yet. I use NVMe-oF both for block devices(ZFS volumes from traditional hard drives) and Intel Optane. It also works with SATA/SAS SSD's. It is not as fast as native, you get some additional latency but still much much better than the alternatives. What is really cool is that Optane is so fast that I can easily create 4+ partitions and share the Optane device to 4+ different systems.