5 ms·

vDPA: Support for block devices in Linux and QEMU

126 points by ingve 3y ago

gonzaloalvarez 3y ago
> One of the significant benefits of vDPA is its strong abstraction, enabling the implementation of virtio devices in both hardware and software—whether in the kernel or user space. This unification under a single framework, where devices appear identical for QEMU facilitates the seamless integration of hardware and software components. I honestly don’t know what this means. Is it faster? Is it more secure? Why would I use this vDPA thing instead of the good’ol virtio blk driver? Looking at the examples it certainly looks more cumbersome to setup…
- rwmj 3y ago
  Software block devices let you do all implement all sorts of disk-like things in userspace. nbdkit is a more flexible version of this (also compatible with qemu), so to get a flavour of the types of things you can do just peruse the list of nbdkit plugins: https://libguestfs.org/nbdkit.1.html#Plugins https://libguestfs.org/nbdkit.1.html#Plugins
- benlwalker 3y ago
  I don't get it either, and I'm a maintainer of SPDK which provides multiple implementations of virtualized devices and is frequently used inside DPUs to present storage devices. If I'm implementing a hardware device anyway, why would I not just use NVMe as the interface? NVMe is superior to virtio-blk in every way that I can think of. Even for a software device in userspace, why not use a technology like vfio-user to present an NVMe device, or just use vhost-user to present the virtio-blk device? I've never really been able to get a clear value proposition for vDPA for storage laid out for me. Maybe I'm missing something critical - it's certainly possible.
  - lathiat 3y ago
    Another top-level comment has covered this at least with regards to networking, hasn't discussed the storage side as much: https://news.ycombinator.com/item?id=39355644 https://news.ycombinator.com/item?id=39355644
    - benlwalker 3y ago
      Yes, I've seen some clearer cases made for networking. In networking there is no standard for the hardware interface. Every vendor does their own thing. Except many can at least handle virtqueues carrying virtio-net messages for the data path, so some framework like vDPA may make sense (I'd prefer to see a full NIC interface standard emerge instead). In storage, however, the industry has agreed on NVMe. This is a full standard for control and data plane. All storage products on the market, including DPUs and SmartNICs, just present NVMe devices. So there's no case to be made for vDPA at all. It just isn't necessary.
      - sgarzare 3y ago
        Yes, I see your point and agree that NVMe can be used for the same purpose. But several HW vendors have implemented virtio-net devices in their SmartNic and may find it convenient to support virtio-blk to reuse most of the building blocks. As for vhost-user, it's perfect for VM use cases, but with containers or applications in the host, it's not easy to use. Whereas, a vDPA device (HW or SW) can easily be attached to the host kernel (using the virtio-vdpa bus) and be managed with the standard virtio-blk driver.
- westurner 3y ago
  Ceph Block Device 3rd Party Integrations: https://docs.ceph.com/en/latest/rbd/rbd-integrations/ https://docs.ceph.com/en/latest/rbd/rbd-integrations/ : Kernel Modules, QEMU, libvirt, Kubernetes, Nomad, OpenStack, CloudStack, LIO iSCSI Gateway, Windows, qemu: https://docs.ceph.com/en/latest/rbd/qemu-rbd/#qemu-and-block-devices https://docs.ceph.com/en/latest/rbd/qemu-rbd/#qemu-and-block... `cephadm bootstrap` requires docker or podman and ssh: https://docs.ceph.com/en/latest/cephadm/install/#bootstrap-a-new-cluster https://docs.ceph.com/en/latest/cephadm/install/#bootstrap-a... Ceph Object Gateway: radosgw: https://docs.ceph.com/en/latest/radosgw/ https://docs.ceph.com/en/latest/radosgw/ : > The Ceph Object Gateway provides interfaces that are compatible with both Amazon S3 and OpenStack Swift, and it has its own user management. Ceph Object Gateway can use a single Ceph Storage cluster to store data from Ceph File System and from Ceph Block device clients. The S3 API and the Swift API share a common namespace, which means that it is possible to write data to a Ceph Storage Cluster with one API and then retrieve that data with the other API. virtio-blk is probably faster, but then do HA redundancy with physically separate nodes and network io anyway; or LocalPersistentVolumes
  - yencabulator 3y ago
    S3 or radosgw are not very relevant to a discussion about block storage devices. Ceph's RBD is "special" in the sense that the client understands the clustering, and talks to multiple servers. If you wanted that in the mix, you'd have to run a local Ceph client -- like the Ceph software stack exposing block devices from kernel does. The only way I can see vDPA being relevant to that is to avoid middlemen layers, VM -> host kernel block device abstraction -> Ceph kernelspace RBD client. But the block device abstraction is pretty thin. The real use case for vDPA, when talking about storage, seems to be standardizing an interface the hardware can provide. And then we're back to "why not NVMe?". (Disclaimer: ex-Ceph-employee)
roomey 3y ago
Good timing considering the esxi news https://kb.vmware.com/s/article/2107518?lang=en_US https://kb.vmware.com/s/article/2107518?lang=en_US
- bpye 3y ago
  > Along with the termination of perpetual licensing, Broadcom has also decided to discontinue the Free ESXi Hypervisor, marking it as EOGA (End of General Availability). So it’s subscription or bust? Owch.
- swozey 3y ago
  No more free at all? I learned virtualization w/ vmware gsx long before I ever had a company to use it at.. Oh well, vmwares irrelevant to me nowadays.
- floating-io 3y ago
  I really wonder how much longer the VMUG licensing is going to be around...
- stephen_g 3y ago
  Wow, I knew things were getting really bad with VMWare, but I didn’t know they had got that bad… I liked ESXi, the free hypervisor was a good stepping stone to get people into the ecosystem and probably sold a lot of vSphere and vCenter licences down the line. Very glad I decided to use Proxmox (as the Broadcom acquisition had been announced) for the small cluster we set up at our business a year ago, it’s been super stable, pretty easy to use but can do way more than I need (we’re not doing much that is that complicated yet)
  - b112 3y ago
    I'm really worried who is going to handle network and raid going forward. LSI was the standard for hardware raid, 3com and others big network names. Now they are Broadcom, which basically means: * do as little as possible * axe most support * screw the little guy * milk milk milk * no real heavy R&D This means, to me, that lots of hardware and software is being handled far less diligently than Boeing. And that scares me.
hello_computer 3y ago
[flagged]
transpute 3y ago
Is vDPA a standardization of the storage interfaces used by AWS Nitro? https://www.allthingsdistributed.com/2020/09/reinventing-virtualization-with-nitro.html https://www.allthingsdistributed.com/2020/09/reinventing-vir... > The Nitro System is comprised of three main parts: the Nitro Cards, the Nitro Security Chip, and the Nitro Hypervisor. The Nitro Cards are a family of cards that offloads and accelerates IO for functions ... The Nitro architecture also enabled us to make the hypervisor layer optional and offer bare metal instances. Bare metal instances provide applications with direct access to the processor and memory resources of the underlying server.
jm1337 3y ago
RH has a blog series which motivates and introduces vDPA. For a start, read their post on virtio-networking and vhost-net [0], continue with virtio-networking and DPDK [1] and finally read "Achieving network wirespeed in an open standard manner: introducing vDPA" [2]. In particular the last link has a lot of diagrams, recaps a lot of the virtio-networking topics and finally has a table which compares the other solutions to vDPA. [0] https://www.redhat.com/en/blog/introduction-virtio-networking-and-vhost-net https://www.redhat.com/en/blog/introduction-virtio-networkin... [1] https://www.redhat.com/en/blog/how-vhost-user-came-being-virtio-networking-and-dpdk https://www.redhat.com/en/blog/how-vhost-user-came-being-vir... [2] https://www.redhat.com/en/blog/achieving-network-wirespeed-open-standard-manner-introducing-vdpa https://www.redhat.com/en/blog/achieving-network-wirespeed-o...
- yencabulator 3y ago
  That seems to talk about networking only, while the blog post talks about storage. For networking I get it, there's no vendor-independent data path standard for easy virtualization there. But for storage, NVMe is supposed to be that.
oohffyvfg 3y ago
ah, RH and docker and their race to poke out of emulation/virtualization for fringe performance gains. we will soon get to a point qemu will just be a job manager for software that plugs directly into local syscalls, like, you know, regular software running on your system.

126 points by ingve 3y ago

> One of the significant benefits of vDPA is its strong abstraction, enabling the implementation of virtio devices in both hardware and software—whether in the kernel or user space. This unification under a single framework, where devices appear identical for QEMU facilitates the seamless integration of hardware and software components. I honestly don’t know what this means. Is it faster? Is it more secure? Why would I use this vDPA thing instead of the good’ol virtio blk driver? Looking at the examples it certainly looks more cumbersome to setup…

rwmj 3y ago
Software block devices let you do all implement all sorts of disk-like things in userspace. nbdkit is a more flexible version of this (also compatible with qemu), so to get a flavour of the types of things you can do just peruse the list of nbdkit plugins: https://libguestfs.org/nbdkit.1.html#Plugins https://libguestfs.org/nbdkit.1.html#Plugins
benlwalker 3y ago
I don't get it either, and I'm a maintainer of SPDK which provides multiple implementations of virtualized devices and is frequently used inside DPUs to present storage devices. If I'm implementing a hardware device anyway, why would I not just use NVMe as the interface? NVMe is superior to virtio-blk in every way that I can think of. Even for a software device in userspace, why not use a technology like vfio-user to present an NVMe device, or just use vhost-user to present the virtio-blk device? I've never really been able to get a clear value proposition for vDPA for storage laid out for me. Maybe I'm missing something critical - it's certainly possible.
- lathiat 3y ago
  Another top-level comment has covered this at least with regards to networking, hasn't discussed the storage side as much: https://news.ycombinator.com/item?id=39355644 https://news.ycombinator.com/item?id=39355644
  - benlwalker 3y ago
    Yes, I've seen some clearer cases made for networking. In networking there is no standard for the hardware interface. Every vendor does their own thing. Except many can at least handle virtqueues carrying virtio-net messages for the data path, so some framework like vDPA may make sense (I'd prefer to see a full NIC interface standard emerge instead). In storage, however, the industry has agreed on NVMe. This is a full standard for control and data plane. All storage products on the market, including DPUs and SmartNICs, just present NVMe devices. So there's no case to be made for vDPA at all. It just isn't necessary.
    - sgarzare 3y ago
      Yes, I see your point and agree that NVMe can be used for the same purpose. But several HW vendors have implemented virtio-net devices in their SmartNic and may find it convenient to support virtio-blk to reuse most of the building blocks. As for vhost-user, it's perfect for VM use cases, but with containers or applications in the host, it's not easy to use. Whereas, a vDPA device (HW or SW) can easily be attached to the host kernel (using the virtio-vdpa bus) and be managed with the standard virtio-blk driver.
westurner 3y ago
Ceph Block Device 3rd Party Integrations: https://docs.ceph.com/en/latest/rbd/rbd-integrations/ https://docs.ceph.com/en/latest/rbd/rbd-integrations/ : Kernel Modules, QEMU, libvirt, Kubernetes, Nomad, OpenStack, CloudStack, LIO iSCSI Gateway, Windows, qemu: https://docs.ceph.com/en/latest/rbd/qemu-rbd/#qemu-and-block-devices https://docs.ceph.com/en/latest/rbd/qemu-rbd/#qemu-and-block... `cephadm bootstrap` requires docker or podman and ssh: https://docs.ceph.com/en/latest/cephadm/install/#bootstrap-a-new-cluster https://docs.ceph.com/en/latest/cephadm/install/#bootstrap-a... Ceph Object Gateway: radosgw: https://docs.ceph.com/en/latest/radosgw/ https://docs.ceph.com/en/latest/radosgw/ : > The Ceph Object Gateway provides interfaces that are compatible with both Amazon S3 and OpenStack Swift, and it has its own user management. Ceph Object Gateway can use a single Ceph Storage cluster to store data from Ceph File System and from Ceph Block device clients. The S3 API and the Swift API share a common namespace, which means that it is possible to write data to a Ceph Storage Cluster with one API and then retrieve that data with the other API. virtio-blk is probably faster, but then do HA redundancy with physically separate nodes and network io anyway; or LocalPersistentVolumes
- yencabulator 3y ago
  S3 or radosgw are not very relevant to a discussion about block storage devices. Ceph's RBD is "special" in the sense that the client understands the clustering, and talks to multiple servers. If you wanted that in the mix, you'd have to run a local Ceph client -- like the Ceph software stack exposing block devices from kernel does. The only way I can see vDPA being relevant to that is to avoid middlemen layers, VM -> host kernel block device abstraction -> Ceph kernelspace RBD client. But the block device abstraction is pretty thin. The real use case for vDPA, when talking about storage, seems to be standardizing an interface the hardware can provide. And then we're back to "why not NVMe?". (Disclaimer: ex-Ceph-employee)

roomey 3y ago

Good timing considering the esxi news https://kb.vmware.com/s/article/2107518?lang=en_US https://kb.vmware.com/s/article/2107518?lang=en_US

bpye 3y ago
> Along with the termination of perpetual licensing, Broadcom has also decided to discontinue the Free ESXi Hypervisor, marking it as EOGA (End of General Availability). So it’s subscription or bust? Owch.
swozey 3y ago
No more free at all? I learned virtualization w/ vmware gsx long before I ever had a company to use it at.. Oh well, vmwares irrelevant to me nowadays.
floating-io 3y ago
I really wonder how much longer the VMUG licensing is going to be around...
stephen_g 3y ago
Wow, I knew things were getting really bad with VMWare, but I didn’t know they had got that bad… I liked ESXi, the free hypervisor was a good stepping stone to get people into the ecosystem and probably sold a lot of vSphere and vCenter licences down the line. Very glad I decided to use Proxmox (as the Broadcom acquisition had been announced) for the small cluster we set up at our business a year ago, it’s been super stable, pretty easy to use but can do way more than I need (we’re not doing much that is that complicated yet)
- b112 3y ago
  I'm really worried who is going to handle network and raid going forward. LSI was the standard for hardware raid, 3com and others big network names. Now they are Broadcom, which basically means: * do as little as possible * axe most support * screw the little guy * milk milk milk * no real heavy R&D This means, to me, that lots of hardware and software is being handled far less diligently than Boeing. And that scares me.

hello_computer 3y ago

[flagged]

transpute 3y ago

Is vDPA a standardization of the storage interfaces used by AWS Nitro? https://www.allthingsdistributed.com/2020/09/reinventing-virtualization-with-nitro.html https://www.allthingsdistributed.com/2020/09/reinventing-vir... > The Nitro System is comprised of three main parts: the Nitro Cards, the Nitro Security Chip, and the Nitro Hypervisor. The Nitro Cards are a family of cards that offloads and accelerates IO for functions ... The Nitro architecture also enabled us to make the hypervisor layer optional and offer bare metal instances. Bare metal instances provide applications with direct access to the processor and memory resources of the underlying server.

jm1337 3y ago

RH has a blog series which motivates and introduces vDPA. For a start, read their post on virtio-networking and vhost-net [0], continue with virtio-networking and DPDK [1] and finally read "Achieving network wirespeed in an open standard manner: introducing vDPA" [2]. In particular the last link has a lot of diagrams, recaps a lot of the virtio-networking topics and finally has a table which compares the other solutions to vDPA. [0] https://www.redhat.com/en/blog/introduction-virtio-networking-and-vhost-net https://www.redhat.com/en/blog/introduction-virtio-networkin... [1] https://www.redhat.com/en/blog/how-vhost-user-came-being-virtio-networking-and-dpdk https://www.redhat.com/en/blog/how-vhost-user-came-being-vir... [2] https://www.redhat.com/en/blog/achieving-network-wirespeed-open-standard-manner-introducing-vdpa https://www.redhat.com/en/blog/achieving-network-wirespeed-o...

yencabulator 3y ago
That seems to talk about networking only, while the blog post talks about storage. For networking I get it, there's no vendor-independent data path standard for easy virtualization there. But for storage, NVMe is supposed to be that.

oohffyvfg 3y ago

ah, RH and docker and their race to poke out of emulation/virtualization for fringe performance gains. we will soon get to a point qemu will just be a job manager for software that plugs directly into local syscalls, like, you know, regular software running on your system.