3 ms·

Do you suppose the elastic block stores of the public cloud providers operate in a non polling manner? The incredibly low level of latency and high levels of i

Do you suppose the elastic block stores of the public cloud providers operate in a non polling manner? The incredibly low level of latency and high levels of iops preclude having it be based on software interrupts, I think even in scenarios where sr-iov is available to the hyper visor.

fdr 3y ago
Yes, SPDK for us sits on adjacent to the VM however, so our use of it for now can be best compared to an alternative to specialized PCIe hardware operating in pass-through mode directly to guests (e.g. AWS Nitro), with enough industry heft to (in the case of AWS especially, and GCP more recently via a probably-Intel collaboration) to get guests to have drivers for those cards. You can see this in action in the Linux drivers for ena (aws) and gve (google). This is how the usual suspects get past software interrupts adjacent to the VMM, on the "client" side of things. Getting the driver stack integrated into Linux is no small task and letting it percolate into distributions in common use is necessarily a slow process. SPDK permits getting decent performance and gradual deepening of our functionality while still relying on virtio, which has already percolated. As I understand it, on the storage side, these cards tend to expose an NVMe interface which is somewhat generic, so you don't see the same kind of driver siloing there. There's a related bit of SPDK and hypervisor functionality, vfio-user (vs vhost-user), but we elected not to use it at this time. They both use a similar shared-memory transport. Azure is an interesting outlier, a large one, in that their reliance on Mellanox (a subsidiary of nvidia) drivers is documented, so they could be considered in a partnership to achieve the same aim. So you could read the mlx drivers in the same fashion as ena and gve. I've been watching the technology "vdpa" with some interest to have a shot to also provide pass-through PCIe devices to the guest that do not add such a driver dependency so far outside our ability to influence: Microsoft is going to have a bit more equal a relationship with nvidia than we would as it comes to problematic changes in the Mellanox drivers. But I suspect it'll be some years before that can possibly happen, if it happens at all. It's not easy to get a Connect-X 6 DX card, for example. So, there are many problems for the foreseeable future trying to get into hardware, though I'd like to avoid precluding it. I liked this blog post in getting a feel for this, but in brief, they're computers plugged into computers: https://www.servethehome.com/zfs-without-a-server-using-the-nvidia-bluefield-2-dpu-nvme-arm-aic-iscsi/ https://www.servethehome.com/zfs-without-a-server-using-the-.... Our alternative is to carve off a core or two instead of plugging a computer into the computer. Maybe at some future time...I suspect, no sooner than five years from now, but probably, should it come to pass, quite a bit later...there will be some commonality and availability in such PCIe cards and Ubicloud or something like it could consider a tangible development theory around them.

fdr 3y ago

Yes, SPDK for us sits on adjacent to the VM however, so our use of it for now can be best compared to an alternative to specialized PCIe hardware operating in pass-through mode directly to guests (e.g. AWS Nitro), with enough industry heft to (in the case of AWS especially, and GCP more recently via a probably-Intel collaboration) to get guests to have drivers for those cards. You can see this in action in the Linux drivers for ena (aws) and gve (google). This is how the usual suspects get past software interrupts adjacent to the VMM, on the "client" side of things. Getting the driver stack integrated into Linux is no small task and letting it percolate into distributions in common use is necessarily a slow process. SPDK permits getting decent performance and gradual deepening of our functionality while still relying on virtio, which has already percolated. As I understand it, on the storage side, these cards tend to expose an NVMe interface which is somewhat generic, so you don't see the same kind of driver siloing there. There's a related bit of SPDK and hypervisor functionality, vfio-user (vs vhost-user), but we elected not to use it at this time. They both use a similar shared-memory transport. Azure is an interesting outlier, a large one, in that their reliance on Mellanox (a subsidiary of nvidia) drivers is documented, so they could be considered in a partnership to achieve the same aim. So you could read the mlx drivers in the same fashion as ena and gve. I've been watching the technology "vdpa" with some interest to have a shot to also provide pass-through PCIe devices to the guest that do not add such a driver dependency so far outside our ability to influence: Microsoft is going to have a bit more equal a relationship with nvidia than we would as it comes to problematic changes in the Mellanox drivers. But I suspect it'll be some years before that can possibly happen, if it happens at all. It's not easy to get a Connect-X 6 DX card, for example. So, there are many problems for the foreseeable future trying to get into hardware, though I'd like to avoid precluding it. I liked this blog post in getting a feel for this, but in brief, they're computers plugged into computers: https://www.servethehome.com/zfs-without-a-server-using-the-nvidia-bluefield-2-dpu-nvme-arm-aic-iscsi/ https://www.servethehome.com/zfs-without-a-server-using-the-.... Our alternative is to carve off a core or two instead of plugging a computer into the computer. Maybe at some future time...I suspect, no sooner than five years from now, but probably, should it come to pass, quite a bit later...there will be some commonality and availability in such PCIe cards and Ubicloud or something like it could consider a tangible development theory around them.