4 ms·

That's very impressive! You may want to do the same for NVMe: creating several namespaces is not supported on most consumer drives, while laptops can rarely ha

by csdvrx 5y ago

That's very impressive! You may want to do the same for NVMe: creating several namespaces is not supported on most consumer drives, while laptops can rarely have more than 1 NVMe (same problem as with the GPUs: a passthrough requires having 2 of them) Being able to split the NVMe drive not by partition but by namespace would let each OS see a "full drive".

ArcVRArthur 5y ago
That's actually a great idea! Is there any documentation you would suggest I read to get started understanding NVMe namespaces? We're also very open to feature additions/pull requests at our repo: https://libvf.io/ https://libvf.io/ I'm very new to this whole thing of building an open source community but I hope if people find value in some of what they built that they might consider helping us with our code. If you have any suggestions on how we could improve and build a good open source community I'd love to hear them!
- csdvrx 5y ago
  > That's actually a great idea! Thanks! Besides being technically interesting and similar to what you do, I think it would also be quite useful for your target audience: if they don't want to do a GPU passthrough, maybe it's because they only have 1 GPU. Maybe they also have only 1 physical drive! > Is there any documentation you would suggest I read to get started understanding NVMe namespaces? Just check the official NVMe specs. You can also grab a drive that supports namespaces to see how it works in practice. It's not very complicated: if you have only 1 NVMe that supports namespaces, and on which you create 3 namespaces, you will have /dev/nvme0n1 /dev/nvme0n2 and /dev/nvme0n3 which can each be partitioned. If you pass /dev/nvme0n2 to a VM1 and /dev/nvme0n3 to another VM, you can be sure that neither will be able to write on the other VM volume, or to the partitions used by the host. It's that simple! So don't waste too much time on the official specs: the goal for your driver would be just to recognize partitions of a special type, intercept them, and present them to the kernel as if they were a namespace (so they can be passed to virtual machines as if they were a namespace...) for people who have NVMe drives that don't support namespaces (the same thing could be used for regular SATA spinning drives and SSD, but the smaller IOPS compared to NVMe would make it less useful in practice) Under the hood, you will also want to implement logical block checks to enforce barriers (just like the IOMMU!) so that VM1 can't write blocks on the VM2 virtual drive, and vice versa. Now that I think about it, it should be quite simple as partitions have a start point and an end point : just check if the write on the virtual disk offset will be within the partition boundaries once you add the offset to map that to the physical disk: for example if /dev/nvme0n2 corresponds to /dev/nvme0n1p6 the offset would simply be the start of this partition. Now, how to do that nicely, in a way that wouldn't confuse the host or cause problems when dual booting? A simple way to do that could be to use a well-known GUID. A good candidate would be 024DEE41-33E7-11D3-9D69-0008C781F39F which was reserved for "MBR partition scheme" but remains mostly unused: a partition under this GUID is expected to contain a full disk image (with partitions etc) but will be ignored: https://en.wikipedia.org/wiki/GUID_Partition_Table#PROTECTIVE-MBR https://en.wikipedia.org/wiki/GUID_Partition_Table#PROTECTIV... On the user side, when not using your driver, each 024DEE41-33E7-11D3-9D69-0008C781F39F type partition could be mounted with losetup -P to read the partition table: https://stackoverflow.com/questions/37227233/having-losetup-read-the-partition-table https://stackoverflow.com/questions/37227233/having-losetup-... After thinking a little more about how it would work, it should be a quite interesting project! I hope you will do that! > We're also very open to feature additions/pull requests Unfortunately, I don't have a lot of time for such projects at the moment but that should be enough to get you started. > If you have any suggestions on how we could improve and build a good open source community I'd love to hear them! Release early, release often, even if it's a bare-mininum MVP. Announce it here. And shoot me an email at my username at outlook.com so I can do the same in other places :)
codetrotter 5y ago
Quick edit: After writing and posting the text below I went to your profile and saw that you mention ZFS in your bio actually. So probably you already know about this then. Leaving it up anyways in case someone finds it interesting. --- Relatedly, in FreeBSD you can attach ZFS file systems to jails. FreeBSD jails is a virtualization mechanism that uses the FreeBSD kernel of the host machine to run processes in isolation. Multiple processes can be isolated together in the same jail, or you can run them in separate jails to isolate them from each others as well. If you are familiar with Linux Namespaces and Docker, I'd say that in principle those are similar to FreeBSD jails. And the way that ZFS works is that you have something called ZFS pools, which sit on top of one or more physical storage media. For example you might have a pool sitting on top of a single NVMe, or on top of multiple NVMe, or on top of one or multiple spinning disks. And with pools that sit on top of multiple storage media you can mirror or stripe the pool across the underlying storage media, depending on your needs (greater capacity vs redundancy). Inside of a pool you have one or more ZFS file systems. These file systems can be snapshotted, and you can roll back to previous snapshots. You can also replicate ZFS file systems and snapshots between pools as well as between different hosts. On my server I have a ZFS pool named "zroot" sitting on top of a single device. Now due to my server itself being a VPS, the device is actually a virtual device that is provided by the hypervisor that the VPS VM is running on. But in the future when I can afford to, and when requirements grow, I can migrate to a physical host where the FreeBSD installation sits directly on physical hardware and the pool has physical devices. Anyways, on that server, inside of the ZFS pool I have some ZFS file systems for the FreeBSD host itself, and I have additional ZFS file systems on the same pool which are attached to FreeBSD jails. In this way, storage is managed from the host at the same time as giving isolated access to portions of it to the FreeBSD jails. Aside from the benefits of being able to snapshot, rollback, send and receive, another great thing about ZFS is that the individual ZFS file systems within a pool can each use how ever much of the available space in the pool they need at any time without dedicating any specific amount of storage to any of the individual ZFS file systems. And furthermore, you are still able to define quotas for the individual ZFS file systems, to limit the max amount of space that they are allowed to consume in the pool. And I think that all of this provides quite closely to what you are describing, although it is using ZFS mechanisms to achieve this and not being based upon NVMe namespaces. So to show what it looks like to use ZFS with FreeBSD jails, here you can see what the FreeBSD host sees: $ zpool status pool: zroot state: ONLINE config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 vtbd0p2 ONLINE 0 0 0 errors: No known data errors $ zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zroot 63.5G 3.56G 59.9G - - 1% 5% 1.00x ONLINE - $ zfs list NAME USED AVAIL REFER MOUNTPOINT zroot 3.56G 58.0G 96K /zroot zroot/ROOT 1.98G 58.0G 96K none zroot/ROOT/default 1.98G 58.0G 1.78G / zroot/jail-data 528K 58.0G 96K none zroot/jail-data/ifee 96K 58.0G 96K /data zroot/jail-data/www 336K 58.0G 336K /data zroot/jail-opt 19.7M 58.0G 96K none zroot/jail-opt/ifee 96K 58.0G 96K /opt zroot/jail-opt/www 19.5M 58.0G 19.5M /opt zroot/tmp 208K 58.0G 120K /tmp zroot/usr 1.54G 58.0G 96K /usr zroot/usr/home 529M 58.0G 529M /usr/home zroot/usr/jail 1.02G 58.0G 96K /usr/jail zroot/usr/jail/fullbase 640M 58.0G 639M /usr/jail/fullbase zroot/usr/jail/ifee 3.63M 58.0G 641M /usr/jail/ifee zroot/usr/jail/svcfw 48.4M 58.0G 686M /usr/jail/svcfw zroot/usr/jail/www 354M 58.0G 991M /usr/jail/www zroot/usr/ports 96K 58.0G 96K /usr/ports zroot/usr/src 96K 58.0G 96K /usr/src zroot/var 1.59M 58.0G 96K /var zroot/var/audit 96K 58.0G 96K /var/audit zroot/var/crash 96K 58.0G 96K /var/crash zroot/var/log 1008K 58.0G 896K /var/log zroot/var/mail 168K 58.0G 104K /var/mail zroot/var/tmp 160K 58.0G 96K /var/tmp And here is what one of the jails sees: $ doas jexec www zfs list NAME USED AVAIL REFER MOUNTPOINT zroot 3.56G 58.0G 96K /zroot zroot/jail-data 528K 58.0G 96K none zroot/jail-data/www 336K 58.0G 336K /data zroot/jail-opt 19.7M 58.0G 96K none zroot/jail-opt/www 19.5M 58.0G 19.5M /opt $ doas jexec www df -h Filesystem Size Used Avail Capacity Mounted on zroot/usr/jail/www 59G 991M 58G 2% / devfs 1.0K 1.0K 0B 100% /dev zroot/jail-data/www 58G 336K 58G 0% /data zroot/jail-opt/www 58G 19M 58G 0% /opt Now let's apply some quotas. $ doas zfs set quota=4G zroot/usr/jail/www $ doas zfs set quota=5G zroot/jail-opt/www $ doas zfs set quota=10G zroot/jail-data/www And we see that these quotas are then reflected inside of the jail. $ doas jexec www zfs list NAME USED AVAIL REFER MOUNTPOINT zroot 3.56G 58.0G 96K /zroot zroot/jail-data 528K 58.0G 96K none zroot/jail-data/www 336K 10.0G 336K /data zroot/jail-opt 19.7M 58.0G 96K none zroot/jail-opt/www 19.5M 4.98G 19.5M /opt $ doas jexec www df -h Filesystem Size Used Avail Capacity Mounted on zroot/usr/jail/www 4.6G 991M 3.7G 21% / devfs 1.0K 1.0K 0B 100% /dev zroot/jail-data/www 10G 336K 10G 0% /data zroot/jail-opt/www 5.0G 19M 5.0G 0% /opt Pretty neat.

by csdvrx 5y ago

ArcVRArthur 5y ago

That's actually a great idea! Is there any documentation you would suggest I read to get started understanding NVMe namespaces? We're also very open to feature additions/pull requests at our repo: https://libvf.io/ https://libvf.io/ I'm very new to this whole thing of building an open source community but I hope if people find value in some of what they built that they might consider helping us with our code. If you have any suggestions on how we could improve and build a good open source community I'd love to hear them!

csdvrx 5y ago
> That's actually a great idea! Thanks! Besides being technically interesting and similar to what you do, I think it would also be quite useful for your target audience: if they don't want to do a GPU passthrough, maybe it's because they only have 1 GPU. Maybe they also have only 1 physical drive! > Is there any documentation you would suggest I read to get started understanding NVMe namespaces? Just check the official NVMe specs. You can also grab a drive that supports namespaces to see how it works in practice. It's not very complicated: if you have only 1 NVMe that supports namespaces, and on which you create 3 namespaces, you will have /dev/nvme0n1 /dev/nvme0n2 and /dev/nvme0n3 which can each be partitioned. If you pass /dev/nvme0n2 to a VM1 and /dev/nvme0n3 to another VM, you can be sure that neither will be able to write on the other VM volume, or to the partitions used by the host. It's that simple! So don't waste too much time on the official specs: the goal for your driver would be just to recognize partitions of a special type, intercept them, and present them to the kernel as if they were a namespace (so they can be passed to virtual machines as if they were a namespace...) for people who have NVMe drives that don't support namespaces (the same thing could be used for regular SATA spinning drives and SSD, but the smaller IOPS compared to NVMe would make it less useful in practice) Under the hood, you will also want to implement logical block checks to enforce barriers (just like the IOMMU!) so that VM1 can't write blocks on the VM2 virtual drive, and vice versa. Now that I think about it, it should be quite simple as partitions have a start point and an end point : just check if the write on the virtual disk offset will be within the partition boundaries once you add the offset to map that to the physical disk: for example if /dev/nvme0n2 corresponds to /dev/nvme0n1p6 the offset would simply be the start of this partition. Now, how to do that nicely, in a way that wouldn't confuse the host or cause problems when dual booting? A simple way to do that could be to use a well-known GUID. A good candidate would be 024DEE41-33E7-11D3-9D69-0008C781F39F which was reserved for "MBR partition scheme" but remains mostly unused: a partition under this GUID is expected to contain a full disk image (with partitions etc) but will be ignored: https://en.wikipedia.org/wiki/GUID_Partition_Table#PROTECTIVE-MBR https://en.wikipedia.org/wiki/GUID_Partition_Table#PROTECTIV... On the user side, when not using your driver, each 024DEE41-33E7-11D3-9D69-0008C781F39F type partition could be mounted with losetup -P to read the partition table: https://stackoverflow.com/questions/37227233/having-losetup-read-the-partition-table https://stackoverflow.com/questions/37227233/having-losetup-... After thinking a little more about how it would work, it should be a quite interesting project! I hope you will do that! > We're also very open to feature additions/pull requests Unfortunately, I don't have a lot of time for such projects at the moment but that should be enough to get you started. > If you have any suggestions on how we could improve and build a good open source community I'd love to hear them! Release early, release often, even if it's a bare-mininum MVP. Announce it here. And shoot me an email at my username at outlook.com so I can do the same in other places :)

codetrotter 5y ago

Quick edit: After writing and posting the text below I went to your profile and saw that you mention ZFS in your bio actually. So probably you already know about this then. Leaving it up anyways in case someone finds it interesting. --- Relatedly, in FreeBSD you can attach ZFS file systems to jails. FreeBSD jails is a virtualization mechanism that uses the FreeBSD kernel of the host machine to run processes in isolation. Multiple processes can be isolated together in the same jail, or you can run them in separate jails to isolate them from each others as well. If you are familiar with Linux Namespaces and Docker, I'd say that in principle those are similar to FreeBSD jails. And the way that ZFS works is that you have something called ZFS pools, which sit on top of one or more physical storage media. For example you might have a pool sitting on top of a single NVMe, or on top of multiple NVMe, or on top of one or multiple spinning disks. And with pools that sit on top of multiple storage media you can mirror or stripe the pool across the underlying storage media, depending on your needs (greater capacity vs redundancy). Inside of a pool you have one or more ZFS file systems. These file systems can be snapshotted, and you can roll back to previous snapshots. You can also replicate ZFS file systems and snapshots between pools as well as between different hosts. On my server I have a ZFS pool named "zroot" sitting on top of a single device. Now due to my server itself being a VPS, the device is actually a virtual device that is provided by the hypervisor that the VPS VM is running on. But in the future when I can afford to, and when requirements grow, I can migrate to a physical host where the FreeBSD installation sits directly on physical hardware and the pool has physical devices. Anyways, on that server, inside of the ZFS pool I have some ZFS file systems for the FreeBSD host itself, and I have additional ZFS file systems on the same pool which are attached to FreeBSD jails. In this way, storage is managed from the host at the same time as giving isolated access to portions of it to the FreeBSD jails. Aside from the benefits of being able to snapshot, rollback, send and receive, another great thing about ZFS is that the individual ZFS file systems within a pool can each use how ever much of the available space in the pool they need at any time without dedicating any specific amount of storage to any of the individual ZFS file systems. And furthermore, you are still able to define quotas for the individual ZFS file systems, to limit the max amount of space that they are allowed to consume in the pool. And I think that all of this provides quite closely to what you are describing, although it is using ZFS mechanisms to achieve this and not being based upon NVMe namespaces. So to show what it looks like to use ZFS with FreeBSD jails, here you can see what the FreeBSD host sees: $ zpool status pool: zroot state: ONLINE config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 vtbd0p2 ONLINE 0 0 0 errors: No known data errors $ zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zroot 63.5G 3.56G 59.9G - - 1% 5% 1.00x ONLINE - $ zfs list NAME USED AVAIL REFER MOUNTPOINT zroot 3.56G 58.0G 96K /zroot zroot/ROOT 1.98G 58.0G 96K none zroot/ROOT/default 1.98G 58.0G 1.78G / zroot/jail-data 528K 58.0G 96K none zroot/jail-data/ifee 96K 58.0G 96K /data zroot/jail-data/www 336K 58.0G 336K /data zroot/jail-opt 19.7M 58.0G 96K none zroot/jail-opt/ifee 96K 58.0G 96K /opt zroot/jail-opt/www 19.5M 58.0G 19.5M /opt zroot/tmp 208K 58.0G 120K /tmp zroot/usr 1.54G 58.0G 96K /usr zroot/usr/home 529M 58.0G 529M /usr/home zroot/usr/jail 1.02G 58.0G 96K /usr/jail zroot/usr/jail/fullbase 640M 58.0G 639M /usr/jail/fullbase zroot/usr/jail/ifee 3.63M 58.0G 641M /usr/jail/ifee zroot/usr/jail/svcfw 48.4M 58.0G 686M /usr/jail/svcfw zroot/usr/jail/www 354M 58.0G 991M /usr/jail/www zroot/usr/ports 96K 58.0G 96K /usr/ports zroot/usr/src 96K 58.0G 96K /usr/src zroot/var 1.59M 58.0G 96K /var zroot/var/audit 96K 58.0G 96K /var/audit zroot/var/crash 96K 58.0G 96K /var/crash zroot/var/log 1008K 58.0G 896K /var/log zroot/var/mail 168K 58.0G 104K /var/mail zroot/var/tmp 160K 58.0G 96K /var/tmp And here is what one of the jails sees: $ doas jexec www zfs list NAME USED AVAIL REFER MOUNTPOINT zroot 3.56G 58.0G 96K /zroot zroot/jail-data 528K 58.0G 96K none zroot/jail-data/www 336K 58.0G 336K /data zroot/jail-opt 19.7M 58.0G 96K none zroot/jail-opt/www 19.5M 58.0G 19.5M /opt $ doas jexec www df -h Filesystem Size Used Avail Capacity Mounted on zroot/usr/jail/www 59G 991M 58G 2% / devfs 1.0K 1.0K 0B 100% /dev zroot/jail-data/www 58G 336K 58G 0% /data zroot/jail-opt/www 58G 19M 58G 0% /opt Now let's apply some quotas. $ doas zfs set quota=4G zroot/usr/jail/www $ doas zfs set quota=5G zroot/jail-opt/www $ doas zfs set quota=10G zroot/jail-data/www And we see that these quotas are then reflected inside of the jail. $ doas jexec www zfs list NAME USED AVAIL REFER MOUNTPOINT zroot 3.56G 58.0G 96K /zroot zroot/jail-data 528K 58.0G 96K none zroot/jail-data/www 336K 10.0G 336K /data zroot/jail-opt 19.7M 58.0G 96K none zroot/jail-opt/www 19.5M 4.98G 19.5M /opt $ doas jexec www df -h Filesystem Size Used Avail Capacity Mounted on zroot/usr/jail/www 4.6G 991M 3.7G 21% / devfs 1.0K 1.0K 0B 100% /dev zroot/jail-data/www 10G 336K 10G 0% /data zroot/jail-opt/www 5.0G 19M 5.0G 0% /opt Pretty neat.