4 ms·

I'm an engineering software contractor and every client has a whole bucket load of outdated IDEs and random USB interface drivers they want me to install on my

I'm an engineering software contractor and every client has a whole bucket load of outdated IDEs and random USB interface drivers they want me to install on my machine. Often this stuff has weird specific version dependencies, kernel level drivers that cause weird things to happen (like the computer to BSOD if you boot it up with other USB devices attached), random system level crashes, and when you go to the driver download page there are always an unsettling list of simple security vulns noted. The #1 thing that I've ever wanted is a Linux that lives between the bootloader and Windows that lets me achieve native performance on the Windows VM, but gives me an environment where I can easily do all the things that you can usually do in HyperV like create snapshots, clone installs, share Sharepoint drives between images, etc. But I do need something that is perfectly stable and just works. This is the right technology (thank you so much for working on it), but just not yet at a maturity that makes me feel comfortable about putting livelihood on the line. Edit: actually, please, if anyone knows something that suits this use-case, even if it costs decent money, please leave a comment.

carlhjerpe 5y ago
I would probably run proxmox if I was you and kit your device with extra peripheral cards that you pass through to the VM, with ZFS you can snapshot your vm volumes and proxmox has builtin support to do full backups and either store on your local machine or ship them off to a remote location. If you pass the GPU through to Windows you're going to be essentially native speed along with great security and flexibility. If you want to have multiple windows machines you could just rdp them unless you need GPU acceleration in them too. If you're working on a laptop I would not recommend ZFS, ZOL doesn't implement freeze and thaw.
- aetherspawn 5y ago
  Would you still recommend this if it was a mobile workstation [1], doing a mix of at-desk work and mobile work, utilizing peripherals like docking stations, and I was running nearly constant compute intensive workloads (Matlab)? 1: https://www.dell.com/en-au/work/shop/workstations-isv-certified/new-precision-7560-mobile-workstation/spd/precision-15-7560-laptop/on7560wm02au_vp https://www.dell.com/en-au/work/shop/workstations-isv-certif...
  - carlhjerpe 5y ago
    I honestly have no idea, I would get another SSD and give it a shot, if it works its really quite great. Not with ZFS on mobile though, go btrfs.
  - acoard 5y ago
    Proxmox is basically Debian plus some VM hypervisor stuff. So your question really is would someone recommend it if they're running Debian, as all that desktop environment stuff would be outside of what proxmox manages. Debian is super reliable - potentially the most reliable linux distro - but this wouldn't be a turnkey solution or anything (for example PopOS, based on Debian via Ubuntu, focuses more on the out-of-the-box experience). I don't know if Debian can handle your particular needs, but my assumption is that if any distro can Debian can - it just might take some time. Personally I love proxmox as a main operating system, as I can get everything from broader Debian/Ubuntu/PopOS environments and learn a lot about Linux too. But, it has taken a lot of my time, so I'd only recommend it if you wanted to invest the time.
- BuildTheRobots 5y ago
  I could be remembering this wrong, but I'm pretty sure proxmox + zfs doesn't let you roll back to a snapshot that has child snapshots. If you're coming from vmware (or probably virtual box, I honestly can't remember), then the ability to be able to jump back and forth between any snapshot and branch off form there easily is really convenient and a bit jarring to loose.
  - carlhjerpe 5y ago
    The thing with proxmox is that it's quite open, you could manage ZFS snapshots outside of the proxmox system if you want, I haven't tried this specific with proxmox, I run NixOS with ZFS root and libvirt for managing the guests (vit-manager and virsh).
wheybags 5y ago
You could netboot off an iscsi target on another machine, which is backed by an image file on a zfs volume. That way you get all the nice zfs features of snapshotting, and windows just thinks it's using a hard drive. You could even skip zfs and just use a qemu image file, with qemu-nbd you can have it present as a block device, and you can then export that block device as an iscsi target. Then you can use qemu-img for snapshotting etc. The machine running the storage could be pretty low spec too, a nuc would do it for sure, maybe even an rpi or similar (if you go the qemu-img route, zfs on rpi is not feasible, ask me how I know :D)
- jeffbee 5y ago
  You don't even have to let the guest know it is netbooting. I use ESXi with one Solaris (Illumos) guest, which runs ZFS over everything and exports iSCSI back to ESXi. The other guests all think they have native storage and I can do whatever I want with the underlying ZFS filesystem.
- compsciphd 5y ago
  I was actually wondering about this a while ago, one can netboot windows? i.e. no "local" storage, all accessed over the network? or does it depend on the hypervisor, i.e. the hypervisor access the storage over the network, but to windows, its a local disk?
  - wheybags 5y ago
    You can, with no hypervisor at all, just running on bare metal, so long as your mobo / nic supports it (which I believe most do). I haven't ever personally done it though, so ymmv.
alisonkisk 5y ago
why don't you require clients to provide a lab machine for all their requirements?
- mpfundstein 5y ago
  hi. i have the same problem. because those lab machines usually SUCK! i have my own custom build servers at home now and virtualize everything. every client has own VM and I usually RDP/ssh into them for work.
- aetherspawn 5y ago
  Because I charge out at $X per hour and the calculations that take my machine 1 hour to do, typically take their machines 3 or 4 hours to do. From an executive level, a lot of clients would rather spend $2k on a cheap throwaway machine than spend say $5K, but then seem fine to have the labor component cost 2-3x as much. I just build the price of the hardware into my hourly rate (which ends up being not very much .. $2 per hour), AU law instant write-off the cost of the machine as tax, tell them they don't need to buy a machine (which they are very happy about), and then everyone is surprised at how fast I can get the job done compared to the in-house teams which have their hands tied.
pizza234 5y ago
The specification you present is a bit unclear. Any modern virtualization has very fast CPU virtualization (it's hard to say near-native, as there are always corner cases), and snapshotting tools. I don't know about Sharepoint; the "clone installs" is a bit fuzzy too, but one can clone installed systems by just copying the underlying image file and ensuring that it has a unique identifier (and updating the guest O/S license, if required). If GPU matters, VFIO definitely is part of the solution. However, if you're trying to achieve a sort-of full system passthrough (eg. because of drivers that have bugs related to certain hardware components, which it seems to be your situation), this will never happen, because certain parts of the guest need to be necessarily emulated (e.g. chipset). Even passing a USB port is not easy - one actually needs to pass the whole hubs (AFAIK a port may belong to two hubs - USB 2 and USB 3).
- matja 5y ago
  For the USB part, would adding a extra USB adaptor card (assuming it's a desktop machine) and using PCI-passthrough/VFIO on that adaptor make it easier? From my knowledge of QEMU, then no USB emulation is involved - the guest OS sees the entire USB adaptor (and the host doesn't) - so one can have all the special USB hardware connected to that.
  - pizza234 5y ago
    I'm not knowledgeable with this, but I think so, as it'd be a standard passthrough. What needs to be kept in mind is that, as the standanrd passthrough, it's subject to the IOMMU groups handling (in best case, there's no other device in the same group; otherwise, one needs some trickery, which AFAIK is not 100% guaranteed to work).
  - acoard 5y ago
    I've read of people doing this setup for exactly this on homelab and VFIO subreddits. PCIe passthrough is easier. I've no personal experience with this, but it seems to work well; the VFIO setups are hardcore anyways.
virteverything 5y ago
KVM? Virt-Manager? It might take an hour to learn virt-manager and some of its quirks, but it absolutely can do everything you request here, I do so very regularly. If you want a low-maintenance version, you could just snag someone's NixOS config - you'd have an easy to reproduce environment that just ensures virt-manager and a really light-weight window manager are installed and then you're done. Virt-Manager does shared directories (9p anyway; there's no UI for virtio-fs, but you can still use it by editting XML). It handles USB2 and USB3 forwarding. It does snapshots, it does clones, you can even leverage Linux filesystems to do far fancier things than possible on HyperV hosts. Etc. (Plus KVM won't trash your plain ole EXT4 partitions like countless people keep reporting under HyperV [and I've personally experienced twice]). It can even do graphics accceleration/virtualization for Linux guests. Virt-Manager is so under-known and under-appreciated, but then again, it's got its rough-edges. With some polish there is really no reason to ever mess with VBox under Linux.
- neop1x 5y ago
  Do you happen to know any tips for good online resources for virt-manager and features like this other than tedious reading of manual pages and trial & error?
  - totony 5y ago
    Libvirt manual is pretty good https://libvirt.org/docs.html https://libvirt.org/docs.html

carlhjerpe 5y ago

I would probably run proxmox if I was you and kit your device with extra peripheral cards that you pass through to the VM, with ZFS you can snapshot your vm volumes and proxmox has builtin support to do full backups and either store on your local machine or ship them off to a remote location. If you pass the GPU through to Windows you're going to be essentially native speed along with great security and flexibility. If you want to have multiple windows machines you could just rdp them unless you need GPU acceleration in them too. If you're working on a laptop I would not recommend ZFS, ZOL doesn't implement freeze and thaw.

aetherspawn 5y ago
Would you still recommend this if it was a mobile workstation [1], doing a mix of at-desk work and mobile work, utilizing peripherals like docking stations, and I was running nearly constant compute intensive workloads (Matlab)? 1: https://www.dell.com/en-au/work/shop/workstations-isv-certified/new-precision-7560-mobile-workstation/spd/precision-15-7560-laptop/on7560wm02au_vp https://www.dell.com/en-au/work/shop/workstations-isv-certif...
- carlhjerpe 5y ago
  I honestly have no idea, I would get another SSD and give it a shot, if it works its really quite great. Not with ZFS on mobile though, go btrfs.
- acoard 5y ago
  Proxmox is basically Debian plus some VM hypervisor stuff. So your question really is would someone recommend it if they're running Debian, as all that desktop environment stuff would be outside of what proxmox manages. Debian is super reliable - potentially the most reliable linux distro - but this wouldn't be a turnkey solution or anything (for example PopOS, based on Debian via Ubuntu, focuses more on the out-of-the-box experience). I don't know if Debian can handle your particular needs, but my assumption is that if any distro can Debian can - it just might take some time. Personally I love proxmox as a main operating system, as I can get everything from broader Debian/Ubuntu/PopOS environments and learn a lot about Linux too. But, it has taken a lot of my time, so I'd only recommend it if you wanted to invest the time.
BuildTheRobots 5y ago
I could be remembering this wrong, but I'm pretty sure proxmox + zfs doesn't let you roll back to a snapshot that has child snapshots. If you're coming from vmware (or probably virtual box, I honestly can't remember), then the ability to be able to jump back and forth between any snapshot and branch off form there easily is really convenient and a bit jarring to loose.
- carlhjerpe 5y ago
  The thing with proxmox is that it's quite open, you could manage ZFS snapshots outside of the proxmox system if you want, I haven't tried this specific with proxmox, I run NixOS with ZFS root and libvirt for managing the guests (vit-manager and virsh).

wheybags 5y ago

You could netboot off an iscsi target on another machine, which is backed by an image file on a zfs volume. That way you get all the nice zfs features of snapshotting, and windows just thinks it's using a hard drive. You could even skip zfs and just use a qemu image file, with qemu-nbd you can have it present as a block device, and you can then export that block device as an iscsi target. Then you can use qemu-img for snapshotting etc. The machine running the storage could be pretty low spec too, a nuc would do it for sure, maybe even an rpi or similar (if you go the qemu-img route, zfs on rpi is not feasible, ask me how I know :D)

jeffbee 5y ago
You don't even have to let the guest know it is netbooting. I use ESXi with one Solaris (Illumos) guest, which runs ZFS over everything and exports iSCSI back to ESXi. The other guests all think they have native storage and I can do whatever I want with the underlying ZFS filesystem.
compsciphd 5y ago
I was actually wondering about this a while ago, one can netboot windows? i.e. no "local" storage, all accessed over the network? or does it depend on the hypervisor, i.e. the hypervisor access the storage over the network, but to windows, its a local disk?
- wheybags 5y ago
  You can, with no hypervisor at all, just running on bare metal, so long as your mobo / nic supports it (which I believe most do). I haven't ever personally done it though, so ymmv.

alisonkisk 5y ago

why don't you require clients to provide a lab machine for all their requirements?

mpfundstein 5y ago
hi. i have the same problem. because those lab machines usually SUCK! i have my own custom build servers at home now and virtualize everything. every client has own VM and I usually RDP/ssh into them for work.
aetherspawn 5y ago
Because I charge out at $X per hour and the calculations that take my machine 1 hour to do, typically take their machines 3 or 4 hours to do. From an executive level, a lot of clients would rather spend $2k on a cheap throwaway machine than spend say $5K, but then seem fine to have the labor component cost 2-3x as much. I just build the price of the hardware into my hourly rate (which ends up being not very much .. $2 per hour), AU law instant write-off the cost of the machine as tax, tell them they don't need to buy a machine (which they are very happy about), and then everyone is surprised at how fast I can get the job done compared to the in-house teams which have their hands tied.

pizza234 5y ago

The specification you present is a bit unclear. Any modern virtualization has very fast CPU virtualization (it's hard to say near-native, as there are always corner cases), and snapshotting tools. I don't know about Sharepoint; the "clone installs" is a bit fuzzy too, but one can clone installed systems by just copying the underlying image file and ensuring that it has a unique identifier (and updating the guest O/S license, if required). If GPU matters, VFIO definitely is part of the solution. However, if you're trying to achieve a sort-of full system passthrough (eg. because of drivers that have bugs related to certain hardware components, which it seems to be your situation), this will never happen, because certain parts of the guest need to be necessarily emulated (e.g. chipset). Even passing a USB port is not easy - one actually needs to pass the whole hubs (AFAIK a port may belong to two hubs - USB 2 and USB 3).

matja 5y ago
For the USB part, would adding a extra USB adaptor card (assuming it's a desktop machine) and using PCI-passthrough/VFIO on that adaptor make it easier? From my knowledge of QEMU, then no USB emulation is involved - the guest OS sees the entire USB adaptor (and the host doesn't) - so one can have all the special USB hardware connected to that.
- pizza234 5y ago
  I'm not knowledgeable with this, but I think so, as it'd be a standard passthrough. What needs to be kept in mind is that, as the standanrd passthrough, it's subject to the IOMMU groups handling (in best case, there's no other device in the same group; otherwise, one needs some trickery, which AFAIK is not 100% guaranteed to work).
- acoard 5y ago
  I've read of people doing this setup for exactly this on homelab and VFIO subreddits. PCIe passthrough is easier. I've no personal experience with this, but it seems to work well; the VFIO setups are hardcore anyways.

virteverything 5y ago

KVM? Virt-Manager? It might take an hour to learn virt-manager and some of its quirks, but it absolutely can do everything you request here, I do so very regularly. If you want a low-maintenance version, you could just snag someone's NixOS config - you'd have an easy to reproduce environment that just ensures virt-manager and a really light-weight window manager are installed and then you're done. Virt-Manager does shared directories (9p anyway; there's no UI for virtio-fs, but you can still use it by editting XML). It handles USB2 and USB3 forwarding. It does snapshots, it does clones, you can even leverage Linux filesystems to do far fancier things than possible on HyperV hosts. Etc. (Plus KVM won't trash your plain ole EXT4 partitions like countless people keep reporting under HyperV [and I've personally experienced twice]). It can even do graphics accceleration/virtualization for Linux guests. Virt-Manager is so under-known and under-appreciated, but then again, it's got its rough-edges. With some polish there is really no reason to ever mess with VBox under Linux.

neop1x 5y ago
Do you happen to know any tips for good online resources for virt-manager and features like this other than tedious reading of manual pages and trial & error?
- totony 5y ago
  Libvirt manual is pretty good https://libvirt.org/docs.html https://libvirt.org/docs.html