9 ms·
From Proxmox to FreeBSD and Sylve in our office lab
- gcifuentes 6mo agoBhyve doesn't feature nested virt though.
- liendolucas 6mo agoHonestly asking, in which cases nested virtualization is useful?
- doubled112 6mo agoWSL2 in a virtual desktop environment.
- zenoprax 6mo agoOne example: when learning Proxmox itself. I was able to set up a multi-node cluster with more complicated networking than I was normally comfortable with and experiment with failures of all sorts (killing a node, disabling NICs, etc.) without needing more hardware or affecting my existing things. Outside of learning and testing I am not sure of what uses there might be but I'm curious to know if there are.
- belthesar 6mo agoNested virtualization can be very handy in both the lab and in production. In the lab, you can try out a new hosting platform by running one atop the other. IE: Proxmox on VMWare, Hyper-V on KVM. This lets you try things out without needing fresh bare metal hardware. In prod, let's say you run workloads in Firecracker VMs. You have plenty of headroom on your existing hardware. Nested virtualization would allow you to set up Firecracker hosts on your existing hardware.
- liendolucas 6mo agoPerhaps I'm misunderstanding, but wouldn't that case be covered by simply putting some vms under a vnet and others on another vnet and make them talk to each other? I can't also understand what you mean by "fresh bare metal hardware". In either case you don't need bare metal, being a top level vm or a nested one.
- mbreese 6mo agoIf you're evaluating VM hosts (proxmox, hyper-V, vmware, etc...) You need to have support for nested virtualization all the way down. Otherwise, if you want to evaluate a VM infrastructure, you need to start with bare-metal. Really, you just need to make sure that your top level support nested virtualization, but I understand their point. However, the point about firecracker VMs in place of containers I think is really a good use-case. Firecracker can provide a better isolation environment, so it would be great to be able to run Firecracker VMs for workloads, which would require that the host (and the VM host above) support nested virtualization.
- wingmanjd 6mo agoWe run Proxmox VMs that are running Hashicorp's Nomad orchestration at $DAYJOB. The Nomad clients are then turning around and running the docker containers (Proxmox -> Nomad VM -> Docker). For us it's easier to manage and segregate duties on the initial metal this way.
- arch1e 6mo agoThat’s true, bhyve doesn’t support nested virtualization right now. In practice though, most setups don’t actually need it if you’re running workloads directly on the host. Also, if your goal is testing or simulating clusters, you can already run Sylve inside jails. That gives you multiple isolated “nodes” on a single machine without needing nested virt. We have a guide for it here: https://sylve.io/guides/advanced-topics/jailing-sylve/ https://sylve.io/guides/advanced-topics/jailing-sylve/ So you can still experiment with things like clustering, networking, failure scenarios, etc., just using jails instead of spinning up hypervisors inside VMs. Nested virt is still useful for specific cases like testing other hypervisors or running Firecracker inside VMs, but for most Sylve-style setups it hasn’t really been a blocker.
- sidkshatriya 6mo agoI love FreeBSD but Linux just provides every feature under the sun when it comes to virtualization. Do you find any missing features on bhyve ? Is bhyve reliable ? I can't imagine its been tested as thoroughly as KVM ...
- gcifuentes 6mo agoBhyve is quite cool but no nested virt which means you cannot nest vm_enter/exit calls with EPT pages so you cannot virtualize within those guests. I found this crucial. For instance Qubes OS won't run in Bhyve by any means.
- MisterTea 6mo agoIf you are unsure of bhyve's abilities then why not test yourself? Speculation and guessing about stability or testing is useless without seeing if it works in your application.
- sidkshatriya 6mo ago> If you are unsure of bhyve's abilities then why not test yourself? It is not possible to come to a conclusion about everything in the world yourself "from scratch". No one has the time to try out everything themselves. Some filteration process needs to be applied to prevent wasting your finite time. That is why you ask for recommendations of hotels, restaurants, travel destinations, good computer brands, software and so on from friends, relatives or other trusted parties/groups. This does not mean your don't form your opinions. You use the opinions of others as a sort of bootstrap or prior which you can always refine. HN is actually the perfect place to ask for opinions. Someone just said bhyve does not support nested virtualization (useful input !). Someone else might chime in and say they have run bhyve for a long time and they trust it (and so on...) So I can't agree with your viewpoint.
- KaiserPro 6mo agoWhat does Sylve provide that proxmox doesn't? Or better, how does it do it better than proxmox? This isn't to say that proxmox is the best thing since sliced bread, I'm curious as to what makes sylve better, is it the API?
- evanjrowley 6mo agoWithout looking at the Sylve docs, I'll conjecture that it has deeper integration with ZFS. With a foundation on FreeBSD, there is a likelihood Sylve can support ZFS-on-root rollbacks better than hacking it into Proxmox. A rollback capability is why I'm looking for Proxmox alternatives. In the Linux world, Talos Linux and IncusOS provide A/B updates which achieve a similar rollback capability. With something based on FreeBSD, your "immutable" OS and all of it's data can be treated equally as ZFS datasets. There's also a higher risk that a Linux kernel update will break ZFS.
- justsomehnguy 6mo ago> Sylve can support ZFS-on-root rollbacks better than hacking it into Proxmo Can you explain your use case when you absolutely can't provide a separate M.2 drive solely for the OS?
- evanjrowley 6mo agoRegardless of the number of drives available, you gain an advantage when your file system can leverage snapshots to roll backwards or forwards. There are other Linux-native filesystems that can provide this capability too, but many admins prefer ZFS because the full range of capabilities is unparelleled.
- Havoc 6mo agoPerhaps I'm missing your point, but proxmox+lxc on zfs storage works fine in proxmox? If just looks like any other storage in proxmox and on commandline you've got all the usual zfs tools
- dizhn 6mo agoSylve looks like a decent project with a promising future but this article really doesn't explain why they picked it over Proxmox at all. They explain a lot of things but I can't see the advantage over prox other than they wanted to use it.
- Cyph0n 6mo agoSometimes unification can be an advantage. I run Proxmox at home, but now that I have been drinking the NixOS koolaid over the past 2 years, all of my homelab problems suddenly look like Nix-shaped nails.
- dizhn 6mo agoI have the same thing with proxmox especially after I realized how well it integrates with proxmox backup server. And I haven't even gotten into clustering yet. It really is a very solid product.
- Cyph0n 6mo agoIndeed, Proxmox VE is an amazing product.
- EnigmaCurry 6mo agoSame. Here's how I scratch the NixOS itch on Proxmox and/or libvirt[1]. One interface for both targets. [1] https://github.com/EnigmaCurry/nixos-vm-template https://github.com/EnigmaCurry/nixos-vm-template
- dizhn 6mo agoThat feature list looks really good. It would actually be really nice to standardize the guest operating systems in such a way. I actually have a few hosts that only run docker. I might be able to test with those.
- craftkiller 6mo agoWell it looks like we might soon be able to have the benefits of NixOS while also having bhyve (and presumably Sylve): https://github.com/nixos-bsd/nixbsd https://github.com/nixos-bsd/nixbsd
- wolvoleo 6mo agoThis is really interesting. I've played with bhyve before but I didn't realise anyone actually used it in anger. And that people had written such great tooling around it. My home lab still uses ESXi 8. But it needs something new and I was looking at proxmox. However I may give this a try first.
- whalesalad 6mo agoWow, TIL! https://github.com/AlchemillaHQ/Sylve https://github.com/AlchemillaHQ/Sylve Also: https://www.youtube.com/watch?v=wo4oD5UON30 https://www.youtube.com/watch?v=wo4oD5UON30
- SoftTalker 6mo agoThanks, I hadn't heard of it either.
- gnabgib 6mo agoThere was a bit of discussion over the weekend (70 points, 21 comments) https://news.ycombinator.com/item?id=47557392 https://news.ycombinator.com/item?id=47557392 Not sure why this copy made the SCP
- ggm 6mo agoI'd love a simple explanation of the virt/bridge interface choices, and also why people pick NAT vs true address for their jails and virtuals. Likewise for disk i/o -some people swear by 9P as a backing mechanism, some by ZVOL.
- h4kunamata 6mo ago>A lot of our week is made up of the same kinds of small tasks: provision a VM, tweak storage settings, pass through a device, replicate a dataset, share a file, test an image, throw the machine away, do it again. None of that is exciting. All I read is that they are still doing ClickOPS over DevSecOps!! At no moment I heard automation, if you aren't using automation in 2026, your future in IT is cooked. I run Proxmox at home for my homelab. I used to use VMs and now I have fully adopted Proxmox LXC containers (I hate Docker). I use Ansible to automate everything. Last night I wanted to setup a notification service called Gotify, the Ansible playbook must: 1. Create a LXC container with specified resources 2. Prepare the system, network and what not 3. Give me a fully operational LXC and service running, go to the browser and voila. All of that by running one command line, so now I can deploy it over and over. I have setup a LXC container running Radarr, qBittorrent, Sonarr, Jackett, WireGuard VPN via Proton VPN, Iptables firewall aka kill-switch. All of what you just read running within a LXC container fully automated via Ansible, OP is doing everything manually. Even if I was running Sylve, Ansible would be doing the whole automation stuff.
- arch1e 6mo agoAuthor of Sylve here, and I helped deploy the setup in the post. > All I read is that they are still doing ClickOPS over DevSecOps!! Their setup is mostly working on embedded stuff, and this involves some amount of moving VM disk images around, sometimes they run different software within the same VM disk, so that means ZFS properties need to be tweaked accordingly (compression, recordsize, etc). This is a lot easier to do with a UI than it is with CLI, and the UI is pretty good at showing you what’s going on. Now I'm all for automating stuff, but there's no clear pattern here to automate away, Now regarding automation in Sylve, you can create a template out of Sylve (with networking, storage, CPU config etc.) and then deploy that template as many times as you want (from the UI), last I checked proxmox only allows you to clone from template one at a time. What I do is pretty similar to what you mention, but I don't really use ansible since on FreeBSD if it's in the ports tree its one command (after base system is set up) which is `pkg install -y <package>`. And your entire stack (from your list), can be done with one command each. The only thing I see that would need a bit setup would be the wireguard vpn, but even that is pretty straightforward under FreeBSD (so you can do it with a jail and no need for a VM).
- meitham 6mo agoThe article promotes the value of UI for the infrastructure by touching on ZFS. But in this age of Ai, what I’m really looking for is a good api or cli one can let LLM drives. I basically care more about using my infrastructure than how to create it. I know proxmox can do this, but I wish there was a nixos like system where all my VMs are in one file I can verify between LLM making the change and deployment
- jollymonATX 6mo agoWhen I first read this I was like wow bad choice vs sticking w proxmox but then I reflected a bit on my rashness. A tight zfs L1 w/o systemd actually does sound interesting. I'm going to wipe a machine and give it a spin and see for myself. Could be interesting!
- ThomIves 6mo agoI was very happy to read this one. I do use ProxMox, and I have very few issues with it, but perhaps it would be worth the investigation.
- matifali 6mo agoAny plans or thoughts for? Terraform Provider? I use Proxmox as my infrastructure provider and heavily rely on Terraform to provision resources.
- arch1e 6mo agoYes! We're planning out terraform provider, It should land in v0.2.5, we're at v0.2.3 now.