3 ms·

Virtualization on TrueNAS/BSD leaves a lot to be desired and in my experience just wasn't nearly as mature as on Linux with KVM/qemu, etc. Likewise, BSD jails s

Virtualization on TrueNAS/BSD leaves a lot to be desired and in my experience just wasn't nearly as mature as on Linux with KVM/qemu, etc. Likewise, BSD jails seemed to me to be really well thought out, but when it came to getting them to work for me to run little apps/services like I could with Docker, or even a full-fat VM, I quickly gave up because the juice didn't seem worth the squeeze. Whereas on Proxmox it's a cinch for me to run like 20 different VMs/linux containers that I use for various projects/jobs/etc. Huge ecosystem and really easy to take advantage of massive community of linux/Debian users/Docker/etc. TrueNAS Scale, which is based on Debian, I think reflects an acknowledgement of the pain points of working around BSD for the types of services a lot of people want to use their NAS for. I don't think I'm ready to switch yet, but maybe in a year or two it will be a great solution.

xoa 5y ago
Don't disagree with you that using TrueNAS Core as a general server/VM platform (some other things as well) has pain points. Some of which are BSDisms, some just as you say due to the plain fact that the community is vastly smaller. Even simple stuff like ports frequently being a bit out of date, or lacking the diversity you'd find elsewhere. Nothing that can't be worked around, but always some extra time on a whole different thing which in more popular environments (be it Proxmox/KVM/ESXi or whatever) would just be there and then get out of your way. That said I do think though a NAS is one of the things, even in a lot of homelab environments, that's actually worth its own hardware separate from the VM if one can swing the extra cost & space. I virtualized NAS for a while too but since I wanted to run VMs themselves off the NAS as well as a lot of other stuff, have a good backup/restore story etc, I found it started to get unpleasantly circular in terms of bootstrapping from a cold start or restore. Also, the hardware requirements for a good NAS are fairly distinct from a lot of what I wanted for virtualizing. It's not impossible to cram lots of hot swap drives and full size PCIe cards including GPUs for passthrough and tons of CPU and tons of RAM into a single 4U by any means, but splitting things out made it a lot simpler to optimize each side better. The NAS could be smaller, cooler, quieter and good at its own thing. After all-in-one I ended up with 3 distinct boxes: router/gateway (OPNsense), VM system, and NAS system. I know others have gone totally different directions, from AIO boxes to wild K8s clusters doing it all, but I feel like those provide a decent balance of flexibility and stability/isolation, I can mess with stuff and feel confidence in basic network connectivity or storage. Isolation also means not feeling quite the same pressure if one platform isn't as good at something as another, so long as it's good at its own specialty. >TrueNAS Scale, which is based on Debian, I think reflects an acknowledgement of the pain points of working around BSD for the types of services a lot of people want to use their NAS for. I'm a bit bummed philosophically there. I've used BSDs for a very long time, like much of the philosophy if not all the warts, and fundamentally losing diversity in software platforms even if open source is concerning. At the same time it's hard to get around the raw power of network effects in tech, and Linux has a ton of resources and eyes thrown at it at this point. I have no plans or interest in moving in the near future either, but it's hard not to feel like the headwinds are building a bit for TrueNAS Core. Though FreeBSD continues to put out excellent releases, and it's gotten this far. I've found myself on the underdog "but I think it's technically better!" losing side of a lot of tech in my life though :). It's always a balancing act in how much it's worth working on a tool itself vs using the tool to do other work.
- SxC97 5y ago
  > I ended up with 3 distinct boxes: router/gateway (OPNsense), VM system, and NAS system. This was the first lesson I learned doing homelabbing. Put your necessary networking stuff on it's own system that won't be affected by your "hobby boxes" going down :)
  - ziml77 5y ago
    I've seen people talk about virtualizing their router and always thought that was a really bad idea. It feels good not having a ton of single-purpose boxes. But it doesn't take much playing around with them to realize the problem with putting all your important stuff on the same machine: it all goes down together. I currently have my stuff split out into one PCEngines box for OPNSense, one Lenovo NUC for HomeAssistant, and one tower for TrueNAS and low importance virtualzation.
    - xoa 5y ago
      >I've seen people talk about virtualizing their router and always thought that was a really bad idea. It feels good not having a ton of single-purpose boxes. Yeah, there is definitely a balance to be struck. Too many single purpose boxes is very inefficient and also results in its own set of problems in terms of backups and restores. >But it doesn't take much playing around with them to realize the problem with putting all your important stuff on the same machine: it all goes down together. For me at least even that wasn't the biggest problem. It was more the extra difficulty with getting stuff back up again, basically things turning into classic spaghetti code for IRL metal. If one thing took down the other stuff that'd be annoying, but if merely booting back up again fixed everything I'd have felt less compulsion. But if one doesn't have a functional network/internet (particularly in more rural areas like mine where there is effectively zero cellular coverage so no emergency fallback there) it can be a lot harder to recover from a real fubar. And if there is some level of circular dependencies it was easily possible to end up in situation where order is critically important just for startup let alone recovery, and things like backups get fragmented. If the NAS is virtualized yet other VMs live on the NAS, the NAS VM must come up first or other stuff just won't work, including the network if the router is virtualized too on the same system. And where does the hypervisor and NAS VM itself get backed up to? How do those tie into remote backup automation? If things go pearshaped, how does bootstrapping off an offsite/cloud restore go? What's downtime look like? It's definitely possible to set up orchestration to do all this, or HA clusters with high enough reliability it wouldn't be a limiting factor or a bunch of other solutions. But for homelab/SOHO/SMB too much for my comfort. On the other hand I'm fine with something like HomeAssistant being virtualized and consider that more reliable. The VM for it runs on the hypervisor box, the data is stored on the NAS box, and the OPNsense box handles the routing. All 3 can stand alone. If any of them go bad, it's clear how I bring each one back up, and I can lean on the the other two. If the whole place burned down, dealing with insurance aside it's still clear how I'd bring the whole thing back up on new hardware at a new location once I had it. And experimentation done in one place isn't likely to leave me in a tricky position to get back to working. If I was really depending on HomeAssistant or the like for critical home functionality maybe I'd give it its own box too, but even then I still feel like the isolation, central backup, protection against data corruption and so on provided by having it virtualized is better. Also easier to rollback the whole thing. Have to say though also feels amazing that we have so many powerful awesome options all on FOSS and standard hardware now. Can do really great stuff on a budget that not that long ago would have been the domain of enterprise.

xoa 5y ago

Don't disagree with you that using TrueNAS Core as a general server/VM platform (some other things as well) has pain points. Some of which are BSDisms, some just as you say due to the plain fact that the community is vastly smaller. Even simple stuff like ports frequently being a bit out of date, or lacking the diversity you'd find elsewhere. Nothing that can't be worked around, but always some extra time on a whole different thing which in more popular environments (be it Proxmox/KVM/ESXi or whatever) would just be there and then get out of your way. That said I do think though a NAS is one of the things, even in a lot of homelab environments, that's actually worth its own hardware separate from the VM if one can swing the extra cost & space. I virtualized NAS for a while too but since I wanted to run VMs themselves off the NAS as well as a lot of other stuff, have a good backup/restore story etc, I found it started to get unpleasantly circular in terms of bootstrapping from a cold start or restore. Also, the hardware requirements for a good NAS are fairly distinct from a lot of what I wanted for virtualizing. It's not impossible to cram lots of hot swap drives and full size PCIe cards including GPUs for passthrough and tons of CPU and tons of RAM into a single 4U by any means, but splitting things out made it a lot simpler to optimize each side better. The NAS could be smaller, cooler, quieter and good at its own thing. After all-in-one I ended up with 3 distinct boxes: router/gateway (OPNsense), VM system, and NAS system. I know others have gone totally different directions, from AIO boxes to wild K8s clusters doing it all, but I feel like those provide a decent balance of flexibility and stability/isolation, I can mess with stuff and feel confidence in basic network connectivity or storage. Isolation also means not feeling quite the same pressure if one platform isn't as good at something as another, so long as it's good at its own specialty. >TrueNAS Scale, which is based on Debian, I think reflects an acknowledgement of the pain points of working around BSD for the types of services a lot of people want to use their NAS for. I'm a bit bummed philosophically there. I've used BSDs for a very long time, like much of the philosophy if not all the warts, and fundamentally losing diversity in software platforms even if open source is concerning. At the same time it's hard to get around the raw power of network effects in tech, and Linux has a ton of resources and eyes thrown at it at this point. I have no plans or interest in moving in the near future either, but it's hard not to feel like the headwinds are building a bit for TrueNAS Core. Though FreeBSD continues to put out excellent releases, and it's gotten this far. I've found myself on the underdog "but I think it's technically better!" losing side of a lot of tech in my life though :). It's always a balancing act in how much it's worth working on a tool itself vs using the tool to do other work.

SxC97 5y ago
> I ended up with 3 distinct boxes: router/gateway (OPNsense), VM system, and NAS system. This was the first lesson I learned doing homelabbing. Put your necessary networking stuff on it's own system that won't be affected by your "hobby boxes" going down :)
- ziml77 5y ago
  I've seen people talk about virtualizing their router and always thought that was a really bad idea. It feels good not having a ton of single-purpose boxes. But it doesn't take much playing around with them to realize the problem with putting all your important stuff on the same machine: it all goes down together. I currently have my stuff split out into one PCEngines box for OPNSense, one Lenovo NUC for HomeAssistant, and one tower for TrueNAS and low importance virtualzation.
  - xoa 5y ago
    >I've seen people talk about virtualizing their router and always thought that was a really bad idea. It feels good not having a ton of single-purpose boxes. Yeah, there is definitely a balance to be struck. Too many single purpose boxes is very inefficient and also results in its own set of problems in terms of backups and restores. >But it doesn't take much playing around with them to realize the problem with putting all your important stuff on the same machine: it all goes down together. For me at least even that wasn't the biggest problem. It was more the extra difficulty with getting stuff back up again, basically things turning into classic spaghetti code for IRL metal. If one thing took down the other stuff that'd be annoying, but if merely booting back up again fixed everything I'd have felt less compulsion. But if one doesn't have a functional network/internet (particularly in more rural areas like mine where there is effectively zero cellular coverage so no emergency fallback there) it can be a lot harder to recover from a real fubar. And if there is some level of circular dependencies it was easily possible to end up in situation where order is critically important just for startup let alone recovery, and things like backups get fragmented. If the NAS is virtualized yet other VMs live on the NAS, the NAS VM must come up first or other stuff just won't work, including the network if the router is virtualized too on the same system. And where does the hypervisor and NAS VM itself get backed up to? How do those tie into remote backup automation? If things go pearshaped, how does bootstrapping off an offsite/cloud restore go? What's downtime look like? It's definitely possible to set up orchestration to do all this, or HA clusters with high enough reliability it wouldn't be a limiting factor or a bunch of other solutions. But for homelab/SOHO/SMB too much for my comfort. On the other hand I'm fine with something like HomeAssistant being virtualized and consider that more reliable. The VM for it runs on the hypervisor box, the data is stored on the NAS box, and the OPNsense box handles the routing. All 3 can stand alone. If any of them go bad, it's clear how I bring each one back up, and I can lean on the the other two. If the whole place burned down, dealing with insurance aside it's still clear how I'd bring the whole thing back up on new hardware at a new location once I had it. And experimentation done in one place isn't likely to leave me in a tricky position to get back to working. If I was really depending on HomeAssistant or the like for critical home functionality maybe I'd give it its own box too, but even then I still feel like the isolation, central backup, protection against data corruption and so on provided by having it virtualized is better. Also easier to rollback the whole thing. Have to say though also feels amazing that we have so many powerful awesome options all on FOSS and standard hardware now. Can do really great stuff on a budget that not that long ago would have been the domain of enterprise.