3 ms·

Out of interest, have any self-hosters here tried one big machine with lots of VMs, as opposed to a stack of NUCs/Pis? The recent Intel scares have got me inte

Out of interest, have any self-hosters here tried one big machine with lots of VMs, as opposed to a stack of NUCs/Pis? The recent Intel scares have got me interested in firmware and Libreboot again, and a HN commenter[0] pointed me at some extremely beefy (e.g. 16 cores!) AMD motherboard/CPU combos, that it’s possible to run fully-free firmware on. Without doing much actual research, Xen on a big box seems like it could be a good way to have a single physical machine where I ‘deploy’ different services in VMs. Anyone gone this route, or can think of any problems with this approach? It seems like it could be a little easier to manage, but I’m sure there are extra considerations for networking and security. [0]: https://news.ycombinator.com/threads?id=153791098c https://news.ycombinator.com/threads?id=153791098c

stefanfisk 8y ago
I’m contemplating something similar! If you want you can even run windows in kvm with pci pass through for native gpu performance.
HappyRobot 8y ago
You might be interested in the https://www.reddit.com/r/homelab https://www.reddit.com/r/homelab subreddit. The prevailing wisdom there is to buy used enterprise hardware, install a hyperadvisor, and set up separate VM's for everything. It's easy to scale, easy to tinker with, and really flexible.
fallenhitokiri 8y ago
OP here - all of my services run in a VM on the Dell server, including some other VMs used for work. Works pretty well, but you pay a slightly higher energy bill. As long as only a few users (in my case between 1 and 6) use the services you'd be surprised with how little resources you can get away. My initial test VM (VMWare) I used had 1 CPU and 1GB memory - it ran Postgres, miniflux, docker registry, gitea and wallabag without problems. CPU mostly idle, memory without a lot of load ~350MB. (Alpine as base for docker)
mindslight 8y ago
There's the security issue of general VM escape, and sidechannel attacks across VMs (eg Spectre). And of course you want redundancy against losing an entire machine (say the PSU fails and overvolts the 12v rail). But more centralized ad-hoc infrastructure should be easier to grok/manage/backup against this. My new router is pfsense running on KVM/Debian on an i5-4430. If I start running more Internet-facing services from home, they'll probably be on that box rather than my Libreboot KGPE. FWIW, Newegg has been clearing out those KGPE boards, occasionally selling them for $80 - https://pcpartpicker.com/product/yvJwrH/asus-motherboard-kgped16 https://pcpartpicker.com/product/yvJwrH/asus-motherboard-kgp... (the last one was on flash.newegg.com and so doesn't show up on PCPP). Non-high-density G34 heatsinks are a bit tough to find these days, but there's a thread where someone details the steps of mounting a 212 Evo. The Raijintek Aidos worked for me just by fashioning the appropriate length of steel bar, drilling four holes (two for the G34 mount, two for securing the existing Raijintek piece), and adding the appropriate screws/springs. The fan mounting on the Aidos does suck though.

stefanfisk 8y ago

I’m contemplating something similar! If you want you can even run windows in kvm with pci pass through for native gpu performance.

HappyRobot 8y ago

You might be interested in the https://www.reddit.com/r/homelab https://www.reddit.com/r/homelab subreddit. The prevailing wisdom there is to buy used enterprise hardware, install a hyperadvisor, and set up separate VM's for everything. It's easy to scale, easy to tinker with, and really flexible.

fallenhitokiri 8y ago

OP here - all of my services run in a VM on the Dell server, including some other VMs used for work. Works pretty well, but you pay a slightly higher energy bill. As long as only a few users (in my case between 1 and 6) use the services you'd be surprised with how little resources you can get away. My initial test VM (VMWare) I used had 1 CPU and 1GB memory - it ran Postgres, miniflux, docker registry, gitea and wallabag without problems. CPU mostly idle, memory without a lot of load ~350MB. (Alpine as base for docker)

mindslight 8y ago

There's the security issue of general VM escape, and sidechannel attacks across VMs (eg Spectre). And of course you want redundancy against losing an entire machine (say the PSU fails and overvolts the 12v rail). But more centralized ad-hoc infrastructure should be easier to grok/manage/backup against this. My new router is pfsense running on KVM/Debian on an i5-4430. If I start running more Internet-facing services from home, they'll probably be on that box rather than my Libreboot KGPE. FWIW, Newegg has been clearing out those KGPE boards, occasionally selling them for $80 - https://pcpartpicker.com/product/yvJwrH/asus-motherboard-kgped16 https://pcpartpicker.com/product/yvJwrH/asus-motherboard-kgp... (the last one was on flash.newegg.com and so doesn't show up on PCPP). Non-high-density G34 heatsinks are a bit tough to find these days, but there's a thread where someone details the steps of mounting a 212 Evo. The Raijintek Aidos worked for me just by fashioning the appropriate length of steel bar, drilling four holes (two for the G34 mount, two for securing the existing Raijintek piece), and adding the appropriate screws/springs. The fan mounting on the Aidos does suck though.