4 ms·

Just get a 12 core+ Ryzen and a bunch of RAM (64GB+). It'll be more efficient/faster GHz than a bunch of old servers and more than capable for running three no

Just get a 12 core+ Ryzen and a bunch of RAM (64GB+). It'll be more efficient/faster GHz than a bunch of old servers and more than capable for running three node clusters.

sgt 5y ago
At what cost though?
- candiddevmike 5y ago
  $500-800 for the CPU and RAM ish? I just built a 16 core Ryzen with 32GB of RAM and a bunch of HDDs for about that a year ago.
  - MrFoof 5y ago
    Pretty much spot on, though the ceiling is a lot lower, at least in the US. -- ----- A 12-core Ryzen 9 5900X is now $395 everywhere in the US, though a patient soul can get one on new eBay for $370-375 before sales tax (but including shipping), best $350 used. DDR4-3200 CL16 from top brands isr $3.50-$3.65/GB new, or under $250 for 64GB. Again, a patient soul can save more still and get it closer to $200. Want 32GB? Cut those prices in half. So ~$530-650 new with sales tax and shipping, with a floor of maybe $455-575. For a tremendously capable processor and 32-64GB of memory.
  - sgt 5y ago
    I guess that's pretty cheap. I've also wondered about a Mac Mini running a bunch of Linux VM's. I've to this date never seen a piece of hardware as reliable as the Mac Mini, based on my own and anecdotal evidence.
- deleted 5y ago
  [deleted]
moondev 5y ago
I went the "prosumer" route too. Threadripper maxed with 256G memory. It runs esxi that auto boots my "workstation" vm on startup. This vm has my GPU, nvme, and usb passthru. This way I can boot my workstation "on metal" if needed, but the vm works great. I then manage my lab infra running on the same machine via vCenter html5 ui.
- candiddevmike 5y ago
  Why ESXi just out of curiousity? Seems like most businesses I work with are ripping it out (some with glee) and going acropolis/kubevirt/QEMU or cloud.
  - bombcar 5y ago
    ESXi free edition is real nice and very easy to setup.
  - ocdtrekkie 5y ago
    For really modern stuff sure, but a lot of enterprises are not using containers, they're still Hyper-V or VMware based. The only reason to go Hyper-V is because it's cheaper, so if you can use ESXi free... you choose ESXi. ESXi remains one of the most ridiculously and stupidly stable platforms of all time. Someday someone will find an ESXi host with a twenty year uptime. Not because you should, but because it can.
  - moondev 5y ago
    vSphere (ESXi + vCenter) is much more enjoyable to use than proxmox in my experience. Compare the UI below but you really notice it when using it. https://i0.wp.com/williamlam.com/wp-content/uploads/2022/04/dfi-ec90a-gh-3.png https://i0.wp.com/williamlam.com/wp-content/uploads/2022/04/... https://pbs.twimg.com/media/E5sm0cjXoAIfLM9?format=png&name=large https://pbs.twimg.com/media/E5sm0cjXoAIfLM9?format=png&name=... Here are some other reasons: * Best support for PCIe passthrough, easily toggle PCIe devices from UI without reboot. * Best networking with virtual distrubuted switches - more intuitive to manage and use * vSAN integration with k8s for block volumes is great, including volume snapshots. Also supports RWX (nfs) volumes * vSphere api is a first class citizen with cloud native tooling such as terraform, ansible and packer. It also has a great golang-based cli via govc. With all that said I actually still experiment with kvm (kubevirt, harvester etc) by running it nested inside esxi which works exactly how you would expect. Much prefer vSphere as the base of my lab though.
MrFoof 5y ago
Honestly, depending on what you're doing? Not even that. * Don't underestimate the number of services you can having running in containers on a Raspberry Pi 4B with 8GB of RAM, sipping just 2.1W (yes, that little!) at idle over PoE or as low as 1.35W over WiFi powered by USB-C. With distributions like DietPi you can have a mere 10 processes at startup (using just 44MiB of RAM), but still all the flexibility in the world. * Also don't forget you can cluster them. Want Kubernetes experience? Perfect opportunity to have a cluster than pulls at most like 17W under extreme loads. * Have an old laptop? Fantastic. Great second life for it. You get the bonus of an integrated keyboard, UPS and display if and when you need it, even if it's normally headless. * Don't forget Mac Minis and 1L form factors. I have an 1L form factor (that weighs about 1.3kg) 8-core Zen 3 Ryzen 7 PRO APU that boosts to 4.6GHz, with 64GB of memory, a 1TB SSD and 2.5GbE that sips a mere 10W at idle, and pulls maybe 45W at full bore. Its silent. takes up as much space as like 25 paper napkins, and I can pass the GPU cores through to VMs. * Don't forget hypervisors as well. For truly ephemeral stuff you can just spin up virtual machines on your main desktop or laptop -- just stuff more RAM in it. -- ----- ----- What's great is you can put the power sipping stuff basically ANYWHERE. It can be easily accessible, but entirely hidden from view. It makes no noise. It's not spewing heat or demanding cooling. There's no blinking lights. You're not driving up the electricity bill. Yet you can do very real work on them. For cost? Raspberry Pis aside (sorry, use rpilocator!) check eBay or other marketplace completed listings to establish a price floor, and just be patient and only grab quality stuff that approaches the price floor. Nothing better than getting new (but open box) or barely used modern hardware for peanuts (60-70% off) because the seller is impatient or ignorant.
- marginalia_nu 5y ago
  > Raspberry Pi > old laptop > Also don't forget you can cluster them. Want Kubernetes experience? Just don't be thinking about mixing ARM and x86 in the same Kubernetes cluster. Like even if you technically can run a multi-arch kubernetes cluster, it's a major footgun. Docker's multi-architecture support leaves a lot to be desired and is a real pain to work with.
- kernelbugs 5y ago
  > 1L Zen 3 Ryzen 7 PRO APU What computer is this, if you don't mind sharing? Sounds great for getting 1 or 2 and hosting a lot of VMs.

Just get a 12 core+ Ryzen and a bunch of RAM (64GB+). It'll be more efficient/faster GHz than a bunch of old servers and more than capable for running three node clusters.

sgt 5y ago

At what cost though?

candiddevmike 5y ago
$500-800 for the CPU and RAM ish? I just built a 16 core Ryzen with 32GB of RAM and a bunch of HDDs for about that a year ago.
- MrFoof 5y ago
  Pretty much spot on, though the ceiling is a lot lower, at least in the US. -- ----- A 12-core Ryzen 9 5900X is now $395 everywhere in the US, though a patient soul can get one on new eBay for $370-375 before sales tax (but including shipping), best $350 used. DDR4-3200 CL16 from top brands isr $3.50-$3.65/GB new, or under $250 for 64GB. Again, a patient soul can save more still and get it closer to $200. Want 32GB? Cut those prices in half. So ~$530-650 new with sales tax and shipping, with a floor of maybe $455-575. For a tremendously capable processor and 32-64GB of memory.
- sgt 5y ago
  I guess that's pretty cheap. I've also wondered about a Mac Mini running a bunch of Linux VM's. I've to this date never seen a piece of hardware as reliable as the Mac Mini, based on my own and anecdotal evidence.
deleted 5y ago
[deleted]

moondev 5y ago

I went the "prosumer" route too. Threadripper maxed with 256G memory. It runs esxi that auto boots my "workstation" vm on startup. This vm has my GPU, nvme, and usb passthru. This way I can boot my workstation "on metal" if needed, but the vm works great. I then manage my lab infra running on the same machine via vCenter html5 ui.

candiddevmike 5y ago
Why ESXi just out of curiousity? Seems like most businesses I work with are ripping it out (some with glee) and going acropolis/kubevirt/QEMU or cloud.
- bombcar 5y ago
  ESXi free edition is real nice and very easy to setup.
- ocdtrekkie 5y ago
  For really modern stuff sure, but a lot of enterprises are not using containers, they're still Hyper-V or VMware based. The only reason to go Hyper-V is because it's cheaper, so if you can use ESXi free... you choose ESXi. ESXi remains one of the most ridiculously and stupidly stable platforms of all time. Someday someone will find an ESXi host with a twenty year uptime. Not because you should, but because it can.
- moondev 5y ago
  vSphere (ESXi + vCenter) is much more enjoyable to use than proxmox in my experience. Compare the UI below but you really notice it when using it. https://i0.wp.com/williamlam.com/wp-content/uploads/2022/04/dfi-ec90a-gh-3.png https://i0.wp.com/williamlam.com/wp-content/uploads/2022/04/... https://pbs.twimg.com/media/E5sm0cjXoAIfLM9?format=png&name=large https://pbs.twimg.com/media/E5sm0cjXoAIfLM9?format=png&name=... Here are some other reasons: * Best support for PCIe passthrough, easily toggle PCIe devices from UI without reboot. * Best networking with virtual distrubuted switches - more intuitive to manage and use * vSAN integration with k8s for block volumes is great, including volume snapshots. Also supports RWX (nfs) volumes * vSphere api is a first class citizen with cloud native tooling such as terraform, ansible and packer. It also has a great golang-based cli via govc. With all that said I actually still experiment with kvm (kubevirt, harvester etc) by running it nested inside esxi which works exactly how you would expect. Much prefer vSphere as the base of my lab though.

MrFoof 5y ago

Honestly, depending on what you're doing? Not even that. * Don't underestimate the number of services you can having running in containers on a Raspberry Pi 4B with 8GB of RAM, sipping just 2.1W (yes, that little!) at idle over PoE or as low as 1.35W over WiFi powered by USB-C. With distributions like DietPi you can have a mere 10 processes at startup (using just 44MiB of RAM), but still all the flexibility in the world. * Also don't forget you can cluster them. Want Kubernetes experience? Perfect opportunity to have a cluster than pulls at most like 17W under extreme loads. * Have an old laptop? Fantastic. Great second life for it. You get the bonus of an integrated keyboard, UPS and display if and when you need it, even if it's normally headless. * Don't forget Mac Minis and 1L form factors. I have an 1L form factor (that weighs about 1.3kg) 8-core Zen 3 Ryzen 7 PRO APU that boosts to 4.6GHz, with 64GB of memory, a 1TB SSD and 2.5GbE that sips a mere 10W at idle, and pulls maybe 45W at full bore. Its silent. takes up as much space as like 25 paper napkins, and I can pass the GPU cores through to VMs. * Don't forget hypervisors as well. For truly ephemeral stuff you can just spin up virtual machines on your main desktop or laptop -- just stuff more RAM in it. -- ----- ----- What's great is you can put the power sipping stuff basically ANYWHERE. It can be easily accessible, but entirely hidden from view. It makes no noise. It's not spewing heat or demanding cooling. There's no blinking lights. You're not driving up the electricity bill. Yet you can do very real work on them. For cost? Raspberry Pis aside (sorry, use rpilocator!) check eBay or other marketplace completed listings to establish a price floor, and just be patient and only grab quality stuff that approaches the price floor. Nothing better than getting new (but open box) or barely used modern hardware for peanuts (60-70% off) because the seller is impatient or ignorant.

marginalia_nu 5y ago
> Raspberry Pi > old laptop > Also don't forget you can cluster them. Want Kubernetes experience? Just don't be thinking about mixing ARM and x86 in the same Kubernetes cluster. Like even if you technically can run a multi-arch kubernetes cluster, it's a major footgun. Docker's multi-architecture support leaves a lot to be desired and is a real pain to work with.
kernelbugs 5y ago
> 1L Zen 3 Ryzen 7 PRO APU What computer is this, if you don't mind sharing? Sounds great for getting 1 or 2 and hosting a lot of VMs.