6 ms·

Rack mount hardware is almost always expensive, loud, and power hungry. I just have never seen the point of building a home lab like this. A single ATX desktop

Rack mount hardware is almost always expensive, loud, and power hungry. I just have never seen the point of building a home lab like this. A single ATX desktop can do almost [1] everything a homelab at a fraction of the cost & power consuption. I think a lot of the reason for homelabs/server hardware was to get access to more CPU cores, now that 8+ cores are very cheap it is actually cheaper to buy a new consumer desktop than it is to run an old server. What makes even less sense is wanting use to use software like vSphere or ESXi since its about 10x more complicated than just using virt-manager/QEMU. It's like using an excavator to dig a fire pit. Server hardware & software makes sense when it's not your home because then you do need a remote access tool like iDRACK. (There are DIY options if you just need something for personal use). That said if you enjoy it as a hobby (or your homelab is actually a business thing) then go for it.

neilv 5y ago
Absolutely, rackmount servers tend to be very loud and often power-hungry. But if you think of rackmount as a form factor, it can make some practical and aesthetic sense. The IKEA CORRAS Birch Effect "rack cabinet" in my living room currently has a nice virtually silent GPU compute server based on a consumer/gaming PC in a 4U short-depth rackmount case, a mostly passively-cooled Supermicro Atom server in which I've replaced the PSU fan with a Noctua, and a sinewave UPS. (There's an ongoing evolution of gear over time, including more exotic stuff, and most toys I might play with in the future will also fit this form factor.) Aesthetically, those stack on the bottom of the cabinet, a Birch Effect shelf sits on top, and the plastic OpenWrt WiFi router with all the pokey antennae sits safely in the shelf cubbyhole. (Once I need detached WiFi APs, I'll probably build a 1U or 2U router with opnSense/Linux/*BSD. And I have a discreet slide-out rackmount console for if I ever get a deeper cabinet.) I like to think it looks like understated black home AV that's not out of place in a living room, which is better than a tangle of assorted non-rackmount PCs and UPS. Practically, besides the tidy organizing and uniform cooling airflow, I have some rack posts I can use if the stack of gear gets too large/tricky for pulling individual boxes without disturbing the others too much. Also, when I lived in a dodgier student apartment, fastening a bunch of rackmount gear together with security-head screws seemed a good way to discourage a burglar from walking off with my data.
luke2m 5y ago
Exactly. The point of homelabbing at that scale is to dig a fire pit with an excavator, in order to learn how to use said excavator.
walshemj 5y ago
You really need the separate physical hardware a home lab is not just a bunch of VM's One of the best things about doing my CCNA courses at a CISCI academy was having 20 switches and routers to play with and find those areas where it does not always work out the way the books say they should.
jjav 5y ago
> Rack mount hardware is almost always expensive, loud, and power hungry. I just have never seen the point of building a home lab like this. Rack mount is just a form factor, so that generalization doesn't make sense. You can get pretty much any capacity systems you need in a rack case, including small cheap ones. When I got tired of having systems in various different case sizes and a mess of wiring and switches etc, I moved everything at home to rack mounted. Now literally everything is one big box (16U rack), with just power and network cables going in. All the computers, network switches, wiring, UPSs, KVM switch and monitor. Very tidy, very clean. The computers themselves are cheap, quiet and efficient, since I don't need any monster servers at home. A couple are Atom-based (<20W total), one Celeron and one desktop AMD-based (forgot model). Same level of hardware I had before rack mounting, but now a lot neater and more organized.
- KingMachiavelli 5y ago
  True, if you build something inside a rackmount case then it can be whatever you want. However, from what I could tell empty rack mount case & rails are still a bit of a premium. Were you able to get rack mount case w/ rails at decent prices? Seems each one would have to be at least a 2U to fit a normal ATX PSU ( I guess you could use an ITX or small PSU but those can be more expensive for the same wattage). How did you handle the system & case fans? Did you rack cases have the standard fans and were you able to configure those to be slow/quiet?
  - jjav 5y ago
    Most of my systems are from Supermicro, so they already came with all the suitable components for 1U.
  - fomine3 5y ago
    Rosewill cases are affordable. 4U cases supports 12cm fan so any silent fan can be installed.
glangdale 5y ago
Agreed. The startup I used to work for (acquired by Intel) both before and after acquisition had a good deal of lab hardware. Between evaluation boards and rack mount hardware our server room was insanely loud - like "it's 25M away and behind a closed door, but you can still tell when the lunchtime QA and benchmarking runs kick off". There's no way I could live with that in my house, or even a fraction of it.
LargoLasskhyfv 5y ago
But excavators are fun!
userbinator 5y ago
Classic cars are also expensive, loud, and power hungry ;-) Sometimes it's not about being practical, it's about having fun playing with stuff you wouldn't otherwise normally use.
MrFoof 5y ago
There's a lot to be said for 1L form factors, which were included in the article, which are basically the size of a Mac Mini. I have an HP ProDesk 405 Mini, with an AMD Ryzen 7 PRO 4750GE (Zen 2 based). 8 cores, 16 threads, with a base clock of 3.1GHz and boosts to 4.3GHz. Includes 8 Vega 64 GPU cores. Tossed 64GB of memory into it and a 1TB Samsung 980 Pro. Supports AMD Dash for basic LOM. Idles at like 12-14W. Always silent. When stuff with the 5750GE hit they'll be better still. It's a great (and cheaper!) alternative to something like a Threadripper 5970X (when it hits in Novemberish) desktop if you wanted to cluster a few and have a low-power Apache Spark cluster that can actually rip through things. I think the only downer would be the lack of 10GbE support unless the next round of 1Ls offer 10GbE cards as options.
p_j_w 5y ago
>What makes even less sense is wanting use to use software like vSphere or ESXi since its about 10x more complicated than just using virt-manager/QEMU. I always just assumed it was as a learning experience. When I was in my 20s, I didn't try to get Apache, qmail, and bind running because it was practical for me, I wanted a marketable skill. There are lucrative jobs out there for people who know these technologies.
- awat 5y ago
  That was my experience as well. Home labs seem to be cyclical for a lot people including myself. Started with a big overkill rack to learn technologies, and now I’m down to a 10 inch desk rack that is ATX case size with just a few things to run my network and small VMs.
- KingMachiavelli 5y ago
  The application side is a more understandable. There are lots of reasons to know and use apache, postfix, etc. As far as learning goes, if you really want to work in enterprise IT then vSphere is good to know but if you are willing to learn things on your own then you might as well learn kubernetes, docker, etc. Knowing things like vSphere is cool & perhaps useful but it also hides the how things work. If you want to know and understand things it is better to stick to the open source & interact directly with KVM and Xen. Like you wouldn't use cPanel to learn how a LAMP stack works.
  - bluedino 5y ago
    What if you just want to spin up servers in an easy to use platform so you can use them to other things, and not learn the integrals of virtualization?
    - easton 5y ago
      Then KVM is still probably the move. Install Ubuntu, then install Cockpit for a web interface to manage KVM and Docker. If you want a little more depth, Proxmox is also good but there’s some mild learning curve there.
jrm4 5y ago
Right? Obviously, everyone should do their thing. But I suppose the tiny tiny "issue" I might have would be - I kind of feel like this reinforces the idea that "having server things in the house is big and complex." So I encourage everyone who does this also tells the newbs, "I mean, you could also just slap linux on that old computer in the corner and do 95% of what I"m doing here, BUT MINE WILL LOOK COOLER."
PaulWaldman 5y ago
>What makes even less sense is wanting use to use software like vSphere or ESXi since its about 10x more complicated than just using virt-manager/QEMU. I disagree. Assuming your example of a single ATX desktop, ESXi really is easy to setup and modern versions provide a graphical web client. This assumes you're staying away from vSAN, vMotion, and iSCSI storage.
- kazen44 5y ago
  but, assuming you are learning the vmware stack because of employment. Not knowing things like vsan, ISCSI and vmotion makes your lab nearly worthless. Learning vsphere and esxi properly requires atleast a decently sized cluster, especially if you start throwing NSX in the mix.
  - devonkim 5y ago
    Forget even iSCSI as the goalpost, a lot of places still use fibrechannel drives and controllers and FCoE makes sense mostly once you’ve hit at least 10 GbE which is a bit pricey for the sake of learning while it’s basically hot garbage in professional environments worth a career in. Half the point of vSphere in a professional capacity requires use of several nodes such as different HA options, how to trunk networks, distinctions between different LUN abstractions (RDMs in physical v virtual compatibility modes) and an endless parade of the ways different vendors’ SANs and physical switches can completely mess up and ruin your entire month if you’re not pedantic about every random switch or flag in configuration. All these other vendor minutiae being so important rather than the general concepts as a practitioner is part of why I don’t do VMware professionally anymore and went quietly to SRE janitoring in major cloud providers and chalked those days up as time in IT. It’s just much more practical to learn AWS than to pore over the Cisco Nexus documentation unless one’s career really is in the lower infrastructure levels of tech.
deeblering4 5y ago
The lenovo “tiny” hardware recommended in the article is really ideal. Its essentially laptop components in a micro case (no hid/battery/screen), they even are powered by a laptop style external DC adapter. They are affordable, quiet, powerful (modern x86_64 with basic gpu) and light on power usage. I run a pair of m92p myself.
- KingMachiavelli 5y ago
  The downside to these small form factor kit PCs is that then you are very connectivity limited. You can't use it to build a NAS directly or GPU connected VMs, etc. They are quite good as a cheap thin client that you use to access your more powerful hardware. As hardware ages it tends to lack some of the nice feature's like dual 4K@60Hz output, thunderbolt, etc. so having a new but cheap/lowpower machine helps.
  - deeblering4 5y ago
    Thats true. This model has just one small expansion slot so you have to be clever. But its fine for general purpose compute. Yeah realistically NAS is out, but thats probably ok. I mean NAS is not a great fit for most general purpose computers. In a pinch you could go usb-3 jbod or something. But personally I think it’d be better to go with either specifically storage oriented hardware, or a scale out filesystem on top of a cheap cluster, something like odroid-hc2. Personally I like to keep things compartmentalized even at home. So my NAS is a dedicated (off the shelf) system, and the lenovo mini servers mount it via NFS/CIFS.
Steltek 5y ago
Yes! Companies refresh Dells pretty often and you can find an i7 for not much money on eBay. Buy two to double up the RAM. If you know the right people at a company, you could get them for free even. Home labs aren't about millions of hits. It's a playground.
kaladin-jasnah 5y ago
> A single ATX desktop can do almost [1] everything a homelab at a fraction of the cost & power consuption. So this is true, but the iDRAC/iLO on my big, loud server has a virtual KVM feature that lets me lazily sit at my desk and click buttons and install my server operating system of choice. It saves a lot of space and effort compared to flashing a USB, going and plugging in a keyboard, (mouse), and monitor, and going through the whole deal. I'd wager that's one of the big things that would compel me to buy a rack server. I recently built a nice ATX desktop, fitted with a 5950X and everything, and I found that the PiKVM project [0] does a pretty good job at replacing that "integral" part of the server for me (you can also look into an ASRock Rack PAUL [1], but good luck finding one for sale right now. > What makes even less sense is wanting use to use software like vSphere or ESXi since its about 10x more complicated than just using virt-manager/QEMU A lot of people (not me, I end up using libvirt/QEMU as it suits my needs) buy homelabs to work towards having hands-on experience for their system administration job, which uses ESXi/vSphere. It might also be for working on getting certifications from VMware, in which case they really don't have any choice but to use ESXi on their servers. > Server hardware & software makes sense when it's not your home because then you do need a remote access tool like iDRACK Now, I addressed this earlier (laziness), but these BMC things are very useful—you can monitor the health of various components of your server, and I believe even update the BIOS without stepping out of your chair. It makes administering a homelab much easier, and even the Pi-KVM, a DIY option, I'm pretty sure, doesn't have monitoring features. Plus, those DIY solutions require wiring stuff into your ATX motherboard, which can get janky and might put off people who want a turnkey solution. [0] https://pi-kvm.org/ https://pi-kvm.org/ [1] https://www.asrockrack.com/general/productdetail.asp?Model=PAUL#Specifications https://www.asrockrack.com/general/productdetail.asp?Model=P...
- PragmaticPulp 5y ago
  > So this is true, but the iDRAC/iLO on my big, loud server has a virtual KVM feature This is available on AMD Ryzen motherboards, too. Just get one of ASRock's motherboards with the BMC controller built-in: https://www.asrockrack.com/general/productdetail.asp?Model=X470D4U#Specifications https://www.asrockrack.com/general/productdetail.asp?Model=X... No need to mess with PiKVM or add-in cards. It's a server board with KVM management that works out of the box with Ryzen processors. It might need a BIOS update to support your 5950X, but it will work.
PragmaticPulp 5y ago
> Rack mount hardware is almost always expensive, loud, and power hungry. Buying old rack-mount server hardware for home use is almost always a mistake. Old server hardware may feel cheap when you see an old dual-socket rack mount server on eBay with hardware that was fast 8 years ago, but you can probably meet or exceed the performance with something like a cheap 8-core Ryzen. Rack mount servers are also exceptionally loud. Unless you love the noise of small, high-RPM server fans, you don't want rack mount server hardware in your house. And don't forget the power bill. Some old servers idle at hundreds of watts, which will add up over the several years you leave it running. 24/7 server hardware is a good example of where it makes sense to be mindful of power consumption. > What makes even less sense is wanting use to use software like vSphere or ESXi since its about 10x more complicated than just using virt-manager/QEMU. I disagree. ESXi is actually extremely easy to use, as long as you pick compatible hardware up front. The GUI isn't perfect, but it's intuitive enough that I feel confident clicking around to accomplish what I need instead of looking up a tutorial first.
- kube-system 5y ago
  I have used consumer hardware and enterprise hardware in my rack. There’s pros and cons either way. Consumer hardware is cheaper, more power efficient, and quiet. But, there are a few reasons I have switched to enterprise hardware: * remote management and redundancy features make it more reliable if I’m traveling and want to remote in to do something * some software works better with enterprise hardware features (i.e. special disk controller modes) * I feel like my development experience is closer to what I can expect in production * 100+ gigs of RAM on one system without a big upfront expense * the occasional PITA of working with enterprise hardware helps me to understand what I might be expecting out of infrastructure team in production, or design ways to make their life easier
  - deleted 5y ago
    [deleted]
- fomine3 5y ago
  Latest Ryzen's CPU perf and max 128GB ECC UDIMM should be enough for most users, but it still lacks homelabby features and flexibilities. I need more than 16x PCIe lanes that supports ACS and IPMI, so I go for EPYC build. It's great except idle power consumption. I with they improve it to Xeon level, but maybe chiplet architecture (8CCD and massive CCD) isn't good about it. For hypervisor, I recommend Proxmox VE.
Lammy 5y ago
> I just have never seen the point of building a home lab like this. https://www.youtube.com/watch?v=38ApYaywLzs https://www.youtube.com/watch?v=38ApYaywLzs I have to pay PG&E rates for power ($$$) so I'm a big fan of lower-power hardware for a system I'm going to leave on 24/7, e.g. my 25W-TDP 1U racked 8-core ECC-equipped Atom server built on this board: https://www.supermicro.com/en/products/motherboard/A2SDi-8C+-HLN4F https://www.supermicro.com/en/products/motherboard/A2SDi-8C+...
- c_o_n_v_e_x 5y ago
  I like the Atom C-series... interesting option for edge deployed equipment.
bradstewart 5y ago
I had more space in my network rack than near my desk, so I bought a cheap Rosewill rackmountable ATX case, and rebuilt my old desktop into it (since I pretty much only use my laptop these days). But I agree that buying old Dell servers for home use is rather silly at this point.
aftbit 5y ago
> Rack mount hardware is almost always expensive, loud, and power hungry. It really depends on the particular hardware. I recently picked up an R720 with 128GB of memory and dual E5-2670v1 for $450 on eBay. It idles at 120W (about the same as my brand new Ryzen 5950X desktop). It is not much louder than my old air-cooled 8700K consumer PC. Of course, it's not much faster either, and it's definitely slower than my Ryzen. I bought it to learn how to use iDRAC and practice ZFS with 10x $20 1GB 10k RPM SAS drives. Also maybe to give Proxmox a try. All of my practical home-prod stuff runs on an old i5 desktop, not my rack servers.
- fomine3 5y ago
  120W at idle on desktop computer looks not good.
  - KingMachiavelli 5y ago
    The issue is that power savings stuff is often bad on Linux... especially on older hardware. 'Idle' in this context means running with near zero load since a server cannot suspend or hibernate. In theory you could setup hibernate/suspend + WoL but that tends to not work well. Some power saving features will really only save 5-10W and can make PCIe cards unstable, poor hard drive performance, laggy keyboard & mouse, etc. In particuler, GPUs often have poor suport for the normal power saving features. I just pulled the dedicated GPU (GTX 760) out of my old machine that is now a headless server and that saved ~20W. What's weird and bad is that that 20W was being consumed without a graphical interface even running.
dheera 5y ago
> expensive This isn't always true. I got a 24-port Aruba 802.3at PoE switch with FOUR 10-gigabit ports, for a grand total of $120. > loud, and power hungry Again, not always true. Enterprises do care about power a lot of the time. I highly recommend the Ubiquiti stuff for home lab use -- most of it is pretty quiet. If you buy other rack-mount hardware, try to buy at least 2U hardware, the bigger fans are much quieter than the 40mm fans in 1U equipement. If you must buy non-Ubiquiti 1U equipment, you can usually change the fans out for Noctua fans. > A single ATX desktop You can build an ATX desktop into a 4U case. I highly recommend the SilverStone RM42-502 for about $250 on Amazon. It takes standard components including standard ATX power supply, standard ATX, micro-ATX, or mini-ITX motherboard, standard fans (or even a CorsAir liquid cooler), it's basically a standard case. If you use quiet components it will be quiet. My ATX rack mount PC is not even noticeable unless I'm running up my GPU doing machine learning stuff. There are much cheaper cases available as well, but the SilverStone case is quality, and will last you forever, you can just keep building new PCs into it for as long as ATX/ITX exist. One of the advantages to building your PC in a rack mount configuration is that it's very easy to stack multiple PCs along with your network routers, switches, NAS, UPS, in one nice rack that's easy to move from apartment to apartment in one piece, and all your cables and connections stay nice and tidy. It's also ideal if you play with a lot of smaller devices. For example if you want to have a cluster of 10 RPis, a rack mount solution is great for keeping the ethernet and power cables tidy, and it isn't going to be loud or any more power hungry than if you had them spread out across the table. You can also 3D print rack mounts for non-rack equipment, just to keep them tidy. My rack: https://i.redd.it/xcss9uassrg71.jpg https://i.redd.it/xcss9uassrg71.jpg
actually_a_dog 5y ago
I agree. My setup is very much like the setup shown near the end of the article (the one that consists of a couple Synology NAS boxes and what looks like a few Mac Minis or other small form factor PCs). I have a file server with a decently large storage array and a few Raspberry Pis and other small electronic gizmos, some of which connect to my wifi, and some which plug into a PC via various cables (ethernet, USB, etc.) The only thing out of this that consumes any amount of power worth mentioning is the file server/storage array. I haven't measured how much power it uses (I probably should), but I'm able to minimize it by allowing the disks to go to sleep and the CPU to run slower than when I'm actively using it. I've never felt limited by this setup at all, but, then again, my home lab isn't really my main hobby.

neilv 5y ago

Absolutely, rackmount servers tend to be very loud and often power-hungry. But if you think of rackmount as a form factor, it can make some practical and aesthetic sense. The IKEA CORRAS Birch Effect "rack cabinet" in my living room currently has a nice virtually silent GPU compute server based on a consumer/gaming PC in a 4U short-depth rackmount case, a mostly passively-cooled Supermicro Atom server in which I've replaced the PSU fan with a Noctua, and a sinewave UPS. (There's an ongoing evolution of gear over time, including more exotic stuff, and most toys I might play with in the future will also fit this form factor.) Aesthetically, those stack on the bottom of the cabinet, a Birch Effect shelf sits on top, and the plastic OpenWrt WiFi router with all the pokey antennae sits safely in the shelf cubbyhole. (Once I need detached WiFi APs, I'll probably build a 1U or 2U router with opnSense/Linux/*BSD. And I have a discreet slide-out rackmount console for if I ever get a deeper cabinet.) I like to think it looks like understated black home AV that's not out of place in a living room, which is better than a tangle of assorted non-rackmount PCs and UPS. Practically, besides the tidy organizing and uniform cooling airflow, I have some rack posts I can use if the stack of gear gets too large/tricky for pulling individual boxes without disturbing the others too much. Also, when I lived in a dodgier student apartment, fastening a bunch of rackmount gear together with security-head screws seemed a good way to discourage a burglar from walking off with my data.

luke2m 5y ago

Exactly. The point of homelabbing at that scale is to dig a fire pit with an excavator, in order to learn how to use said excavator.

walshemj 5y ago

You really need the separate physical hardware a home lab is not just a bunch of VM's One of the best things about doing my CCNA courses at a CISCI academy was having 20 switches and routers to play with and find those areas where it does not always work out the way the books say they should.

jjav 5y ago

> Rack mount hardware is almost always expensive, loud, and power hungry. I just have never seen the point of building a home lab like this. Rack mount is just a form factor, so that generalization doesn't make sense. You can get pretty much any capacity systems you need in a rack case, including small cheap ones. When I got tired of having systems in various different case sizes and a mess of wiring and switches etc, I moved everything at home to rack mounted. Now literally everything is one big box (16U rack), with just power and network cables going in. All the computers, network switches, wiring, UPSs, KVM switch and monitor. Very tidy, very clean. The computers themselves are cheap, quiet and efficient, since I don't need any monster servers at home. A couple are Atom-based (<20W total), one Celeron and one desktop AMD-based (forgot model). Same level of hardware I had before rack mounting, but now a lot neater and more organized.

KingMachiavelli 5y ago
True, if you build something inside a rackmount case then it can be whatever you want. However, from what I could tell empty rack mount case & rails are still a bit of a premium. Were you able to get rack mount case w/ rails at decent prices? Seems each one would have to be at least a 2U to fit a normal ATX PSU ( I guess you could use an ITX or small PSU but those can be more expensive for the same wattage). How did you handle the system & case fans? Did you rack cases have the standard fans and were you able to configure those to be slow/quiet?
- jjav 5y ago
  Most of my systems are from Supermicro, so they already came with all the suitable components for 1U.
- fomine3 5y ago
  Rosewill cases are affordable. 4U cases supports 12cm fan so any silent fan can be installed.

glangdale 5y ago

Agreed. The startup I used to work for (acquired by Intel) both before and after acquisition had a good deal of lab hardware. Between evaluation boards and rack mount hardware our server room was insanely loud - like "it's 25M away and behind a closed door, but you can still tell when the lunchtime QA and benchmarking runs kick off". There's no way I could live with that in my house, or even a fraction of it.

LargoLasskhyfv 5y ago

But excavators are fun!

userbinator 5y ago

Classic cars are also expensive, loud, and power hungry ;-) Sometimes it's not about being practical, it's about having fun playing with stuff you wouldn't otherwise normally use.

MrFoof 5y ago

There's a lot to be said for 1L form factors, which were included in the article, which are basically the size of a Mac Mini. I have an HP ProDesk 405 Mini, with an AMD Ryzen 7 PRO 4750GE (Zen 2 based). 8 cores, 16 threads, with a base clock of 3.1GHz and boosts to 4.3GHz. Includes 8 Vega 64 GPU cores. Tossed 64GB of memory into it and a 1TB Samsung 980 Pro. Supports AMD Dash for basic LOM. Idles at like 12-14W. Always silent. When stuff with the 5750GE hit they'll be better still. It's a great (and cheaper!) alternative to something like a Threadripper 5970X (when it hits in Novemberish) desktop if you wanted to cluster a few and have a low-power Apache Spark cluster that can actually rip through things. I think the only downer would be the lack of 10GbE support unless the next round of 1Ls offer 10GbE cards as options.

p_j_w 5y ago

>What makes even less sense is wanting use to use software like vSphere or ESXi since its about 10x more complicated than just using virt-manager/QEMU. I always just assumed it was as a learning experience. When I was in my 20s, I didn't try to get Apache, qmail, and bind running because it was practical for me, I wanted a marketable skill. There are lucrative jobs out there for people who know these technologies.

awat 5y ago
That was my experience as well. Home labs seem to be cyclical for a lot people including myself. Started with a big overkill rack to learn technologies, and now I’m down to a 10 inch desk rack that is ATX case size with just a few things to run my network and small VMs.
KingMachiavelli 5y ago
The application side is a more understandable. There are lots of reasons to know and use apache, postfix, etc. As far as learning goes, if you really want to work in enterprise IT then vSphere is good to know but if you are willing to learn things on your own then you might as well learn kubernetes, docker, etc. Knowing things like vSphere is cool & perhaps useful but it also hides the how things work. If you want to know and understand things it is better to stick to the open source & interact directly with KVM and Xen. Like you wouldn't use cPanel to learn how a LAMP stack works.
- bluedino 5y ago
  What if you just want to spin up servers in an easy to use platform so you can use them to other things, and not learn the integrals of virtualization?
  - easton 5y ago
    Then KVM is still probably the move. Install Ubuntu, then install Cockpit for a web interface to manage KVM and Docker. If you want a little more depth, Proxmox is also good but there’s some mild learning curve there.

jrm4 5y ago

Right? Obviously, everyone should do their thing. But I suppose the tiny tiny "issue" I might have would be - I kind of feel like this reinforces the idea that "having server things in the house is big and complex." So I encourage everyone who does this also tells the newbs, "I mean, you could also just slap linux on that old computer in the corner and do 95% of what I"m doing here, BUT MINE WILL LOOK COOLER."

PaulWaldman 5y ago

>What makes even less sense is wanting use to use software like vSphere or ESXi since its about 10x more complicated than just using virt-manager/QEMU. I disagree. Assuming your example of a single ATX desktop, ESXi really is easy to setup and modern versions provide a graphical web client. This assumes you're staying away from vSAN, vMotion, and iSCSI storage.

kazen44 5y ago
but, assuming you are learning the vmware stack because of employment. Not knowing things like vsan, ISCSI and vmotion makes your lab nearly worthless. Learning vsphere and esxi properly requires atleast a decently sized cluster, especially if you start throwing NSX in the mix.
- devonkim 5y ago
  Forget even iSCSI as the goalpost, a lot of places still use fibrechannel drives and controllers and FCoE makes sense mostly once you’ve hit at least 10 GbE which is a bit pricey for the sake of learning while it’s basically hot garbage in professional environments worth a career in. Half the point of vSphere in a professional capacity requires use of several nodes such as different HA options, how to trunk networks, distinctions between different LUN abstractions (RDMs in physical v virtual compatibility modes) and an endless parade of the ways different vendors’ SANs and physical switches can completely mess up and ruin your entire month if you’re not pedantic about every random switch or flag in configuration. All these other vendor minutiae being so important rather than the general concepts as a practitioner is part of why I don’t do VMware professionally anymore and went quietly to SRE janitoring in major cloud providers and chalked those days up as time in IT. It’s just much more practical to learn AWS than to pore over the Cisco Nexus documentation unless one’s career really is in the lower infrastructure levels of tech.

deeblering4 5y ago

The lenovo “tiny” hardware recommended in the article is really ideal. Its essentially laptop components in a micro case (no hid/battery/screen), they even are powered by a laptop style external DC adapter. They are affordable, quiet, powerful (modern x86_64 with basic gpu) and light on power usage. I run a pair of m92p myself.

KingMachiavelli 5y ago
The downside to these small form factor kit PCs is that then you are very connectivity limited. You can't use it to build a NAS directly or GPU connected VMs, etc. They are quite good as a cheap thin client that you use to access your more powerful hardware. As hardware ages it tends to lack some of the nice feature's like dual 4K@60Hz output, thunderbolt, etc. so having a new but cheap/lowpower machine helps.
- deeblering4 5y ago
  Thats true. This model has just one small expansion slot so you have to be clever. But its fine for general purpose compute. Yeah realistically NAS is out, but thats probably ok. I mean NAS is not a great fit for most general purpose computers. In a pinch you could go usb-3 jbod or something. But personally I think it’d be better to go with either specifically storage oriented hardware, or a scale out filesystem on top of a cheap cluster, something like odroid-hc2. Personally I like to keep things compartmentalized even at home. So my NAS is a dedicated (off the shelf) system, and the lenovo mini servers mount it via NFS/CIFS.

Steltek 5y ago

Yes! Companies refresh Dells pretty often and you can find an i7 for not much money on eBay. Buy two to double up the RAM. If you know the right people at a company, you could get them for free even. Home labs aren't about millions of hits. It's a playground.

kaladin-jasnah 5y ago

> A single ATX desktop can do almost [1] everything a homelab at a fraction of the cost & power consuption. So this is true, but the iDRAC/iLO on my big, loud server has a virtual KVM feature that lets me lazily sit at my desk and click buttons and install my server operating system of choice. It saves a lot of space and effort compared to flashing a USB, going and plugging in a keyboard, (mouse), and monitor, and going through the whole deal. I'd wager that's one of the big things that would compel me to buy a rack server. I recently built a nice ATX desktop, fitted with a 5950X and everything, and I found that the PiKVM project [0] does a pretty good job at replacing that "integral" part of the server for me (you can also look into an ASRock Rack PAUL [1], but good luck finding one for sale right now. > What makes even less sense is wanting use to use software like vSphere or ESXi since its about 10x more complicated than just using virt-manager/QEMU A lot of people (not me, I end up using libvirt/QEMU as it suits my needs) buy homelabs to work towards having hands-on experience for their system administration job, which uses ESXi/vSphere. It might also be for working on getting certifications from VMware, in which case they really don't have any choice but to use ESXi on their servers. > Server hardware & software makes sense when it's not your home because then you do need a remote access tool like iDRACK Now, I addressed this earlier (laziness), but these BMC things are very useful—you can monitor the health of various components of your server, and I believe even update the BIOS without stepping out of your chair. It makes administering a homelab much easier, and even the Pi-KVM, a DIY option, I'm pretty sure, doesn't have monitoring features. Plus, those DIY solutions require wiring stuff into your ATX motherboard, which can get janky and might put off people who want a turnkey solution. [0] https://pi-kvm.org/ https://pi-kvm.org/ [1] https://www.asrockrack.com/general/productdetail.asp?Model=PAUL#Specifications https://www.asrockrack.com/general/productdetail.asp?Model=P...

PragmaticPulp 5y ago
> So this is true, but the iDRAC/iLO on my big, loud server has a virtual KVM feature This is available on AMD Ryzen motherboards, too. Just get one of ASRock's motherboards with the BMC controller built-in: https://www.asrockrack.com/general/productdetail.asp?Model=X470D4U#Specifications https://www.asrockrack.com/general/productdetail.asp?Model=X... No need to mess with PiKVM or add-in cards. It's a server board with KVM management that works out of the box with Ryzen processors. It might need a BIOS update to support your 5950X, but it will work.

PragmaticPulp 5y ago

> Rack mount hardware is almost always expensive, loud, and power hungry. Buying old rack-mount server hardware for home use is almost always a mistake. Old server hardware may feel cheap when you see an old dual-socket rack mount server on eBay with hardware that was fast 8 years ago, but you can probably meet or exceed the performance with something like a cheap 8-core Ryzen. Rack mount servers are also exceptionally loud. Unless you love the noise of small, high-RPM server fans, you don't want rack mount server hardware in your house. And don't forget the power bill. Some old servers idle at hundreds of watts, which will add up over the several years you leave it running. 24/7 server hardware is a good example of where it makes sense to be mindful of power consumption. > What makes even less sense is wanting use to use software like vSphere or ESXi since its about 10x more complicated than just using virt-manager/QEMU. I disagree. ESXi is actually extremely easy to use, as long as you pick compatible hardware up front. The GUI isn't perfect, but it's intuitive enough that I feel confident clicking around to accomplish what I need instead of looking up a tutorial first.

kube-system 5y ago
I have used consumer hardware and enterprise hardware in my rack. There’s pros and cons either way. Consumer hardware is cheaper, more power efficient, and quiet. But, there are a few reasons I have switched to enterprise hardware: * remote management and redundancy features make it more reliable if I’m traveling and want to remote in to do something * some software works better with enterprise hardware features (i.e. special disk controller modes) * I feel like my development experience is closer to what I can expect in production * 100+ gigs of RAM on one system without a big upfront expense * the occasional PITA of working with enterprise hardware helps me to understand what I might be expecting out of infrastructure team in production, or design ways to make their life easier
- deleted 5y ago
  [deleted]
fomine3 5y ago
Latest Ryzen's CPU perf and max 128GB ECC UDIMM should be enough for most users, but it still lacks homelabby features and flexibilities. I need more than 16x PCIe lanes that supports ACS and IPMI, so I go for EPYC build. It's great except idle power consumption. I with they improve it to Xeon level, but maybe chiplet architecture (8CCD and massive CCD) isn't good about it. For hypervisor, I recommend Proxmox VE.

Lammy 5y ago

> I just have never seen the point of building a home lab like this. https://www.youtube.com/watch?v=38ApYaywLzs https://www.youtube.com/watch?v=38ApYaywLzs I have to pay PG&E rates for power ($$$) so I'm a big fan of lower-power hardware for a system I'm going to leave on 24/7, e.g. my 25W-TDP 1U racked 8-core ECC-equipped Atom server built on this board: https://www.supermicro.com/en/products/motherboard/A2SDi-8C+-HLN4F https://www.supermicro.com/en/products/motherboard/A2SDi-8C+...

c_o_n_v_e_x 5y ago
I like the Atom C-series... interesting option for edge deployed equipment.

bradstewart 5y ago

I had more space in my network rack than near my desk, so I bought a cheap Rosewill rackmountable ATX case, and rebuilt my old desktop into it (since I pretty much only use my laptop these days). But I agree that buying old Dell servers for home use is rather silly at this point.

aftbit 5y ago

> Rack mount hardware is almost always expensive, loud, and power hungry. It really depends on the particular hardware. I recently picked up an R720 with 128GB of memory and dual E5-2670v1 for $450 on eBay. It idles at 120W (about the same as my brand new Ryzen 5950X desktop). It is not much louder than my old air-cooled 8700K consumer PC. Of course, it's not much faster either, and it's definitely slower than my Ryzen. I bought it to learn how to use iDRAC and practice ZFS with 10x $20 1GB 10k RPM SAS drives. Also maybe to give Proxmox a try. All of my practical home-prod stuff runs on an old i5 desktop, not my rack servers.

fomine3 5y ago
120W at idle on desktop computer looks not good.
- KingMachiavelli 5y ago
  The issue is that power savings stuff is often bad on Linux... especially on older hardware. 'Idle' in this context means running with near zero load since a server cannot suspend or hibernate. In theory you could setup hibernate/suspend + WoL but that tends to not work well. Some power saving features will really only save 5-10W and can make PCIe cards unstable, poor hard drive performance, laggy keyboard & mouse, etc. In particuler, GPUs often have poor suport for the normal power saving features. I just pulled the dedicated GPU (GTX 760) out of my old machine that is now a headless server and that saved ~20W. What's weird and bad is that that 20W was being consumed without a graphical interface even running.

dheera 5y ago

> expensive This isn't always true. I got a 24-port Aruba 802.3at PoE switch with FOUR 10-gigabit ports, for a grand total of $120. > loud, and power hungry Again, not always true. Enterprises do care about power a lot of the time. I highly recommend the Ubiquiti stuff for home lab use -- most of it is pretty quiet. If you buy other rack-mount hardware, try to buy at least 2U hardware, the bigger fans are much quieter than the 40mm fans in 1U equipement. If you must buy non-Ubiquiti 1U equipment, you can usually change the fans out for Noctua fans. > A single ATX desktop You can build an ATX desktop into a 4U case. I highly recommend the SilverStone RM42-502 for about $250 on Amazon. It takes standard components including standard ATX power supply, standard ATX, micro-ATX, or mini-ITX motherboard, standard fans (or even a CorsAir liquid cooler), it's basically a standard case. If you use quiet components it will be quiet. My ATX rack mount PC is not even noticeable unless I'm running up my GPU doing machine learning stuff. There are much cheaper cases available as well, but the SilverStone case is quality, and will last you forever, you can just keep building new PCs into it for as long as ATX/ITX exist. One of the advantages to building your PC in a rack mount configuration is that it's very easy to stack multiple PCs along with your network routers, switches, NAS, UPS, in one nice rack that's easy to move from apartment to apartment in one piece, and all your cables and connections stay nice and tidy. It's also ideal if you play with a lot of smaller devices. For example if you want to have a cluster of 10 RPis, a rack mount solution is great for keeping the ethernet and power cables tidy, and it isn't going to be loud or any more power hungry than if you had them spread out across the table. You can also 3D print rack mounts for non-rack equipment, just to keep them tidy. My rack: https://i.redd.it/xcss9uassrg71.jpg https://i.redd.it/xcss9uassrg71.jpg

actually_a_dog 5y ago

I agree. My setup is very much like the setup shown near the end of the article (the one that consists of a couple Synology NAS boxes and what looks like a few Mac Minis or other small form factor PCs). I have a file server with a decently large storage array and a few Raspberry Pis and other small electronic gizmos, some of which connect to my wifi, and some which plug into a PC via various cables (ethernet, USB, etc.) The only thing out of this that consumes any amount of power worth mentioning is the file server/storage array. I haven't measured how much power it uses (I probably should), but I'm able to minimize it by allowing the disks to go to sleep and the CPU to run slower than when I'm actively using it. I've never felt limited by this setup at all, but, then again, my home lab isn't really my main hobby.