4 ms·

From what I've seen on reddit's /r/homelab, the neat thing about racked systems is you can buy very cheap used hardware that are discards from enterprise uses.

From what I've seen on reddit's /r/homelab, the neat thing about racked systems is you can buy very cheap used hardware that are discards from enterprise uses. There's very little market for an old server or rackmount network switch so you can get very good bargains. Personally I buy new cheap mini PCs and consumer networking stuff. That's another way to get bargains, since consumer hardware is so cheap. But I'm a bit jealous of the enterprise capabilities like proper managed switches or remote managed servers.

mech422 3y ago
The down side to the old enterprise servers is the power draw. I had 2 Dell blade server chassis and 1 HP blade server chassis that used Xeon based blades. The electric bill was UNREAL with 10+ blades running at once. I do use Dell PowerConnect manged switches from EBay, but tbh for what I do, TP Link switches work really well and are cheap. They support VLANs, MAC Lans, SNMP, SSH/Web configuration, etc etc - basically fully managed switches. And they have been rock solid as 'edge' switches for each room in the house. The powerconnects basically work as top-of-rack switches for pulling all the different rooms/rack into 1 feed to the cable modem.
- blackfawn 3y ago
  Agreed. Power draw and often noise on enterprise equipment is ridiculous in a home setting. I try to stick to the small form factor (SFF) or ultra-small form factor (USFF) computers whenever I can, which is what the author is using. I used to use Raspberry Pi boards for some things but always ran into problems w/ microSD cards dying, even with log2ram and buying industrial cards. I haven't had a single issue after switching to tiny x64 machines w/ real SSDs in them. I picked up a HP managed switch that I thought I might use but it draws more power than the ThinkCentre M93p Tiny computers do and it's actively cooled w/ super noisy 1U fan(s) so I'm going to ditch it.
  - nvy 3y ago
    Which x64 machines have you been using? I'm looking for a successor to PC Engines APUs
    - blackfawn 3y ago
      I've been sticking w/ very compact units, mainly ThinkCentre M series "Tiny" and Dell OptiPlex "Micro" form factor, which are both practically identical in size. Both use a laptop power supply. You can get current versions for many hundreds of dollars or generation(s) old ones for as cheap as $30 USD. My most recent ones were older 3rd & 4th gen i5 & i7 CPUs but they still blow current Raspberry Pi, etc. out of the water, have M.2 for network, etc. & SATA for storage. A Dell I have that's a generation newer than these adds NVMe too. It's hard finding these small PCs that have multiple Ethernet but I just added a gigabit Ethernet M.2 card to the one.
    - vially 3y ago
      I was also looking for a successor to the PC Engines APUs and came across https://teklager.se/en/ https://teklager.se/en/ that lists some possible alternatives that you might find interesting. Personally I was looking to build a router so I ended up buying a fanless N100 based mini PC from Aliexpress (e.g.: search term is "N100 firewall appliance") and have been very satisfied with it so far (Proxmox homelab with OPNsense running as a VM).
    - mech422 3y ago
      I love the Odroid H series [0]. cheap, quiet and only like 4w idle draw 0) https://ameridroid.com/products/odroid-h3 https://ameridroid.com/products/odroid-h3
    - transpute 3y ago
      On paper, these look promising, no direct experience. Also on Amazon US/CA. $170, N100, 16GB LPDDR5, dual 2.5GbE, https://aoostar.com/products/aoostar-n-box-pro-intel-n100-mini-pc-with-lpddr5-16g-ram-m-2-ssd-w11-pro-2-5g-lan-full-featured-type-c-port https://aoostar.com/products/aoostar-n-box-pro-intel-n100-mi... $210, N100, dual 3.5" SATA, M.2 NVME, dual 2.5GbE, https://aoostar.com/products/aoostar-r1-2bay-nas-intel-n100-mini-pc-with-w11-pro-lpddr4-16gb-ram-512gb-ssd https://aoostar.com/products/aoostar-r1-2bay-nas-intel-n100-... $250, Ryzen 5700U, https://aoostar.com/products/aoostar-cyber-amd-ryzen-7-5700u-rgb-mini-pc-with-8c-16t-up-to-4-3ghz-with-w11-pro-ddr4-16gb-32gb-512gb-1t-nvme-ssd?variant=47230207557930 https://aoostar.com/products/aoostar-cyber-amd-ryzen-7-5700u...
  - mech422 3y ago
    Yeah - I skipped rPi because back when I started replacing stuff ARM still had some gotcha's with stuff that hadn't been ported. I do use the rPi's for the 3d printers though. I've found the TP Link stuff to be good for home switches. cheap, quiet, and basically fully featured/managed. Stuff like SNMP, VLAN/Mac Lan, port mirroring/monitoring, etc. Also SSH and web interfaces... I have 2-3 currently, 1 per room feeding back to the PowerConnects (NOISY) and haven't had a single issue in 3-5 years of use :-)
    - doawoo 3y ago
      One of the coolest additions to my home lab recently was an Orange Pi 5 plus/pro - 8 core ARM 64-bit cpu and 32GB of RAM all with awesome power usage and zero noise, plus an up to date kernel. I skipped the rpi as well a while back but these newer boards are worth lookin at! :)
      - mech422 3y ago
        Oh nice! - I'll have to take a look Thanks!
    - vGPU 3y ago
      The Omada line makes pretty decent AP’s too. A step above ubiquiti in terms of reliability but also dead simple. I’d have gone ruckus if I could have but the mailman stole my $200 eBay R550 score and I didn’t feel like shelling out $500 for one.
- EvanAnderson 3y ago
  > The down side to the old enterprise servers is the power draw. It depends on the generation of the gear. Stuff coming off-lease in the last 3 - 4 years has been of a generation where lower power draw became a major selling point. In Dell land, for example, the power draw differences are dramatic between the Rx10, Rx20, and newer series. (Basically, throw Rx10 servers in the trash. Rx20 servers are better but still not great.)
- buybackoff 3y ago
  Power draw also comes with terrible noise. They were designed to be in a server room, not a living/bed room.
doublepg23 3y ago
I feel like it’s a harder sell than it used to be because post-Ryzen there’s been real increases in CPU performance. Modern NUCs and prebuilt also having native NVMe is a boon too. All depends on what you’re going for though.
internet101010 3y ago
The caveat to this is noise. People who have never seen a rack server in person don't understand how loud they really are. PSUs, case fans, cpu fans (if applicable) all gotta go. You kind of have to go into it knowing you are going to spend an additional $100-200 replacing it all, and that's assuming you will not have to 3D print a custom bracket to house the new fans.
- freedomben 3y ago
  Yep, the noise is serious. I may not even seem that bad at first, but if you are going to hear it prolonged, it starts to get really unbearable after a 20 minutes of it. I don't know anybody who isn't irritable after a couple hours. And the heat is enough to make a room very uncomfortable. I started my home lab in my office, and it didn't take long to realize that was not an option. Luckily I was able to find a nook under the stairs where I could stash it, but the heat and noise are polar opposite of a productive office environment.
bayindirh 3y ago
What homelab people who do not work with/in a data center daily doesn't realize is how noisy and power hungry is the equipment they procure. They got a managed switch, nice. It has VLANs, stacking, etc. nice. How many 48 port switches you can possibly stack in a home? You got a Sun ZFS7420. Great. That's a rack with two controllers with 1TB RAM each. That thing screams. An entry level 7320 is maybe workable, but you can't do 7420 at home. Same for HPE/Dell/Netapp storage boxen. Doing Infiniband is home is another thing I can't understand. Fiber cables are extremely expensive. Copper is range limited and thick, plus unless you do HPC, its latency and speed doesn't mean much. Go fiber Ethernet. Much more practical. I understand the curiosity and willingness to play, but it's not that practical to play with these at home (noise, space, heat, power wise). I also buy high end consumer hardware. TP-Link has 4/8 port smart managed SOHO switches for example, they even have PoE. Power your Raspi cluster via PoE with less cables and get some respectable computing power with no noise. Neat! No? Dell's Rack servers are generally not noisy at idle, but a 2U server under some real load is unbearable. What's funny that an Intel N95 has the same power with an i7-3770K with better SIMD support while being 4x more efficient, and that thing is not slow by any means.
- isopede 3y ago
  I don't think fiber cables are expensive. I ran CAT6 in my house and at the same time pulled an OS2 single-mode 30m fiber cable to my office for core switching. The 30m fiber cable was $17 on Amazon, plus two 10G-LR modules at $18 each. The equivalent CAT6 is more power hungry, has less range, the cable is the same damn price, and the 10GBase-T SFP->RJ45 transceivers are twice the price of the LR transceivers at $50 each.
  - bayindirh 3y ago
    Infiniband uses its own cables with dedicated/integrated transceivers. It's not compatible with Ethernet and can't be swapped (unless you plan to talk plain Ethernet with your IB card, which is possible). A 5m, 200gbps cable is $1,149.00 a pop [0]. Mind you, this is neither the longest, nor the fastest cable. [0]: https://esaitech.com/products/mellanoxmfs1s00-h005v-200gbps-infiniband-qsfp56-active-optical-cable https://esaitech.com/products/mellanoxmfs1s00-h005v-200gbps-... More cables: https://esaitech.com/search?type=product&options%5Bprefix%5D=last&options%5Bunavailable_products%5D=last&q=product_type%3ACable+AND+Mellanox https://esaitech.com/search?type=product&options%5Bprefix%5D...
yummypaint 3y ago
I haven't seen any mention of going to scrap yards in this thread, but just about every general-purpose yard ive been in has had racks of some kind. Hard to beat scrap price as racks themselves tend to be light when empty.

mech422 3y ago

The down side to the old enterprise servers is the power draw. I had 2 Dell blade server chassis and 1 HP blade server chassis that used Xeon based blades. The electric bill was UNREAL with 10+ blades running at once. I do use Dell PowerConnect manged switches from EBay, but tbh for what I do, TP Link switches work really well and are cheap. They support VLANs, MAC Lans, SNMP, SSH/Web configuration, etc etc - basically fully managed switches. And they have been rock solid as 'edge' switches for each room in the house. The powerconnects basically work as top-of-rack switches for pulling all the different rooms/rack into 1 feed to the cable modem.

blackfawn 3y ago
Agreed. Power draw and often noise on enterprise equipment is ridiculous in a home setting. I try to stick to the small form factor (SFF) or ultra-small form factor (USFF) computers whenever I can, which is what the author is using. I used to use Raspberry Pi boards for some things but always ran into problems w/ microSD cards dying, even with log2ram and buying industrial cards. I haven't had a single issue after switching to tiny x64 machines w/ real SSDs in them. I picked up a HP managed switch that I thought I might use but it draws more power than the ThinkCentre M93p Tiny computers do and it's actively cooled w/ super noisy 1U fan(s) so I'm going to ditch it.
- nvy 3y ago
  Which x64 machines have you been using? I'm looking for a successor to PC Engines APUs
  - blackfawn 3y ago
    I've been sticking w/ very compact units, mainly ThinkCentre M series "Tiny" and Dell OptiPlex "Micro" form factor, which are both practically identical in size. Both use a laptop power supply. You can get current versions for many hundreds of dollars or generation(s) old ones for as cheap as $30 USD. My most recent ones were older 3rd & 4th gen i5 & i7 CPUs but they still blow current Raspberry Pi, etc. out of the water, have M.2 for network, etc. & SATA for storage. A Dell I have that's a generation newer than these adds NVMe too. It's hard finding these small PCs that have multiple Ethernet but I just added a gigabit Ethernet M.2 card to the one.
  - vially 3y ago
    I was also looking for a successor to the PC Engines APUs and came across https://teklager.se/en/ https://teklager.se/en/ that lists some possible alternatives that you might find interesting. Personally I was looking to build a router so I ended up buying a fanless N100 based mini PC from Aliexpress (e.g.: search term is "N100 firewall appliance") and have been very satisfied with it so far (Proxmox homelab with OPNsense running as a VM).
  - mech422 3y ago
    I love the Odroid H series [0]. cheap, quiet and only like 4w idle draw 0) https://ameridroid.com/products/odroid-h3 https://ameridroid.com/products/odroid-h3
  - transpute 3y ago
    On paper, these look promising, no direct experience. Also on Amazon US/CA. $170, N100, 16GB LPDDR5, dual 2.5GbE, https://aoostar.com/products/aoostar-n-box-pro-intel-n100-mini-pc-with-lpddr5-16g-ram-m-2-ssd-w11-pro-2-5g-lan-full-featured-type-c-port https://aoostar.com/products/aoostar-n-box-pro-intel-n100-mi... $210, N100, dual 3.5" SATA, M.2 NVME, dual 2.5GbE, https://aoostar.com/products/aoostar-r1-2bay-nas-intel-n100-mini-pc-with-w11-pro-lpddr4-16gb-ram-512gb-ssd https://aoostar.com/products/aoostar-r1-2bay-nas-intel-n100-... $250, Ryzen 5700U, https://aoostar.com/products/aoostar-cyber-amd-ryzen-7-5700u-rgb-mini-pc-with-8c-16t-up-to-4-3ghz-with-w11-pro-ddr4-16gb-32gb-512gb-1t-nvme-ssd?variant=47230207557930 https://aoostar.com/products/aoostar-cyber-amd-ryzen-7-5700u...
- mech422 3y ago
  Yeah - I skipped rPi because back when I started replacing stuff ARM still had some gotcha's with stuff that hadn't been ported. I do use the rPi's for the 3d printers though. I've found the TP Link stuff to be good for home switches. cheap, quiet, and basically fully featured/managed. Stuff like SNMP, VLAN/Mac Lan, port mirroring/monitoring, etc. Also SSH and web interfaces... I have 2-3 currently, 1 per room feeding back to the PowerConnects (NOISY) and haven't had a single issue in 3-5 years of use :-)
  - doawoo 3y ago
    One of the coolest additions to my home lab recently was an Orange Pi 5 plus/pro - 8 core ARM 64-bit cpu and 32GB of RAM all with awesome power usage and zero noise, plus an up to date kernel. I skipped the rpi as well a while back but these newer boards are worth lookin at! :)
    - mech422 3y ago
      Oh nice! - I'll have to take a look Thanks!
  - vGPU 3y ago
    The Omada line makes pretty decent AP’s too. A step above ubiquiti in terms of reliability but also dead simple. I’d have gone ruckus if I could have but the mailman stole my $200 eBay R550 score and I didn’t feel like shelling out $500 for one.
EvanAnderson 3y ago
> The down side to the old enterprise servers is the power draw. It depends on the generation of the gear. Stuff coming off-lease in the last 3 - 4 years has been of a generation where lower power draw became a major selling point. In Dell land, for example, the power draw differences are dramatic between the Rx10, Rx20, and newer series. (Basically, throw Rx10 servers in the trash. Rx20 servers are better but still not great.)
buybackoff 3y ago
Power draw also comes with terrible noise. They were designed to be in a server room, not a living/bed room.

doublepg23 3y ago

I feel like it’s a harder sell than it used to be because post-Ryzen there’s been real increases in CPU performance. Modern NUCs and prebuilt also having native NVMe is a boon too. All depends on what you’re going for though.

internet101010 3y ago

The caveat to this is noise. People who have never seen a rack server in person don't understand how loud they really are. PSUs, case fans, cpu fans (if applicable) all gotta go. You kind of have to go into it knowing you are going to spend an additional $100-200 replacing it all, and that's assuming you will not have to 3D print a custom bracket to house the new fans.

freedomben 3y ago
Yep, the noise is serious. I may not even seem that bad at first, but if you are going to hear it prolonged, it starts to get really unbearable after a 20 minutes of it. I don't know anybody who isn't irritable after a couple hours. And the heat is enough to make a room very uncomfortable. I started my home lab in my office, and it didn't take long to realize that was not an option. Luckily I was able to find a nook under the stairs where I could stash it, but the heat and noise are polar opposite of a productive office environment.

bayindirh 3y ago

What homelab people who do not work with/in a data center daily doesn't realize is how noisy and power hungry is the equipment they procure. They got a managed switch, nice. It has VLANs, stacking, etc. nice. How many 48 port switches you can possibly stack in a home? You got a Sun ZFS7420. Great. That's a rack with two controllers with 1TB RAM each. That thing screams. An entry level 7320 is maybe workable, but you can't do 7420 at home. Same for HPE/Dell/Netapp storage boxen. Doing Infiniband is home is another thing I can't understand. Fiber cables are extremely expensive. Copper is range limited and thick, plus unless you do HPC, its latency and speed doesn't mean much. Go fiber Ethernet. Much more practical. I understand the curiosity and willingness to play, but it's not that practical to play with these at home (noise, space, heat, power wise). I also buy high end consumer hardware. TP-Link has 4/8 port smart managed SOHO switches for example, they even have PoE. Power your Raspi cluster via PoE with less cables and get some respectable computing power with no noise. Neat! No? Dell's Rack servers are generally not noisy at idle, but a 2U server under some real load is unbearable. What's funny that an Intel N95 has the same power with an i7-3770K with better SIMD support while being 4x more efficient, and that thing is not slow by any means.

isopede 3y ago
I don't think fiber cables are expensive. I ran CAT6 in my house and at the same time pulled an OS2 single-mode 30m fiber cable to my office for core switching. The 30m fiber cable was $17 on Amazon, plus two 10G-LR modules at $18 each. The equivalent CAT6 is more power hungry, has less range, the cable is the same damn price, and the 10GBase-T SFP->RJ45 transceivers are twice the price of the LR transceivers at $50 each.
- bayindirh 3y ago
  Infiniband uses its own cables with dedicated/integrated transceivers. It's not compatible with Ethernet and can't be swapped (unless you plan to talk plain Ethernet with your IB card, which is possible). A 5m, 200gbps cable is $1,149.00 a pop [0]. Mind you, this is neither the longest, nor the fastest cable. [0]: https://esaitech.com/products/mellanoxmfs1s00-h005v-200gbps-infiniband-qsfp56-active-optical-cable https://esaitech.com/products/mellanoxmfs1s00-h005v-200gbps-... More cables: https://esaitech.com/search?type=product&options%5Bprefix%5D=last&options%5Bunavailable_products%5D=last&q=product_type%3ACable+AND+Mellanox https://esaitech.com/search?type=product&options%5Bprefix%5D...

yummypaint 3y ago

I haven't seen any mention of going to scrap yards in this thread, but just about every general-purpose yard ive been in has had racks of some kind. Hard to beat scrap price as racks themselves tend to be light when empty.