19 ms·

My Home Lab Server with 20 Cores / 40 Threads and 128 GB Memory

239 points by louwrentius 7y ago

hrangozz 7y ago
My home lab system of choice is the lenovo m92p tiny (or updated version) It's a headless system in a tiny enclosure. i5 processor with up to 16GB ram. Power consumption is nice and low. And they feature remote management via serial over IP, remote power cycle, etc. They can be had for less than $100 on ebay, and for me have been rock solid.
eropple 7y ago
Neat machine, but 96W idle is a lot for a home server, IMO. Maybe you're somewhere where power's super cheap (and hopefully clean), but a lot of folks aren't. I run my old desktop (a i7-6700K) in a rack in my basement, now, with 64GB of RAM, a Mellanox Connect-X for 10G networking, and half a dozen disks, and it idles under 15W. The entire rack, UniFi stuff/POE wifi APs and all, sits around 50W. 96W just for a single machine is A Lot.
- stordoff 7y ago
  Worth noting: > By default, this server is turned off. When I want to do some work, I turn the server on using wake-on-lan. I do this with all my equipment except for my router/firewall. If it is only on when it is doing work, idle power doesn't matter _too_ much.
  - dr_zoidberg 7y ago
    The Intel ark site says the TDP for that xeon is at 115W, and that's two of them. When you're running at full power (which could happen) you're looking at the CPUs alone drawing over 200W. Another pointer to high power consumption is the 750W PSU. All in all we're not looking at a particularly efficient machine (even if it's unfair comparing it to 2019 hardware).
    - dredmorbius 7y ago
      The PSU's rating is for its potential to supply power, whilst draw components (CPUs, disk, etc.) are rated by use. The actual efficiency is defined by capacity, load, and miscellaneous other factors, but should be in the neighbourhood of 20% of delivered power. If the system idles at ~100 watts, the PSU might add another 20-30 to the mix, not 750. https://www.howtogeek.com/129667/what-exactly-does-the-wattage-rating-on-a-power-supply-unit-mean/ https://www.howtogeek.com/129667/what-exactly-does-the-watta...
      - brokenmachine 7y ago
        Agree with all of that, but I think OP was referring to the fact that higher-rated PSUs will use more power than a lower rated PSU when idle.
        dredmorbius 7y ago
        Running a PSU at ~50% of its rated max is generally recommendable. At 115W * 2, plus other draws (disk, mobos, blinkenlights), that's around 300W. Again: peak draw, when on. I'm not arguing that this system is particularly efficient, only that you don't want to add 750W for the PSU to the draw. I've not specced out low-power systems myself. I doubt you could get 40 hyperthreads running way below this, though there are definitely some low-power systems which might have a total budget below 50W. Reddit's HomeLabPorn may have some more useful guidance: https://old.reddit.com/r/HomeLabPorn/ https://old.reddit.com/r/HomeLabPorn/ (Not my area of expertise, I've never really stayed current in HW. Though I'm aware power/thermal budget has been a major focus, both mobile and server, for the past decade or so.)
        dr_zoidberg 7y ago
        The 24 cores/48 threads EPYCs that come with 128mb L3 cache have a TDP that's between 155W and 180W. If you want to go with a lower core count (because you have a more modern architecture), you could even go with the 16c/32t EPYC 7282 (64MB L3) that sits at 120W. All in all, you're looking at almost half the power draw (AMD measures it differently than Intel, so aboutish the same) for the same performance. I specifically mention the L3 cache size because, while not being substantial, a large cache can get you 10 to 20% improved performance due less CPU stalling from cache misses. For comparison, the Xeon in question has 25MB L3, so we'd be looking at 50MB cache, split in two dies (so it doesn't quite work as a whole block of 50mb cache). greatjack613 also mentioned the fact that an AMD Zen 3900x matches the multicore performance (and has better single core perf) for a fraction of the TDP, see https://news.ycombinator.com/item?id=20687932 https://news.ycombinator.com/item?id=20687932. All in all that was the general idea: yes, you can get a used server for a good price, but we shouldn't forget the efficiency aspect of it, compared to new hardware.
- rhn_mk1 7y ago
  Did you do anything particular to get 15W? I have recently measured my AMD AM1 system, which sits at 40W with 8GiB RAM, 2 HDDs and a couple mostly idle VMs on it.
  - rhinoceraptor 7y ago
    I'd check the BIOS to see if there are any power saving options, and/or unclocking/undervolting the CPU. 2.5" HDDs and SSDs use about the same amount of power, but 3.5" HDDs use 2-3x more. So you could potentially save 12 watts if you're using 3.5" drives. Make sure your PSU is a high efficiency one, and isn't too low, or or too high wattage. From what I've read the maximum efficiency is in the 40-80% range.
    - scns 7y ago
      Maximum efficiency is at 50%
  - eropple 7y ago
    I do undervolt it and I aggressively spin down the drives. (My concern is mostly storage and redundancy, not performance.)
- atonse 7y ago
  In addition to wattage, my other concern for a home server was fan noise. If it's anywhere that I am, I really don't want to hear it. So I got these amazing Noctua fans and simply can't hear the server. (Updating language based on replies) A datacenter-like server probably isn't designed for that. So it wouldn't necessarily work out. I have an 8 core i5. I bought Intel (18 months ago) because I didn't want to deal with any AMD incompatibility especially since I wanted to also run it as a gaming machine using VT-d. If I were doing it again, I'd definitely go with Ryzen.
  - skunkworker 7y ago
    I’m not sure what you’re getting at with the home server fans and not working out. I’ve been using a noctua on my Xeon home server without and issues, and reconfigured the fan sensors and control using ipmi. There’s no problem making a quiet Xeon home server.
    - snuxoll 7y ago
      I’ve got four servers and two switches in my home office, noise is around 45dBA and with the door closed it’s inaudible. Total power draw idle is around 340W (3.1A measure by my PDU @ 110V).
    - atonse 7y ago
      Sorry I wasn't clear. I meant something like a Rack mounted server designed for a datacenter. Would it become whisper quiet if you just replaced the fan? Is it that simple?
      - snuxoll 7y ago
        Replacing the fans on server systems is tricky, many “quiet” fans are such because they have a lower max RPM which will freak the hell out of your management controller. Also, there’s only so much optimization you can do to 40mm fans in a 1U chassis. The fans in my 1U dell servers are fairly quiet when idle around 3600RPM. They make noise, you wouldn’t want to be sleeping or watching TV in the same room - but with the door to my office closed they can’t be heard in the hallway.
- wil421 7y ago
  I have a UniFi USG, UniFi 150W switch (using maybe 30w), a freeNAS with a i3-8100, a mATX super micro board and 6 4TB WD Red drives. My backup power supply says maybe 120w.
  - CSDude 7y ago
    My USG runs very hot, do you experience it?
    - atonse 7y ago
      All my unifi stuff runs hot. The ERLite 3 and switches, etc. I think they count on using the cases as heatsinks.
      - eropple 7y ago
        Same here. Those things get toasty. I accidentally placed a RPi3 (passively cooled but with a metal case) on top of my USG once and it got hot enough not just to throttle it, but also to cause HDMI artifacting.
    - wil421 7y ago
      Not really. I have a NanoHD and my father in law has one too. His runs really hot and mine is cool to the touch. Mine has more traffic and was made before his NanoHD.
      - CSDude 7y ago
        I also have a NanoHD and its acceptable but USG (non-pro) runs so hot that I fear leaving it in a non airconditoned cupboard that my internet exit resides
    - michaelper22 7y ago
      Yup. My USG and US-8-60W both run hot enough to not want my hand on for very long (the power supplies are pretty warm too).
- fnord77 7y ago
  would newer hardware idle at even lower wattage than 15W? Or is that about the best one could expect?
  - rhinoceraptor 7y ago
    For a full desktop class chip, I think that's about as low as it can go. You could go lower with something like a NUC, but 15 watts is really good. 15 watts idle would cost something like $1.50 to $2 to run 24/7.
  - eropple 7y ago
    Newer, maybe slightly but not hugely. (It's all still kinda warmed-over Skylake.) You could get cheaper with lower-TDP chips, but there are definitely diminishing returns on the floor.
- AdrianB1 7y ago
  Probably an error in measuring the power consumption, there is no way to go to 15W, double that or more.
  - rhinoceraptor 7y ago
    It's believable (if only obtained with a lot of BIOS tweaking), I run an older Dell 1u server as my router. It's got an E3-1220 V2 (2012 era, 69 watt TDP), one 2.5" SSD, 8gb of RAM and after a bit of tweaking it sits at around 30 watts.
    - eropple 7y ago
      It wasn't a lot of BIOS tweaking, but yes, some undervolting was required. ;) I also hdparm'd it such that my spinning disks park pretty aggressively.
- tbyehl 7y ago
  Or it's a ton of compute in a single system for just 96W. Average US rates are roughly $1/yr per continuous watt.
- egwynn 7y ago
  To contextualize this, ~100W will probably cost between $10-$15/mo to run 24/7. EDIT: Assuming a US household paying between $0.10-$0.20 per kwh.
  - jdnenej 7y ago
    Where I am it costs 37c per kWh. It's no wonder almost everyone has solar.
- Havoc 7y ago
  >idles under 15W Damn...meanwhile my fairly modern Asus laptop idles at 27W. (Probably the GPU...think I forced it to dedicated only)
- louwrentius 7y ago
  I turn this server off when I'm done experimenting. I turn it on remotely using wake-on-lan.
  - apexalpha 7y ago
    What exactly do you use it for if it isn on 24/7? I bought a used intel Nuc a while back with 15w idle. And it runs 24/7 and does everything I need from downloading to nextcloud to plex, my website, my emails etc... I really wonder what you do with this kind of power.
- abstractbarista 7y ago
  Some people like me will justify it due to being a huge hobby for them. It takes the place of what other people would spend going out often, traveling, streaming, etc... I've got a rack that pulls ~600W 24/7 but that's across 4 machines, 3 UPS's, >30 HDD's yielding >20TB across multiple zpools, >256GB RAM. And I'm actually using most of that capacity, not just idling away. I do hope to upgrade soon to power-sipping platforms like you mention but currently I'm still on R710/R810 stuff (Westmere Xeon's).
  - Dayshine 7y ago
    I have to ask, using it to do what?
    - berbec 7y ago
      Linux ISO plex server.
    - abstractbarista 7y ago
      I wish I had something written up to share. But it's honestly a mix of things I love to play with. Several full blockchain nodes (especially uploading to light clients), bittorrent client (seeding tons), plex, home assistant, boinc, tor bridge, grafana, web and email servers, zoneminder CCTV, mandelbulber (clustered rendering of 3d fractals), four different Minecraft servers (some modded), ADS-B plane tracking, weather station and radiation monitoring, and a bunch more. It's really just one big toy for me. I'm using Proxmox for high availability of over 30 VM's (still want to play with containers soon). EDIT: OPNSense firewall says I uploaded over 15TB in the last month. Fortunately Google fiber doesn't care!
      - louwrentius 7y ago
        That's awesome!
  - louwrentius 7y ago
    That is awesome. Do you have something written up about your setup online?
- CommieBobDole 7y ago
  You know, that raises a question - how expensive does power get, anyway? Last time I posted my home lab setup (couple of old R610s, Cisco small business switch, ASA 5510), I got a lot of pushback on how much power that consumes and how expensive it is to run. I did the math, and I was looking at maybe $20-$30/month depending on load. Which is not really a lot of money considering how much use I get out of it. I'm definitely getting cheaper power than a lot of people at $.08/kWh, but it looks like the US average is only about $.12 - are there places where a couple hundred watts is going to be a significant financial burden on the average IT worker?
  - eropple 7y ago
    To me it's less a financial burden and more the discomfort of burning electricity just to run a few Docker containers and some databases for personal projects. I expect that once, next summer, we get the house kitted with solar and batteries, I will feel a little more okay about it.
- louwrentius 7y ago
  I've done a measurement with a power meter and it's even 110W. (it includes ilo).
- Aeolun 7y ago
  That costs you maybe $.5 dollars a day? Maybe $1 if you are unlucky. This seems fairly reasonable?
greatjack613 7y ago
To put this in perspective, The new $500 ryzen 9 3900x matches this $1700 xeon server in multi core geekbench, and kicks it in single core perf with a fraction of the power usage. https://browser.geekbench.com/processors/2587 https://browser.geekbench.com/processors/2587
- btgeekboy 7y ago
  That $1500 includes the rest of the computer, though. That adds up.
  - greatjack613 7y ago
    correct, but you can get build a full system with the Ryzen 9 plus 128gb of ram etc for definitely less than $1700. Especially given the drop in ram and ssd's these days
    - pedrocr 7y ago
      128GB of ECC RAM itself is going to be ~800-1000$ as far as I can find some to fit in a Ryzen build. So it won't be trivial to reach the price point for a full build although Ryzen is clearly great value anyway.
      - greatjack613 7y ago
        True, but only if ECC is necessary. In this case where he is mainly using the server for experiments and to try software, regular ram will do absolutely fine. Here is a 128gb kit for under $600 dollars. https://www.newegg.com/corsair-128gb-288-pin-ddr4-sdram/p/N82E16820233946 https://www.newegg.com/corsair-128gb-288-pin-ddr4-sdram/p/N8...
        close04 7y ago
        > only if ECC is necessary If your workloads require 128GB of RAM then you probably want ECC. You'd be skimping in the wrong place.
sigmonsays 7y ago
Where do you store this machine? How is the noise? also, I wonder about the heat generated. Does it keep a room toastier than the rest of the house?
- deleted 7y ago
  [deleted]
- OJFord 7y ago
  Many examples on reddit.com/r/homelab - including even more hungry (or several of them in a rack(s)) systems. The short answer is Yes, servers built to be servers are designed to get the heat out and keep internal temp. down - so noisy fans and as much heat as you generate outside of the box. Of course, nothing stops you replacing fans with quieter ones (at least one of more expensive or air movement) or putting consumer hardware (which has different design goals that you might prefer in the home) in a rack-mount chassis.
- neogodless 7y ago
  The article has some answers to these questions: > My very subjective opinion is that at 50 dB the sound level is reasonable for a server like this, but it's definitely not quiet. I would not be able to work, relax or sleep with this server in the same room. > Although this server is fairly quiet at idle, it does need it's own dedicated room. When the door is closed, you won't hear it at idle, but under sustained load, you will hear high pitched fan noise even with a closed door.
- sannee 7y ago
  I live in a dorm room and keep an R610 about three meters from my bed. You can use some magical IPMI commands to disable the internal fan feedback loop and replace it with a custom one (usually the defaults cool the server to around 30C, which is not really necessary). This makes the server more quiet then the small fridge I also have in the room. The 100W power consumption definitely makes the room warmer. It is approximately like having another person present.
eanzenberg 7y ago
But can it play Crysis?
holy_city 7y ago
I think for $1700 with the memory/storage capacity that's a great deal, but those benchmarks can be topped by the higher-tier consumer CPUs today. The point on people being happy with slower CPU cores is kind of weird to bring up with a server. Most games don't push CPUs that hard, you usually need a really expensive GPU before you see noticeable benefits in gaming from faster processors. Having done some core critical work for the last few years (media processing/systems programming), my recent upgrade from a 4th gen i7 to a Zen2 CPU is paying in spades. If I was building a server to do some of the batch processing stuff I'd like, I would definitely invest in a faster, cooler, more power efficient machine. But that's just me. I don't think I could beat that price point though.
- whalesalad 7y ago
  Dunno... I have an R720 with similar specs (half the ram, though) and it was only $400. I also idle around ~100w because I ripped all the SAS 10k drives out and put SSD's in. It sits powered-off most of the time though because I haven't been able to put it to good use, yet. For a while it was running my Unifi controller + Pihole ... but you don't need the Unifi controller unless you are actively performing maintenance, and Pihole happily hums along on a Rpi 3 that uses far less power.
blackflame7000 7y ago
I built a very similar computer and tbh I prefer to use an 8th gen core-i7 at 4.8ghz as it is faster and less power-hungry for most everyday applications. Javascript, for example, is mostly single-threaded and you can see a noticeable difference in web loading times.
zokier 7y ago
I got bitten by homelab fever few years back and got myself a small server. I had such grand dreams with it that never materialized. Now it sits unplugged in the corner deprecating itself :( Stuff that I was planning to do: * Managed VM platform (~"EC2") * Centralized auth (FreeIPA) * ZFS NAS (also possibly ceph) + backuping * Container platform * Your typical web/email stuff * Monitoring/alerting/log management * VPN endpoint (and other more advanced networking stuff) * Probably something more I have already forgotten I realized that building a private cloud actually takes serious effort and not just putting some lego pieces together. There is also bit of circulatory stuff there that makes bootstrapping more difficult, especially on one single box.
- roel_v 7y ago
  If you want something that does these things and mostly just works, get a Qnap. More expensive, of course.
- mey 7y ago
  To make it simple, since you already have the hardware, I would suggest setting up the server as a VM Host to allow for experimentation with little effort. VMware vSphere Hypervisor, Proxmox, or Microsoft Hyper-V Server 2019 are all free options. It makes it easy to experiment with the above in parallel. My current VM Host has * One VM as a docker host (turtles all the way down) for development tools. Build server, bug tracker, private artifact repo (Sonatype Nexus). * 3 VMs as a Kubernetes cluster for experimentation Building a NAS is the only thing that would take significant effort and is a project unto itself. Unless you have a strong desire to experiment with failure modes (network dropping out, killing iSCSI, online VM fail over), stick to a single box.
- vorpalhex 7y ago
  I've been pretty successful in meeting my private cloud dreams. I use FreeNAS as the bare metal OS, and run Ubuntu VMs which host Docker Containers (using NFS to keep actual persistent data on the underlying FreeNAS box). Docker/Docker-Compose isn't quite lego.. but it's awfully close.
- bsder 7y ago
vorpalhex 7y ago
This is a neat writeup, thanks! Do you use a proper server rack and if so, where do you keep it installed? Right now my homelab is in a tower, but that's obviously not ideal for density.
- louwrentius 7y ago
  Thank you. I use a lack rack setup. https://gathering.tweakers.net/forum/list_messages/1656249/last https://gathering.tweakers.net/forum/list_messages/1656249/l...
kart23 7y ago
Boot time of 4 minutes! Is that normal for servers? And why would it be this long if so?
- whalesalad 7y ago
  Yes and no. There is a longer boot time because the POST process is a bit more intense. ECC RAM is a big part of that. Chances are this particular caes is likely becuase the boot order needs to be reconfigured and it is hunting for PXE/network stuff before hitting a timeout? If you yield control to some kind of Broadcom controller it'll do all kinds of shit before giving up and handing you back to boot a disk.
  - louwrentius 7y ago
    I boot straight from HDD so no waiting on PXE boot timeouts or other shenanigans (boot media, etc).
- edwintorok 7y ago
  Unfortunately it is quite common that servers are slower to boot than desktops if it has a lot of CPUs, or a lot of memory, or just a very slow BIOS impl that spends a lot of time probing and initializing everything. Servers with hundreds of cores are a lot slower to boot than my laptop too.
- nodja 7y ago
  Yes. It's just not a priority for servers. The reason it takes this long is that it does a bunch of self tests, and then it has to load all the ROMs for the components (NICs, HBAs, etc.) which often trigger messages like "X Loaded, press CTRL+L to configure" which stay on screen for 5-10 seconds each.
- sh-run 7y ago
  It's been a while since I was in a position where I regularly watched servers boot, but 2015ish I was doing a lot of 'boots on the ground' sysadmin and virtualization work. Servers do a lot of additional testing during boot-up, temperature sensors, RAID cache batteries, memory and RAID arrays are all checked. Some of those checks can be disabled, but you don't really reboot production servers regularly so you typically wouldn't want to. The extra 3 minutes of boot time is much easier to deal with than a bad host coming online. On top of all that it's pretty standard (at least in the VMware world) to store the OS on an SD card. So the OS has to be read into memory and ESX is kinda slow to boot even if installed to a disk.
parkaboy 7y ago
When I was looking at Monero mining ages ago, I built a rig using the Dell Poweredge R810. It has 4x CPU sockets in which I have Xeon E7-4860 (for mining that uses AES) that each have 10 cores/20 threads -- 40 cores/80 threads total. I found cheap deals on both scouring eBay. The entire setup cost me < $500. However...it was noisy AF and consumed something like a few hundred Watts. Maybe as much as 500. Needless to say, I have not been running it. Does anyone have some tips on where to get cheap power? ;)
- OrangeMango 7y ago
  > Does anyone have some tips on where to get cheap power? Do you live in a location that offers real-time pricing? Where I live, you can opt-in to such a scheme and then monitor an API from the power company and adjust/schedule your power usage to favor times in which electricity is very inexpensive. [1] Sometimes, you might even get paid to consume electricity: > Negative Prices: With real-time hourly market prices, it is possible for the price of electricity to be negative for short periods of time. This typically occurs in the middle of the night and under certain circumstances when electricity supply is far greater than demand. In the market, some types of electricity generators cannot or prefer not to reduce electricity output for short periods of time when demand is insufficient, and as a result some generators may provide electricity to the market at prices below zero. Since Hourly Pricing participants pay the market price of electricity, they are actually being paid to use electricity during negative priced hours. Delivery charges still apply. [1] https://hourlypricing.comed.com/live-prices/?date=20190813 https://hourlypricing.comed.com/live-prices/?date=20190813
- semi-extrinsic 7y ago
  Depends on where you live, but round here, I use both the GPU mining rig and the home server (Dell R630) to heat my basement. For >70% of the year I would be running electric heaters down there anyway, so it's "free" electricity.
briffle 7y ago
I bought a 2013 Era Workstation last year. (Dell Precision T3600) It had a 6 core xeon, and 32GB of ECC ram, and was $300 (I had my own hard drives, and got a decent video card) it works great as a linux workstation, but its just so hot. My whole room is the warmest room in the house. I actually have to run a window air conditioner in the room to make it comfortable (I work from home, so use the office all day). I imagine its probably one of the largest power consumers (including the cost to cool with the window ac) in the house. I look forward to tax season, I am going to replace it with a new AMD 3700x based system next year.
sgt 7y ago
Been there, done that. Today it's obvious that your home server should be one place only; in the Amazon cloud, running as individual lambdas and perhaps some m4.large EC2 instances. Jokes aside, I ran a DL160 server at home for a couple of years until the motherboard started acting up. The fans would all go to 100%, and then back down again, then remain at 100% for a while. Then there were intermittent crashes. The only solution was to replace the entire motherboard. At this point I stopped looking and replaced the whole darn thing with an old headless Macbook Pro running Linux. For my purposes it is fine, and I haven't looked back. The power savings are great too.
- louwrentius 7y ago
  I haven't touched upon this in the article but for $1700 you can get quite some cloud stuff. Still, even with power cost factored in, I think I'll be better of with my own hardware.
  - sgt 7y ago
    I also prefer having something local. Then you're also completely immune to any network issues outside of your house.
jagger27 7y ago
I picked up a Dell R820 with quad 8-core (E5-4650L) and 96GB (24x4GB) RAM for USD$700 in March this year. Because of the memory mezzanines, it’s only half full in this configuration. And if I find a decent deal on E5-46xx V2s at some point I could get up to 96 threads. I even managed to get it to boot from a PCIe NVME drive with an internal USB stick running the Clover bootloader (yes, the Hackintosh bootloader) to bootstrap into Ubuntu. It makes for a great VM server. Also helps that iDRAC 7 is aeons ahead of the horrible iDRAC 6 servers I was using before.
- whalesalad 7y ago
  > iDRAC 7 Being able to completely control every aspect of the thing from my Mac via a simple Browser + VNC combo is incredible (even when it is powered off)
dev_dull 7y ago
> The server reports around 98 Watt of power ... by default, this server is turned off. This is a poor tradeoff to me. A low power computer can be left on all the time, and be there for you when you need it for things like openvpn or ssh tunnels etc. Leaving this on and idle would cost around $20 a month in electricity here in the bay area.
- louwrentius 7y ago
  It would probably cost me 2 euros per year per watt. So I think it's about the same cost. I have other low-power hardware for the OpenVPN / firewall /routing stuff so I don't need this machine turned on.
ChuckMcM 7y ago
For lab set ups "last year's" enterprise hardware can be a good deal. A couple of notes on the article 1) Setting up a RAID array isn't that difficult and it makes for more reliable storage. 2) Using dual supplies actually lowers the fan noise because the supply running at half power generates less heat than one running at full power. You can plug them both into the same outlet strip :-) 3) These things have a "lifetime" which is the point where things are easily found on the web which support them. And then they become "anchors" without all that support. Very carefully and diligently download and archive all of the necessary software, drivers, manuals, and extra cables for the system so that in another 5 years when it breaks you can reconstruct it successfully.
- louwrentius 7y ago
  1) That is true. My storage is all SSD and I'm deploying everything through Ansible, I can always redeploy. If I really need data backup, I can use one of the other SSDs or even the single spinning disk I put in as a backup target. 2) That's interesting, I don't think the noise comes from the PSU though, mainly the six case fans. 3) True point. I'm not too worried. The machine is fully supported by Linux (no drivers required) and the latest SPP is applied. I never expect to do hardware changes/upgrades down the road. And in five year's who then lives, deals with the shit ;-)
otter-in-a-suit 7y ago
Sometimes I wonder whether going the "homelab" route would have been easier/cheaper for me. I built my server a couple of months ago, from scratch.[1] However, being forced to use a proprietary tool (ssacli) and limited drive compatibility do not sound desirable. This seems like an odd limitation - is this a normal thing with these types of projects/machines? [1] https://chollinger.com/blog/2019/04/building-a-home-server/ https://chollinger.com/blog/2019/04/building-a-home-server/
- louwrentius 7y ago
  Nice blogpost! The tool is not mandatory. You can fully configure storage by booting into the storage utility of the RAID controller. It just saves you a ton of time.
old-gregg 7y ago
Ugh... that's crazy. Instead, look at ASRock X470D4U AM4 server motherboard (with IPMI and ECC) paired with something Ryzen 2700, which should give you near complete silence, 25W idle power usage and 8 modern cores boosting up to 4Ghz, for well under $1K Link: https://www.asrockrack.com/general/productdetail.asp?Model=X470D4U#Specifications https://www.asrockrack.com/general/productdetail.asp?Model=X...
- whalesalad 7y ago
  Where can you find 128gb of ECC memory for that board for under 1k (w/ processor and board)?
867-5309 7y ago
$1700 seems like an awful lot to spend on this setup, even with 1yr warranty. I recently bought the following from eBay on a £500 budget: Asrock Rack c602 mobo 2x Xeon e3-2650Lv2 (20c/40t) 8x8GB Samsung DDR3 ECC generic EATX case with fans 2x CPU heatsinks 3x case fans XFX fanless modular PSU if the price and specs alone aren't compelling enough: it runs idle at ~50w it has a similar passmark of >15000 it has 4x GbE ports it has 4x PCIE3 slots (!) I'd never heard of iLO, which other commenters mention as a selling point, but a quick search leads me to believe this is HP's take on IPMI, which this mobo has. originally built as an HTPC server, I had 2 main criteria for my build: cool and quiet. hence opting for low powered processor versions, 0db PSU and PWM case fans. if you don't require these criteria you can knock ~20% off the budget. there was so much power being unused that I binned a few other devices (namely crappy ISP-provided router and tv box) and made this build the heart of my home network. it is now my family's router, firewall, adBlocker, movie and tv server, game server, free cloud storage manager (synced to every household device), OS updates cache, music streamer, torrent client/server, VM server, web server, database server, VPN client, proxy server, etc., the list is virtually endless. these are all run simultaneously with ample resources leftover for frequent workstation usage. I should admit that I thoroughly researched every component's specifications and price, and as such it took me around 3 months of waiting to source them. I also admit this use case and learning curve is not for everyone, but it was ultimately a rewarding experience for both my brain and wallet.
- louwrentius 7y ago
  That's the first good example I've seen that really undercuts my price, but it also has 60% of the single threaded performance. Probably still enough, I can see that, it works for you. I admit I was more lazy, I have a separate room for this machine so I don't mind the noise that much. I just configured what I wanted and ordered it. I only did some research on the CPU performance.
  - 867-5309 7y ago
    yes I only really notice the single core performance on OpenVPN streams which, despite the CPUs' AES-NI and offloading to mobo's Intel-branded Ethernet chips, still cap out at ~80Mbps. it's fine for most internet needs and a few simultaneous video streams, but really bottlenecks torrenting and shifting large files around the web, on 1GBps FTTH. I toy with the idea of buying a generic fanless Chinese 4-port i7-8th Gen 15W 'U' variant to handle most networking duties, thus rendering the behemoth to be purely on-demand, WoL, semi-idle, etc. which would cut costs on electricity long term, but with such devices currently priced ~$300 and with Brexit threatening to bloat that, I am not in any rush. plus it gives me more time to research / discover / await / implement a multi-core VPN solution
lostlogin 7y ago
For anyone keen to see and read more, there is a helpful community and great write ups at https://www.reddit.com/r/homelab/ https://www.reddit.com/r/homelab/
Havoc 7y ago
This is the reason I browser /r/homelab on reddit. It doesn't make any sense to me, but is somehow cool anyway
Havoc 7y ago
I wonder if a old server can be used for heating...like literally exhaust the air into a duct and pipe it into the desired room ..In that context crazy inefficient boxes make way more sense. Esp with the reduced e-waste contribution
- gdubs 7y ago
  They make water heaters now that use ambient heat in a room to transfer that heat to the water — I believe they’re essentially heat pumps. Maybe you could put your water heater in a closet with the server and make use of some of the wasted energy.
- opportune 7y ago
  Yep, it can. Although you would want the server to be spending time doing something productive rather than just idling or spinning worthless cycles There are people who use GPUs and ASICs mining cryptocurrency as heaters for greenhouses. Spending the energy on crypto mining is basically a way to recoup some of your energy costs. https://cointelegraph.com/news/cryptomatoes-grows-5-acres-of-fruit-from-bitcoin-mining-heat https://cointelegraph.com/news/cryptomatoes-grows-5-acres-of...
  - Havoc 7y ago
    >doing something productive rather than just idling or spinning worthless cycles Saw a site recently that loans out home GPUs for tensorflow airbnb style. Nifty idea (don't recall the name sorry)
    - dest 7y ago
      Qarnot?
      - deleted 7y ago
        [deleted]
- jdnenej 7y ago
  Using it only as a heater would be a bad idea. Heat pumps are much much more efficient than resistance heaters which your server would be.
muro 7y ago
I didn't buy a single computer currently at home new - a few year old hardware is almost new and undistinguishable in performance. However I would stay away from servers - workstations (e.g. Z440) can be bought with almost the same hardware and for similar cost, yet are quiet.
- davidgerard 7y ago
  been eyeing up workstations for a gaming rig. Second-hand, 6-12 months' warranty, a ton of CPUs, a ton of memory, Nvidia Quadro cards aren't directly equivalent to the same generation of GTX but your game will run just fine ...
  - davidgerard 7y ago
    aaand we just ordered a Dell T3610 (from 2014) with four-core Xeon E5-1620v2 and Quadro K4200 4GB (also from 2014) and 32GB RAM. Just under £600. The loved one is looking forward to her new video production workstation and gaming rig.
    - muro 7y ago
      Maybe a newer graphics card would make sense (depending on the games), otherwise nice find!
      - davidgerard 7y ago
        oh yeah, a K4200 is about £100-200 second-hand. About equivalent to a GTX 960/970 for gaming. Hot stuff for 2014! The loved one spent about five minutes yesterday evening just watching the rendering of the water in WoW. Quite the improvement over her HP Envy laptop, l33t as it was for a laptop ... The PC is a huge black monolith. Easy to open, add cards to, etc - extremely maintainable. Also just about silent, hauntingly so. But really - everyone after a new box should go to eBay, PCs, search on "workstation". There's lots of ex-corporate beast machines just waiting for a home. Put 20 threads to work on your compilation!
adolph 7y ago
It would be interesting to compare this approach with an updated Atwood's Scooter Computer. https://blog.codinghorror.com/the-scooter-computer/ https://blog.codinghorror.com/the-scooter-computer/
- louwrentius 7y ago
  sysbench --test=cpu --cpu-max-prime=20000 run I got 10 seconds single thread on the DL380p. The scooter is indeed the opposite of this (awesome too btw). So to have fun: 1 core is ~3 scooters. So this box can do the work of 60 scooters ;) (only counting real cores).
timw4mail 7y ago
So...what's the real advantage of using server-grade hardware for this use-case? Wouldn't a desktop/workstation with consumer parts work just as well, and give better performance for a similar price?
- louwrentius 7y ago
  Some examples are reported by people in this post. Where I live, I could not get 20c/40t + 128GB for this price point any other way.
wodenokoto 7y ago
In this context, what is a lab setup? A LAMP stack? A tensor flow training server?
- louwrentius 7y ago
  In my case a KVM server to spin up virtual machines in which I can test anything from playing with Kubernetes to Grafana, Elasticsearch, Ubuntu MAAS, or whatever I want to toy with.
zantana 7y ago
I'm always wondering what are people doing in these homelabs that they need this type of hardware. I worked for an MSP and sometimes people would take home an old DL380 or something, but it always seemed like a waste. With a 16GB NUC I can easily provision 6-7 small vms without an issues which is enough for general self-host and exam prep. With docker you can run a simple instance of just about anything and leave it up all the time.
- louwrentius 7y ago
  Maybe I have to admit my purchase is not entirely rational. But it's fun!
  - zantana 7y ago
    Well I can't argue with that.
aussieguy1234 7y ago
How good would this box be at mining CPU based cryptocurrencies, like Monero?

239 points by louwrentius 7y ago

My home lab system of choice is the lenovo m92p tiny (or updated version) It's a headless system in a tiny enclosure. i5 processor with up to 16GB ram. Power consumption is nice and low. And they feature remote management via serial over IP, remote power cycle, etc. They can be had for less than $100 on ebay, and for me have been rock solid.

eropple 7y ago

Neat machine, but 96W idle is a lot for a home server, IMO. Maybe you're somewhere where power's super cheap (and hopefully clean), but a lot of folks aren't. I run my old desktop (a i7-6700K) in a rack in my basement, now, with 64GB of RAM, a Mellanox Connect-X for 10G networking, and half a dozen disks, and it idles under 15W. The entire rack, UniFi stuff/POE wifi APs and all, sits around 50W. 96W just for a single machine is A Lot.

stordoff 7y ago
Worth noting: > By default, this server is turned off. When I want to do some work, I turn the server on using wake-on-lan. I do this with all my equipment except for my router/firewall. If it is only on when it is doing work, idle power doesn't matter _too_ much.
- dr_zoidberg 7y ago
  The Intel ark site says the TDP for that xeon is at 115W, and that's two of them. When you're running at full power (which could happen) you're looking at the CPUs alone drawing over 200W. Another pointer to high power consumption is the 750W PSU. All in all we're not looking at a particularly efficient machine (even if it's unfair comparing it to 2019 hardware).
  - dredmorbius 7y ago
    The PSU's rating is for its potential to supply power, whilst draw components (CPUs, disk, etc.) are rated by use. The actual efficiency is defined by capacity, load, and miscellaneous other factors, but should be in the neighbourhood of 20% of delivered power. If the system idles at ~100 watts, the PSU might add another 20-30 to the mix, not 750. https://www.howtogeek.com/129667/what-exactly-does-the-wattage-rating-on-a-power-supply-unit-mean/ https://www.howtogeek.com/129667/what-exactly-does-the-watta...
    - brokenmachine 7y ago
      Agree with all of that, but I think OP was referring to the fact that higher-rated PSUs will use more power than a lower rated PSU when idle.
      - dredmorbius 7y ago
        Running a PSU at ~50% of its rated max is generally recommendable. At 115W * 2, plus other draws (disk, mobos, blinkenlights), that's around 300W. Again: peak draw, when on. I'm not arguing that this system is particularly efficient, only that you don't want to add 750W for the PSU to the draw. I've not specced out low-power systems myself. I doubt you could get 40 hyperthreads running way below this, though there are definitely some low-power systems which might have a total budget below 50W. Reddit's HomeLabPorn may have some more useful guidance: https://old.reddit.com/r/HomeLabPorn/ https://old.reddit.com/r/HomeLabPorn/ (Not my area of expertise, I've never really stayed current in HW. Though I'm aware power/thermal budget has been a major focus, both mobile and server, for the past decade or so.)
        dr_zoidberg 7y ago
        The 24 cores/48 threads EPYCs that come with 128mb L3 cache have a TDP that's between 155W and 180W. If you want to go with a lower core count (because you have a more modern architecture), you could even go with the 16c/32t EPYC 7282 (64MB L3) that sits at 120W. All in all, you're looking at almost half the power draw (AMD measures it differently than Intel, so aboutish the same) for the same performance. I specifically mention the L3 cache size because, while not being substantial, a large cache can get you 10 to 20% improved performance due less CPU stalling from cache misses. For comparison, the Xeon in question has 25MB L3, so we'd be looking at 50MB cache, split in two dies (so it doesn't quite work as a whole block of 50mb cache). greatjack613 also mentioned the fact that an AMD Zen 3900x matches the multicore performance (and has better single core perf) for a fraction of the TDP, see https://news.ycombinator.com/item?id=20687932 https://news.ycombinator.com/item?id=20687932. All in all that was the general idea: yes, you can get a used server for a good price, but we shouldn't forget the efficiency aspect of it, compared to new hardware.
rhn_mk1 7y ago
Did you do anything particular to get 15W? I have recently measured my AMD AM1 system, which sits at 40W with 8GiB RAM, 2 HDDs and a couple mostly idle VMs on it.
- rhinoceraptor 7y ago
  I'd check the BIOS to see if there are any power saving options, and/or unclocking/undervolting the CPU. 2.5" HDDs and SSDs use about the same amount of power, but 3.5" HDDs use 2-3x more. So you could potentially save 12 watts if you're using 3.5" drives. Make sure your PSU is a high efficiency one, and isn't too low, or or too high wattage. From what I've read the maximum efficiency is in the 40-80% range.
  - scns 7y ago
    Maximum efficiency is at 50%
- eropple 7y ago
  I do undervolt it and I aggressively spin down the drives. (My concern is mostly storage and redundancy, not performance.)
atonse 7y ago
In addition to wattage, my other concern for a home server was fan noise. If it's anywhere that I am, I really don't want to hear it. So I got these amazing Noctua fans and simply can't hear the server. (Updating language based on replies) A datacenter-like server probably isn't designed for that. So it wouldn't necessarily work out. I have an 8 core i5. I bought Intel (18 months ago) because I didn't want to deal with any AMD incompatibility especially since I wanted to also run it as a gaming machine using VT-d. If I were doing it again, I'd definitely go with Ryzen.
- skunkworker 7y ago
  I’m not sure what you’re getting at with the home server fans and not working out. I’ve been using a noctua on my Xeon home server without and issues, and reconfigured the fan sensors and control using ipmi. There’s no problem making a quiet Xeon home server.
  - snuxoll 7y ago
    I’ve got four servers and two switches in my home office, noise is around 45dBA and with the door closed it’s inaudible. Total power draw idle is around 340W (3.1A measure by my PDU @ 110V).
  - atonse 7y ago
    Sorry I wasn't clear. I meant something like a Rack mounted server designed for a datacenter. Would it become whisper quiet if you just replaced the fan? Is it that simple?
    - snuxoll 7y ago
      Replacing the fans on server systems is tricky, many “quiet” fans are such because they have a lower max RPM which will freak the hell out of your management controller. Also, there’s only so much optimization you can do to 40mm fans in a 1U chassis. The fans in my 1U dell servers are fairly quiet when idle around 3600RPM. They make noise, you wouldn’t want to be sleeping or watching TV in the same room - but with the door to my office closed they can’t be heard in the hallway.
wil421 7y ago
I have a UniFi USG, UniFi 150W switch (using maybe 30w), a freeNAS with a i3-8100, a mATX super micro board and 6 4TB WD Red drives. My backup power supply says maybe 120w.
- CSDude 7y ago
  My USG runs very hot, do you experience it?
  - atonse 7y ago
    All my unifi stuff runs hot. The ERLite 3 and switches, etc. I think they count on using the cases as heatsinks.
    - eropple 7y ago
      Same here. Those things get toasty. I accidentally placed a RPi3 (passively cooled but with a metal case) on top of my USG once and it got hot enough not just to throttle it, but also to cause HDMI artifacting.
  - wil421 7y ago
    Not really. I have a NanoHD and my father in law has one too. His runs really hot and mine is cool to the touch. Mine has more traffic and was made before his NanoHD.
    - CSDude 7y ago
      I also have a NanoHD and its acceptable but USG (non-pro) runs so hot that I fear leaving it in a non airconditoned cupboard that my internet exit resides
  - michaelper22 7y ago
    Yup. My USG and US-8-60W both run hot enough to not want my hand on for very long (the power supplies are pretty warm too).
fnord77 7y ago
would newer hardware idle at even lower wattage than 15W? Or is that about the best one could expect?
- rhinoceraptor 7y ago
  For a full desktop class chip, I think that's about as low as it can go. You could go lower with something like a NUC, but 15 watts is really good. 15 watts idle would cost something like $1.50 to $2 to run 24/7.
- eropple 7y ago
  Newer, maybe slightly but not hugely. (It's all still kinda warmed-over Skylake.) You could get cheaper with lower-TDP chips, but there are definitely diminishing returns on the floor.
AdrianB1 7y ago
Probably an error in measuring the power consumption, there is no way to go to 15W, double that or more.
- rhinoceraptor 7y ago
  It's believable (if only obtained with a lot of BIOS tweaking), I run an older Dell 1u server as my router. It's got an E3-1220 V2 (2012 era, 69 watt TDP), one 2.5" SSD, 8gb of RAM and after a bit of tweaking it sits at around 30 watts.
  - eropple 7y ago
    It wasn't a lot of BIOS tweaking, but yes, some undervolting was required. ;) I also hdparm'd it such that my spinning disks park pretty aggressively.
tbyehl 7y ago
Or it's a ton of compute in a single system for just 96W. Average US rates are roughly $1/yr per continuous watt.
egwynn 7y ago
To contextualize this, ~100W will probably cost between $10-$15/mo to run 24/7. EDIT: Assuming a US household paying between $0.10-$0.20 per kwh.
- jdnenej 7y ago
  Where I am it costs 37c per kWh. It's no wonder almost everyone has solar.
Havoc 7y ago
>idles under 15W Damn...meanwhile my fairly modern Asus laptop idles at 27W. (Probably the GPU...think I forced it to dedicated only)
louwrentius 7y ago
I turn this server off when I'm done experimenting. I turn it on remotely using wake-on-lan.
- apexalpha 7y ago
  What exactly do you use it for if it isn on 24/7? I bought a used intel Nuc a while back with 15w idle. And it runs 24/7 and does everything I need from downloading to nextcloud to plex, my website, my emails etc... I really wonder what you do with this kind of power.
abstractbarista 7y ago
Some people like me will justify it due to being a huge hobby for them. It takes the place of what other people would spend going out often, traveling, streaming, etc... I've got a rack that pulls ~600W 24/7 but that's across 4 machines, 3 UPS's, >30 HDD's yielding >20TB across multiple zpools, >256GB RAM. And I'm actually using most of that capacity, not just idling away. I do hope to upgrade soon to power-sipping platforms like you mention but currently I'm still on R710/R810 stuff (Westmere Xeon's).
- Dayshine 7y ago
  I have to ask, using it to do what?
  - berbec 7y ago
    Linux ISO plex server.
  - abstractbarista 7y ago
    I wish I had something written up to share. But it's honestly a mix of things I love to play with. Several full blockchain nodes (especially uploading to light clients), bittorrent client (seeding tons), plex, home assistant, boinc, tor bridge, grafana, web and email servers, zoneminder CCTV, mandelbulber (clustered rendering of 3d fractals), four different Minecraft servers (some modded), ADS-B plane tracking, weather station and radiation monitoring, and a bunch more. It's really just one big toy for me. I'm using Proxmox for high availability of over 30 VM's (still want to play with containers soon). EDIT: OPNSense firewall says I uploaded over 15TB in the last month. Fortunately Google fiber doesn't care!
    - louwrentius 7y ago
      That's awesome!
- louwrentius 7y ago
  That is awesome. Do you have something written up about your setup online?
CommieBobDole 7y ago
You know, that raises a question - how expensive does power get, anyway? Last time I posted my home lab setup (couple of old R610s, Cisco small business switch, ASA 5510), I got a lot of pushback on how much power that consumes and how expensive it is to run. I did the math, and I was looking at maybe $20-$30/month depending on load. Which is not really a lot of money considering how much use I get out of it. I'm definitely getting cheaper power than a lot of people at $.08/kWh, but it looks like the US average is only about $.12 - are there places where a couple hundred watts is going to be a significant financial burden on the average IT worker?
- eropple 7y ago
  To me it's less a financial burden and more the discomfort of burning electricity just to run a few Docker containers and some databases for personal projects. I expect that once, next summer, we get the house kitted with solar and batteries, I will feel a little more okay about it.
louwrentius 7y ago
I've done a measurement with a power meter and it's even 110W. (it includes ilo).
Aeolun 7y ago
That costs you maybe $.5 dollars a day? Maybe $1 if you are unlucky. This seems fairly reasonable?

greatjack613 7y ago

To put this in perspective, The new $500 ryzen 9 3900x matches this $1700 xeon server in multi core geekbench, and kicks it in single core perf with a fraction of the power usage. https://browser.geekbench.com/processors/2587 https://browser.geekbench.com/processors/2587

btgeekboy 7y ago
That $1500 includes the rest of the computer, though. That adds up.
- greatjack613 7y ago
  correct, but you can get build a full system with the Ryzen 9 plus 128gb of ram etc for definitely less than $1700. Especially given the drop in ram and ssd's these days
  - pedrocr 7y ago
    128GB of ECC RAM itself is going to be ~800-1000$ as far as I can find some to fit in a Ryzen build. So it won't be trivial to reach the price point for a full build although Ryzen is clearly great value anyway.
    - greatjack613 7y ago
      True, but only if ECC is necessary. In this case where he is mainly using the server for experiments and to try software, regular ram will do absolutely fine. Here is a 128gb kit for under $600 dollars. https://www.newegg.com/corsair-128gb-288-pin-ddr4-sdram/p/N82E16820233946 https://www.newegg.com/corsair-128gb-288-pin-ddr4-sdram/p/N8...
      - close04 7y ago
        > only if ECC is necessary If your workloads require 128GB of RAM then you probably want ECC. You'd be skimping in the wrong place.

sigmonsays 7y ago

Where do you store this machine? How is the noise? also, I wonder about the heat generated. Does it keep a room toastier than the rest of the house?

deleted 7y ago
[deleted]
OJFord 7y ago
Many examples on reddit.com/r/homelab - including even more hungry (or several of them in a rack(s)) systems. The short answer is Yes, servers built to be servers are designed to get the heat out and keep internal temp. down - so noisy fans and as much heat as you generate outside of the box. Of course, nothing stops you replacing fans with quieter ones (at least one of more expensive or air movement) or putting consumer hardware (which has different design goals that you might prefer in the home) in a rack-mount chassis.
neogodless 7y ago
The article has some answers to these questions: > My very subjective opinion is that at 50 dB the sound level is reasonable for a server like this, but it's definitely not quiet. I would not be able to work, relax or sleep with this server in the same room. > Although this server is fairly quiet at idle, it does need it's own dedicated room. When the door is closed, you won't hear it at idle, but under sustained load, you will hear high pitched fan noise even with a closed door.
sannee 7y ago
I live in a dorm room and keep an R610 about three meters from my bed. You can use some magical IPMI commands to disable the internal fan feedback loop and replace it with a custom one (usually the defaults cool the server to around 30C, which is not really necessary). This makes the server more quiet then the small fridge I also have in the room. The 100W power consumption definitely makes the room warmer. It is approximately like having another person present.

eanzenberg 7y ago

But can it play Crysis?

holy_city 7y ago

I think for $1700 with the memory/storage capacity that's a great deal, but those benchmarks can be topped by the higher-tier consumer CPUs today. The point on people being happy with slower CPU cores is kind of weird to bring up with a server. Most games don't push CPUs that hard, you usually need a really expensive GPU before you see noticeable benefits in gaming from faster processors. Having done some core critical work for the last few years (media processing/systems programming), my recent upgrade from a 4th gen i7 to a Zen2 CPU is paying in spades. If I was building a server to do some of the batch processing stuff I'd like, I would definitely invest in a faster, cooler, more power efficient machine. But that's just me. I don't think I could beat that price point though.

whalesalad 7y ago
Dunno... I have an R720 with similar specs (half the ram, though) and it was only $400. I also idle around ~100w because I ripped all the SAS 10k drives out and put SSD's in. It sits powered-off most of the time though because I haven't been able to put it to good use, yet. For a while it was running my Unifi controller + Pihole ... but you don't need the Unifi controller unless you are actively performing maintenance, and Pihole happily hums along on a Rpi 3 that uses far less power.

blackflame7000 7y ago

I built a very similar computer and tbh I prefer to use an 8th gen core-i7 at 4.8ghz as it is faster and less power-hungry for most everyday applications. Javascript, for example, is mostly single-threaded and you can see a noticeable difference in web loading times.

zokier 7y ago

I got bitten by homelab fever few years back and got myself a small server. I had such grand dreams with it that never materialized. Now it sits unplugged in the corner deprecating itself :( Stuff that I was planning to do: * Managed VM platform (~"EC2") * Centralized auth (FreeIPA) * ZFS NAS (also possibly ceph) + backuping * Container platform * Your typical web/email stuff * Monitoring/alerting/log management * VPN endpoint (and other more advanced networking stuff) * Probably something more I have already forgotten I realized that building a private cloud actually takes serious effort and not just putting some lego pieces together. There is also bit of circulatory stuff there that makes bootstrapping more difficult, especially on one single box.

roel_v 7y ago
If you want something that does these things and mostly just works, get a Qnap. More expensive, of course.
mey 7y ago
To make it simple, since you already have the hardware, I would suggest setting up the server as a VM Host to allow for experimentation with little effort. VMware vSphere Hypervisor, Proxmox, or Microsoft Hyper-V Server 2019 are all free options. It makes it easy to experiment with the above in parallel. My current VM Host has * One VM as a docker host (turtles all the way down) for development tools. Build server, bug tracker, private artifact repo (Sonatype Nexus). * 3 VMs as a Kubernetes cluster for experimentation Building a NAS is the only thing that would take significant effort and is a project unto itself. Unless you have a strong desire to experiment with failure modes (network dropping out, killing iSCSI, online VM fail over), stick to a single box.
vorpalhex 7y ago
I've been pretty successful in meeting my private cloud dreams. I use FreeNAS as the bare metal OS, and run Ubuntu VMs which host Docker Containers (using NFS to keep actual persistent data on the underlying FreeNAS box). Docker/Docker-Compose isn't quite lego.. but it's awfully close.
bsder 7y ago

vorpalhex 7y ago

This is a neat writeup, thanks! Do you use a proper server rack and if so, where do you keep it installed? Right now my homelab is in a tower, but that's obviously not ideal for density.

louwrentius 7y ago
Thank you. I use a lack rack setup. https://gathering.tweakers.net/forum/list_messages/1656249/last https://gathering.tweakers.net/forum/list_messages/1656249/l...

kart23 7y ago

Boot time of 4 minutes! Is that normal for servers? And why would it be this long if so?

whalesalad 7y ago
Yes and no. There is a longer boot time because the POST process is a bit more intense. ECC RAM is a big part of that. Chances are this particular caes is likely becuase the boot order needs to be reconfigured and it is hunting for PXE/network stuff before hitting a timeout? If you yield control to some kind of Broadcom controller it'll do all kinds of shit before giving up and handing you back to boot a disk.
- louwrentius 7y ago
  I boot straight from HDD so no waiting on PXE boot timeouts or other shenanigans (boot media, etc).
edwintorok 7y ago
Unfortunately it is quite common that servers are slower to boot than desktops if it has a lot of CPUs, or a lot of memory, or just a very slow BIOS impl that spends a lot of time probing and initializing everything. Servers with hundreds of cores are a lot slower to boot than my laptop too.
nodja 7y ago
Yes. It's just not a priority for servers. The reason it takes this long is that it does a bunch of self tests, and then it has to load all the ROMs for the components (NICs, HBAs, etc.) which often trigger messages like "X Loaded, press CTRL+L to configure" which stay on screen for 5-10 seconds each.
sh-run 7y ago
It's been a while since I was in a position where I regularly watched servers boot, but 2015ish I was doing a lot of 'boots on the ground' sysadmin and virtualization work. Servers do a lot of additional testing during boot-up, temperature sensors, RAID cache batteries, memory and RAID arrays are all checked. Some of those checks can be disabled, but you don't really reboot production servers regularly so you typically wouldn't want to. The extra 3 minutes of boot time is much easier to deal with than a bad host coming online. On top of all that it's pretty standard (at least in the VMware world) to store the OS on an SD card. So the OS has to be read into memory and ESX is kinda slow to boot even if installed to a disk.

parkaboy 7y ago

When I was looking at Monero mining ages ago, I built a rig using the Dell Poweredge R810. It has 4x CPU sockets in which I have Xeon E7-4860 (for mining that uses AES) that each have 10 cores/20 threads -- 40 cores/80 threads total. I found cheap deals on both scouring eBay. The entire setup cost me < $500. However...it was noisy AF and consumed something like a few hundred Watts. Maybe as much as 500. Needless to say, I have not been running it. Does anyone have some tips on where to get cheap power? ;)

OrangeMango 7y ago
> Does anyone have some tips on where to get cheap power? Do you live in a location that offers real-time pricing? Where I live, you can opt-in to such a scheme and then monitor an API from the power company and adjust/schedule your power usage to favor times in which electricity is very inexpensive. [1] Sometimes, you might even get paid to consume electricity: > Negative Prices: With real-time hourly market prices, it is possible for the price of electricity to be negative for short periods of time. This typically occurs in the middle of the night and under certain circumstances when electricity supply is far greater than demand. In the market, some types of electricity generators cannot or prefer not to reduce electricity output for short periods of time when demand is insufficient, and as a result some generators may provide electricity to the market at prices below zero. Since Hourly Pricing participants pay the market price of electricity, they are actually being paid to use electricity during negative priced hours. Delivery charges still apply. [1] https://hourlypricing.comed.com/live-prices/?date=20190813 https://hourlypricing.comed.com/live-prices/?date=20190813
semi-extrinsic 7y ago
Depends on where you live, but round here, I use both the GPU mining rig and the home server (Dell R630) to heat my basement. For >70% of the year I would be running electric heaters down there anyway, so it's "free" electricity.

briffle 7y ago

I bought a 2013 Era Workstation last year. (Dell Precision T3600) It had a 6 core xeon, and 32GB of ECC ram, and was $300 (I had my own hard drives, and got a decent video card) it works great as a linux workstation, but its just so hot. My whole room is the warmest room in the house. I actually have to run a window air conditioner in the room to make it comfortable (I work from home, so use the office all day). I imagine its probably one of the largest power consumers (including the cost to cool with the window ac) in the house. I look forward to tax season, I am going to replace it with a new AMD 3700x based system next year.

sgt 7y ago

Been there, done that. Today it's obvious that your home server should be one place only; in the Amazon cloud, running as individual lambdas and perhaps some m4.large EC2 instances. Jokes aside, I ran a DL160 server at home for a couple of years until the motherboard started acting up. The fans would all go to 100%, and then back down again, then remain at 100% for a while. Then there were intermittent crashes. The only solution was to replace the entire motherboard. At this point I stopped looking and replaced the whole darn thing with an old headless Macbook Pro running Linux. For my purposes it is fine, and I haven't looked back. The power savings are great too.

louwrentius 7y ago
I haven't touched upon this in the article but for $1700 you can get quite some cloud stuff. Still, even with power cost factored in, I think I'll be better of with my own hardware.
- sgt 7y ago
  I also prefer having something local. Then you're also completely immune to any network issues outside of your house.

jagger27 7y ago

I picked up a Dell R820 with quad 8-core (E5-4650L) and 96GB (24x4GB) RAM for USD$700 in March this year. Because of the memory mezzanines, it’s only half full in this configuration. And if I find a decent deal on E5-46xx V2s at some point I could get up to 96 threads. I even managed to get it to boot from a PCIe NVME drive with an internal USB stick running the Clover bootloader (yes, the Hackintosh bootloader) to bootstrap into Ubuntu. It makes for a great VM server. Also helps that iDRAC 7 is aeons ahead of the horrible iDRAC 6 servers I was using before.

whalesalad 7y ago
> iDRAC 7 Being able to completely control every aspect of the thing from my Mac via a simple Browser + VNC combo is incredible (even when it is powered off)

dev_dull 7y ago

> The server reports around 98 Watt of power ... by default, this server is turned off. This is a poor tradeoff to me. A low power computer can be left on all the time, and be there for you when you need it for things like openvpn or ssh tunnels etc. Leaving this on and idle would cost around $20 a month in electricity here in the bay area.

louwrentius 7y ago
It would probably cost me 2 euros per year per watt. So I think it's about the same cost. I have other low-power hardware for the OpenVPN / firewall /routing stuff so I don't need this machine turned on.

ChuckMcM 7y ago

For lab set ups "last year's" enterprise hardware can be a good deal. A couple of notes on the article 1) Setting up a RAID array isn't that difficult and it makes for more reliable storage. 2) Using dual supplies actually lowers the fan noise because the supply running at half power generates less heat than one running at full power. You can plug them both into the same outlet strip :-) 3) These things have a "lifetime" which is the point where things are easily found on the web which support them. And then they become "anchors" without all that support. Very carefully and diligently download and archive all of the necessary software, drivers, manuals, and extra cables for the system so that in another 5 years when it breaks you can reconstruct it successfully.

louwrentius 7y ago
1) That is true. My storage is all SSD and I'm deploying everything through Ansible, I can always redeploy. If I really need data backup, I can use one of the other SSDs or even the single spinning disk I put in as a backup target. 2) That's interesting, I don't think the noise comes from the PSU though, mainly the six case fans. 3) True point. I'm not too worried. The machine is fully supported by Linux (no drivers required) and the latest SPP is applied. I never expect to do hardware changes/upgrades down the road. And in five year's who then lives, deals with the shit ;-)

otter-in-a-suit 7y ago

Sometimes I wonder whether going the "homelab" route would have been easier/cheaper for me. I built my server a couple of months ago, from scratch.[1] However, being forced to use a proprietary tool (ssacli) and limited drive compatibility do not sound desirable. This seems like an odd limitation - is this a normal thing with these types of projects/machines? [1] https://chollinger.com/blog/2019/04/building-a-home-server/ https://chollinger.com/blog/2019/04/building-a-home-server/

louwrentius 7y ago
Nice blogpost! The tool is not mandatory. You can fully configure storage by booting into the storage utility of the RAID controller. It just saves you a ton of time.

old-gregg 7y ago

Ugh... that's crazy. Instead, look at ASRock X470D4U AM4 server motherboard (with IPMI and ECC) paired with something Ryzen 2700, which should give you near complete silence, 25W idle power usage and 8 modern cores boosting up to 4Ghz, for well under $1K Link: https://www.asrockrack.com/general/productdetail.asp?Model=X470D4U#Specifications https://www.asrockrack.com/general/productdetail.asp?Model=X...

whalesalad 7y ago
Where can you find 128gb of ECC memory for that board for under 1k (w/ processor and board)?

867-5309 7y ago

$1700 seems like an awful lot to spend on this setup, even with 1yr warranty. I recently bought the following from eBay on a £500 budget: Asrock Rack c602 mobo 2x Xeon e3-2650Lv2 (20c/40t) 8x8GB Samsung DDR3 ECC generic EATX case with fans 2x CPU heatsinks 3x case fans XFX fanless modular PSU if the price and specs alone aren't compelling enough: it runs idle at ~50w it has a similar passmark of >15000 it has 4x GbE ports it has 4x PCIE3 slots (!) I'd never heard of iLO, which other commenters mention as a selling point, but a quick search leads me to believe this is HP's take on IPMI, which this mobo has. originally built as an HTPC server, I had 2 main criteria for my build: cool and quiet. hence opting for low powered processor versions, 0db PSU and PWM case fans. if you don't require these criteria you can knock ~20% off the budget. there was so much power being unused that I binned a few other devices (namely crappy ISP-provided router and tv box) and made this build the heart of my home network. it is now my family's router, firewall, adBlocker, movie and tv server, game server, free cloud storage manager (synced to every household device), OS updates cache, music streamer, torrent client/server, VM server, web server, database server, VPN client, proxy server, etc., the list is virtually endless. these are all run simultaneously with ample resources leftover for frequent workstation usage. I should admit that I thoroughly researched every component's specifications and price, and as such it took me around 3 months of waiting to source them. I also admit this use case and learning curve is not for everyone, but it was ultimately a rewarding experience for both my brain and wallet.

louwrentius 7y ago
That's the first good example I've seen that really undercuts my price, but it also has 60% of the single threaded performance. Probably still enough, I can see that, it works for you. I admit I was more lazy, I have a separate room for this machine so I don't mind the noise that much. I just configured what I wanted and ordered it. I only did some research on the CPU performance.
- 867-5309 7y ago
  yes I only really notice the single core performance on OpenVPN streams which, despite the CPUs' AES-NI and offloading to mobo's Intel-branded Ethernet chips, still cap out at ~80Mbps. it's fine for most internet needs and a few simultaneous video streams, but really bottlenecks torrenting and shifting large files around the web, on 1GBps FTTH. I toy with the idea of buying a generic fanless Chinese 4-port i7-8th Gen 15W 'U' variant to handle most networking duties, thus rendering the behemoth to be purely on-demand, WoL, semi-idle, etc. which would cut costs on electricity long term, but with such devices currently priced ~$300 and with Brexit threatening to bloat that, I am not in any rush. plus it gives me more time to research / discover / await / implement a multi-core VPN solution

lostlogin 7y ago

For anyone keen to see and read more, there is a helpful community and great write ups at https://www.reddit.com/r/homelab/ https://www.reddit.com/r/homelab/

Havoc 7y ago

This is the reason I browser /r/homelab on reddit. It doesn't make any sense to me, but is somehow cool anyway

Havoc 7y ago

I wonder if a old server can be used for heating...like literally exhaust the air into a duct and pipe it into the desired room ..In that context crazy inefficient boxes make way more sense. Esp with the reduced e-waste contribution

gdubs 7y ago
They make water heaters now that use ambient heat in a room to transfer that heat to the water — I believe they’re essentially heat pumps. Maybe you could put your water heater in a closet with the server and make use of some of the wasted energy.
opportune 7y ago
Yep, it can. Although you would want the server to be spending time doing something productive rather than just idling or spinning worthless cycles There are people who use GPUs and ASICs mining cryptocurrency as heaters for greenhouses. Spending the energy on crypto mining is basically a way to recoup some of your energy costs. https://cointelegraph.com/news/cryptomatoes-grows-5-acres-of-fruit-from-bitcoin-mining-heat https://cointelegraph.com/news/cryptomatoes-grows-5-acres-of...
- Havoc 7y ago
  >doing something productive rather than just idling or spinning worthless cycles Saw a site recently that loans out home GPUs for tensorflow airbnb style. Nifty idea (don't recall the name sorry)
  - dest 7y ago
    Qarnot?
    - deleted 7y ago
      [deleted]
jdnenej 7y ago
Using it only as a heater would be a bad idea. Heat pumps are much much more efficient than resistance heaters which your server would be.

muro 7y ago

I didn't buy a single computer currently at home new - a few year old hardware is almost new and undistinguishable in performance. However I would stay away from servers - workstations (e.g. Z440) can be bought with almost the same hardware and for similar cost, yet are quiet.

davidgerard 7y ago
been eyeing up workstations for a gaming rig. Second-hand, 6-12 months' warranty, a ton of CPUs, a ton of memory, Nvidia Quadro cards aren't directly equivalent to the same generation of GTX but your game will run just fine ...
- davidgerard 7y ago
  aaand we just ordered a Dell T3610 (from 2014) with four-core Xeon E5-1620v2 and Quadro K4200 4GB (also from 2014) and 32GB RAM. Just under £600. The loved one is looking forward to her new video production workstation and gaming rig.
  - muro 7y ago
    Maybe a newer graphics card would make sense (depending on the games), otherwise nice find!
    - davidgerard 7y ago
      oh yeah, a K4200 is about £100-200 second-hand. About equivalent to a GTX 960/970 for gaming. Hot stuff for 2014! The loved one spent about five minutes yesterday evening just watching the rendering of the water in WoW. Quite the improvement over her HP Envy laptop, l33t as it was for a laptop ... The PC is a huge black monolith. Easy to open, add cards to, etc - extremely maintainable. Also just about silent, hauntingly so. But really - everyone after a new box should go to eBay, PCs, search on "workstation". There's lots of ex-corporate beast machines just waiting for a home. Put 20 threads to work on your compilation!

adolph 7y ago

It would be interesting to compare this approach with an updated Atwood's Scooter Computer. https://blog.codinghorror.com/the-scooter-computer/ https://blog.codinghorror.com/the-scooter-computer/

louwrentius 7y ago
sysbench --test=cpu --cpu-max-prime=20000 run I got 10 seconds single thread on the DL380p. The scooter is indeed the opposite of this (awesome too btw). So to have fun: 1 core is ~3 scooters. So this box can do the work of 60 scooters ;) (only counting real cores).

timw4mail 7y ago

So...what's the real advantage of using server-grade hardware for this use-case? Wouldn't a desktop/workstation with consumer parts work just as well, and give better performance for a similar price?

louwrentius 7y ago
Some examples are reported by people in this post. Where I live, I could not get 20c/40t + 128GB for this price point any other way.

wodenokoto 7y ago

In this context, what is a lab setup? A LAMP stack? A tensor flow training server?

louwrentius 7y ago
In my case a KVM server to spin up virtual machines in which I can test anything from playing with Kubernetes to Grafana, Elasticsearch, Ubuntu MAAS, or whatever I want to toy with.

zantana 7y ago

I'm always wondering what are people doing in these homelabs that they need this type of hardware. I worked for an MSP and sometimes people would take home an old DL380 or something, but it always seemed like a waste. With a 16GB NUC I can easily provision 6-7 small vms without an issues which is enough for general self-host and exam prep. With docker you can run a simple instance of just about anything and leave it up all the time.

louwrentius 7y ago
Maybe I have to admit my purchase is not entirely rational. But it's fun!
- zantana 7y ago
  Well I can't argue with that.

aussieguy1234 7y ago

How good would this box be at mining CPU based cryptocurrencies, like Monero?