8 ms·

Neat machine, but 96W idle is a lot for a home server, IMO. Maybe you're somewhere where power's super cheap (and hopefully clean), but a lot of folks aren't.

Neat machine, but 96W idle is a lot for a home server, IMO. Maybe you're somewhere where power's super cheap (and hopefully clean), but a lot of folks aren't. I run my old desktop (a i7-6700K) in a rack in my basement, now, with 64GB of RAM, a Mellanox Connect-X for 10G networking, and half a dozen disks, and it idles under 15W. The entire rack, UniFi stuff/POE wifi APs and all, sits around 50W. 96W just for a single machine is A Lot.

stordoff 7y ago
Worth noting: > By default, this server is turned off. When I want to do some work, I turn the server on using wake-on-lan. I do this with all my equipment except for my router/firewall. If it is only on when it is doing work, idle power doesn't matter _too_ much.
- dr_zoidberg 7y ago
  The Intel ark site says the TDP for that xeon is at 115W, and that's two of them. When you're running at full power (which could happen) you're looking at the CPUs alone drawing over 200W. Another pointer to high power consumption is the 750W PSU. All in all we're not looking at a particularly efficient machine (even if it's unfair comparing it to 2019 hardware).
  - dredmorbius 7y ago
    The PSU's rating is for its potential to supply power, whilst draw components (CPUs, disk, etc.) are rated by use. The actual efficiency is defined by capacity, load, and miscellaneous other factors, but should be in the neighbourhood of 20% of delivered power. If the system idles at ~100 watts, the PSU might add another 20-30 to the mix, not 750. https://www.howtogeek.com/129667/what-exactly-does-the-wattage-rating-on-a-power-supply-unit-mean/ https://www.howtogeek.com/129667/what-exactly-does-the-watta...
    - brokenmachine 7y ago
      Agree with all of that, but I think OP was referring to the fact that higher-rated PSUs will use more power than a lower rated PSU when idle.
      - dredmorbius 7y ago
        Running a PSU at ~50% of its rated max is generally recommendable. At 115W * 2, plus other draws (disk, mobos, blinkenlights), that's around 300W. Again: peak draw, when on. I'm not arguing that this system is particularly efficient, only that you don't want to add 750W for the PSU to the draw. I've not specced out low-power systems myself. I doubt you could get 40 hyperthreads running way below this, though there are definitely some low-power systems which might have a total budget below 50W. Reddit's HomeLabPorn may have some more useful guidance: https://old.reddit.com/r/HomeLabPorn/ https://old.reddit.com/r/HomeLabPorn/ (Not my area of expertise, I've never really stayed current in HW. Though I'm aware power/thermal budget has been a major focus, both mobile and server, for the past decade or so.)
        dr_zoidberg 7y ago
        The 24 cores/48 threads EPYCs that come with 128mb L3 cache have a TDP that's between 155W and 180W. If you want to go with a lower core count (because you have a more modern architecture), you could even go with the 16c/32t EPYC 7282 (64MB L3) that sits at 120W. All in all, you're looking at almost half the power draw (AMD measures it differently than Intel, so aboutish the same) for the same performance. I specifically mention the L3 cache size because, while not being substantial, a large cache can get you 10 to 20% improved performance due less CPU stalling from cache misses. For comparison, the Xeon in question has 25MB L3, so we'd be looking at 50MB cache, split in two dies (so it doesn't quite work as a whole block of 50mb cache). greatjack613 also mentioned the fact that an AMD Zen 3900x matches the multicore performance (and has better single core perf) for a fraction of the TDP, see https://news.ycombinator.com/item?id=20687932 https://news.ycombinator.com/item?id=20687932. All in all that was the general idea: yes, you can get a used server for a good price, but we shouldn't forget the efficiency aspect of it, compared to new hardware.
rhn_mk1 7y ago
Did you do anything particular to get 15W? I have recently measured my AMD AM1 system, which sits at 40W with 8GiB RAM, 2 HDDs and a couple mostly idle VMs on it.
- rhinoceraptor 7y ago
  I'd check the BIOS to see if there are any power saving options, and/or unclocking/undervolting the CPU. 2.5" HDDs and SSDs use about the same amount of power, but 3.5" HDDs use 2-3x more. So you could potentially save 12 watts if you're using 3.5" drives. Make sure your PSU is a high efficiency one, and isn't too low, or or too high wattage. From what I've read the maximum efficiency is in the 40-80% range.
  - scns 7y ago
    Maximum efficiency is at 50%
- eropple 7y ago
  I do undervolt it and I aggressively spin down the drives. (My concern is mostly storage and redundancy, not performance.)
atonse 7y ago
In addition to wattage, my other concern for a home server was fan noise. If it's anywhere that I am, I really don't want to hear it. So I got these amazing Noctua fans and simply can't hear the server. (Updating language based on replies) A datacenter-like server probably isn't designed for that. So it wouldn't necessarily work out. I have an 8 core i5. I bought Intel (18 months ago) because I didn't want to deal with any AMD incompatibility especially since I wanted to also run it as a gaming machine using VT-d. If I were doing it again, I'd definitely go with Ryzen.
- skunkworker 7y ago
  I’m not sure what you’re getting at with the home server fans and not working out. I’ve been using a noctua on my Xeon home server without and issues, and reconfigured the fan sensors and control using ipmi. There’s no problem making a quiet Xeon home server.
  - snuxoll 7y ago
    I’ve got four servers and two switches in my home office, noise is around 45dBA and with the door closed it’s inaudible. Total power draw idle is around 340W (3.1A measure by my PDU @ 110V).
  - atonse 7y ago
    Sorry I wasn't clear. I meant something like a Rack mounted server designed for a datacenter. Would it become whisper quiet if you just replaced the fan? Is it that simple?
    - snuxoll 7y ago
      Replacing the fans on server systems is tricky, many “quiet” fans are such because they have a lower max RPM which will freak the hell out of your management controller. Also, there’s only so much optimization you can do to 40mm fans in a 1U chassis. The fans in my 1U dell servers are fairly quiet when idle around 3600RPM. They make noise, you wouldn’t want to be sleeping or watching TV in the same room - but with the door to my office closed they can’t be heard in the hallway.
wil421 7y ago
I have a UniFi USG, UniFi 150W switch (using maybe 30w), a freeNAS with a i3-8100, a mATX super micro board and 6 4TB WD Red drives. My backup power supply says maybe 120w.
- CSDude 7y ago
  My USG runs very hot, do you experience it?
  - atonse 7y ago
    All my unifi stuff runs hot. The ERLite 3 and switches, etc. I think they count on using the cases as heatsinks.
    - eropple 7y ago
      Same here. Those things get toasty. I accidentally placed a RPi3 (passively cooled but with a metal case) on top of my USG once and it got hot enough not just to throttle it, but also to cause HDMI artifacting.
  - wil421 7y ago
    Not really. I have a NanoHD and my father in law has one too. His runs really hot and mine is cool to the touch. Mine has more traffic and was made before his NanoHD.
    - CSDude 7y ago
      I also have a NanoHD and its acceptable but USG (non-pro) runs so hot that I fear leaving it in a non airconditoned cupboard that my internet exit resides
  - michaelper22 7y ago
    Yup. My USG and US-8-60W both run hot enough to not want my hand on for very long (the power supplies are pretty warm too).
fnord77 7y ago
would newer hardware idle at even lower wattage than 15W? Or is that about the best one could expect?
- rhinoceraptor 7y ago
  For a full desktop class chip, I think that's about as low as it can go. You could go lower with something like a NUC, but 15 watts is really good. 15 watts idle would cost something like $1.50 to $2 to run 24/7.
- eropple 7y ago
  Newer, maybe slightly but not hugely. (It's all still kinda warmed-over Skylake.) You could get cheaper with lower-TDP chips, but there are definitely diminishing returns on the floor.
AdrianB1 7y ago
Probably an error in measuring the power consumption, there is no way to go to 15W, double that or more.
- rhinoceraptor 7y ago
  It's believable (if only obtained with a lot of BIOS tweaking), I run an older Dell 1u server as my router. It's got an E3-1220 V2 (2012 era, 69 watt TDP), one 2.5" SSD, 8gb of RAM and after a bit of tweaking it sits at around 30 watts.
  - eropple 7y ago
    It wasn't a lot of BIOS tweaking, but yes, some undervolting was required. ;) I also hdparm'd it such that my spinning disks park pretty aggressively.
tbyehl 7y ago
Or it's a ton of compute in a single system for just 96W. Average US rates are roughly $1/yr per continuous watt.
egwynn 7y ago
To contextualize this, ~100W will probably cost between $10-$15/mo to run 24/7. EDIT: Assuming a US household paying between $0.10-$0.20 per kwh.
- jdnenej 7y ago
  Where I am it costs 37c per kWh. It's no wonder almost everyone has solar.
Havoc 7y ago
>idles under 15W Damn...meanwhile my fairly modern Asus laptop idles at 27W. (Probably the GPU...think I forced it to dedicated only)
louwrentius 7y ago
I turn this server off when I'm done experimenting. I turn it on remotely using wake-on-lan.
- apexalpha 7y ago
  What exactly do you use it for if it isn on 24/7? I bought a used intel Nuc a while back with 15w idle. And it runs 24/7 and does everything I need from downloading to nextcloud to plex, my website, my emails etc... I really wonder what you do with this kind of power.
abstractbarista 7y ago
Some people like me will justify it due to being a huge hobby for them. It takes the place of what other people would spend going out often, traveling, streaming, etc... I've got a rack that pulls ~600W 24/7 but that's across 4 machines, 3 UPS's, >30 HDD's yielding >20TB across multiple zpools, >256GB RAM. And I'm actually using most of that capacity, not just idling away. I do hope to upgrade soon to power-sipping platforms like you mention but currently I'm still on R710/R810 stuff (Westmere Xeon's).
- Dayshine 7y ago
  I have to ask, using it to do what?
  - berbec 7y ago
    Linux ISO plex server.
  - abstractbarista 7y ago
    I wish I had something written up to share. But it's honestly a mix of things I love to play with. Several full blockchain nodes (especially uploading to light clients), bittorrent client (seeding tons), plex, home assistant, boinc, tor bridge, grafana, web and email servers, zoneminder CCTV, mandelbulber (clustered rendering of 3d fractals), four different Minecraft servers (some modded), ADS-B plane tracking, weather station and radiation monitoring, and a bunch more. It's really just one big toy for me. I'm using Proxmox for high availability of over 30 VM's (still want to play with containers soon). EDIT: OPNSense firewall says I uploaded over 15TB in the last month. Fortunately Google fiber doesn't care!
    - louwrentius 7y ago
      That's awesome!
- louwrentius 7y ago
  That is awesome. Do you have something written up about your setup online?
CommieBobDole 7y ago
You know, that raises a question - how expensive does power get, anyway? Last time I posted my home lab setup (couple of old R610s, Cisco small business switch, ASA 5510), I got a lot of pushback on how much power that consumes and how expensive it is to run. I did the math, and I was looking at maybe $20-$30/month depending on load. Which is not really a lot of money considering how much use I get out of it. I'm definitely getting cheaper power than a lot of people at $.08/kWh, but it looks like the US average is only about $.12 - are there places where a couple hundred watts is going to be a significant financial burden on the average IT worker?
- eropple 7y ago
  To me it's less a financial burden and more the discomfort of burning electricity just to run a few Docker containers and some databases for personal projects. I expect that once, next summer, we get the house kitted with solar and batteries, I will feel a little more okay about it.
louwrentius 7y ago
I've done a measurement with a power meter and it's even 110W. (it includes ilo).
Aeolun 7y ago
That costs you maybe $.5 dollars a day? Maybe $1 if you are unlucky. This seems fairly reasonable?

stordoff 7y ago

Worth noting: > By default, this server is turned off. When I want to do some work, I turn the server on using wake-on-lan. I do this with all my equipment except for my router/firewall. If it is only on when it is doing work, idle power doesn't matter _too_ much.

dr_zoidberg 7y ago
The Intel ark site says the TDP for that xeon is at 115W, and that's two of them. When you're running at full power (which could happen) you're looking at the CPUs alone drawing over 200W. Another pointer to high power consumption is the 750W PSU. All in all we're not looking at a particularly efficient machine (even if it's unfair comparing it to 2019 hardware).
- dredmorbius 7y ago
  The PSU's rating is for its potential to supply power, whilst draw components (CPUs, disk, etc.) are rated by use. The actual efficiency is defined by capacity, load, and miscellaneous other factors, but should be in the neighbourhood of 20% of delivered power. If the system idles at ~100 watts, the PSU might add another 20-30 to the mix, not 750. https://www.howtogeek.com/129667/what-exactly-does-the-wattage-rating-on-a-power-supply-unit-mean/ https://www.howtogeek.com/129667/what-exactly-does-the-watta...
  - brokenmachine 7y ago
    Agree with all of that, but I think OP was referring to the fact that higher-rated PSUs will use more power than a lower rated PSU when idle.
    - dredmorbius 7y ago
      Running a PSU at ~50% of its rated max is generally recommendable. At 115W * 2, plus other draws (disk, mobos, blinkenlights), that's around 300W. Again: peak draw, when on. I'm not arguing that this system is particularly efficient, only that you don't want to add 750W for the PSU to the draw. I've not specced out low-power systems myself. I doubt you could get 40 hyperthreads running way below this, though there are definitely some low-power systems which might have a total budget below 50W. Reddit's HomeLabPorn may have some more useful guidance: https://old.reddit.com/r/HomeLabPorn/ https://old.reddit.com/r/HomeLabPorn/ (Not my area of expertise, I've never really stayed current in HW. Though I'm aware power/thermal budget has been a major focus, both mobile and server, for the past decade or so.)
      - dr_zoidberg 7y ago
        The 24 cores/48 threads EPYCs that come with 128mb L3 cache have a TDP that's between 155W and 180W. If you want to go with a lower core count (because you have a more modern architecture), you could even go with the 16c/32t EPYC 7282 (64MB L3) that sits at 120W. All in all, you're looking at almost half the power draw (AMD measures it differently than Intel, so aboutish the same) for the same performance. I specifically mention the L3 cache size because, while not being substantial, a large cache can get you 10 to 20% improved performance due less CPU stalling from cache misses. For comparison, the Xeon in question has 25MB L3, so we'd be looking at 50MB cache, split in two dies (so it doesn't quite work as a whole block of 50mb cache). greatjack613 also mentioned the fact that an AMD Zen 3900x matches the multicore performance (and has better single core perf) for a fraction of the TDP, see https://news.ycombinator.com/item?id=20687932 https://news.ycombinator.com/item?id=20687932. All in all that was the general idea: yes, you can get a used server for a good price, but we shouldn't forget the efficiency aspect of it, compared to new hardware.

rhn_mk1 7y ago

Did you do anything particular to get 15W? I have recently measured my AMD AM1 system, which sits at 40W with 8GiB RAM, 2 HDDs and a couple mostly idle VMs on it.

rhinoceraptor 7y ago
I'd check the BIOS to see if there are any power saving options, and/or unclocking/undervolting the CPU. 2.5" HDDs and SSDs use about the same amount of power, but 3.5" HDDs use 2-3x more. So you could potentially save 12 watts if you're using 3.5" drives. Make sure your PSU is a high efficiency one, and isn't too low, or or too high wattage. From what I've read the maximum efficiency is in the 40-80% range.
- scns 7y ago
  Maximum efficiency is at 50%
eropple 7y ago
I do undervolt it and I aggressively spin down the drives. (My concern is mostly storage and redundancy, not performance.)

atonse 7y ago

In addition to wattage, my other concern for a home server was fan noise. If it's anywhere that I am, I really don't want to hear it. So I got these amazing Noctua fans and simply can't hear the server. (Updating language based on replies) A datacenter-like server probably isn't designed for that. So it wouldn't necessarily work out. I have an 8 core i5. I bought Intel (18 months ago) because I didn't want to deal with any AMD incompatibility especially since I wanted to also run it as a gaming machine using VT-d. If I were doing it again, I'd definitely go with Ryzen.

skunkworker 7y ago
I’m not sure what you’re getting at with the home server fans and not working out. I’ve been using a noctua on my Xeon home server without and issues, and reconfigured the fan sensors and control using ipmi. There’s no problem making a quiet Xeon home server.
- snuxoll 7y ago
  I’ve got four servers and two switches in my home office, noise is around 45dBA and with the door closed it’s inaudible. Total power draw idle is around 340W (3.1A measure by my PDU @ 110V).
- atonse 7y ago
  Sorry I wasn't clear. I meant something like a Rack mounted server designed for a datacenter. Would it become whisper quiet if you just replaced the fan? Is it that simple?
  - snuxoll 7y ago
    Replacing the fans on server systems is tricky, many “quiet” fans are such because they have a lower max RPM which will freak the hell out of your management controller. Also, there’s only so much optimization you can do to 40mm fans in a 1U chassis. The fans in my 1U dell servers are fairly quiet when idle around 3600RPM. They make noise, you wouldn’t want to be sleeping or watching TV in the same room - but with the door to my office closed they can’t be heard in the hallway.

wil421 7y ago

I have a UniFi USG, UniFi 150W switch (using maybe 30w), a freeNAS with a i3-8100, a mATX super micro board and 6 4TB WD Red drives. My backup power supply says maybe 120w.

CSDude 7y ago
My USG runs very hot, do you experience it?
- atonse 7y ago
  All my unifi stuff runs hot. The ERLite 3 and switches, etc. I think they count on using the cases as heatsinks.
  - eropple 7y ago
    Same here. Those things get toasty. I accidentally placed a RPi3 (passively cooled but with a metal case) on top of my USG once and it got hot enough not just to throttle it, but also to cause HDMI artifacting.
- wil421 7y ago
  Not really. I have a NanoHD and my father in law has one too. His runs really hot and mine is cool to the touch. Mine has more traffic and was made before his NanoHD.
  - CSDude 7y ago
    I also have a NanoHD and its acceptable but USG (non-pro) runs so hot that I fear leaving it in a non airconditoned cupboard that my internet exit resides
- michaelper22 7y ago
  Yup. My USG and US-8-60W both run hot enough to not want my hand on for very long (the power supplies are pretty warm too).

fnord77 7y ago

would newer hardware idle at even lower wattage than 15W? Or is that about the best one could expect?

rhinoceraptor 7y ago
For a full desktop class chip, I think that's about as low as it can go. You could go lower with something like a NUC, but 15 watts is really good. 15 watts idle would cost something like $1.50 to $2 to run 24/7.
eropple 7y ago
Newer, maybe slightly but not hugely. (It's all still kinda warmed-over Skylake.) You could get cheaper with lower-TDP chips, but there are definitely diminishing returns on the floor.

AdrianB1 7y ago

Probably an error in measuring the power consumption, there is no way to go to 15W, double that or more.

rhinoceraptor 7y ago
It's believable (if only obtained with a lot of BIOS tweaking), I run an older Dell 1u server as my router. It's got an E3-1220 V2 (2012 era, 69 watt TDP), one 2.5" SSD, 8gb of RAM and after a bit of tweaking it sits at around 30 watts.
- eropple 7y ago
  It wasn't a lot of BIOS tweaking, but yes, some undervolting was required. ;) I also hdparm'd it such that my spinning disks park pretty aggressively.

tbyehl 7y ago

Or it's a ton of compute in a single system for just 96W. Average US rates are roughly $1/yr per continuous watt.

egwynn 7y ago

To contextualize this, ~100W will probably cost between $10-$15/mo to run 24/7. EDIT: Assuming a US household paying between $0.10-$0.20 per kwh.

jdnenej 7y ago
Where I am it costs 37c per kWh. It's no wonder almost everyone has solar.

Havoc 7y ago

>idles under 15W Damn...meanwhile my fairly modern Asus laptop idles at 27W. (Probably the GPU...think I forced it to dedicated only)

louwrentius 7y ago

I turn this server off when I'm done experimenting. I turn it on remotely using wake-on-lan.

apexalpha 7y ago
What exactly do you use it for if it isn on 24/7? I bought a used intel Nuc a while back with 15w idle. And it runs 24/7 and does everything I need from downloading to nextcloud to plex, my website, my emails etc... I really wonder what you do with this kind of power.

abstractbarista 7y ago

Some people like me will justify it due to being a huge hobby for them. It takes the place of what other people would spend going out often, traveling, streaming, etc... I've got a rack that pulls ~600W 24/7 but that's across 4 machines, 3 UPS's, >30 HDD's yielding >20TB across multiple zpools, >256GB RAM. And I'm actually using most of that capacity, not just idling away. I do hope to upgrade soon to power-sipping platforms like you mention but currently I'm still on R710/R810 stuff (Westmere Xeon's).

Dayshine 7y ago
I have to ask, using it to do what?
- berbec 7y ago
  Linux ISO plex server.
- abstractbarista 7y ago
  I wish I had something written up to share. But it's honestly a mix of things I love to play with. Several full blockchain nodes (especially uploading to light clients), bittorrent client (seeding tons), plex, home assistant, boinc, tor bridge, grafana, web and email servers, zoneminder CCTV, mandelbulber (clustered rendering of 3d fractals), four different Minecraft servers (some modded), ADS-B plane tracking, weather station and radiation monitoring, and a bunch more. It's really just one big toy for me. I'm using Proxmox for high availability of over 30 VM's (still want to play with containers soon). EDIT: OPNSense firewall says I uploaded over 15TB in the last month. Fortunately Google fiber doesn't care!
  - louwrentius 7y ago
    That's awesome!
louwrentius 7y ago
That is awesome. Do you have something written up about your setup online?

CommieBobDole 7y ago

You know, that raises a question - how expensive does power get, anyway? Last time I posted my home lab setup (couple of old R610s, Cisco small business switch, ASA 5510), I got a lot of pushback on how much power that consumes and how expensive it is to run. I did the math, and I was looking at maybe $20-$30/month depending on load. Which is not really a lot of money considering how much use I get out of it. I'm definitely getting cheaper power than a lot of people at $.08/kWh, but it looks like the US average is only about $.12 - are there places where a couple hundred watts is going to be a significant financial burden on the average IT worker?

eropple 7y ago
To me it's less a financial burden and more the discomfort of burning electricity just to run a few Docker containers and some databases for personal projects. I expect that once, next summer, we get the house kitted with solar and batteries, I will feel a little more okay about it.

louwrentius 7y ago

I've done a measurement with a power meter and it's even 110W. (it includes ilo).

Aeolun 7y ago

That costs you maybe $.5 dollars a day? Maybe $1 if you are unlucky. This seems fairly reasonable?