14 ms·

13th gen

332 points by ryangibb 3y ago

dividedcomet 3y ago
My simple home server setup that runs on almost no electricity is a M2 Mac Mini plugged into a multi-bay drive enclosure. I’ve got like 12TB of storage on it, and it consumes almost no power. It’s also got enough umph to run Plex, gitea, and some personal web services without breaking a sweat. Also, if you’re horrible on time, they’re on sale for $400 all the time and are so easy to setup. Apple even has a remote login tool now, so you can access it headless from a laptop too.
ggm 3y ago
Sort of not surprising how variant divergent chipsets go with power states, and other things. How does he get raidz2 to spin down without busting the raidset? Putting drives into sleep states isn't usually good for in-CPU zfs is it? Is the l2arc doing heavy lifting here? Good comments about ECC memory in the feedback discussion too.
- phil21 3y ago
  I've found ZFS to be extremely forgiving for hard drives having random slow response times. So long as you are getting a response within the OS I/O timeout period, it's simply a matter of blocking I/O until the drives spin up. This can honestly cause a lot of issues on production systems with a drive that wants to half fail vs. outright fail. I believe this is on the order of 30-60s from memory. l2arc likely works quite well for a home NAS setup allowing for the drives to be kept spun down most of the time. Strangely I also built (about 10 years ago now) a home NAS utilizing a bunch of 2.5" 1TB Seagate drives. I would not repeat the experiment as the downsides in performance was simply not worth the space/power savings. Then again, I also built a ZFS pool out of daisy chained USB hubs and 256 (255?) free vendor schwag USB thumb drives. Take any advice with a grain of salt.
  - paulmd 3y ago
    yup. the problem is really with the SMR drives where they can (seemingly) hang for minutes at a time as they flush out the buffer track. ordinary spin-down isn't really a problem, as long as the drives spin up within a reasonable amount of time, ZFS won't drop the disk from the array. ZFS is designed for HDD-based systems after all. actually it works notably kinda poorly for SSDs in general - a lot of the design+tuning decisions were made under the assumptions of HDD-level disk latency and aren't necessarily optimal when you can just go look at the SSD! however, tons and tons of drive spin-up cycles are not good for HDDs. Aggressive idle timeout for power management was famously the problem with the WD Green series (wdidle3.exe lol). Best practice is leave the drives spinning all the time, it's better for the drives and doesn't consume all that much power overall. Or I would certainly think about, say, a 1-hour timeout at least. https://www.truenas.com/community/threads/hacking-wd-greens-and-reds-with-wdidle3-exe.18171/ https://www.truenas.com/community/threads/hacking-wd-greens-... However, block-level striping like ZFS/BTRFS/Storage Spaces is not very good for spinning down anyway. Essentially all files will have to hit all disks, so you have to spin up the whole array. L2ARC with a SSD behind it might be able to serve a lot of these requests, but as soon as any block isn't in cache you will probably be spinning up all the disks very shortly (unless it's literally 1 block). Unraid is better at this since it's a file-level striping - newer releases can even use ZFS as a backend but a file always lives on a single unraid volume, so with 1-disk ZFS pools underneath you will only be spinning up one disk. This can also be used with ZFS ARC/L2ARC or Unraid might have its own setup for tiering hot data on cache drives or hot-data drives. (1-disk ZFS pools as Unraid volumes fits the consumer use-case very nicely imo, and that's going to be my advice for friends and family setting up NASs going forward. If ZFS loses any vdev from the pool the whole pool dies, so you want to add at least 2-disk mirrors if not 4-disk RAIDZ vdevs, but since Unraid works at a file-stripe level (with file mirroring) you just add extra disks and let it manage the file layout (and mirrors/balancing). Also, if you lose a disk, you only lose those files (or mirrors of files) but all the other files remain intact, you don't lose 1/8th of every file or whatever, and that's a failure mode that aligns a lot better with consumer expectations/needs and consumer-level janitoring. And you still retain all the benefits of ZFS in terms of ARC caching, file integrity, etc. It's not without flaws, in the naive case the performance will degrade to 1- or 2-disk read speeds (since 1 file is on 1 disk, with eg 1 mirror copy) and writes will probably be 1-disk speed, and a file or volume/image cannot exceed the size of a single disk and must have sufficient contiguous free space, and snapshots/versioning will consume more data than block-level versioning, etc. All the usual consequences of having 1 file backed by 1 disk will apply. But for "average" use-cases it seems pretty ideal and ZFS is an absolutely rock-stable backend for unraid to throw files into.) anyway it's a little surprising that having a bunch of individual disks gave you problems with ZFS. I run 8x8TB shucked drives (looking to upgrade soon) in RAIDZ2 and I get basically 8x single-disk speed over 10gbe, ZFS amortizes out the performance very nicely. But there are definitely risks/downsides, and power costs, to having a ton of small drives, agreed. Definitely use raidz or mirrors for sure.
louwrentius 3y ago
I have the same amount of storage available in a ~9-year-old 24-bay NAS chassis that does 150 Watt idle (with drives spinning). My NAS is powered down most of the time for this reason, only booted (IPMI) remotely when needed. Although the actual idle power consumption in the article seems to be a tad higher than 7 watts, it's so much lower, it's not such a big deal to run it 24/7 and enjoy the convenience. Loved the write-up!
newsclues 3y ago
I had the same issue of picking a motherboard with limited SATA ports and then having to deal with extra expansion cards. 4 is not enough for homelab type servers.
- fomine3 3y ago
  MiniPCIe or M.2(PCIe) to SATA board may help you http://www.iocrest.com/index.php?id=2349 http://www.iocrest.com/index.php?id=2349
chx 3y ago
Why not the N100? Even an N305 fits the purpose, the N100 would be even less https://www.reddit.com/r/MiniPCs/comments/12fv7fh/beelink_eq12_notes_is_the_ultraefficient_6w_tdp/kdtw9to/ https://www.reddit.com/r/MiniPCs/comments/12fv7fh/beelink_eq...
- cjdell 3y ago
  I'm very impressed with my N100 mini PC (fits in your palm) that I bought from AliExpress. Takes between 2-8W and uses just a plain old 12V plug-style power supply with a DC barrel. Perfect for Home Assistant and light virtualisation. Performance is actually better than my quad core i5-6500 mini PC. Definitely no slouch.
- imglorp 3y ago
  Because author said they wanted a bunch of disks.
- hrdwdmrbl 3y ago
  +1 for the Nx00 series of chips. I just bought myself a pre-built mini pc with an N100. Low power, good price, great performance. I wonder if in a few years they might not eat the whole mini PC market. If the price can come down such that they’re competitive with the various kinds of Pis…
- arp242 3y ago
  N100 wasn't yet released when this was written in May (or was only just released). Also the N100 only supports 16G RAM, and this guy has 64G. Number of pcix lanes (9 vs. 20) probably matter for their use case as well. And the i5 does seem quite a bit faster in general. Comparison: https://ark.intel.com/content/www/us/en/ark/compare.html?productIds=134586,231803 https://ark.intel.com/content/www/us/en/ark/compare.html?pro...
  - s800 3y ago
    I'm running a home server on an N100 (ITX model) with a 32GB DIMM, works well.
MochaDen 3y ago
Low-power is great but running a big RAID long-term without ECC gives me the heebee jeebies! Any good solutions for a similar system but more robust over 5+ years?
- ianai 3y ago
  Agree. Didn’t even see ECC discussed. Apparently this board supports ecc with this chip: Supermicro X13SAE W680 LGA1700 ATX Motherboard Costs 550. One option is building around that and having some pcie 4.0 to nvme boards hosting as many nvme drives as needed. Not cheap though but around home affordable.
  - ThatMedicIsASpy 3y ago
    You need workstation chipsets to have ECC on intel desktop CPUs. And yes they start at around 500.
    - philjohn 3y ago
      If you go back a few generations, the C246 chipset can be had on boards costing 200, and if you pair it with an i3-9100T you get ECC as well as pretty damn low power usage.
      - ianai 3y ago
        You are limited to pcie 3.0 speeds there though. But good suggestion.
        philjohn 3y ago
        That's true, but if your goal is low power, that's not necessarily going to be a bottleneck - even if you dedicate all 16 PCIe lanes to NVMe storage it's going to be more than fast enough for 99% of home server needs.
- faeriechangling 3y ago
Dachande663 3y ago
Is there not an element of penny-wise, pound-foolish here where you end up optimizing the cpu/mono side of things but then run 6+ drives vs fewer larger ones?
- bluGill 3y ago
  You should buy drives in multiples of 5 or 6. Of course this is subject to much debate. Drives fail so you need more than one extra for redundency - I suggest 2 so when (not if!) one fails you still have procuction while repacing the bad drive. 3 drives in a raid-1 mirror for the price of one is spendy so most start looking at raid-5 with dual parity. However putting more than 6 drives in those starts to run into performance issues better handled by striping across two raids. (if you do try more than 6 your odds of 3 drives failing become reasonable so start adding more paricy stripes) which is why I say 6 drives at once is the sweet spot, but others come up with other answers that are not unreasonable. of course one input is how much data do you have. for many 1 modern disk is plenty of space, so you go raid 1 and redundancy is only so you don't need to wait for off site backups to restore after failure.
  - dannyw 3y ago
    It also depends on your use case. If your NAS is write-once-heavy and just used for archiving data, and isn't being used to run a database or something like that; then you can even do a 20 drive zpool3 if you want. That's my setup for nearly 1/3 PB.
    - bluGill 3y ago
      You should replace thet system with off site backups. Local NAS is useful for fast local access which isn't what you get.
jnsaff2 3y ago
I have a 5-node ceph cluster built out of Fujitsu desktops that I got for 50 euro a piece. 4 nodes have 8gb ram and one has 16gb. CPU in each is i5-6500. Each has an NVMe that is split for OS and journal and a spinning HDD. The cluster idles at 75W and full load about 120W. That is intense ceph traffic not other workloads.
- Throw839 3y ago
  That fujitsu part is important. Many mainstream brands do not implement power states correctly, Fujitsu seems to be focused on power consumption quite lot.
  - skippyboxedhero 3y ago
    The NUC and Optiplex aren't bad either. There are also very good AsRock boards (I can't remember what the modern ones are called but H110T is one, I used this for a bit, idled at 6W, laptop memory and power brick). But Fujitsu is the S-tier. In practice, I found I needed a bit more power but you can get some of the Fujitsu boards with a CPU for $30-40, which is hard to beat.
  - ThatMedicIsASpy 3y ago
    I have a HP ProDesk powertop shows up to C10 which I never reach. Must be the SSD or NVMe I have. But yeah The BIOS in those are super cut down and there are hardly any energy settings I can change.
  - paulmd 3y ago
    fujitsu has always been underappreciated in the mainstream tbh. there has always been a thinkpad-style cult following (although much smaller) but japanese companies often do a pretty terrible job at marketing in the west (fujifilm being another fantastic example). my university issued T4220 convertible laptops, with wacom digitizers in the screens. I rarely used it but the pivot in the screen made it indestructible, it survived numerous falls hitting the corner of the screen/etc because the screen simply flops out of the way and pivots to absorb the energy. I later got a ST6012 slate PC that my uni bookstore was clearing out (also with a wacom digitizer, and a Core2Solo ULV!). Both of them are extremely well-thought-out and competently designed/built hardware. Doesn't "feel" thinkpad grade, but it absolutely is underneath, and featured PCMCIA and bay batteries and other power-user features. https://www.notebookcheck.net/Fujitsu-Siemens-Lifebook-T4220.3694.0.html https://www.notebookcheck.net/Fujitsu-Siemens-Lifebook-T4220... https://www.ruggedpcreview.com/3_slates_fujitsu_st6012.html https://www.ruggedpcreview.com/3_slates_fujitsu_st6012.html They also did a ton of HPC stuff for Riken and the other japanese research labs, they did a whole family of SPARC processors for mainframes and HPC stuff, and pivoted into ARM after that wound down. Very cool stuff that receives almost no attention from mainstream tech media, less than POWER even. https://www.youtube.com/watch?v=m0GqCxMmyF4 https://www.youtube.com/watch?v=m0GqCxMmyF4 Anyway back on topic but my personal cheat-code for power is Intel NUCs. Intel, too, paid far more attention to idle power and power-states than the average system-integrator. The NUCs are really really good at idle even considering they're using standalone bricks (my experience is laptop bricks are much less efficient and rarely meet 80+ cert etc). A ton of people use them as building blocks in other cases (like HDPlex H1 or Akasa cases), they don't have a ton of IO normally but they have a SATA and a M.2 and you can use a riser cable on the M.2 slot to attach any pcie card you want. People would do this with skull canyon f.ex (and HDPlex H1 explicitly supports this with the square ones). The "enthusiast" style NUCs often have multiple M.2s or even actual pcie slots and are nice for this. https://www.amazon.com/ADT-Link-Extender-Graphics-Adapter-PCI-Express/dp/B07YDH8KW9 https://www.amazon.com/ADT-Link-Extender-Graphics-Adapter-PC... And don't forget that once you have engineered your way to pcie card formfactor, you can throw a Highpoint Rocket R1104 or a SAS controller card in there and run multiple SSDs (up to 8x NVMe) on a single pcie slot, without bifurcation. Or there are numerous other "cheat code" m.2 devices for breaking the intended limits of your system - GPUs (Innodisk EPV-1101/Asrock M2_GPU), SATA controllers, etc. https://www.youtube.com/watch?v=M9TcL9aY004 https://www.youtube.com/watch?v=M9TcL9aY004 (actually this makes the good point that CFExpress is a thing and is very optimized for power. No idea how durable they are in practice, and they are definitely very expensive, but they also might help in some extreme low power situations.) Personally I never found AMD is that efficient at idle. Even with a monolithic apu you will want to dig up an X300TM-ITX from aliexpress, since this allows you to forgo the chipset. Sadly AMD does not allow X300 to be marketed directly as a standalone product, only as an integrated system like a nuc or laptop or industrial pc, despite the onboard/SOC IO being already quite adequate for beige box usage. Gotta sell those chipsets (hey, how about two chipsets per board!?). But OP article is completely right that AMD’s chipsets just are not very efficient.
jeffbee 3y ago
I wonder if this system is connected with wired ethernet or wifi. I found that it makes a large difference on my NAS. With a wired link the SoC can't reach a deep sleep state because the ethernet peripheral demands low-latency wakeup from the PCIe root port. This is power management policy that is flowing from the link peer all the way to your CPU! I found that wifi doesn't have this problem, and gives better-than-gigabit performance, sometimes.
- skippyboxedhero 3y ago
  If you have an network card over PCIe then there may be an issue with the card. I have never had an issue reaching low sleep state, you can modify WoL behaviour too. Wifi is, again in my experience, uses significantly more power. I have seen 3-5W and usually switch if off.
  - jeffbee 3y ago
    I don't think it's an issue with the card. It's a combination of ethernet and PCIe features that make this happen. There is a standard called "energy efficient ethernet" that makes it not happen, but my switch doesn't do it.
jauntywundrkind 3y ago
I feel like I see a good number of nas builds go by, but rarely are they anywhere as technical. Nice.
squarefoot 3y ago
For those interested in repurposing a small mini-PC with no Mini PCI ports available as NAS, I recently purchased a ICY IB-3780-C31 enclosure (USB3.1 to 8xSATA), and although I still have to put it in operation (will order new disks soon), I tested it with a pair from my spares and can confirm it works out of the box with both Linux and XigmaNAS (FreeBSD). Just beware that although it can turn back on after the connected PC goes to sleep and then wakes up (the oddly named "sync" button on front panel does that), it doesn't after a accidental loss of power or power outage even if the connected PC is set up to boot automatically, which to me is quite bad, therefore having recently moved to a place where power outages aren't uncommon and can last longer than a normal UPS could handle, I'll probably modify the enclosure by adding a switchable monostable circuit that emulates a short press of the power button after power is restored. That would mean a big goodbye to warranty, so I'll have to think about it, but the problem can indeed be solved.
ThatMedicIsASpy 3y ago
7950X3D, X670E Taichi, 96GB 6400MHz CL32, 2x4TB Lexar, 4x18TB Seagate Exos X18, RX570 8G, Proxmox. Idle no VM ~60-70W. Idle TrueNAS VM drives spinning ~90-100W. Idle TrueNAS & Fedora Desktop with GPU passthrough ~150W In a few weeks the 570 is replaced by 7900 xtx. The RAM adds a lot of W. 3-5W per 8GB of RAM depending on the frequency is common for DDR5. I was expecting around 50-100W for Proxmox+TrueNAS. I did not consider the power draw of the RAM when I went for 96GB.
- eurekin 3y ago
  What about networking? Did you go over 1gbit?
  - ThatMedicIsASpy 3y ago
    It has 2.5G. There are X670E with 10G if you desire more. My home net is 1G with two MikroTik hAP ax3 which are connected with the single 2.5G poe port they have (and one powers the other).
- hypercube33 3y ago
  I really want the Ryzen Embedded and or Epyc 3000(?) series that has dual 10gbe on package for something like a NAS but both are super expensive or impossible to find.
  - ThatMedicIsASpy 3y ago
    AsRock Rack B650D4U-2L2T/BCM, 2x10G, 2x1G, IPMI For less power consumption Ryzen 8000 is coming up (wait for Jan 8th, CES) and the APU tend to be monolithic and draw a lot less power than the chiplets.
    - tw04 3y ago
      Even that uses Broadcom 10gbe, not the embedded AMD Ethernet. It’s really strange, I can only assume there’s something fatally wrong with the AMD Ethernet.
1letterunixname 3y ago
I feel like an petrochem refinery with my 44 spinning rust units NAS 847E16-RJBOD, 48 port POE+ 10 GbE switch, 2 lights-out and environmental monitoring UPSes, and DECISO OPNsense router using a combined average of 1264W. ]: One UPS is at least reporting a power factor with an efficiency of 98%, while the other one isn't as great at 91%. APM is disabled on all HDDs because it just leads to delay and wear for mythological power savings that isn't going to happen in this setup. Note that SMART rarely/never predicts failures, but one of the strongest signals of drive failures is slightly elevated temperatures (usually as a result of bearing wear). This creates enough waste heat such that one room never needs heating, but cooling isn't strictly needed either because there's no point to reducing datacenter ambient below 27 C.
- syntheticnature 3y ago
  I was looking into water heaters that use heat pumps recently, and a lot of them function by sucking heat out of the room. While water and computers don't mix, might be an even better use for all that waste heat...
  - sgarland 3y ago
    My plan for when I build a house (so far with two moves during my serious homelab period, the room layout hasn’t worked) is to build a small server room that vents out to the heat pump water heater.
Palomides 3y ago
amusing to read this very detailed article and not have any idea what OP actually does with 72TB of online storage 1gbe seems a bit anemic for a NAS
- dannyw 3y ago
  it's usually p0rn
  - jmnicolas 3y ago
    Try to be charitable, I'm sure there are a couple Linux ISO in there.
alphabettsy 3y ago
Very cool write up. Good timing too as I find myself attempting to reduce the power consumption of my homelab.
vermaden 3y ago
Good read. Tried something similar in the past: - https://vermaden.wordpress.com/2019/04/03/silent-fanless-freebsd-server-redundant-backup/ https://vermaden.wordpress.com/2019/04/03/silent-fanless-fre... - https://vermaden.wordpress.com/2023/04/10/silent-fanless-dell-wyse-3030-lt-freebsd-server/ https://vermaden.wordpress.com/2023/04/10/silent-fanless-del...
uxp8u61q 3y ago
I know nothing about building NASs so maybe my question has an obvious answer. But my impression is that most x64 CPUs are thoroughly beaten by Arm or RISC-V CPUs when it comes to power consumption. Is there a specific need for the x64 architecture here? I couldn't find an answer in TFA.
- hmottestad 3y ago
  You can very easily run docker containers on it. That’s why I went with a ryzen chip in mine. You could always use an rpi if you want to go with ARM, and you’ll want something with ARMv8.
- adrian_b 3y ago
  Most Arm or RISC-V CPUs (with the exception of a few server-oriented models that are much more expensive than x86) have very few PCIe lanes and SATA ports, so you cannot make a high throughput NAS with any of them. There are some NAS models with Arm-based CPUs and multiple SSDs/HDDs, but those have a very low throughput due to using e.g. only one PCIe lane per socket, with at most PCIe 3 speed.
- arp242 3y ago
  > my impression is that most x64 CPUs are thoroughly beaten by Arm or RISC-V CPUs when it comes to power consumption Not really. ARM (and to lesser degree, RISC-V) are often used and optimized for low-power usage and/or low-heat. x64 is often more optimized for maximum performance, at the expense of higher power usage and more heat. For many x64 CPUs you can drastically reduce the power usage if you underclock the CPU just a little bit (~10% slower), especially desktop CPUs but also laptops. There are ARM and RISC-V CPUs that consume much less power, but they're also much slower and have a much more limited feature-set. You do need to compare like to like, and when you do the power usage differences are usually small to non-existent in modern CPUs. ARM today is no longer the ARM that Wilson et al. design 40 years ago. And for something connected to mains, even doubling the efficiency and going from 7W to 3.5W doesn't really make all that much difference. It's just not a big impact on your energy bill or climate change.
- pmontra 3y ago
sandreas 3y ago
There is a german forum thread with a google docs document listing different configurations below 30W[1]. Since there are very different requirements, this might be interesting for many homeserver / NAS builders. For me personally I found my ideal price-performance config to be the following hardware: Board: Fujitsu D3417-B2 CPU: Intel Xeon 1225 V5 (better the also compatible 1275v6, but its way more expensive) RAM: 64GB ECC RAM (4x16GB) SSD: WD SN850x 2TB (consumer SSD) Case: Fractal Design Define Mini C Cooling: Big block no name, passively cooled by case fan Power: Pico PSU 120W + 120W Leicke power supply Remote Administration via Intel AMT + MeshCommander using a DP Dummy Plug I bought this config used VERY CHEAP and I am running Proxmox - it draws 9.3W idle (without HDDs). There are 6 SATA ports and a PCIe port, if anyone would like to add more space or passthrough a dedicated GPU. It may be hard to get, but I paid €380,00 in total. Does not work very well for Media Encoding, here you should go for a Core i3 8100 or above. Alternatively you could go for the following changes, but these might be even harder to get for a reasonable price: Boards: GIGABYTE C246N-WU2 (ITX), Gigabyte C246-WU4 (mATX), Fujitsu D3517-B (mATX), Fujitsu D3644 (mATX) Power: Corsair RM550x (2021 Version) Cheap used Workstations that are good servers are Dell T30 or Fujitsu Celsius W550. The Fujitsu ones have D3417(-A!) boards (not -B) having proprietary power supplies with 16 power pins (no 24pin ATX but 16pin). There are Adapters on Aliexpress for 24PIN to 16pin (Bojiadafast), but this is a bit risky - I'm validating that atm. Ryzen possibilities are pretty rare, but there are reports that the AMD Ryzen 5 PRO 4650G with a Asus PRIME B550M-A Board is drawing about 16W Idle. Hope I could help :-) [1]: https://goo.gl/z8nt3A https://goo.gl/z8nt3A
- manmal 3y ago
  For anybody reading this - I think it’s a great config, but would be careful around pico PSUs in case you want to run a bunch of good old spinning disks. HDDs have a sharp power peak when spinning up, and if you have a couple of them in a RAID, they might do so synchronously, potentially exceeding the envelope.
  - LargoLasskhyfv 3y ago
    In times long gone there's been a setting. Staggered spin-up it was called. Turned on one disk after another. So the PSU didn't bother, or stalled. Done.
ulnarkressty 3y ago
As exciting as it is to design a low power system, it's kind of pointless in the case of a NAS that uses spinning rust as storage media - as the author later writes, the HDD power consumption dwarfs the other system components. If one uses SSD or M.2 drives, there are some solutions on the market that provide high speed hardware RAID in a separate external enclosure. Coupled with a laptop board they could make for a decent low power system. Not sure how reliable USB or Thunderbolt is compared to internal SATA or PCIe connections though... would be interesting to find out.
- V__ 3y ago
  Don't they stop spinning when idle?
  - layer8 3y ago
    Not by default, but you can have the OS have them spin down after a certain idle period. Doing that too frequently can affect the life time of the drive though. You save maybe 4 Watts per drive by spinning them down.
    - Dalewyn 3y ago
      Personally, I make sure my HDDs (regardless use case) never spin down when idle. The cost in life span isn't worth the electricity bill saved.
      - Arn_Thor 3y ago
        Same. Also I’ve had some software error out as it waits for drives to spin up (in an NFS share context).
  - orthoxerox 3y ago
    They are never idle if the NAS is seeding torrents.
- nabla9 3y ago
  You can shut down HDD's when you don't use them. sudo hdparm -Y /dev/sdX
homero 3y ago
Crucial Force GT supposed to say Corsair
treprinum 3y ago
My NAS has Pentium J 4-core and is way under 7W idle, inside some small Fractal case with 6x20TB HDD. Why would you need 12th/13th gen for file transfers?
- wffurr 3y ago
  For encoding maybe? OP says “reasonable CPU performance for compression” and also it was a CPU they already had from a desktop build.
- dbeley 3y ago
  Interesting I assume it's with all drives off, how many Watts with some disk usage?
  - treprinum 3y ago
    Each drive adds around 7W non-idle.
jepler 3y ago
Author seems to have built 5 systems from 2016 to 2023, or around every other year. Some parts (e.g., RAM) are re-used across multiple builds It's interesting to wonder: How much $$ is the hardware cost vs the lifetime energy costs? Is a more power-hungry machine that would operate for 4 years better than one that would operate for 2 years? The motherboard + CPU is USD 322 right now on pcpartpicker. At USD 0.25/kWh (well above my local rate but below the highest rates in the US), 36W continuous over 4 years is also about $315. So, a ~43W, 4-year system might well be cheaper to buy and operate than a 7W, 2-year system.
- hrdwdmrbl 3y ago
  Fun tinkering like that is not about saving money. It's about the journey, not the destination.
  - 1-6 3y ago
    At least the author documents their journey well. If many people who read this save energy by learning good fundamentals of system build, that alone goes a long way. CPU/mobo makers are also probably taking notes.
- buro9 3y ago
  I average a fully loaded NAS every 8 years apparently. I don't really expand their storage, as by this time the network and software is out of date, so I buy a new one and make the current the backup to the new, and then dispose of the one that was the old backup. It's interesting, the upgrade interval has held for 20 years already and so it becomes easy to understand the amortisation, 16 years of use, but really only 8 as the primary and 8 as the backup.
CommanderData 3y ago
Great hardware but when the software is a job to administer I have a hard time justifying builds like these. My Synology NAS for example has 8 GB RAM and a J4150 processor. Runs about 15 containers, Wireguard and on top DSM (which is Synology's OS). Usually idles around 1-3%. Software makes all the difference - DSM has been by far the biggest benefit and surprise to me and I would be deprived of time with anything else. I'm running TrueNas as a second backup server but it no way compares to DSM. Sometimes I don't want to trawl through logs and trial and error just to get a basic CRON setup to backup a file off another server, there's countless examples where DSM has just worked. I really think Synology is missing a trick here, they clearly have software that is miles ahead of everything else and customizable should you need to. They should be more like Microsoft of the NAS world, making DSM run on non Synology platforms, or at least making it easier to do yourself. It's a great OS and it sells itself and can easily be a way to up sell stuff like Active backup for business.
- matthewfcarlson 3y ago
  I suspect focusing on a set of hardware and making it work really well is part of what makes the OS so good. Hardware configs explode in a combinatorial fashion, making it impossible to test everything once you have more than a few options.
- thelittleone 3y ago
  Agree, its a super nice UI and UX. First time I used them was around 2010 and it was already a joy to use. Would be kind of nice to have a Synology DSM front end to cloud IaaS.
- agumonkey 3y ago
  dsm being diskstation manager ?
  - CommanderData 3y ago
    Yes, it's the OS that ships with their hardware. DSM 7 being their latest.
- 3y ago
neilv 3y ago
Great work, for lots of storage. If you can fit your storage on an SSD (or RAID-mirrored pair), and don't need much compute, you can do a low-power server (like to run little services for yourself) using an SBC like a RasPi, or something like a NUC. Personally, I currently have a couple 1U Atom servers that run fanless except for the Noctua fans that I swapped into the PSUs. Advantages over RasPi include SATA and ECC RAM, and were also easier to buy over Covid. (I also have a 4U GPU server, which is currently off when not in use, because I haven't invested in figuring out how to low-power idle it like the article writer has.)
mattbillenstein 3y ago
Everyone's needs are different, but over time with the loss of a couple of disks, I started to hate running RAID5 or 6 with HDDs. It became an exercise in how fast could I replace a disk before the next one died and would the rebuild actually work or not - although, it always did. Also, the hot-swap case/cage I had with all the SATA connectors and power connectors seemed kinda flaky - it was very cheap. So a couple years ago, I downsized to a single 2TB SSD in a smallish ATX case - and another in a completely different machine that gets rsync'd every 4 hours. My nas is now just my last desktop's hardware with two SSDs - one boot, one larger storage running plex, smbd, backups of misc stuff on cron, duplicity backup to the cloud, etc on Ubuntu. If I didn't have this extra hardware, I'd probably just run two medium powered NUCs with a nvme boot disk and a bigger SSD for storage. It's all very very simple and I have it setup so I can run some LXC containers should I want to do some dev work there, but I usually have other hardware for that anyway.
- at_a_remove 3y ago
  I am imagining a NAS appliance designed for high availability and low-touch, like a RAID 10 box with a ton of hot spares, designed to automatically rebuild in case of failure. The hot spares are spun up once a month to prevent stiction.
- nine_k 3y ago
  Good thing that 2TB suffices for you! I think RAID NAS boxes seem to make sense at sizes several times this.
- CPLX 3y ago
  I have done a bunch of stuff in the past but most recently I just bought two synology arrays and followed the instructions to set them up in less than an hour and they just worked with literally zero hassle.
vardump 3y ago
What would be the cheapest and lowest power option to get 100G networking (50-100G actual performance is good enough) and at least one 8 lane PCI-e GPU? Plus at least 2x M.2.
deleted 3y ago
[deleted]
pomatic 3y ago
7 watts idle is unimpressive out of context If you have a NAS that is idle most of the time, and want to minimise power consumption, how about an embedded-cpu based WoL generator? Sniff packets destined for the fileserver, which is otherwise in deep sleep, and automagically wake it up when relevant traffic is detected, with the relevant WoL packet. You'd get say <300mA consumption at 3v3, and full performance on wake. It would be transparent to the machines attempting to access the server. If idle power measurement is your criteria, this might be a way forwards?
- getwiththeprog 3y ago
  Cool idea. I have no idea how to automagically do that.
  - graphe 3y ago
    PoE or power over Ethernet.
    - allset_ 3y ago
      You don't need PoE to do WoL.
Paradigma11 3y ago
How viable would it be to run a home server in a VM on your main PC that is occasionally also used for high performance activities like gaming. I do have 12 cores and 64GB Ram and I would not mind parting with 4C/16GB and maybe the onboard GPU (video decoding) but I dont know how the rest of the system might be influenced if there is heavy IO or other stuff.
- marcosscriven 3y ago
  Proxmox is good for this. You can have a Windows VM with GPU passthrough. Then you can add any other VMs or LXCs to your heart’s content.
jmnicolas 3y ago
I have a 4 Cores Intel i3 13100 and I can't reach these low numbers. The machine idle with just 1 nvme is 16 watts. I managed to go to 12 watts with the powersaver mode in tuned but I have random ssh freezes and I didn't find any relevant doc to solve the problem. But as soon as the machine is in light use, just a network transfer for example the power usage go through the roof: 30 watts. Now I have loaded it with 2 internal 3.5 hard drives, I have a VM with Docker running, a torrent client and Syncthing (so nothing terribly intensive). The thing never goes below 40 watts and is currently at 60 (I'm copying data onto the drives). I would never have bought a desktop class CPU if it wasn't for these articles that promise unreachable low power draw.

332 points by ryangibb 3y ago

My simple home server setup that runs on almost no electricity is a M2 Mac Mini plugged into a multi-bay drive enclosure. I’ve got like 12TB of storage on it, and it consumes almost no power. It’s also got enough umph to run Plex, gitea, and some personal web services without breaking a sweat. Also, if you’re horrible on time, they’re on sale for $400 all the time and are so easy to setup. Apple even has a remote login tool now, so you can access it headless from a laptop too.

ggm 3y ago

Sort of not surprising how variant divergent chipsets go with power states, and other things. How does he get raidz2 to spin down without busting the raidset? Putting drives into sleep states isn't usually good for in-CPU zfs is it? Is the l2arc doing heavy lifting here? Good comments about ECC memory in the feedback discussion too.

phil21 3y ago
I've found ZFS to be extremely forgiving for hard drives having random slow response times. So long as you are getting a response within the OS I/O timeout period, it's simply a matter of blocking I/O until the drives spin up. This can honestly cause a lot of issues on production systems with a drive that wants to half fail vs. outright fail. I believe this is on the order of 30-60s from memory. l2arc likely works quite well for a home NAS setup allowing for the drives to be kept spun down most of the time. Strangely I also built (about 10 years ago now) a home NAS utilizing a bunch of 2.5" 1TB Seagate drives. I would not repeat the experiment as the downsides in performance was simply not worth the space/power savings. Then again, I also built a ZFS pool out of daisy chained USB hubs and 256 (255?) free vendor schwag USB thumb drives. Take any advice with a grain of salt.
- paulmd 3y ago
  yup. the problem is really with the SMR drives where they can (seemingly) hang for minutes at a time as they flush out the buffer track. ordinary spin-down isn't really a problem, as long as the drives spin up within a reasonable amount of time, ZFS won't drop the disk from the array. ZFS is designed for HDD-based systems after all. actually it works notably kinda poorly for SSDs in general - a lot of the design+tuning decisions were made under the assumptions of HDD-level disk latency and aren't necessarily optimal when you can just go look at the SSD! however, tons and tons of drive spin-up cycles are not good for HDDs. Aggressive idle timeout for power management was famously the problem with the WD Green series (wdidle3.exe lol). Best practice is leave the drives spinning all the time, it's better for the drives and doesn't consume all that much power overall. Or I would certainly think about, say, a 1-hour timeout at least. https://www.truenas.com/community/threads/hacking-wd-greens-and-reds-with-wdidle3-exe.18171/ https://www.truenas.com/community/threads/hacking-wd-greens-... However, block-level striping like ZFS/BTRFS/Storage Spaces is not very good for spinning down anyway. Essentially all files will have to hit all disks, so you have to spin up the whole array. L2ARC with a SSD behind it might be able to serve a lot of these requests, but as soon as any block isn't in cache you will probably be spinning up all the disks very shortly (unless it's literally 1 block). Unraid is better at this since it's a file-level striping - newer releases can even use ZFS as a backend but a file always lives on a single unraid volume, so with 1-disk ZFS pools underneath you will only be spinning up one disk. This can also be used with ZFS ARC/L2ARC or Unraid might have its own setup for tiering hot data on cache drives or hot-data drives. (1-disk ZFS pools as Unraid volumes fits the consumer use-case very nicely imo, and that's going to be my advice for friends and family setting up NASs going forward. If ZFS loses any vdev from the pool the whole pool dies, so you want to add at least 2-disk mirrors if not 4-disk RAIDZ vdevs, but since Unraid works at a file-stripe level (with file mirroring) you just add extra disks and let it manage the file layout (and mirrors/balancing). Also, if you lose a disk, you only lose those files (or mirrors of files) but all the other files remain intact, you don't lose 1/8th of every file or whatever, and that's a failure mode that aligns a lot better with consumer expectations/needs and consumer-level janitoring. And you still retain all the benefits of ZFS in terms of ARC caching, file integrity, etc. It's not without flaws, in the naive case the performance will degrade to 1- or 2-disk read speeds (since 1 file is on 1 disk, with eg 1 mirror copy) and writes will probably be 1-disk speed, and a file or volume/image cannot exceed the size of a single disk and must have sufficient contiguous free space, and snapshots/versioning will consume more data than block-level versioning, etc. All the usual consequences of having 1 file backed by 1 disk will apply. But for "average" use-cases it seems pretty ideal and ZFS is an absolutely rock-stable backend for unraid to throw files into.) anyway it's a little surprising that having a bunch of individual disks gave you problems with ZFS. I run 8x8TB shucked drives (looking to upgrade soon) in RAIDZ2 and I get basically 8x single-disk speed over 10gbe, ZFS amortizes out the performance very nicely. But there are definitely risks/downsides, and power costs, to having a ton of small drives, agreed. Definitely use raidz or mirrors for sure.

louwrentius 3y ago

I have the same amount of storage available in a ~9-year-old 24-bay NAS chassis that does 150 Watt idle (with drives spinning). My NAS is powered down most of the time for this reason, only booted (IPMI) remotely when needed. Although the actual idle power consumption in the article seems to be a tad higher than 7 watts, it's so much lower, it's not such a big deal to run it 24/7 and enjoy the convenience. Loved the write-up!

newsclues 3y ago

I had the same issue of picking a motherboard with limited SATA ports and then having to deal with extra expansion cards. 4 is not enough for homelab type servers.

fomine3 3y ago
MiniPCIe or M.2(PCIe) to SATA board may help you http://www.iocrest.com/index.php?id=2349 http://www.iocrest.com/index.php?id=2349

chx 3y ago

Why not the N100? Even an N305 fits the purpose, the N100 would be even less https://www.reddit.com/r/MiniPCs/comments/12fv7fh/beelink_eq12_notes_is_the_ultraefficient_6w_tdp/kdtw9to/ https://www.reddit.com/r/MiniPCs/comments/12fv7fh/beelink_eq...

cjdell 3y ago
I'm very impressed with my N100 mini PC (fits in your palm) that I bought from AliExpress. Takes between 2-8W and uses just a plain old 12V plug-style power supply with a DC barrel. Perfect for Home Assistant and light virtualisation. Performance is actually better than my quad core i5-6500 mini PC. Definitely no slouch.
imglorp 3y ago
Because author said they wanted a bunch of disks.
hrdwdmrbl 3y ago
+1 for the Nx00 series of chips. I just bought myself a pre-built mini pc with an N100. Low power, good price, great performance. I wonder if in a few years they might not eat the whole mini PC market. If the price can come down such that they’re competitive with the various kinds of Pis…
arp242 3y ago
N100 wasn't yet released when this was written in May (or was only just released). Also the N100 only supports 16G RAM, and this guy has 64G. Number of pcix lanes (9 vs. 20) probably matter for their use case as well. And the i5 does seem quite a bit faster in general. Comparison: https://ark.intel.com/content/www/us/en/ark/compare.html?productIds=134586,231803 https://ark.intel.com/content/www/us/en/ark/compare.html?pro...
- s800 3y ago
  I'm running a home server on an N100 (ITX model) with a 32GB DIMM, works well.

MochaDen 3y ago

Low-power is great but running a big RAID long-term without ECC gives me the heebee jeebies! Any good solutions for a similar system but more robust over 5+ years?

ianai 3y ago
Agree. Didn’t even see ECC discussed. Apparently this board supports ecc with this chip: Supermicro X13SAE W680 LGA1700 ATX Motherboard Costs 550. One option is building around that and having some pcie 4.0 to nvme boards hosting as many nvme drives as needed. Not cheap though but around home affordable.
- ThatMedicIsASpy 3y ago
  You need workstation chipsets to have ECC on intel desktop CPUs. And yes they start at around 500.
  - philjohn 3y ago
    If you go back a few generations, the C246 chipset can be had on boards costing 200, and if you pair it with an i3-9100T you get ECC as well as pretty damn low power usage.
    - ianai 3y ago
      You are limited to pcie 3.0 speeds there though. But good suggestion.
      - philjohn 3y ago
        That's true, but if your goal is low power, that's not necessarily going to be a bottleneck - even if you dedicate all 16 PCIe lanes to NVMe storage it's going to be more than fast enough for 99% of home server needs.
faeriechangling 3y ago

Dachande663 3y ago

Is there not an element of penny-wise, pound-foolish here where you end up optimizing the cpu/mono side of things but then run 6+ drives vs fewer larger ones?

bluGill 3y ago
You should buy drives in multiples of 5 or 6. Of course this is subject to much debate. Drives fail so you need more than one extra for redundency - I suggest 2 so when (not if!) one fails you still have procuction while repacing the bad drive. 3 drives in a raid-1 mirror for the price of one is spendy so most start looking at raid-5 with dual parity. However putting more than 6 drives in those starts to run into performance issues better handled by striping across two raids. (if you do try more than 6 your odds of 3 drives failing become reasonable so start adding more paricy stripes) which is why I say 6 drives at once is the sweet spot, but others come up with other answers that are not unreasonable. of course one input is how much data do you have. for many 1 modern disk is plenty of space, so you go raid 1 and redundancy is only so you don't need to wait for off site backups to restore after failure.
- dannyw 3y ago
  It also depends on your use case. If your NAS is write-once-heavy and just used for archiving data, and isn't being used to run a database or something like that; then you can even do a 20 drive zpool3 if you want. That's my setup for nearly 1/3 PB.
  - bluGill 3y ago
    You should replace thet system with off site backups. Local NAS is useful for fast local access which isn't what you get.

jnsaff2 3y ago

I have a 5-node ceph cluster built out of Fujitsu desktops that I got for 50 euro a piece. 4 nodes have 8gb ram and one has 16gb. CPU in each is i5-6500. Each has an NVMe that is split for OS and journal and a spinning HDD. The cluster idles at 75W and full load about 120W. That is intense ceph traffic not other workloads.

Throw839 3y ago
That fujitsu part is important. Many mainstream brands do not implement power states correctly, Fujitsu seems to be focused on power consumption quite lot.
- skippyboxedhero 3y ago
  The NUC and Optiplex aren't bad either. There are also very good AsRock boards (I can't remember what the modern ones are called but H110T is one, I used this for a bit, idled at 6W, laptop memory and power brick). But Fujitsu is the S-tier. In practice, I found I needed a bit more power but you can get some of the Fujitsu boards with a CPU for $30-40, which is hard to beat.
- ThatMedicIsASpy 3y ago
  I have a HP ProDesk powertop shows up to C10 which I never reach. Must be the SSD or NVMe I have. But yeah The BIOS in those are super cut down and there are hardly any energy settings I can change.
- paulmd 3y ago
  fujitsu has always been underappreciated in the mainstream tbh. there has always been a thinkpad-style cult following (although much smaller) but japanese companies often do a pretty terrible job at marketing in the west (fujifilm being another fantastic example). my university issued T4220 convertible laptops, with wacom digitizers in the screens. I rarely used it but the pivot in the screen made it indestructible, it survived numerous falls hitting the corner of the screen/etc because the screen simply flops out of the way and pivots to absorb the energy. I later got a ST6012 slate PC that my uni bookstore was clearing out (also with a wacom digitizer, and a Core2Solo ULV!). Both of them are extremely well-thought-out and competently designed/built hardware. Doesn't "feel" thinkpad grade, but it absolutely is underneath, and featured PCMCIA and bay batteries and other power-user features. https://www.notebookcheck.net/Fujitsu-Siemens-Lifebook-T4220.3694.0.html https://www.notebookcheck.net/Fujitsu-Siemens-Lifebook-T4220... https://www.ruggedpcreview.com/3_slates_fujitsu_st6012.html https://www.ruggedpcreview.com/3_slates_fujitsu_st6012.html They also did a ton of HPC stuff for Riken and the other japanese research labs, they did a whole family of SPARC processors for mainframes and HPC stuff, and pivoted into ARM after that wound down. Very cool stuff that receives almost no attention from mainstream tech media, less than POWER even. https://www.youtube.com/watch?v=m0GqCxMmyF4 https://www.youtube.com/watch?v=m0GqCxMmyF4 Anyway back on topic but my personal cheat-code for power is Intel NUCs. Intel, too, paid far more attention to idle power and power-states than the average system-integrator. The NUCs are really really good at idle even considering they're using standalone bricks (my experience is laptop bricks are much less efficient and rarely meet 80+ cert etc). A ton of people use them as building blocks in other cases (like HDPlex H1 or Akasa cases), they don't have a ton of IO normally but they have a SATA and a M.2 and you can use a riser cable on the M.2 slot to attach any pcie card you want. People would do this with skull canyon f.ex (and HDPlex H1 explicitly supports this with the square ones). The "enthusiast" style NUCs often have multiple M.2s or even actual pcie slots and are nice for this. https://www.amazon.com/ADT-Link-Extender-Graphics-Adapter-PCI-Express/dp/B07YDH8KW9 https://www.amazon.com/ADT-Link-Extender-Graphics-Adapter-PC... And don't forget that once you have engineered your way to pcie card formfactor, you can throw a Highpoint Rocket R1104 or a SAS controller card in there and run multiple SSDs (up to 8x NVMe) on a single pcie slot, without bifurcation. Or there are numerous other "cheat code" m.2 devices for breaking the intended limits of your system - GPUs (Innodisk EPV-1101/Asrock M2_GPU), SATA controllers, etc. https://www.youtube.com/watch?v=M9TcL9aY004 https://www.youtube.com/watch?v=M9TcL9aY004 (actually this makes the good point that CFExpress is a thing and is very optimized for power. No idea how durable they are in practice, and they are definitely very expensive, but they also might help in some extreme low power situations.) Personally I never found AMD is that efficient at idle. Even with a monolithic apu you will want to dig up an X300TM-ITX from aliexpress, since this allows you to forgo the chipset. Sadly AMD does not allow X300 to be marketed directly as a standalone product, only as an integrated system like a nuc or laptop or industrial pc, despite the onboard/SOC IO being already quite adequate for beige box usage. Gotta sell those chipsets (hey, how about two chipsets per board!?). But OP article is completely right that AMD’s chipsets just are not very efficient.

jeffbee 3y ago

I wonder if this system is connected with wired ethernet or wifi. I found that it makes a large difference on my NAS. With a wired link the SoC can't reach a deep sleep state because the ethernet peripheral demands low-latency wakeup from the PCIe root port. This is power management policy that is flowing from the link peer all the way to your CPU! I found that wifi doesn't have this problem, and gives better-than-gigabit performance, sometimes.

skippyboxedhero 3y ago
If you have an network card over PCIe then there may be an issue with the card. I have never had an issue reaching low sleep state, you can modify WoL behaviour too. Wifi is, again in my experience, uses significantly more power. I have seen 3-5W and usually switch if off.
- jeffbee 3y ago
  I don't think it's an issue with the card. It's a combination of ethernet and PCIe features that make this happen. There is a standard called "energy efficient ethernet" that makes it not happen, but my switch doesn't do it.

jauntywundrkind 3y ago

I feel like I see a good number of nas builds go by, but rarely are they anywhere as technical. Nice.

squarefoot 3y ago

For those interested in repurposing a small mini-PC with no Mini PCI ports available as NAS, I recently purchased a ICY IB-3780-C31 enclosure (USB3.1 to 8xSATA), and although I still have to put it in operation (will order new disks soon), I tested it with a pair from my spares and can confirm it works out of the box with both Linux and XigmaNAS (FreeBSD). Just beware that although it can turn back on after the connected PC goes to sleep and then wakes up (the oddly named "sync" button on front panel does that), it doesn't after a accidental loss of power or power outage even if the connected PC is set up to boot automatically, which to me is quite bad, therefore having recently moved to a place where power outages aren't uncommon and can last longer than a normal UPS could handle, I'll probably modify the enclosure by adding a switchable monostable circuit that emulates a short press of the power button after power is restored. That would mean a big goodbye to warranty, so I'll have to think about it, but the problem can indeed be solved.

ThatMedicIsASpy 3y ago

7950X3D, X670E Taichi, 96GB 6400MHz CL32, 2x4TB Lexar, 4x18TB Seagate Exos X18, RX570 8G, Proxmox. Idle no VM ~60-70W. Idle TrueNAS VM drives spinning ~90-100W. Idle TrueNAS & Fedora Desktop with GPU passthrough ~150W In a few weeks the 570 is replaced by 7900 xtx. The RAM adds a lot of W. 3-5W per 8GB of RAM depending on the frequency is common for DDR5. I was expecting around 50-100W for Proxmox+TrueNAS. I did not consider the power draw of the RAM when I went for 96GB.

eurekin 3y ago
What about networking? Did you go over 1gbit?
- ThatMedicIsASpy 3y ago
  It has 2.5G. There are X670E with 10G if you desire more. My home net is 1G with two MikroTik hAP ax3 which are connected with the single 2.5G poe port they have (and one powers the other).
hypercube33 3y ago
I really want the Ryzen Embedded and or Epyc 3000(?) series that has dual 10gbe on package for something like a NAS but both are super expensive or impossible to find.
- ThatMedicIsASpy 3y ago
  AsRock Rack B650D4U-2L2T/BCM, 2x10G, 2x1G, IPMI For less power consumption Ryzen 8000 is coming up (wait for Jan 8th, CES) and the APU tend to be monolithic and draw a lot less power than the chiplets.
  - tw04 3y ago
    Even that uses Broadcom 10gbe, not the embedded AMD Ethernet. It’s really strange, I can only assume there’s something fatally wrong with the AMD Ethernet.

1letterunixname 3y ago

I feel like an petrochem refinery with my 44 spinning rust units NAS 847E16-RJBOD, 48 port POE+ 10 GbE switch, 2 lights-out and environmental monitoring UPSes, and DECISO OPNsense router using a combined average of 1264W. ]: One UPS is at least reporting a power factor with an efficiency of 98%, while the other one isn't as great at 91%. APM is disabled on all HDDs because it just leads to delay and wear for mythological power savings that isn't going to happen in this setup. Note that SMART rarely/never predicts failures, but one of the strongest signals of drive failures is slightly elevated temperatures (usually as a result of bearing wear). This creates enough waste heat such that one room never needs heating, but cooling isn't strictly needed either because there's no point to reducing datacenter ambient below 27 C.

syntheticnature 3y ago
I was looking into water heaters that use heat pumps recently, and a lot of them function by sucking heat out of the room. While water and computers don't mix, might be an even better use for all that waste heat...
- sgarland 3y ago
  My plan for when I build a house (so far with two moves during my serious homelab period, the room layout hasn’t worked) is to build a small server room that vents out to the heat pump water heater.

Palomides 3y ago

amusing to read this very detailed article and not have any idea what OP actually does with 72TB of online storage 1gbe seems a bit anemic for a NAS

dannyw 3y ago
it's usually p0rn
- jmnicolas 3y ago
  Try to be charitable, I'm sure there are a couple Linux ISO in there.

alphabettsy 3y ago

Very cool write up. Good timing too as I find myself attempting to reduce the power consumption of my homelab.

vermaden 3y ago

Good read. Tried something similar in the past: - https://vermaden.wordpress.com/2019/04/03/silent-fanless-freebsd-server-redundant-backup/ https://vermaden.wordpress.com/2019/04/03/silent-fanless-fre... - https://vermaden.wordpress.com/2023/04/10/silent-fanless-dell-wyse-3030-lt-freebsd-server/ https://vermaden.wordpress.com/2023/04/10/silent-fanless-del...

uxp8u61q 3y ago

I know nothing about building NASs so maybe my question has an obvious answer. But my impression is that most x64 CPUs are thoroughly beaten by Arm or RISC-V CPUs when it comes to power consumption. Is there a specific need for the x64 architecture here? I couldn't find an answer in TFA.

hmottestad 3y ago
You can very easily run docker containers on it. That’s why I went with a ryzen chip in mine. You could always use an rpi if you want to go with ARM, and you’ll want something with ARMv8.
adrian_b 3y ago
Most Arm or RISC-V CPUs (with the exception of a few server-oriented models that are much more expensive than x86) have very few PCIe lanes and SATA ports, so you cannot make a high throughput NAS with any of them. There are some NAS models with Arm-based CPUs and multiple SSDs/HDDs, but those have a very low throughput due to using e.g. only one PCIe lane per socket, with at most PCIe 3 speed.
arp242 3y ago
> my impression is that most x64 CPUs are thoroughly beaten by Arm or RISC-V CPUs when it comes to power consumption Not really. ARM (and to lesser degree, RISC-V) are often used and optimized for low-power usage and/or low-heat. x64 is often more optimized for maximum performance, at the expense of higher power usage and more heat. For many x64 CPUs you can drastically reduce the power usage if you underclock the CPU just a little bit (~10% slower), especially desktop CPUs but also laptops. There are ARM and RISC-V CPUs that consume much less power, but they're also much slower and have a much more limited feature-set. You do need to compare like to like, and when you do the power usage differences are usually small to non-existent in modern CPUs. ARM today is no longer the ARM that Wilson et al. design 40 years ago. And for something connected to mains, even doubling the efficiency and going from 7W to 3.5W doesn't really make all that much difference. It's just not a big impact on your energy bill or climate change.
pmontra 3y ago

sandreas 3y ago

There is a german forum thread with a google docs document listing different configurations below 30W[1]. Since there are very different requirements, this might be interesting for many homeserver / NAS builders. For me personally I found my ideal price-performance config to be the following hardware: Board: Fujitsu D3417-B2 CPU: Intel Xeon 1225 V5 (better the also compatible 1275v6, but its way more expensive) RAM: 64GB ECC RAM (4x16GB) SSD: WD SN850x 2TB (consumer SSD) Case: Fractal Design Define Mini C Cooling: Big block no name, passively cooled by case fan Power: Pico PSU 120W + 120W Leicke power supply Remote Administration via Intel AMT + MeshCommander using a DP Dummy Plug I bought this config used VERY CHEAP and I am running Proxmox - it draws 9.3W idle (without HDDs). There are 6 SATA ports and a PCIe port, if anyone would like to add more space or passthrough a dedicated GPU. It may be hard to get, but I paid €380,00 in total. Does not work very well for Media Encoding, here you should go for a Core i3 8100 or above. Alternatively you could go for the following changes, but these might be even harder to get for a reasonable price: Boards: GIGABYTE C246N-WU2 (ITX), Gigabyte C246-WU4 (mATX), Fujitsu D3517-B (mATX), Fujitsu D3644 (mATX) Power: Corsair RM550x (2021 Version) Cheap used Workstations that are good servers are Dell T30 or Fujitsu Celsius W550. The Fujitsu ones have D3417(-A!) boards (not -B) having proprietary power supplies with 16 power pins (no 24pin ATX but 16pin). There are Adapters on Aliexpress for 24PIN to 16pin (Bojiadafast), but this is a bit risky - I'm validating that atm. Ryzen possibilities are pretty rare, but there are reports that the AMD Ryzen 5 PRO 4650G with a Asus PRIME B550M-A Board is drawing about 16W Idle. Hope I could help :-) [1]: https://goo.gl/z8nt3A https://goo.gl/z8nt3A

manmal 3y ago
For anybody reading this - I think it’s a great config, but would be careful around pico PSUs in case you want to run a bunch of good old spinning disks. HDDs have a sharp power peak when spinning up, and if you have a couple of them in a RAID, they might do so synchronously, potentially exceeding the envelope.
- LargoLasskhyfv 3y ago
  In times long gone there's been a setting. Staggered spin-up it was called. Turned on one disk after another. So the PSU didn't bother, or stalled. Done.

ulnarkressty 3y ago

As exciting as it is to design a low power system, it's kind of pointless in the case of a NAS that uses spinning rust as storage media - as the author later writes, the HDD power consumption dwarfs the other system components. If one uses SSD or M.2 drives, there are some solutions on the market that provide high speed hardware RAID in a separate external enclosure. Coupled with a laptop board they could make for a decent low power system. Not sure how reliable USB or Thunderbolt is compared to internal SATA or PCIe connections though... would be interesting to find out.

V__ 3y ago
Don't they stop spinning when idle?
- layer8 3y ago
  Not by default, but you can have the OS have them spin down after a certain idle period. Doing that too frequently can affect the life time of the drive though. You save maybe 4 Watts per drive by spinning them down.
  - Dalewyn 3y ago
    Personally, I make sure my HDDs (regardless use case) never spin down when idle. The cost in life span isn't worth the electricity bill saved.
    - Arn_Thor 3y ago
      Same. Also I’ve had some software error out as it waits for drives to spin up (in an NFS share context).
- orthoxerox 3y ago
  They are never idle if the NAS is seeding torrents.
nabla9 3y ago
You can shut down HDD's when you don't use them. sudo hdparm -Y /dev/sdX

homero 3y ago

Crucial Force GT supposed to say Corsair

treprinum 3y ago

My NAS has Pentium J 4-core and is way under 7W idle, inside some small Fractal case with 6x20TB HDD. Why would you need 12th/13th gen for file transfers?

wffurr 3y ago
For encoding maybe? OP says “reasonable CPU performance for compression” and also it was a CPU they already had from a desktop build.
dbeley 3y ago
Interesting I assume it's with all drives off, how many Watts with some disk usage?
- treprinum 3y ago
  Each drive adds around 7W non-idle.

jepler 3y ago

Author seems to have built 5 systems from 2016 to 2023, or around every other year. Some parts (e.g., RAM) are re-used across multiple builds It's interesting to wonder: How much $$ is the hardware cost vs the lifetime energy costs? Is a more power-hungry machine that would operate for 4 years better than one that would operate for 2 years? The motherboard + CPU is USD 322 right now on pcpartpicker. At USD 0.25/kWh (well above my local rate but below the highest rates in the US), 36W continuous over 4 years is also about $315. So, a ~43W, 4-year system might well be cheaper to buy and operate than a 7W, 2-year system.

hrdwdmrbl 3y ago
Fun tinkering like that is not about saving money. It's about the journey, not the destination.
- 1-6 3y ago
  At least the author documents their journey well. If many people who read this save energy by learning good fundamentals of system build, that alone goes a long way. CPU/mobo makers are also probably taking notes.
buro9 3y ago
I average a fully loaded NAS every 8 years apparently. I don't really expand their storage, as by this time the network and software is out of date, so I buy a new one and make the current the backup to the new, and then dispose of the one that was the old backup. It's interesting, the upgrade interval has held for 20 years already and so it becomes easy to understand the amortisation, 16 years of use, but really only 8 as the primary and 8 as the backup.

CommanderData 3y ago

Great hardware but when the software is a job to administer I have a hard time justifying builds like these. My Synology NAS for example has 8 GB RAM and a J4150 processor. Runs about 15 containers, Wireguard and on top DSM (which is Synology's OS). Usually idles around 1-3%. Software makes all the difference - DSM has been by far the biggest benefit and surprise to me and I would be deprived of time with anything else. I'm running TrueNas as a second backup server but it no way compares to DSM. Sometimes I don't want to trawl through logs and trial and error just to get a basic CRON setup to backup a file off another server, there's countless examples where DSM has just worked. I really think Synology is missing a trick here, they clearly have software that is miles ahead of everything else and customizable should you need to. They should be more like Microsoft of the NAS world, making DSM run on non Synology platforms, or at least making it easier to do yourself. It's a great OS and it sells itself and can easily be a way to up sell stuff like Active backup for business.

matthewfcarlson 3y ago
I suspect focusing on a set of hardware and making it work really well is part of what makes the OS so good. Hardware configs explode in a combinatorial fashion, making it impossible to test everything once you have more than a few options.
thelittleone 3y ago
Agree, its a super nice UI and UX. First time I used them was around 2010 and it was already a joy to use. Would be kind of nice to have a Synology DSM front end to cloud IaaS.
agumonkey 3y ago
dsm being diskstation manager ?
- CommanderData 3y ago
  Yes, it's the OS that ships with their hardware. DSM 7 being their latest.
3y ago

neilv 3y ago

Great work, for lots of storage. If you can fit your storage on an SSD (or RAID-mirrored pair), and don't need much compute, you can do a low-power server (like to run little services for yourself) using an SBC like a RasPi, or something like a NUC. Personally, I currently have a couple 1U Atom servers that run fanless except for the Noctua fans that I swapped into the PSUs. Advantages over RasPi include SATA and ECC RAM, and were also easier to buy over Covid. (I also have a 4U GPU server, which is currently off when not in use, because I haven't invested in figuring out how to low-power idle it like the article writer has.)

mattbillenstein 3y ago

Everyone's needs are different, but over time with the loss of a couple of disks, I started to hate running RAID5 or 6 with HDDs. It became an exercise in how fast could I replace a disk before the next one died and would the rebuild actually work or not - although, it always did. Also, the hot-swap case/cage I had with all the SATA connectors and power connectors seemed kinda flaky - it was very cheap. So a couple years ago, I downsized to a single 2TB SSD in a smallish ATX case - and another in a completely different machine that gets rsync'd every 4 hours. My nas is now just my last desktop's hardware with two SSDs - one boot, one larger storage running plex, smbd, backups of misc stuff on cron, duplicity backup to the cloud, etc on Ubuntu. If I didn't have this extra hardware, I'd probably just run two medium powered NUCs with a nvme boot disk and a bigger SSD for storage. It's all very very simple and I have it setup so I can run some LXC containers should I want to do some dev work there, but I usually have other hardware for that anyway.

at_a_remove 3y ago
I am imagining a NAS appliance designed for high availability and low-touch, like a RAID 10 box with a ton of hot spares, designed to automatically rebuild in case of failure. The hot spares are spun up once a month to prevent stiction.
nine_k 3y ago
Good thing that 2TB suffices for you! I think RAID NAS boxes seem to make sense at sizes several times this.
CPLX 3y ago
I have done a bunch of stuff in the past but most recently I just bought two synology arrays and followed the instructions to set them up in less than an hour and they just worked with literally zero hassle.

vardump 3y ago

What would be the cheapest and lowest power option to get 100G networking (50-100G actual performance is good enough) and at least one 8 lane PCI-e GPU? Plus at least 2x M.2.

deleted 3y ago

[deleted]

pomatic 3y ago

7 watts idle is unimpressive out of context If you have a NAS that is idle most of the time, and want to minimise power consumption, how about an embedded-cpu based WoL generator? Sniff packets destined for the fileserver, which is otherwise in deep sleep, and automagically wake it up when relevant traffic is detected, with the relevant WoL packet. You'd get say <300mA consumption at 3v3, and full performance on wake. It would be transparent to the machines attempting to access the server. If idle power measurement is your criteria, this might be a way forwards?

getwiththeprog 3y ago
Cool idea. I have no idea how to automagically do that.
- graphe 3y ago
  PoE or power over Ethernet.
  - allset_ 3y ago
    You don't need PoE to do WoL.

Paradigma11 3y ago

How viable would it be to run a home server in a VM on your main PC that is occasionally also used for high performance activities like gaming. I do have 12 cores and 64GB Ram and I would not mind parting with 4C/16GB and maybe the onboard GPU (video decoding) but I dont know how the rest of the system might be influenced if there is heavy IO or other stuff.

marcosscriven 3y ago
Proxmox is good for this. You can have a Windows VM with GPU passthrough. Then you can add any other VMs or LXCs to your heart’s content.

jmnicolas 3y ago

I have a 4 Cores Intel i3 13100 and I can't reach these low numbers. The machine idle with just 1 nvme is 16 watts. I managed to go to 12 watts with the powersaver mode in tuned but I have random ssh freezes and I didn't find any relevant doc to solve the problem. But as soon as the machine is in light use, just a network transfer for example the power usage go through the roof: 30 watts. Now I have loaded it with 2 internal 3.5 hard drives, I have a VM with Docker running, a torrent client and Syncthing (so nothing terribly intensive). The thing never goes below 40 watts and is currently at 60 (I'm copying data onto the drives). I would never have bought a desktop class CPU if it wasn't for these articles that promise unreachable low power draw.