7 ms·

While it may be tempting to go "mini" and NVMe, for a normal use case I think this is hardly cost effective. You give up so much by using an all in mini device

While it may be tempting to go "mini" and NVMe, for a normal use case I think this is hardly cost effective. You give up so much by using an all in mini device... No Upgrades, no ECC, harder cooling, less I/O. I have had a Proxmox Server with a used Fujitsu D3417 and 64gb ecc for roughly 5 years now, paid 350 bucks for the whole thing and upgraded the storage once from 1tb to 2tb. It draws 12-14W in normal day use and has 10 docker containers and 1 windows VM running. So I would prefer a mATX board with ECC, IPMI 4xNVMe and 2.5GB over these toy boxes... However, Jeff's content is awesome like always

deleted 1y ago
[deleted]
samhclark 1y ago
I think you're right generally, but I wanna call out the ODROID H4 models as an exception to a lot of what you said. They are mostly upgradable (SODIMM RAM, SATA ports, M.2 2280 slots), and it does support in-band ECC which kinda checks the ECC box. They've got a Mini-ITX adapter for $15 so it can fit into existing cases too. No IPMI and not very many NVME slots. So I think you're right that a good mATX board could be better.
- sandreas 1y ago
  Well, if you would like to go mini (with ECC and 2.5G) you could take a look at this one: https://www.aliexpress.com/item/1005006369887180.html https://www.aliexpress.com/item/1005006369887180.html Not totally upgradable, but at least pretty low cost and modern with an optional SATA + NVMe combination for Proxmox. Shovel in an enterprise SATA and a consumer 8TB WD SN850x and this should work pretty good. Even Optane is supported. IPMI could be replaced with NanoKVM or JetKVM...
  - a012 1y ago
    That looks pretty slick with a standard hsf for the CPU, thanks for sharing
  - smartbit 1y ago
    Nice indeed. With only 2 nvme slots, what drive configuration do you have in your mind? Backup from nvme to HDD locally and another device remote?
    - sandreas 1y ago
      For my personal purpose i would go 2 x wd sn850x 2tb RAID1 nvme(consumer) You could also go 32gb+ Intel optane Boot and enterprise SATA data, depending on your use case
- geek_at 1y ago
  Not sure about the odroid but I got myself the nas kit from friendly elec. With the largest ram it was about 150 bucks and comes with 2,5g ethernet and 4 NVME slots. No fan and keeps fairly cool even under load. Running it with encrypted zfs volumes and even with a 5bay 3.5 Inch HDD dock attached via USB https://wiki.friendlyelec.com/wiki/index.php/CM3588_NAS_Kit https://wiki.friendlyelec.com/wiki/index.php/CM3588_NAS_Kit
- ilkhan4 1y ago
  You can get a 1 -> 4 M.2 adapter for these as well which would give each one a 1x PCIe lane (same as all these other boards). If you still want spinning rust, these also have built-in power for those and SATA ports so you only need a 12-19v power supply. No idea why these aren't more popular as a basis for a NAS.
fnord77 1y ago
these little boxes are perfect for my home My use case is a backup server for my macs and cold storage for movies. 6x2Tb drives will give me a 9Tb raid-5 for $809 ($100 each for the drives, $209 for the nas). Very quiet so I can have it in my living room plugged into my TV. < 10W power. I have no room for a big noisy server.
- sandreas 1y ago
  While I get your point about size, I'd not use RAID-5 for my personal homelab. I'd also say that 6x2TB drives are not the optimal solution for low power consumption. You're also missing out server quality BIOS, Design/Stability/x64 and remote management. However, not bad. While my Server is quite big compared to a "mini" device, it's silent. No CPU Fan only 120mm case fans spinning around 500rpm, maybe 900rpm on load - hardly noticable. I've also a completely passive backup solution with a Streacom FC5, but I don't really trust it for the chipsets, so I also installed a low rpm 120mm fan. How did you fit 6 drives in a "mini" case? Using Asus Flashstor or beelink?
  - epistasis 1y ago
    I'm interested in learning more about your setup. What sort of system did you put together for $350? Is it a normal ATX case? I really like the idea of running proxmox but I don't know how to get something cheap!
    - sandreas 1y ago
      My current config: Fujitsu D3417-B12 Intel Xeon 1225 64GB ecc WD SN850x 2TB mATX case Pico PSU 150 For backup I use a 2TB enterprise HDD and ZFS send For snapshotting i use zfs-auto-snapshot So really nothing recommendable for buying today. You could go for this https://www.aliexpress.com/item/1005006369887180.html https://www.aliexpress.com/item/1005006369887180.html Or an old Fujitsu Celsius W580 Workstation with a Bojiadafast ATX Power Supply Adapter, if you need harddisks. Unfortunately there is no silver bullet these days. The old stuff is... well too old or no longer available and the new stuff is either to pricey, lacks features (ECC and 2.5G mainly) or to power hungry. A year ago there were bargains for Gigabyte MC12-LE0 board available for < 50bucks, but nowadays these cost about 250 again. These boards also had the problem of drawing too much power for an ultra low power homelab. If I HAD to buy one today, I'd probably go for a Ryzen Pro 5700 with a gaming board (like ASUS ROG Strix B550-F Gaming) with ECC RAM, which is supported on some boards.
cyanydeez 1y ago
I've had a synology since 2015. Why, besides the drives themselves, would most home labs need to upgrade? I don't really understand the general public, or even most usages, requiring upgrade paths beyond get a new device. By the time the need to upgrade comes, the tech stack is likely faster and you're basically just talking about gutting the PC and doing everything over again, except maybe power supply.
- sandreas 1y ago
  Understandable... Well, the bottleneck for a Proxmox Server often is RAM - sometimes CPU cores (to share between VMs). This might not be the case for a NAS-only device. Another upgrade path is to keep the case, fans, cooling solution and only switch Mainboard, CPU and RAM. I'm also not a huge fan of non x64 devices, because they still often require jumping through some hoops regarding boot order, external device boot or power loss struggle.
- deleted 1y ago
  [deleted]
- dragontamer 1y ago
  > except maybe power supply. Modern Power MOSFETs are cheaper and more efficient. 10 Years ago 80Gold efficiency was a bit expensive and 80Bronze was common. Today, 80Gold is cheap and common and only 80Platinum reaches into the exotic level.
  - sandreas 1y ago
    A 80Bronze 300W can still be more efficient than a 750W 80Platinum on mainly low loads. Additionally, some of the devices are way more efficient than they are certified for. A well known example is the Corsair RM550x (2021). If your peak power draw is <200W, I would recommend an efficient <450W power supply. Another aspect: Buying a 120 bucks power supply that is 1.2% more efficient than a 60 bucks one is just a waste of money.
ndiddy 1y ago
Another thing is that unless you have a very specific need for SSDs (such as heavily random access focused workloads, very tight space constraints, or working in a bumpy environment), mechanical hard drives are still way more cost effective for storing lots of data than NVMe. You can get a manufacturer refurbished 12TB hard drive with a multi-year warranty for ~$120, while even an 8TB NVMe drive goes for at least $500. Of course for general-purpose internal drives, NVMe is a far better experience than a mechanical HDD, but my NAS with 6 hard drives in RAIDz2 still gets bottlenecked by my 2.5GBit LAN, not the speeds of the drives.
- acranox 1y ago
  Don’t forget about power. If you’re trying to build a low power NAS, those hdds idle around 5w each, while the ssd is closer to 5mw. Once you’ve got a few disks, the HDDs can account for half the power or more. The cost penalty for 2TB or 4TB ssds is still big, but not as bad as at the 8TB level.
  - markhahn 1y ago
    such power claims are problematic - you're not letting the HDs spin down, for instance, and not crediting the fact that an SSD may easily dissipate more power than an HD under load. (in this thread, the host and network are slow, so it's not relevant that SSDs are far faster when active.)
    - philjohn 1y ago
      There's a lot of "never let your drive spin down! They need to be running 24/7 or they'll die in no time at all!" voices in the various homelab communities sadly. Even the lower tier IronWolf drives from Seagate specify 600k load/unload cycles (not spin down, granted, but gives an idea of the longevity).
      - sandreas 1y ago
        Is there any (semi-)scientific proof to that (serious question)? I did search a lot to this topic but found nothing...
deleted 1y ago
[deleted]
layoric 1y ago
No ECC is the biggest trade off for me, but the C236 express chipset has very little choice for CPUs, they are all 4 core 8 thread. Ive got multiple x99 platform systems and for a long time they were the king of cost efficiency, but lately the ryzen laptop chips are becoming too good to pass up, even without ECC. Eg Ryzen 5825u minis
- mytailorisrich 1y ago
  For a home NAS, ECC is as needed as it is on your laptop.
  - vbezhenar 1y ago
    ECC is essential indeed for any computer. But the laptop situation is truly dire, while it's possible to find some NAS with ECC support.
    - mytailorisrich 1y ago
      Most computers don't have ECC. So it might be essential in theory but in practice things work fine without (for standard personal, even work, use cases).
asciimov 1y ago
The selling point for the people in the Plex community is the N100/N150 include Intel’s Quicksync which gives you video hardware transcoding without a dedicated video card. It’ll handle 3 to 4 4K transcoded streams. There are several sub $150 units that allow you to upgrade the ram, limited to one 32gb stick max. You can use an nvme to sata adapter to add plenty of spinning rust or connect it to a das. While I wouldn’t throw any vms on these, you have enough headroom for non-ai home sever apps.
- MrDarcy 1y ago
  We’ve been able to buy used OptiPlex 3060 or 3070’s for about $100 for years now and they tick all the boxes for Plex and QuickSync. Only two NVME and one SATA slot though, so maybe not ideal for a NAS but definitely fits the power and thermal profile, and it’s nice to reuse perfectly good hardware.
- sandreas 1y ago
  The device I linked below (https://www.aliexpress.com/item/1005006369887180.html https://www.aliexpress.com/item/1005006369887180.html) has a XEON, ECC, 2xNVMe, SATA, 2.5GBit and everything else you need in a very small box. Intel also means it has QuickSync, so you won't need to buy a N150. However, I tend to be sceptic about these aliexpress-boxes, too. Reliable server manufacturers, like Dell, HP, Lenovo or Fujitsu (RIP) are way more reliable.
  - asciimov 1y ago
    I agree, if your linked pc wasn't from aliexpress it would be a compelling alternative. Not that these n100 mini-pcs are any better, but I can get one tomorrow off amazon for under $150. I do think they bridge the server gap between using a rpi and a full server. Likely for the hobbyist that doesn't yet have the need (or space) for better hardware.
dvdkon 1y ago
Where are you measuring the power consumption? I've recently started measuring the wattage of all the various electronics in my collection, and I haven't found any computer that's not underpowered and draws under 25W from the wall when idle, and that's with no HDDs and minimal RAM. Turns out I actually have power supplies that alone draw over 30W with zero load; when trying for the lowest idle power consumption I've found that the choice of power supply matters a lot,
- sandreas 1y ago
  I use a cheap power meter for initial testing and a tasmota plug for having a monitoring solution as well as a second measure and a possibility ro remote hard reset in case of a freeze or something. Turns out that power supply and motherboard are the most important to save power - besides low C-states (powertop). I had best results with Fujitsu D3x17 / d3644 and Gigabyte C246 wu2. Today these are unicorns not worth hunting for. Like I said: No modern server grade board is that good while being cheap. You could take a look at Kontron K3851-R ATX If i remember correctly. Kontron bought Fujitsus Mainboard segment a while ago.
justinclift 1y ago
Gen 11 HPE Microservers seem pretty decent: https://buy.hpe.com/us/en/compute/tower-servers/proliant-microserver/proliant-microserver/hpe-proliant-microserver-gen11/p/1014826370 https://buy.hpe.com/us/en/compute/tower-servers/proliant-mic... Pity they're Intel cpu's though. :( HPE have announced 12th Gen servers for their other lines recently, so maybe the Microservers will get a 12th Gen update this year too. Hopefully with AMD cpus rather than the Intel crap.
- sandreas 1y ago
  Personally i find these too expensive compared to the good old N54L and Dell T20 era. The Hardware is great but >1000 bucks is pretty hard even for enthusiasts
  - justinclift 1y ago
    The Gen8 stuff is pretty good, though the 16GB ram maximum for those is what's causing me to look at upgrades. :)

deleted 1y ago

[deleted]

samhclark 1y ago

I think you're right generally, but I wanna call out the ODROID H4 models as an exception to a lot of what you said. They are mostly upgradable (SODIMM RAM, SATA ports, M.2 2280 slots), and it does support in-band ECC which kinda checks the ECC box. They've got a Mini-ITX adapter for $15 so it can fit into existing cases too. No IPMI and not very many NVME slots. So I think you're right that a good mATX board could be better.

sandreas 1y ago
Well, if you would like to go mini (with ECC and 2.5G) you could take a look at this one: https://www.aliexpress.com/item/1005006369887180.html https://www.aliexpress.com/item/1005006369887180.html Not totally upgradable, but at least pretty low cost and modern with an optional SATA + NVMe combination for Proxmox. Shovel in an enterprise SATA and a consumer 8TB WD SN850x and this should work pretty good. Even Optane is supported. IPMI could be replaced with NanoKVM or JetKVM...
- a012 1y ago
  That looks pretty slick with a standard hsf for the CPU, thanks for sharing
- smartbit 1y ago
  Nice indeed. With only 2 nvme slots, what drive configuration do you have in your mind? Backup from nvme to HDD locally and another device remote?
  - sandreas 1y ago
    For my personal purpose i would go 2 x wd sn850x 2tb RAID1 nvme(consumer) You could also go 32gb+ Intel optane Boot and enterprise SATA data, depending on your use case
geek_at 1y ago
Not sure about the odroid but I got myself the nas kit from friendly elec. With the largest ram it was about 150 bucks and comes with 2,5g ethernet and 4 NVME slots. No fan and keeps fairly cool even under load. Running it with encrypted zfs volumes and even with a 5bay 3.5 Inch HDD dock attached via USB https://wiki.friendlyelec.com/wiki/index.php/CM3588_NAS_Kit https://wiki.friendlyelec.com/wiki/index.php/CM3588_NAS_Kit
ilkhan4 1y ago
You can get a 1 -> 4 M.2 adapter for these as well which would give each one a 1x PCIe lane (same as all these other boards). If you still want spinning rust, these also have built-in power for those and SATA ports so you only need a 12-19v power supply. No idea why these aren't more popular as a basis for a NAS.

fnord77 1y ago

these little boxes are perfect for my home My use case is a backup server for my macs and cold storage for movies. 6x2Tb drives will give me a 9Tb raid-5 for $809 ($100 each for the drives, $209 for the nas). Very quiet so I can have it in my living room plugged into my TV. < 10W power. I have no room for a big noisy server.

sandreas 1y ago
While I get your point about size, I'd not use RAID-5 for my personal homelab. I'd also say that 6x2TB drives are not the optimal solution for low power consumption. You're also missing out server quality BIOS, Design/Stability/x64 and remote management. However, not bad. While my Server is quite big compared to a "mini" device, it's silent. No CPU Fan only 120mm case fans spinning around 500rpm, maybe 900rpm on load - hardly noticable. I've also a completely passive backup solution with a Streacom FC5, but I don't really trust it for the chipsets, so I also installed a low rpm 120mm fan. How did you fit 6 drives in a "mini" case? Using Asus Flashstor or beelink?
- epistasis 1y ago
  I'm interested in learning more about your setup. What sort of system did you put together for $350? Is it a normal ATX case? I really like the idea of running proxmox but I don't know how to get something cheap!
  - sandreas 1y ago
    My current config: Fujitsu D3417-B12 Intel Xeon 1225 64GB ecc WD SN850x 2TB mATX case Pico PSU 150 For backup I use a 2TB enterprise HDD and ZFS send For snapshotting i use zfs-auto-snapshot So really nothing recommendable for buying today. You could go for this https://www.aliexpress.com/item/1005006369887180.html https://www.aliexpress.com/item/1005006369887180.html Or an old Fujitsu Celsius W580 Workstation with a Bojiadafast ATX Power Supply Adapter, if you need harddisks. Unfortunately there is no silver bullet these days. The old stuff is... well too old or no longer available and the new stuff is either to pricey, lacks features (ECC and 2.5G mainly) or to power hungry. A year ago there were bargains for Gigabyte MC12-LE0 board available for < 50bucks, but nowadays these cost about 250 again. These boards also had the problem of drawing too much power for an ultra low power homelab. If I HAD to buy one today, I'd probably go for a Ryzen Pro 5700 with a gaming board (like ASUS ROG Strix B550-F Gaming) with ECC RAM, which is supported on some boards.

cyanydeez 1y ago

I've had a synology since 2015. Why, besides the drives themselves, would most home labs need to upgrade? I don't really understand the general public, or even most usages, requiring upgrade paths beyond get a new device. By the time the need to upgrade comes, the tech stack is likely faster and you're basically just talking about gutting the PC and doing everything over again, except maybe power supply.

sandreas 1y ago
Understandable... Well, the bottleneck for a Proxmox Server often is RAM - sometimes CPU cores (to share between VMs). This might not be the case for a NAS-only device. Another upgrade path is to keep the case, fans, cooling solution and only switch Mainboard, CPU and RAM. I'm also not a huge fan of non x64 devices, because they still often require jumping through some hoops regarding boot order, external device boot or power loss struggle.
deleted 1y ago
[deleted]
dragontamer 1y ago
> except maybe power supply. Modern Power MOSFETs are cheaper and more efficient. 10 Years ago 80Gold efficiency was a bit expensive and 80Bronze was common. Today, 80Gold is cheap and common and only 80Platinum reaches into the exotic level.
- sandreas 1y ago
  A 80Bronze 300W can still be more efficient than a 750W 80Platinum on mainly low loads. Additionally, some of the devices are way more efficient than they are certified for. A well known example is the Corsair RM550x (2021). If your peak power draw is <200W, I would recommend an efficient <450W power supply. Another aspect: Buying a 120 bucks power supply that is 1.2% more efficient than a 60 bucks one is just a waste of money.

ndiddy 1y ago

Another thing is that unless you have a very specific need for SSDs (such as heavily random access focused workloads, very tight space constraints, or working in a bumpy environment), mechanical hard drives are still way more cost effective for storing lots of data than NVMe. You can get a manufacturer refurbished 12TB hard drive with a multi-year warranty for ~$120, while even an 8TB NVMe drive goes for at least $500. Of course for general-purpose internal drives, NVMe is a far better experience than a mechanical HDD, but my NAS with 6 hard drives in RAIDz2 still gets bottlenecked by my 2.5GBit LAN, not the speeds of the drives.

acranox 1y ago
Don’t forget about power. If you’re trying to build a low power NAS, those hdds idle around 5w each, while the ssd is closer to 5mw. Once you’ve got a few disks, the HDDs can account for half the power or more. The cost penalty for 2TB or 4TB ssds is still big, but not as bad as at the 8TB level.
- markhahn 1y ago
  such power claims are problematic - you're not letting the HDs spin down, for instance, and not crediting the fact that an SSD may easily dissipate more power than an HD under load. (in this thread, the host and network are slow, so it's not relevant that SSDs are far faster when active.)
  - philjohn 1y ago
    There's a lot of "never let your drive spin down! They need to be running 24/7 or they'll die in no time at all!" voices in the various homelab communities sadly. Even the lower tier IronWolf drives from Seagate specify 600k load/unload cycles (not spin down, granted, but gives an idea of the longevity).
    - sandreas 1y ago
      Is there any (semi-)scientific proof to that (serious question)? I did search a lot to this topic but found nothing...

deleted 1y ago

[deleted]

layoric 1y ago

No ECC is the biggest trade off for me, but the C236 express chipset has very little choice for CPUs, they are all 4 core 8 thread. Ive got multiple x99 platform systems and for a long time they were the king of cost efficiency, but lately the ryzen laptop chips are becoming too good to pass up, even without ECC. Eg Ryzen 5825u minis

mytailorisrich 1y ago
For a home NAS, ECC is as needed as it is on your laptop.
- vbezhenar 1y ago
  ECC is essential indeed for any computer. But the laptop situation is truly dire, while it's possible to find some NAS with ECC support.
  - mytailorisrich 1y ago
    Most computers don't have ECC. So it might be essential in theory but in practice things work fine without (for standard personal, even work, use cases).

asciimov 1y ago

The selling point for the people in the Plex community is the N100/N150 include Intel’s Quicksync which gives you video hardware transcoding without a dedicated video card. It’ll handle 3 to 4 4K transcoded streams. There are several sub $150 units that allow you to upgrade the ram, limited to one 32gb stick max. You can use an nvme to sata adapter to add plenty of spinning rust or connect it to a das. While I wouldn’t throw any vms on these, you have enough headroom for non-ai home sever apps.

MrDarcy 1y ago
We’ve been able to buy used OptiPlex 3060 or 3070’s for about $100 for years now and they tick all the boxes for Plex and QuickSync. Only two NVME and one SATA slot though, so maybe not ideal for a NAS but definitely fits the power and thermal profile, and it’s nice to reuse perfectly good hardware.
sandreas 1y ago
The device I linked below (https://www.aliexpress.com/item/1005006369887180.html https://www.aliexpress.com/item/1005006369887180.html) has a XEON, ECC, 2xNVMe, SATA, 2.5GBit and everything else you need in a very small box. Intel also means it has QuickSync, so you won't need to buy a N150. However, I tend to be sceptic about these aliexpress-boxes, too. Reliable server manufacturers, like Dell, HP, Lenovo or Fujitsu (RIP) are way more reliable.
- asciimov 1y ago
  I agree, if your linked pc wasn't from aliexpress it would be a compelling alternative. Not that these n100 mini-pcs are any better, but I can get one tomorrow off amazon for under $150. I do think they bridge the server gap between using a rpi and a full server. Likely for the hobbyist that doesn't yet have the need (or space) for better hardware.

dvdkon 1y ago

Where are you measuring the power consumption? I've recently started measuring the wattage of all the various electronics in my collection, and I haven't found any computer that's not underpowered and draws under 25W from the wall when idle, and that's with no HDDs and minimal RAM. Turns out I actually have power supplies that alone draw over 30W with zero load; when trying for the lowest idle power consumption I've found that the choice of power supply matters a lot,

sandreas 1y ago
I use a cheap power meter for initial testing and a tasmota plug for having a monitoring solution as well as a second measure and a possibility ro remote hard reset in case of a freeze or something. Turns out that power supply and motherboard are the most important to save power - besides low C-states (powertop). I had best results with Fujitsu D3x17 / d3644 and Gigabyte C246 wu2. Today these are unicorns not worth hunting for. Like I said: No modern server grade board is that good while being cheap. You could take a look at Kontron K3851-R ATX If i remember correctly. Kontron bought Fujitsus Mainboard segment a while ago.

justinclift 1y ago

Gen 11 HPE Microservers seem pretty decent: https://buy.hpe.com/us/en/compute/tower-servers/proliant-microserver/proliant-microserver/hpe-proliant-microserver-gen11/p/1014826370 https://buy.hpe.com/us/en/compute/tower-servers/proliant-mic... Pity they're Intel cpu's though. :( HPE have announced 12th Gen servers for their other lines recently, so maybe the Microservers will get a 12th Gen update this year too. Hopefully with AMD cpus rather than the Intel crap.

sandreas 1y ago
Personally i find these too expensive compared to the good old N54L and Dell T20 era. The Hardware is great but >1000 bucks is pretty hard even for enthusiasts
- justinclift 1y ago
  The Gen8 stuff is pretty good, though the 16GB ram maximum for those is what's causing me to look at upgrades. :)