8 ms·

Raspberry Pi 2.5 Gbps OMV NAS Performance

89 points by geerlingguy 6y ago

raarts 6y ago
The primary reason by far why I haven't done a lot more with the Pi is the lack of large enclosures.
- geerlingguy 6y ago
  I mention the Wiretrustee SATA towards the end of the post; it looks like they'll have a laser-cut acrylic case in two sizes, for either 2.5" or 3.5" drives. I'm hopeful one of the mini ITX or micro ATX boards goes into production soon; at that point, you could mount the Pi securely into any bog-standard ITX/ATX case and even make use of PSU connectors and the PCIe slot!
  - tssva 6y ago
    I mentioned it in a comment for your original hardware video but I'll repeat it here. I think another setup worth trying is an eSATA 4 drive enclosure connected to the SATA controller board via a SATA to eSATA cable.
- guilhas 6y ago
  And for 50euros you can get a second hand HP microserver on eBay
  - xnyan 6y ago
    Power consumption quickly becomes relevant in terms of cost when you are running it 24/7. I measure about 33 kWh a year per RasPi I run, my undervolted x86 2014 intel platform is easily over 15x that.
deleted 6y ago
[deleted]
dshep 6y ago
Pretty neat. I have been thinking about how I can build a tiny NAS, but I would like ECC so that rules out the pi, etc. Best option I have found so far might be this ASRock 4x4-v2000 https://www.asrockind.com/en-gb/4X4-V2000M https://www.asrockind.com/en-gb/4X4-V2000M , it has an 8-core cpu and supports ECC. Would need to get a M.2 to 4x SATA adapter. The hard part seems to be figuring out how to buy the board itself...
- Scene_Cast2 6y ago
  I've been using a Helios4 for this; it's tiny, has ECC, and plenty of speed for a NAS. Their new board, the helios64, is bigger, and ECC support was in-progress last time I checked.
  - sbierwagen 6y ago
    Doh, wish I knew this existed last month. Just built a NAS around an ITX board.
  - vladvasiliu 6y ago
    The Helios64 "Full Bundle" looks nice, but only 4 GB RAM seems limited for ZFS. Plus, for 300 USD, there should be a way to set up a similar x86 system, possibly from second-hand parts. I'm thinking some older AMD Zen with 8-16 GB RAM and a Fractal Design Node or similar case. Yes, I know that would likely use more electricity, but in my particular case (France), electricity isn't that big of a cost, plus I wouldn't be running this 24/7. More like one or two evenings a week. The only reason why I'm not running recycled "enterprise" servers at home is that my apartment is small and I absolutely cannot stand the noise they make.
    - MivLives 6y ago
      I've had good luck buying old gaming pc parts off my friends. If they're upgrading their cpu, they normally have a mobo/ram/cpu to sell.
      - 6y ago
geerlingguy 6y ago
One of the little gems I found while I was putting together the video for this post was "Shouting in the datacenter", an older clip (from the pre-HD era) demonstrating the consequence of doing exactly what the title says: https://youtu.be/tDacjrSCeq4 https://youtu.be/tDacjrSCeq4 Warning: loud!
- sliken 6y ago
  I graphed latency to load 4KB or somesuch across the entire disk for an enterprise disk and a consumer disk while on a bench (no vibration). Then I did the same benchmark with the disk mounted in a 1U chassis with a bunch of those annoying 15k rpm fans. Both showed a big difference, but the enterprise drive was 90+% of the no vibration case, the consumer drive was like 30%.
ww520 6y ago
Great info in the article. Had something similar. The firmware update of my sse4200 NAS has stopped and its NetBios version is badly outdated. I was looking for an inexpensive way to reuse the disks to build another NAS. After looking around a bit, ended up getting a RAID capable DAS enclosure for the disks, repurposed an old Chromebox to connect to it via usb, installed Linux, and shared over SAMBA. It works well.
Youden 6y ago
Does anybody know why support for NBase-T is so uncommon and expensive in networking equipment? These 2.5Gbps and 5Gbps adapters look really nice, especially for laptops and other devices with USB 3.2 ports but it appears to be near impossible to affordably connect them to a more-than-trivial network.
- PragmaticPulp 6y ago
  1G has been around so long that it's cheap commodity hardware now. 10G was too much of a leap, IMO, so the hardware remains expensive. 10G over regular Cat5e is also relatively power hungry. 2.5G and 5G arrived relatively recently to fill the gap with the stalled jump to 10G. You can get started with something like a $110 QNAP 2.5G 5-port switch and some cheap 2.5G adapters on the computers you care about.
  - tutfbhuf 6y ago
    > 10G was too much of a leap, IMO, so the hardware remains expensive. Switches are still quiet expensive, but 10G cards are quiet affordable nowadays. Notebooks can use a Thunderbolt 3 to 10G ethernet, but I think we will see 10G ports one consumer notebooks soon. I wouldn't invest in a bridge technology like 2.5G since it introduces non-trivial costs an offers only 2.5x speed up instead of 10x.
    - PragmaticPulp 6y ago
      > but I think we will see 10G ports one consumer notebooks soon Consumer notebooks are moving away from having ethernet ports in general. 10GBASE-T is relatively power hungry, requiring a couple of watts per port. I don't think we'll be seeing this in consumer notebooks any time soon. Also, I have one of those Thuderbolt to 10G adapters. It's not small. It's only useful when I'm doing bulk file transfers. 2.5G is fine for most work. Many consumer NAS boxes can barely saturate 2.5G anyway.
      - hutrdvnj 6y ago
        > Consumer notebooks are moving away from having ethernet ports in general. I might be wrong, but I don't think that every notebook will be as slim as the height of a USB-C port and with only one or two USB-C ports as only ports. This might be true for Apple and some companies that copy Apple. But there are still consumers notebooks that don't follow that trend and build some bulkier books with more ports and better thermals. > 10GBASE-T is relatively power hungry, requiring a couple of watts per port. I don't think we'll be seeing this in consumer notebooks any time soon. It all depends on customers needs. I would rather have a 10GBASE-T port than a dedicated GPU in my book, and I think the latter even consumes more watts.
WhatIsDukkha 6y ago
Are there performance issues for this with BTRFS as well or just ZFS?
- atmosx 6y ago
  I don't know about BTRFS but ZFS is expensive in terms of RAM and since ram is scarce on RPis, I would expect ZFS to underperform under RPis.
  - tpxl 6y ago
    RPi 4 goes up to 8G RAM which is more than enough.
    - nuker 6y ago
      I found for 5x 10TB disks TrueNAS needs more than 8GB.
      - magicalhippo 6y ago
        TrueNAS eats a lot of memory for itself though. Just the web frontend thing eats 3GB on my box, and "services" eats about 12-16GB total (most of which I have no idea where is going). Since ZFS uses memory for caching, it will perform much better with more memory, but it should work[1] with less. [1]: https://www.reddit.com/r/DataHoarder/comments/5u3385/linus_tech_tips_unboxes_1_pb_of_seagate/ddrngar/ https://www.reddit.com/r/DataHoarder/comments/5u3385/linus_t...
    - atmosx 6y ago
      is it? $ zpool status pool: storage state: ONLINE scan: resilvered 854G in 04:18:32 with 0 errors on Sat Jun 20 18:30:06 2020 config: NAME STATE READ WRITE CKSUM storage ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sdc ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 sdb ONLINE 0 0 0 sde ONLINE 0 0 0 errors: No known data errors $ grep -e '^size' /proc/spl/kstat/zfs/arcstats size 4 8604931056 $ bc -q 8604931056 / (1024*1024*1024) 8 On could tune ARC opts and whatnot, but I'm not sure how reliable and fast the system will turn out to be.
rasz 6y ago
tldr: still worse than store bought NAS, or even 10 year old Lenovo x220 with two mpcie/Expresscard to pcie adapters.
- Naac 6y ago
  Store bought NAS doesn't run open source software.
  - tyingq 6y ago
    Some of them can be coaxed into running normal distros. Buffalo Linkstations, for example.
- walterbell 6y ago
  Any recommendations on PCIe adapters which work with both slots on the X220, since only one of them is ExpressCard?
  - rasz 6y ago
    I just solder directly to the motherboard, there are 3 usable PCIEx1 interfaces on x220, one at the missing USB 3.0 controller footprint, second one mpcie, third ExpressCard. You could also desolder memory card controller for fourth pcie lane. Then you have 3x SATA controllers, one HDD, one on mpcie socket, and third on the Dock connector. You also get over 10 individual USB 2.0 channels. There was a time before Sandy Bridge you could configure pcie bifurcation and freely join separate lanes, but I dont think anyone figured out how to do it on x220.
    - walterbell 6y ago
      Thanks for the pointers on dock/SATA: https://forum.thinkpads.com/viewtopic.php?t=119738 https://forum.thinkpads.com/viewtopic.php?t=119738 & https://www.reddit.com/r/thinkpad/comments/6qomsx/51nb_extra_msata_30_in_the_x230/ https://www.reddit.com/r/thinkpad/comments/6qomsx/51nb_extra... Have you considered writing a forum/blog post or Tell HN on your X220 NAS mods?
ggm 6y ago
8gb pi4, radxa quad sata HAT and zfs, working fine. The memory thing is a historical misunderstanding about how the arc works. Sure, it's slow compared to direct pci attached disc controllers, but as a vault, it's working fine. https://wiki.radxa.com/Dual_Quad_SATA_HAT https://wiki.radxa.com/Dual_Quad_SATA_HAT
- 0x0000000 6y ago
  Where do you buy this? I've been searching periodically for months and it always seems unavailable.
  - ggm 6y ago
    I too want to buy (another!) And it looks like all supplies worldwide are dried up. The rockpi 5 port variant may still be available. I bought mine in 2020 from allnet, direct to Australia from China, worked fine.

89 points by geerlingguy 6y ago

raarts 6y ago

The primary reason by far why I haven't done a lot more with the Pi is the lack of large enclosures.

geerlingguy 6y ago
I mention the Wiretrustee SATA towards the end of the post; it looks like they'll have a laser-cut acrylic case in two sizes, for either 2.5" or 3.5" drives. I'm hopeful one of the mini ITX or micro ATX boards goes into production soon; at that point, you could mount the Pi securely into any bog-standard ITX/ATX case and even make use of PSU connectors and the PCIe slot!
- tssva 6y ago
  I mentioned it in a comment for your original hardware video but I'll repeat it here. I think another setup worth trying is an eSATA 4 drive enclosure connected to the SATA controller board via a SATA to eSATA cable.
guilhas 6y ago
And for 50euros you can get a second hand HP microserver on eBay
- xnyan 6y ago
  Power consumption quickly becomes relevant in terms of cost when you are running it 24/7. I measure about 33 kWh a year per RasPi I run, my undervolted x86 2014 intel platform is easily over 15x that.

deleted 6y ago

[deleted]

dshep 6y ago

Pretty neat. I have been thinking about how I can build a tiny NAS, but I would like ECC so that rules out the pi, etc. Best option I have found so far might be this ASRock 4x4-v2000 https://www.asrockind.com/en-gb/4X4-V2000M https://www.asrockind.com/en-gb/4X4-V2000M , it has an 8-core cpu and supports ECC. Would need to get a M.2 to 4x SATA adapter. The hard part seems to be figuring out how to buy the board itself...

Scene_Cast2 6y ago
I've been using a Helios4 for this; it's tiny, has ECC, and plenty of speed for a NAS. Their new board, the helios64, is bigger, and ECC support was in-progress last time I checked.
- sbierwagen 6y ago
  Doh, wish I knew this existed last month. Just built a NAS around an ITX board.
- vladvasiliu 6y ago
  The Helios64 "Full Bundle" looks nice, but only 4 GB RAM seems limited for ZFS. Plus, for 300 USD, there should be a way to set up a similar x86 system, possibly from second-hand parts. I'm thinking some older AMD Zen with 8-16 GB RAM and a Fractal Design Node or similar case. Yes, I know that would likely use more electricity, but in my particular case (France), electricity isn't that big of a cost, plus I wouldn't be running this 24/7. More like one or two evenings a week. The only reason why I'm not running recycled "enterprise" servers at home is that my apartment is small and I absolutely cannot stand the noise they make.
  - MivLives 6y ago
    I've had good luck buying old gaming pc parts off my friends. If they're upgrading their cpu, they normally have a mobo/ram/cpu to sell.
    - 6y ago

geerlingguy 6y ago

One of the little gems I found while I was putting together the video for this post was "Shouting in the datacenter", an older clip (from the pre-HD era) demonstrating the consequence of doing exactly what the title says: https://youtu.be/tDacjrSCeq4 https://youtu.be/tDacjrSCeq4 Warning: loud!

sliken 6y ago
I graphed latency to load 4KB or somesuch across the entire disk for an enterprise disk and a consumer disk while on a bench (no vibration). Then I did the same benchmark with the disk mounted in a 1U chassis with a bunch of those annoying 15k rpm fans. Both showed a big difference, but the enterprise drive was 90+% of the no vibration case, the consumer drive was like 30%.

ww520 6y ago

Great info in the article. Had something similar. The firmware update of my sse4200 NAS has stopped and its NetBios version is badly outdated. I was looking for an inexpensive way to reuse the disks to build another NAS. After looking around a bit, ended up getting a RAID capable DAS enclosure for the disks, repurposed an old Chromebox to connect to it via usb, installed Linux, and shared over SAMBA. It works well.

Youden 6y ago

Does anybody know why support for NBase-T is so uncommon and expensive in networking equipment? These 2.5Gbps and 5Gbps adapters look really nice, especially for laptops and other devices with USB 3.2 ports but it appears to be near impossible to affordably connect them to a more-than-trivial network.

PragmaticPulp 6y ago
1G has been around so long that it's cheap commodity hardware now. 10G was too much of a leap, IMO, so the hardware remains expensive. 10G over regular Cat5e is also relatively power hungry. 2.5G and 5G arrived relatively recently to fill the gap with the stalled jump to 10G. You can get started with something like a $110 QNAP 2.5G 5-port switch and some cheap 2.5G adapters on the computers you care about.
- tutfbhuf 6y ago
  > 10G was too much of a leap, IMO, so the hardware remains expensive. Switches are still quiet expensive, but 10G cards are quiet affordable nowadays. Notebooks can use a Thunderbolt 3 to 10G ethernet, but I think we will see 10G ports one consumer notebooks soon. I wouldn't invest in a bridge technology like 2.5G since it introduces non-trivial costs an offers only 2.5x speed up instead of 10x.
  - PragmaticPulp 6y ago
    > but I think we will see 10G ports one consumer notebooks soon Consumer notebooks are moving away from having ethernet ports in general. 10GBASE-T is relatively power hungry, requiring a couple of watts per port. I don't think we'll be seeing this in consumer notebooks any time soon. Also, I have one of those Thuderbolt to 10G adapters. It's not small. It's only useful when I'm doing bulk file transfers. 2.5G is fine for most work. Many consumer NAS boxes can barely saturate 2.5G anyway.
    - hutrdvnj 6y ago
      > Consumer notebooks are moving away from having ethernet ports in general. I might be wrong, but I don't think that every notebook will be as slim as the height of a USB-C port and with only one or two USB-C ports as only ports. This might be true for Apple and some companies that copy Apple. But there are still consumers notebooks that don't follow that trend and build some bulkier books with more ports and better thermals. > 10GBASE-T is relatively power hungry, requiring a couple of watts per port. I don't think we'll be seeing this in consumer notebooks any time soon. It all depends on customers needs. I would rather have a 10GBASE-T port than a dedicated GPU in my book, and I think the latter even consumes more watts.

WhatIsDukkha 6y ago

Are there performance issues for this with BTRFS as well or just ZFS?

atmosx 6y ago
I don't know about BTRFS but ZFS is expensive in terms of RAM and since ram is scarce on RPis, I would expect ZFS to underperform under RPis.
- tpxl 6y ago
  RPi 4 goes up to 8G RAM which is more than enough.
  - nuker 6y ago
    I found for 5x 10TB disks TrueNAS needs more than 8GB.
    - magicalhippo 6y ago
      TrueNAS eats a lot of memory for itself though. Just the web frontend thing eats 3GB on my box, and "services" eats about 12-16GB total (most of which I have no idea where is going). Since ZFS uses memory for caching, it will perform much better with more memory, but it should work[1] with less. [1]: https://www.reddit.com/r/DataHoarder/comments/5u3385/linus_tech_tips_unboxes_1_pb_of_seagate/ddrngar/ https://www.reddit.com/r/DataHoarder/comments/5u3385/linus_t...
  - atmosx 6y ago
    is it? $ zpool status pool: storage state: ONLINE scan: resilvered 854G in 04:18:32 with 0 errors on Sat Jun 20 18:30:06 2020 config: NAME STATE READ WRITE CKSUM storage ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sdc ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 sdb ONLINE 0 0 0 sde ONLINE 0 0 0 errors: No known data errors $ grep -e '^size' /proc/spl/kstat/zfs/arcstats size 4 8604931056 $ bc -q 8604931056 / (1024*1024*1024) 8 On could tune ARC opts and whatnot, but I'm not sure how reliable and fast the system will turn out to be.

rasz 6y ago

tldr: still worse than store bought NAS, or even 10 year old Lenovo x220 with two mpcie/Expresscard to pcie adapters.

Naac 6y ago
Store bought NAS doesn't run open source software.
- tyingq 6y ago
  Some of them can be coaxed into running normal distros. Buffalo Linkstations, for example.
walterbell 6y ago
Any recommendations on PCIe adapters which work with both slots on the X220, since only one of them is ExpressCard?
- rasz 6y ago
  I just solder directly to the motherboard, there are 3 usable PCIEx1 interfaces on x220, one at the missing USB 3.0 controller footprint, second one mpcie, third ExpressCard. You could also desolder memory card controller for fourth pcie lane. Then you have 3x SATA controllers, one HDD, one on mpcie socket, and third on the Dock connector. You also get over 10 individual USB 2.0 channels. There was a time before Sandy Bridge you could configure pcie bifurcation and freely join separate lanes, but I dont think anyone figured out how to do it on x220.
  - walterbell 6y ago
    Thanks for the pointers on dock/SATA: https://forum.thinkpads.com/viewtopic.php?t=119738 https://forum.thinkpads.com/viewtopic.php?t=119738 & https://www.reddit.com/r/thinkpad/comments/6qomsx/51nb_extra_msata_30_in_the_x230/ https://www.reddit.com/r/thinkpad/comments/6qomsx/51nb_extra... Have you considered writing a forum/blog post or Tell HN on your X220 NAS mods?

ggm 6y ago

8gb pi4, radxa quad sata HAT and zfs, working fine. The memory thing is a historical misunderstanding about how the arc works. Sure, it's slow compared to direct pci attached disc controllers, but as a vault, it's working fine. https://wiki.radxa.com/Dual_Quad_SATA_HAT https://wiki.radxa.com/Dual_Quad_SATA_HAT

0x0000000 6y ago
Where do you buy this? I've been searching periodically for months and it always seems unavailable.
- ggm 6y ago
  I too want to buy (another!) And it looks like all supplies worldwide are dried up. The rockpi 5 port variant may still be available. I bought mine in 2020 from allnet, direct to Australia from China, worked fine.