7 ms·

My 2023 all-flash ZFS NAS (Network Storage) build

158 points by signa11 3y ago

zamadatix 3y ago
I did something similar this year with 12x 4tb NVMe drives. I had a 12x 2tb setup is an fashstor 12 but the CPU and RAM limitations led me to traditional RAID there. For the second NAS I just went standard parts but with extreme performance, 13900K and the like. To get the necessary PCIe lanes (since not every port would bifurcate) I used 3 4 way switches from aliexpress. Choosing a motherboard with 2 x8 cpu slots made the bandwidth not a problem, the third does hang off the demo 4.0 lanes though. The boot drive hangs directly off the cpu in the x4 slot. The 13900K actually supports ECC in the workstation motherboards as well. I just expose everything via NFS and call it a day All in all I'm happy with it. I still use spinning rust for the backup though, no sense worrying about that. At the time it was cheaper to get 12x4 TB budget NVMe and the extra parts than go for any reasonable count of SATA SSDs, not sure if that's still true or not. SATA definitely would have been easier, e.g. stuff like in the Flashstor 12x2TB build I had to submit a kernel patch because the cheap drives were duplicating their nsids.
- magicalhippo 3y ago
  > the CPU and RAM limitations led me to traditional RAID there IIRC there was some talk that RAID-Z lead to write amplification compared to mirrored drives, thus not good for cheaper SSDs. Haven't had time to sit down and think it through, does anyone know if thats right or wrong? > I used 3 4 way switches from aliexpress Did you find any that were significantly cheaper than ~100 USD?
  - zamadatix 3y ago
    In general I've found cheap SSDs these days have better endurances compared to cheap spinning disks anyways. https://www.servethehome.com/discussing-low-wd-red-pro-nas-hard-drive-endurance-ratings/2/ https://www.servethehome.com/discussing-low-wd-red-pro-nas-h.... I think the drives I got were 1.6 PBW rated, no errors or failures yet. Beyond that, my workload isn't write heavy enough for this to be a concern of mine whether or not there is amplification. That SSDs don't count reads or active drive time in the endurance rating was more important. For the Flashstor build I did RAID4 instead of RAID5 just because I could though. I want to say that sounds about right on price. If I didn't also have a desire for high single core performance at the same time a used/old Threadripper/Epyc build and bifurcation would probably make more sense. I also disconnected the onboard tiny "definitely going to be noisy as hell in 3 months" low quality fans from them and just rested a 140mm blowing down across the top of the 3 cards at 30% speed. Temps of the controllers and SSDs became better and it's dead silent.
- gigel82 3y ago
  Can you expand more on the hardware used? Specifically, which 4 way switches you used? I have a bunch of unused M.2 drives from decommissioned servers that I'd love to use as additional storage.
  - zamadatix 3y ago
    I started with this page for adapters people had tried and recommended here https://forums.servethehome.com/index.php?threads/multi-nvme-m-2-u-2-adapters-that-do-not-require-bifurcation.31172/ https://forums.servethehome.com/index.php?threads/multi-nvme... but I think I ultimately went with the LinkReal x8 to quad. If you don't specifically want the high per core performance of something like a 13900k a used/old Threadripper or Epyc system and bifurcation might make more sense. It'll also enable you to get maximum per drive bandwidth, if that's a concern for your (when you have 12 drives in some form of stripe and parity the per drive bandwidth ends up not being that important for most sane workloads though).
- robbintt 3y ago
  What brand of 4tb nvmes are you using?
  - zamadatix 3y ago
    I scooped up some MP34s on sale. The have something like a 1.6 PBW endurance rating and I haven't had a single one fail or start throwing errors yet (though my workload isn't as write heavy as many might have).
    - robbintt 3y ago
      That's cool. I use some 4 tb silicon power brand drives. I didn't research them much. Prices are definitely going to fall soon on 8 tb, and I'll likely retrofit things pretty quickly with those.
temp0826 3y ago
Why disable swap? And...only 8gb of ram (am I misreading that?) with zfs? It's been years since I've needed a bunch of storage but I remember zfs being much happier with gobs of ram.
- e12e 3y ago
  IIRCC it's not recommend to have swap on a zvol. So you would need a non-zfs managed partition/device for swap. Not sure about current status? https://github.com/openzfs/zfs/issues/7734 https://github.com/openzfs/zfs/issues/7734 https://github.com/openzfs/zfs/issues/342 https://github.com/openzfs/zfs/issues/342 https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSForSwapMyViews https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSForSwapM...
  - temp0826 3y ago
    Swap on a zvol I understand, disabling it entirely seems questionable, no?
    - psanford 3y ago
      This is a dedicated NAS. What workload do you expect to be paging to disk? Your primary workload is already reading and writing bits off disk.
      - temp0826 3y ago
        Because we live in the real world and no system is truly in a bubble if it's connected to a network. But mostly because seeing oom-killer messages in dmesg makes me sad :).
        Dylan16807 3y ago
        If something gets leaky enough then it's still going to hurt performance for a while and die. Swap delays that but extra RAM delays it too. If you take a use case where 2GB of memory is fine, and give it 8GB, then you already solved the problem swap would solve. You can always add more but you're past the point of diminishing returns.
Bluecobra 3y ago
This is really cool. I did a similar thing years ago with FreeBSD + 1TB WD Green drives on an HP Microserver. It had a AMD CPU similar to an Intel atom so power consumption was pretty low. Overall I was really happy with ZFS and it worked well. I ended up using jails to have it perform various tasks like using pf as a firewall.
aborsy 3y ago
An off the shelf NAS might be better. Some of them allow installing your own operating system such as TrueNAS. These boxes come with the right form factors, CPU, RAM, number of NICs, power consumption, etc. Solutions such synology provide web interface, phone apps, and applications for syncing, photo management and backup. The units come with software for domain name, SSL certificates, monitoring, drive management, WoL, etc. There is a lot of software functionality useful for NAS built in.
moondev 3y ago
Has anyone built or attempted to build their own flash array, as in 20+ nvme u.2 disks in a hot swap chassis and consume via nvme-of or nvme-tcp I see there are tri-mode backplanes but looking for something that skips the hba and connects via something like occulink directly?
- rtp4me 3y ago
  We are currently testing a number of systems with 12x 30TB NVMe drives with Debian 12 and ZFS 2.2.0. Each of our systems have 2x 128G EPYC CPUs, 1.5TB of RAM, and a dual-port 100GbE NICs. These systems will be used to run KVM VMs plus general ZFS data storage. The goal is to add another 12x NVMe drives and create an additional storage pool. I have spent an enormous amount of time over the past couple of weeks tuning ZFS to give us the best balance of reads-vs-writes, but the biggest problem is trying to find the right benchmark tool to properly reflect real-world usage. We are currently using FIO but sheer number of options (depth queue, numjobs, libaio vs io_uring) makes the tool unreliable. For example, comparing libaio vs io_uring with the same options (numbjobs, etc) makes a HUGE different. In some cases, io_uring gives us double (or more) performance than libaio, however, io_uring can produce numbers that don't make any sense (eg: 105GB/sec reads for a system that maxes out at 72B/sec). That said, we were able to push > 70GB/secs large-block reads (1M) from 12x NVMe drives which seems to validate ZFS can perform well on these servers. OpenZFS has come a long way from the 0.8 days, and the new O_DIRECT option coming out soon should give us even better performance for the flash arrays.
  - mgerdts 3y ago
    If you are seeing unreasonably fast read throughput, it is likely that reads are being served from the ARC. If your workload will benefit from the ARC, you may be seeing valid numbers. If your workload will not benefit from the ARC, set primarycache=metadata on the dataset and rerun your test, potentially with a pool export/import or reboot to be sure the cache is cleared. The fact that fio has a bunch of options doesn’t make the tool unreliable. Not understanding the tool or what you are testing makes you unreliable as a tester. The tool is reliable. As you learn you will become a more reliable tester with it.
willis936 3y ago
A few notes: 1. A raspberry pi was picked as a single point failure point. This will be a headache within 24 months. 2. Daily power/thermal cycles might age some things more than a steady thermal load of always being on. Not a big deal and optimizing power consumption is totally reasonable, but a trade to track.
praseodym 3y ago
I noticed that the author is using ZFS-native encryption, which in my experience is not particularly stable. I've even managed to corrupt an entire pool with ZFS send/receives when trying to make backups. I'd strongly recommend using ZFS-on-LUKS instead if encryption is required. The list of open issues on GitHub in the native encryption component is quite telling: https://github.com/openzfs/zfs/issues?q=is%3Aopen+is%3Aissue+label%3A%22Component%3A+Encryption%22 https://github.com/openzfs/zfs/issues?q=is%3Aopen+is%3Aissue...
- Modified3019 3y ago
  I have to agree, the leadership’s response and handling of errors in regards to encryption has been… honestly pretty disappointing to see as a zfs fan and user. While it’s not common, the amount of people running into edge cases and killing both their sending and sometimes even receiving pools (better have another backup!) is frankly unacceptable. “Raw sends” seem to be especially at risk, though sending in general seems to be where the issues mostly lay. My thoughts mirror the comments here: https://discourse.practicalzfs.com/t/is-native-encryption-ready-for-production-use/532/2 https://discourse.practicalzfs.com/t/is-native-encryption-re... Here’s a currently unmaintained document by Rincebrain that he was using to try to track things before he got burned out by the lack or response. https://docs.google.com/spreadsheets/d/1OfRSXibZ2nIE9DGK6swwBZXgXwdCPKgp4SbPZwTexCg/htmlview https://docs.google.com/spreadsheets/d/1OfRSXibZ2nIE9DGK6sww...
- aborsy 3y ago
  I use it with snapshots and ZFS send, and so far it’s been fine. Truenas users also may use it. But I’m concerned reading these comments. Anyone else experiencing issues?
  - Fnoord 3y ago
    I've been using ZFS with native encryption (Ubuntu Server) but also ZFS with LUKS (Arch, Ubuntu Desktop). Zero issues (though inability to run latest kernel can be annoying, esp on rolling distro). Wouldn't surprise me if write cache has a role in this issue though.
  - 3y ago
ciupicri 3y ago
> My main motivation for using ZFS instead of ext4 is that ZFS does data checksumming, whereas ext4 only checksums metadata and the journal, but not data at rest. There's also dm-integrity [1]: > The dm-integrity target emulates a block device that has additional per-sector tags that can be used for storing integrity information. One more thing: where is the ECC RAM? [1]: https://docs.kernel.org/admin-guide/device-mapper/dm-integrity.html https://docs.kernel.org/admin-guide/device-mapper/dm-integri...
SergeAx 3y ago
Every time I visit Switzerland, I notice how wealthy this country is indeed.
TwoNineA 3y ago
Why not ECC ram? Lot of AM4 boards support it.
- resonanttoe 3y ago
  Doesn't look like the 3000G supports it. https://en.wikichip.org/wiki/amd/athlon/3000g https://en.wikichip.org/wiki/amd/athlon/3000g
  - yasuocidal 3y ago
    AFAIK, all ryzen cpus UNofficial support ECC, its up to the motherboard IIRC.
    - dannyw 3y ago
      No, the 3XXXG series do not support ECC; which simply do not work. This is because the iGPU doesn't support ECC RAM.
      - Dylan16807 3y ago
        > This is because the iGPU doesn't support ECC RAM. What, how? The only part of the chip that touches the ECC bytes should be the memory controller itself. The reads and writes going across the internal fabric should be exactly the same. Also the PRO versions support ECC and seem to have the same iGPU.
- cassianoleal 3y ago
  Only PRO CPUs do though, and they're only available as OEM parts of complete systems, or second-hand that's been ripped out of them. :) On top of that, finding compatible ECC sticks is not easy and they can be quite pricey.
  - TwoNineA 3y ago
    My Ryzen 3900X supports it fine, ASUS Prime Pro 370 and 570 boards. Using 64 gigs of this: https://media.kingston.com/pcn/PCN_KSM26ED8_16ME_C1.pdf https://media.kingston.com/pcn/PCN_KSM26ED8_16ME_C1.pdf
resonanttoe 3y ago
I'd be interested to see a follow up in time to see what the IO wear-n-tear looks like. Also "Using gokrazy instead of Ubuntu Server would get rid of a lot of moving parts. The current blocker is that ZFS is not available on gokrazy. Unfortunately that’s not easy to change, in particular also from a licensing perspective." I don't pretend to get how licensing works, but is OpenZFS and their licensing not an option here? I know its been really tricky with ZFS in general and I don't think its fully answered? but I'm not up to date with it.
quux 3y ago
Am I misreading or did he create a non-redundant pool spanning the 2 SSD drives? I don't think scrubbing will keep him from losing data when one of those drives fails or gets corrupted. Edit: Looked again and he's getting redundancy by running multiple NASes and rsyncing between them. Still seems like a risky setup though.
- prmoustache 3y ago
  On the other hand, I myself use a similar approach. For home purpose I prefer having redundancy across several cheaper machines without raid1,10,5,6,z than a more expensive single machine with disk redundancy. And I'd rather pay the additionnal money on putting ECC ram on all machines.
  - helf 3y ago
    [dead]
- dpedu 3y ago
  It wasn't mentioned in the blog, but you can set `copies=N` on ZFS filesystems and ZFS will keep N copies of user data. This provides redundancy that protects against bitrot or minor corruption, even if your zpool isn't mirrored. Of course, it provides no protection against complete drive failure.
- Too 3y ago
  For home-use this is a reasonable tradeoff. Imagine my non tech-savvy wife for some reason has to get access to the data when one of the NAS has malfunctioned because the zfs pool encryption was badly configured. Explaining "zfs receive" or let alone what zfs, linux and ssh is going to be grounds for divorce. Heck, i don't even want to read man pages about zfs myself on weekends, i have enough such problems at work. Besides, you still want 2 physical locations. It's going to be less optimal and less professional. That's ok, for something as important as backups, keep it boring. Simply starting up the second box is simple, stupid and has well-understood failure modes. Maybe someone like me should just buy an off-the shelf NAS.
- eternityforest 3y ago
spdy 3y ago
And why not buy server grade ssd`s from ebay? You can get them new at the same price point only caveat that you need a motherboard that supports bifurcation.
- formerly_proven 3y ago
  Depending on the vendor you might end up with drives that run custom firmware, which cannot be crossflashed, that might not even work at all in a non-OEM system, that might not be compatible with normal block storage use, that might not support stuff like power management etc. (and allegedly might have reset SMART logs)
  - wil421 3y ago
    You can easily buy the correct price of hardware by searching the forums and following the advice. The Freenas/Truenas forum is full of hardware guides and threads on what 10gb NICs or controllers to buy and when you need it flashed into IT Mode. Most of the resellers will answers questions or even flash firmware. Local Craigslist guys even offered to do it but they don’t know why you’re doing it, the eBay people understand.
    - formerly_proven 3y ago
      GP is specifically about Enterprise SSDs from eBay, which comes with specific caveats that don’t apply to OEM Mellanox cards.
      - wil421 3y ago
        It does look harder than I thought for SSDs. It looks like a 5 year old thread was commented on this year with some details about Samsung drives.[1] [1]https://forums.servethehome.com/index.php?threads/firmware-enterprise-ssds-sm1625-pm853t-and-others.14014/page-2 https://forums.servethehome.com/index.php?threads/firmware-e...
deleted 3y ago
[deleted]
NKosmatos 3y ago
Oh I remember this guy! Michael has a nice blog where he writes interesting posts about hardware, networks and all sort of computer stuff. Sometimes I'm feeling a little bit jealous with all these "toys" he has, but he writes them in a good way (without wanting to show off). If you haven't seen it, read about when he upgraded his internet to 25 Gbit/s fiber: https://michael.stapelberg.ch/posts/2022-04-23-fiber7-25gbit-upgrade/ https://michael.stapelberg.ch/posts/2022-04-23-fiber7-25gbit... Coming back on topic, the idea of having 3 custom made NASes is surely interesting, but apart from learning and experimenting with all these, I don't see a very big advantage from backup/security point of view compared to commecrial NASes (Synology, QNAP). For sure we can all argue here about selection of filesystem (ZFS, btrfs, ext4...), selection of CPU, RAM type and all the others, but it all boils down to what each one wants (and has the money to spare). IMHO I wouldn't go with QVO SSDs and non redundant volumes (especially spanning 2 disks), but hey that's just me :-)
throw0101c 3y ago
At EuroBSDcon 2022, Allan Jude gave the presentation "Scaling ZFS for NVMe": > Learn about how ZFS is being adapted to the ways the rules of storage are being changed by NVMe. In the past, storage was slow relative to the CPU so requests were preprocessed, sorted, and coalesced to improve performance. Modern NVMe is so low latency that we must avoid as much of this preprocessing as possible to maintain the performance this new storage paradigm has to offer. > An overview of the work Klara has done to improve performance of multiple ZFS pools of large numbers of NVMe disks on high thread count machines. […] > A walkthrough of how we improved performance from 3 GB/sec to over 7 GB/sec of writes. * https://www.youtube.com/watch?v=v8sl8gj9UnA https://www.youtube.com/watch?v=v8sl8gj9UnA When ZFS was created spinning rust was still the main thing, and SSDs were gaining popularity, so ZFS created "hybrid storage" pools: * https://cacm.acm.org/magazines/2008/7/5377-flash-storage-memory/fulltext https://cacm.acm.org/magazines/2008/7/5377-flash-storage-mem... * https://web.archive.org/web/20080613124922/http://blogs.sun.com/jonathan/entry/not_a_flash_in_the https://web.archive.org/web/20080613124922/http://blogs.sun.... * https://web.archive.org/web/20080615042818/http://blogs.sun.com/ahl/entry/flash_hybrid_pools_and_future https://web.archive.org/web/20080615042818/http://blogs.sun.... * https://www.brendangregg.com/blog/2009-10-08/hybrid-storage-pool-top-speeds.html https://www.brendangregg.com/blog/2009-10-08/hybrid-storage-... * https://en.wikipedia.org/wiki/Hybrid_array https://en.wikipedia.org/wiki/Hybrid_array Still useful for use cases that lean towards bulk storage (versus IOps).
kubb 3y ago
This guy’s apartment must be packed with servers, PC’s, network cables and the like.
Fnoord 3y ago
> The goals of my NAS setup are: > When hardware breaks, I can get replacements from the local PC store the same day. That wouldn't be my M.O. I don't want to go to a local PC store. The ones remaining are all either expensive, sell shit, or do both. I'd order from Amazon or whoever is cheapest according to a local price comparison website, and receive a replacement next day. It also costs less time having to do shopping (heck I even order groceries online, saving time). Besides, with warranty, it'd take longer than same day replacement. Now, if the guy was running mission critical data solely I'd say 'OK', but nope: > For over 10 years now, I run two self-built NAS (Network Storage) devices which serve media (currently via Jellyfin) and run daily backups of all my PCs and servers. As such, it seems a silly requirement for the average content someone serves with Jellyfin. I use 4x 4 TB HDDs in ZFS RAIDZ2 (running Ubuntu under Proxmox) with one enterprise grade SSD for OS and ZFS cache. All that Jellyfin data doesn't have to reside on any SSD. Not in this setup, and in most setups for home users: neither. You also don't require HA for such, to the point where even RAID is kind of silly. I use restic for daily backups to a NAS in the same city, a Synology with RAID1 btrfs. But I don't back up any media served via Jellyfin. Seems silly, and I don't have fiber upload speed as of yet. The internet has backups of that. Usenet servers, for example. None of these servers have a UPS, but in case a power cut the enterprise SSD (a Samsung NVMe costing more than 2x as much as a consumer version of same amount of storage) ensures the data is consistent. Both systems use FDE. I also have an offsite, offline backup of our most important data. It is encrypted at rest but not I don't regularly rebase which is kind of stupid. But that is on me. Either way, this fellow wants to have some kind of silent setup (for reasons) and yeah then SSDs make sense. But if you want value for GB/EUR or akin, even with the extremely low SSD prices of recent and even with the higher energy prices, then HDDs are still worth it. Especially for data as content served via Jellyfin. I'm pondering about upgrading to more than 1 gbit for LAN but for now I don't see it is worth it. Also given my server (MicroServer 10 Plus) only has two PCIe and one is used for PCIe to NVMe and other for iLO I don't have the bandwidth available, a disadvantage of my current setup. I suppose I could use a USB to 2.5 gbit converter if USB could saturate it but then I'd use USB for high data transfers. It appears to me that because my switch is managed, pair bonding would be a better approach. Which the Synology also supports.
- gosub100 3y ago

158 points by signa11 3y ago

I did something similar this year with 12x 4tb NVMe drives. I had a 12x 2tb setup is an fashstor 12 but the CPU and RAM limitations led me to traditional RAID there. For the second NAS I just went standard parts but with extreme performance, 13900K and the like. To get the necessary PCIe lanes (since not every port would bifurcate) I used 3 4 way switches from aliexpress. Choosing a motherboard with 2 x8 cpu slots made the bandwidth not a problem, the third does hang off the demo 4.0 lanes though. The boot drive hangs directly off the cpu in the x4 slot. The 13900K actually supports ECC in the workstation motherboards as well. I just expose everything via NFS and call it a day All in all I'm happy with it. I still use spinning rust for the backup though, no sense worrying about that. At the time it was cheaper to get 12x4 TB budget NVMe and the extra parts than go for any reasonable count of SATA SSDs, not sure if that's still true or not. SATA definitely would have been easier, e.g. stuff like in the Flashstor 12x2TB build I had to submit a kernel patch because the cheap drives were duplicating their nsids.

magicalhippo 3y ago
> the CPU and RAM limitations led me to traditional RAID there IIRC there was some talk that RAID-Z lead to write amplification compared to mirrored drives, thus not good for cheaper SSDs. Haven't had time to sit down and think it through, does anyone know if thats right or wrong? > I used 3 4 way switches from aliexpress Did you find any that were significantly cheaper than ~100 USD?
- zamadatix 3y ago
  In general I've found cheap SSDs these days have better endurances compared to cheap spinning disks anyways. https://www.servethehome.com/discussing-low-wd-red-pro-nas-hard-drive-endurance-ratings/2/ https://www.servethehome.com/discussing-low-wd-red-pro-nas-h.... I think the drives I got were 1.6 PBW rated, no errors or failures yet. Beyond that, my workload isn't write heavy enough for this to be a concern of mine whether or not there is amplification. That SSDs don't count reads or active drive time in the endurance rating was more important. For the Flashstor build I did RAID4 instead of RAID5 just because I could though. I want to say that sounds about right on price. If I didn't also have a desire for high single core performance at the same time a used/old Threadripper/Epyc build and bifurcation would probably make more sense. I also disconnected the onboard tiny "definitely going to be noisy as hell in 3 months" low quality fans from them and just rested a 140mm blowing down across the top of the 3 cards at 30% speed. Temps of the controllers and SSDs became better and it's dead silent.
gigel82 3y ago
Can you expand more on the hardware used? Specifically, which 4 way switches you used? I have a bunch of unused M.2 drives from decommissioned servers that I'd love to use as additional storage.
- zamadatix 3y ago
  I started with this page for adapters people had tried and recommended here https://forums.servethehome.com/index.php?threads/multi-nvme-m-2-u-2-adapters-that-do-not-require-bifurcation.31172/ https://forums.servethehome.com/index.php?threads/multi-nvme... but I think I ultimately went with the LinkReal x8 to quad. If you don't specifically want the high per core performance of something like a 13900k a used/old Threadripper or Epyc system and bifurcation might make more sense. It'll also enable you to get maximum per drive bandwidth, if that's a concern for your (when you have 12 drives in some form of stripe and parity the per drive bandwidth ends up not being that important for most sane workloads though).
robbintt 3y ago
What brand of 4tb nvmes are you using?
- zamadatix 3y ago
  I scooped up some MP34s on sale. The have something like a 1.6 PBW endurance rating and I haven't had a single one fail or start throwing errors yet (though my workload isn't as write heavy as many might have).
  - robbintt 3y ago
    That's cool. I use some 4 tb silicon power brand drives. I didn't research them much. Prices are definitely going to fall soon on 8 tb, and I'll likely retrofit things pretty quickly with those.

temp0826 3y ago

Why disable swap? And...only 8gb of ram (am I misreading that?) with zfs? It's been years since I've needed a bunch of storage but I remember zfs being much happier with gobs of ram.

e12e 3y ago
IIRCC it's not recommend to have swap on a zvol. So you would need a non-zfs managed partition/device for swap. Not sure about current status? https://github.com/openzfs/zfs/issues/7734 https://github.com/openzfs/zfs/issues/7734 https://github.com/openzfs/zfs/issues/342 https://github.com/openzfs/zfs/issues/342 https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSForSwapMyViews https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSForSwapM...
- temp0826 3y ago
  Swap on a zvol I understand, disabling it entirely seems questionable, no?
  - psanford 3y ago
    This is a dedicated NAS. What workload do you expect to be paging to disk? Your primary workload is already reading and writing bits off disk.
    - temp0826 3y ago
      Because we live in the real world and no system is truly in a bubble if it's connected to a network. But mostly because seeing oom-killer messages in dmesg makes me sad :).
      - Dylan16807 3y ago
        If something gets leaky enough then it's still going to hurt performance for a while and die. Swap delays that but extra RAM delays it too. If you take a use case where 2GB of memory is fine, and give it 8GB, then you already solved the problem swap would solve. You can always add more but you're past the point of diminishing returns.

Bluecobra 3y ago

This is really cool. I did a similar thing years ago with FreeBSD + 1TB WD Green drives on an HP Microserver. It had a AMD CPU similar to an Intel atom so power consumption was pretty low. Overall I was really happy with ZFS and it worked well. I ended up using jails to have it perform various tasks like using pf as a firewall.

aborsy 3y ago

An off the shelf NAS might be better. Some of them allow installing your own operating system such as TrueNAS. These boxes come with the right form factors, CPU, RAM, number of NICs, power consumption, etc. Solutions such synology provide web interface, phone apps, and applications for syncing, photo management and backup. The units come with software for domain name, SSL certificates, monitoring, drive management, WoL, etc. There is a lot of software functionality useful for NAS built in.

moondev 3y ago

Has anyone built or attempted to build their own flash array, as in 20+ nvme u.2 disks in a hot swap chassis and consume via nvme-of or nvme-tcp I see there are tri-mode backplanes but looking for something that skips the hba and connects via something like occulink directly?

rtp4me 3y ago
We are currently testing a number of systems with 12x 30TB NVMe drives with Debian 12 and ZFS 2.2.0. Each of our systems have 2x 128G EPYC CPUs, 1.5TB of RAM, and a dual-port 100GbE NICs. These systems will be used to run KVM VMs plus general ZFS data storage. The goal is to add another 12x NVMe drives and create an additional storage pool. I have spent an enormous amount of time over the past couple of weeks tuning ZFS to give us the best balance of reads-vs-writes, but the biggest problem is trying to find the right benchmark tool to properly reflect real-world usage. We are currently using FIO but sheer number of options (depth queue, numjobs, libaio vs io_uring) makes the tool unreliable. For example, comparing libaio vs io_uring with the same options (numbjobs, etc) makes a HUGE different. In some cases, io_uring gives us double (or more) performance than libaio, however, io_uring can produce numbers that don't make any sense (eg: 105GB/sec reads for a system that maxes out at 72B/sec). That said, we were able to push > 70GB/secs large-block reads (1M) from 12x NVMe drives which seems to validate ZFS can perform well on these servers. OpenZFS has come a long way from the 0.8 days, and the new O_DIRECT option coming out soon should give us even better performance for the flash arrays.
- mgerdts 3y ago
  If you are seeing unreasonably fast read throughput, it is likely that reads are being served from the ARC. If your workload will benefit from the ARC, you may be seeing valid numbers. If your workload will not benefit from the ARC, set primarycache=metadata on the dataset and rerun your test, potentially with a pool export/import or reboot to be sure the cache is cleared. The fact that fio has a bunch of options doesn’t make the tool unreliable. Not understanding the tool or what you are testing makes you unreliable as a tester. The tool is reliable. As you learn you will become a more reliable tester with it.

willis936 3y ago

A few notes: 1. A raspberry pi was picked as a single point failure point. This will be a headache within 24 months. 2. Daily power/thermal cycles might age some things more than a steady thermal load of always being on. Not a big deal and optimizing power consumption is totally reasonable, but a trade to track.

praseodym 3y ago

I noticed that the author is using ZFS-native encryption, which in my experience is not particularly stable. I've even managed to corrupt an entire pool with ZFS send/receives when trying to make backups. I'd strongly recommend using ZFS-on-LUKS instead if encryption is required. The list of open issues on GitHub in the native encryption component is quite telling: https://github.com/openzfs/zfs/issues?q=is%3Aopen+is%3Aissue+label%3A%22Component%3A+Encryption%22 https://github.com/openzfs/zfs/issues?q=is%3Aopen+is%3Aissue...

Modified3019 3y ago
I have to agree, the leadership’s response and handling of errors in regards to encryption has been… honestly pretty disappointing to see as a zfs fan and user. While it’s not common, the amount of people running into edge cases and killing both their sending and sometimes even receiving pools (better have another backup!) is frankly unacceptable. “Raw sends” seem to be especially at risk, though sending in general seems to be where the issues mostly lay. My thoughts mirror the comments here: https://discourse.practicalzfs.com/t/is-native-encryption-ready-for-production-use/532/2 https://discourse.practicalzfs.com/t/is-native-encryption-re... Here’s a currently unmaintained document by Rincebrain that he was using to try to track things before he got burned out by the lack or response. https://docs.google.com/spreadsheets/d/1OfRSXibZ2nIE9DGK6swwBZXgXwdCPKgp4SbPZwTexCg/htmlview https://docs.google.com/spreadsheets/d/1OfRSXibZ2nIE9DGK6sww...
aborsy 3y ago
I use it with snapshots and ZFS send, and so far it’s been fine. Truenas users also may use it. But I’m concerned reading these comments. Anyone else experiencing issues?
- Fnoord 3y ago
  I've been using ZFS with native encryption (Ubuntu Server) but also ZFS with LUKS (Arch, Ubuntu Desktop). Zero issues (though inability to run latest kernel can be annoying, esp on rolling distro). Wouldn't surprise me if write cache has a role in this issue though.
- 3y ago

ciupicri 3y ago

> My main motivation for using ZFS instead of ext4 is that ZFS does data checksumming, whereas ext4 only checksums metadata and the journal, but not data at rest. There's also dm-integrity [1]: > The dm-integrity target emulates a block device that has additional per-sector tags that can be used for storing integrity information. One more thing: where is the ECC RAM? [1]: https://docs.kernel.org/admin-guide/device-mapper/dm-integrity.html https://docs.kernel.org/admin-guide/device-mapper/dm-integri...

SergeAx 3y ago

Every time I visit Switzerland, I notice how wealthy this country is indeed.

TwoNineA 3y ago

Why not ECC ram? Lot of AM4 boards support it.

resonanttoe 3y ago
Doesn't look like the 3000G supports it. https://en.wikichip.org/wiki/amd/athlon/3000g https://en.wikichip.org/wiki/amd/athlon/3000g
- yasuocidal 3y ago
  AFAIK, all ryzen cpus UNofficial support ECC, its up to the motherboard IIRC.
  - dannyw 3y ago
    No, the 3XXXG series do not support ECC; which simply do not work. This is because the iGPU doesn't support ECC RAM.
    - Dylan16807 3y ago
      > This is because the iGPU doesn't support ECC RAM. What, how? The only part of the chip that touches the ECC bytes should be the memory controller itself. The reads and writes going across the internal fabric should be exactly the same. Also the PRO versions support ECC and seem to have the same iGPU.
cassianoleal 3y ago
Only PRO CPUs do though, and they're only available as OEM parts of complete systems, or second-hand that's been ripped out of them. :) On top of that, finding compatible ECC sticks is not easy and they can be quite pricey.
- TwoNineA 3y ago
  My Ryzen 3900X supports it fine, ASUS Prime Pro 370 and 570 boards. Using 64 gigs of this: https://media.kingston.com/pcn/PCN_KSM26ED8_16ME_C1.pdf https://media.kingston.com/pcn/PCN_KSM26ED8_16ME_C1.pdf

resonanttoe 3y ago

I'd be interested to see a follow up in time to see what the IO wear-n-tear looks like. Also "Using gokrazy instead of Ubuntu Server would get rid of a lot of moving parts. The current blocker is that ZFS is not available on gokrazy. Unfortunately that’s not easy to change, in particular also from a licensing perspective." I don't pretend to get how licensing works, but is OpenZFS and their licensing not an option here? I know its been really tricky with ZFS in general and I don't think its fully answered? but I'm not up to date with it.

quux 3y ago

Am I misreading or did he create a non-redundant pool spanning the 2 SSD drives? I don't think scrubbing will keep him from losing data when one of those drives fails or gets corrupted. Edit: Looked again and he's getting redundancy by running multiple NASes and rsyncing between them. Still seems like a risky setup though.

prmoustache 3y ago
On the other hand, I myself use a similar approach. For home purpose I prefer having redundancy across several cheaper machines without raid1,10,5,6,z than a more expensive single machine with disk redundancy. And I'd rather pay the additionnal money on putting ECC ram on all machines.
- helf 3y ago
  [dead]
dpedu 3y ago
It wasn't mentioned in the blog, but you can set `copies=N` on ZFS filesystems and ZFS will keep N copies of user data. This provides redundancy that protects against bitrot or minor corruption, even if your zpool isn't mirrored. Of course, it provides no protection against complete drive failure.
Too 3y ago
For home-use this is a reasonable tradeoff. Imagine my non tech-savvy wife for some reason has to get access to the data when one of the NAS has malfunctioned because the zfs pool encryption was badly configured. Explaining "zfs receive" or let alone what zfs, linux and ssh is going to be grounds for divorce. Heck, i don't even want to read man pages about zfs myself on weekends, i have enough such problems at work. Besides, you still want 2 physical locations. It's going to be less optimal and less professional. That's ok, for something as important as backups, keep it boring. Simply starting up the second box is simple, stupid and has well-understood failure modes. Maybe someone like me should just buy an off-the shelf NAS.
eternityforest 3y ago

spdy 3y ago

And why not buy server grade ssd`s from ebay? You can get them new at the same price point only caveat that you need a motherboard that supports bifurcation.

formerly_proven 3y ago
Depending on the vendor you might end up with drives that run custom firmware, which cannot be crossflashed, that might not even work at all in a non-OEM system, that might not be compatible with normal block storage use, that might not support stuff like power management etc. (and allegedly might have reset SMART logs)
- wil421 3y ago
  You can easily buy the correct price of hardware by searching the forums and following the advice. The Freenas/Truenas forum is full of hardware guides and threads on what 10gb NICs or controllers to buy and when you need it flashed into IT Mode. Most of the resellers will answers questions or even flash firmware. Local Craigslist guys even offered to do it but they don’t know why you’re doing it, the eBay people understand.
  - formerly_proven 3y ago
    GP is specifically about Enterprise SSDs from eBay, which comes with specific caveats that don’t apply to OEM Mellanox cards.
    - wil421 3y ago
      It does look harder than I thought for SSDs. It looks like a 5 year old thread was commented on this year with some details about Samsung drives.[1] [1]https://forums.servethehome.com/index.php?threads/firmware-enterprise-ssds-sm1625-pm853t-and-others.14014/page-2 https://forums.servethehome.com/index.php?threads/firmware-e...

deleted 3y ago

[deleted]

NKosmatos 3y ago

Oh I remember this guy! Michael has a nice blog where he writes interesting posts about hardware, networks and all sort of computer stuff. Sometimes I'm feeling a little bit jealous with all these "toys" he has, but he writes them in a good way (without wanting to show off). If you haven't seen it, read about when he upgraded his internet to 25 Gbit/s fiber: https://michael.stapelberg.ch/posts/2022-04-23-fiber7-25gbit-upgrade/ https://michael.stapelberg.ch/posts/2022-04-23-fiber7-25gbit... Coming back on topic, the idea of having 3 custom made NASes is surely interesting, but apart from learning and experimenting with all these, I don't see a very big advantage from backup/security point of view compared to commecrial NASes (Synology, QNAP). For sure we can all argue here about selection of filesystem (ZFS, btrfs, ext4...), selection of CPU, RAM type and all the others, but it all boils down to what each one wants (and has the money to spare). IMHO I wouldn't go with QVO SSDs and non redundant volumes (especially spanning 2 disks), but hey that's just me :-)

throw0101c 3y ago

At EuroBSDcon 2022, Allan Jude gave the presentation "Scaling ZFS for NVMe": > Learn about how ZFS is being adapted to the ways the rules of storage are being changed by NVMe. In the past, storage was slow relative to the CPU so requests were preprocessed, sorted, and coalesced to improve performance. Modern NVMe is so low latency that we must avoid as much of this preprocessing as possible to maintain the performance this new storage paradigm has to offer. > An overview of the work Klara has done to improve performance of multiple ZFS pools of large numbers of NVMe disks on high thread count machines. […] > A walkthrough of how we improved performance from 3 GB/sec to over 7 GB/sec of writes. * https://www.youtube.com/watch?v=v8sl8gj9UnA https://www.youtube.com/watch?v=v8sl8gj9UnA When ZFS was created spinning rust was still the main thing, and SSDs were gaining popularity, so ZFS created "hybrid storage" pools: * https://cacm.acm.org/magazines/2008/7/5377-flash-storage-memory/fulltext https://cacm.acm.org/magazines/2008/7/5377-flash-storage-mem... * https://web.archive.org/web/20080613124922/http://blogs.sun.com/jonathan/entry/not_a_flash_in_the https://web.archive.org/web/20080613124922/http://blogs.sun.... * https://web.archive.org/web/20080615042818/http://blogs.sun.com/ahl/entry/flash_hybrid_pools_and_future https://web.archive.org/web/20080615042818/http://blogs.sun.... * https://www.brendangregg.com/blog/2009-10-08/hybrid-storage-pool-top-speeds.html https://www.brendangregg.com/blog/2009-10-08/hybrid-storage-... * https://en.wikipedia.org/wiki/Hybrid_array https://en.wikipedia.org/wiki/Hybrid_array Still useful for use cases that lean towards bulk storage (versus IOps).

kubb 3y ago

This guy’s apartment must be packed with servers, PC’s, network cables and the like.

Fnoord 3y ago

> The goals of my NAS setup are: > When hardware breaks, I can get replacements from the local PC store the same day. That wouldn't be my M.O. I don't want to go to a local PC store. The ones remaining are all either expensive, sell shit, or do both. I'd order from Amazon or whoever is cheapest according to a local price comparison website, and receive a replacement next day. It also costs less time having to do shopping (heck I even order groceries online, saving time). Besides, with warranty, it'd take longer than same day replacement. Now, if the guy was running mission critical data solely I'd say 'OK', but nope: > For over 10 years now, I run two self-built NAS (Network Storage) devices which serve media (currently via Jellyfin) and run daily backups of all my PCs and servers. As such, it seems a silly requirement for the average content someone serves with Jellyfin. I use 4x 4 TB HDDs in ZFS RAIDZ2 (running Ubuntu under Proxmox) with one enterprise grade SSD for OS and ZFS cache. All that Jellyfin data doesn't have to reside on any SSD. Not in this setup, and in most setups for home users: neither. You also don't require HA for such, to the point where even RAID is kind of silly. I use restic for daily backups to a NAS in the same city, a Synology with RAID1 btrfs. But I don't back up any media served via Jellyfin. Seems silly, and I don't have fiber upload speed as of yet. The internet has backups of that. Usenet servers, for example. None of these servers have a UPS, but in case a power cut the enterprise SSD (a Samsung NVMe costing more than 2x as much as a consumer version of same amount of storage) ensures the data is consistent. Both systems use FDE. I also have an offsite, offline backup of our most important data. It is encrypted at rest but not I don't regularly rebase which is kind of stupid. But that is on me. Either way, this fellow wants to have some kind of silent setup (for reasons) and yeah then SSDs make sense. But if you want value for GB/EUR or akin, even with the extremely low SSD prices of recent and even with the higher energy prices, then HDDs are still worth it. Especially for data as content served via Jellyfin. I'm pondering about upgrading to more than 1 gbit for LAN but for now I don't see it is worth it. Also given my server (MicroServer 10 Plus) only has two PCIe and one is used for PCIe to NVMe and other for iLO I don't have the bandwidth available, a disadvantage of my current setup. I suppose I could use a USB to 2.5 gbit converter if USB could saturate it but then I'd use USB for high data transfers. It appears to me that because my switch is managed, pair bonding would be a better approach. Which the Synology also supports.

gosub100 3y ago