21 ms·

Building a Budget Homelab NAS Server

• sch00lb0y 4y ago
  Home labs are always fun to build
• amluto 4y ago
  As far as I can tell, building a similar system using NVMe is vastly more complicated. If you can fit everything into M.2 slots, it’s easy. Otherwise you need barely-standardized PCIe cards, backplanes, connectors ordered off a menu of enterprise things with bizarre model numbers, and possibly “RAID” cards even though you don’t actually want hardware RAID.
  • zekica 4y ago
    You can use pcie switches on pcie x4 to 4xM.2x4 drives.
    • willis936 4y ago
      If your motherboard is new enough. I run my home NAS+server on 2013-era enterprise hardware and use a Supermicro AOC-SHG3-4M2P to make it work.
    • loeg 4y ago
      The active pcie switches are a lot more expensive than the passive splitters that only work on systems with pcie bifurcation.
      • amluto 4y ago
        For a large scale storage system (more NVMe devices than PCIe lanes), a switch is mandatory. Or a “RAID” card or another switch-like device.
        • deleted 4y ago
          [deleted]
      • fomine3 4y ago
        Nowadays some recent consumer platforms supports PCIe x16 to x4/x4/x4/x4 bifurcation that should be enough. For RAID, switching (dynamically shares bandwidth between SSDs) is nonsense, isn't it?
  • jandrese 4y ago
    22TB of NVMe drives is going to be a bit more expensive than the system in the article however. I do wonder what the power consumption figures would be though. His system was drawing an annoyingly large amount of power and I suspect that was mostly those HDDs.
• nicolaslem 4y ago
  I also run a home NAS in a Node 304. I went with a supermicro mainboard for ECC support which means I had to swap the three fans that come with the case because the mainboard only supports PWM fans. Non-PWM fans would only spin at full speed otherwise. Regarding the SLOG device, you probably don't need it for a file server, but if you do you can definitely free a drive bay for an HDD by just using double sided tape somewhere like on the PSU. I'm sure it's also possible to put three more HDDs above the CPU, right in front of the exhaust fan. If I had a 3D printer I would try to build something bringing the total to nine HDDs. If you need more SATA ports but are running out of PCIe slots, you may be able to reuse an empty M.2 slot. An M.2 with two lanes of PCIe 3 gives you 5 SATA ports with an adapter[0]. [0] https://www.aliexpress.com/item/1005004223466933.html https://www.aliexpress.com/item/1005004223466933.html
  • mrslave 4y ago
    I occasionally look at the PCIe-to-SATA market and find it confusing. It appears polarized into either cards from a reputable brand and very expensive, even with few SATA ports, or cards from an unknown brand and relatively affordable. What's your experience with this and what can you recommend (2-port or 4-port)? Are the cheap cards safe & reliable or are they to be avoided?
    • magicalhippo 4y ago
      I've been running older SAS cards for years and it's been doing just fine. They go for cheap on eBay. Each SAS port serve four SATA drives, using SAS-to-SATA cables. Just make sure to get one that runs in IT mode or you have to mess with the firmware.
      • vladvasiliu 4y ago
        > Just make sure to get one that runs in IT mode or you have to mess with the firmware. In case some people wonder what "IT mode" is, as I used to some years ago, what you basically want is a card that will expose the drives directly to the OS, as opposed to "volumes". In other terms, if the card is a RAID controller, it may insist on you creating arrays and only expose those. You can circumvent it by creating single-drive arrays, but it's a pain. Some cards can do both, but it's usually not advertised. Non-RAID cards also tend to be cheaper. Others (usually LSI) can be flashed with a non-RAID firmware, but again, it's less of a hassle to not have to do it.
• hddherman 4y ago
  > I see people talking about snapshotting, but I haven’t found a need for it. I already have snapshots in my restic backup solution. They’re not especially convenient, but I’ve been using restic for two years, and I only recall needing to recover data from a snapshot once. The ease at which you can revert mistakes using ZFS snapshots is much better compared to restic. You can pretty much navigate to the correct snapshot on your live filesystem and restore whatever you need to restore. It also makes backups easier as you can just send the snapshots to the backup device (another server or external storage device).
  • linsomniac 4y ago
    This. I'm always going into my backup server and looking in the ".zfs/snapshots" directory to look at the history of how files have changed over the backups. Love restic, but the ZFS snapshots are fantastic.
  • eminence32 4y ago
    I have personally been saved by ZFS snapshots (multiple times!) because sometimes I do dumb things, like running: rm -rf tmp * Instead of: rm -rf tmp*
    • tenken 4y ago
      I never do "rm prefix" in a dir. I always do "rm ./aDir/prefix" for example. This assures I'm not globbing outside a directory (or just a directory) and tries to help assure I'm not shooting my own foot. Yea, i love up 1 directory before I delete anything.
      • mmastrac 4y ago
        > i love up 1 directory It took me a minute, but I assume this should be "move up". This seems like a good habit.
        • tenken 4y ago
          Yes, i typed the reply on my phone and it autocorrected to that. :(
• gravypod 4y ago
  I wish there was a good guide for buying JBOD hba cards. I want to replace my drobo with all SATs ssds.
  • NoNotTheDuo 4y ago
    Is this what you’re looking for? https://forums.serverbuilds.net/t/official-recommended-sas2-hba-internal-external/4581 https://forums.serverbuilds.net/t/official-recommended-sas2-...
  • dsr_ 4y ago
    Get LSI 2000 or 3000 series SATA cards. Several manufacturers make them approximately to the reference spec. The drivers are in the Linux kernel. If they don't come flashed to the IT spec firmware (no RAID capabilities), do that, but the cheap ones usually do. The 4i models sometimes come with ordinary individual SATA connectors; the 8i will have one of two kinds of combo connectors that can accept cables that go to a backplace or breakout cables to ordinary SATA connectors. There you go.
• rajandatta 4y ago
  Thinking of doing the same for a similar scale! Thanks for sharing.
  • mtlynch 4y ago
    Thanks for reading! Glad to hear it was helpful.
• eminence32 4y ago
  Nice build. I recently built my second NAS (from a used R720 from ebay). The total (without disks) is pretty similar to the build documented in this article. Having a large NAS has had an interesting (though predictable) impact on all the computers around it: Pretty much every bit of data lives on the NAS (accessed either by CIFS, NFS, or iSCSI). When I had to reinstall Windows, it was mostly painless because all my important data and Steam games library was on a remote iSCSI disk. When I replaced the drives on my linux servers, I didn't have to backup hardly anything, as I worked almost exclusivly on NFS-mounted directories. When bringing up a new raspberry pi for projects, it also has instant access to more terabytes of storage than it could ever need. Also, for a homelab, getting 10GBe fiber between two machines is surpringly cheap and easy. For certain workloads, it can be a noticable speed boost over 1GBe.
  • Macha 4y ago
    How is performance, especially with regards to load times, if your steam library is mounted remotely? I ask because the difference between a SSD and a hard drive can be massive in this regards, so I'd be really interested to know if the network latency is also a comparable hit.
    • vladvasiliu 4y ago
      I'm not a hardcore gamer by any means, but I really wonder how much influence drives actually have on games. My gaming computer had an old SATA SSD (Samsung 840 Evo IIRC). Some games took ages to load (particularly Fallout 4). I switched to a much faster NVME drive, and subjectively, it's not any faster loading games. I'd say this was a very underwhelming purchase.
      • Macha 4y ago
        There's certainly diminishing returns between "an SSD" and "a faster SSD" (unless the slower one is DRAMless or QLC), but hard drive to SSD is still a big gulf
    • eminence32 4y ago
• jen20 4y ago
  Nice article - though one misstatement is that ZFS dies not allow you add disks to a pool. It does [1] [2], by adding new vdevs. The linked issue is about adding support for expanding existing vdevs instead. [1]: https://openzfs.github.io/openzfs-docs/man/8/zpool-add.8.html https://openzfs.github.io/openzfs-docs/man/8/zpool-add.8.htm... [2]: https://docs.oracle.com/cd/E53394_01/html/E54801/gayrd.html https://docs.oracle.com/cd/E53394_01/html/E54801/gayrd.html
  • linsomniac 4y ago
    It's a hard pill to swallow, adding two more drives in a vdev to get one more drive worth of storage (authors case maxes out at 6 drives, currently has 4). So often you will bite the bullet and just completely rebuild. True RAIDz expansion is something that's supposed to be coming, possibly in Q3 2022, so it may be that by the time one needs to expand a volume, that ability will have landed. That'll be a game changer.
• 627467 4y ago
  How do you handle backups of such amount of data?
  • willis936 4y ago
    I use B2 + E2EE. TrueNAS can push and pull pools to many different options but Backblaze is the cheapest I've found.
  • throw0101a 4y ago
    A 22TB pool can perhaps be backed up to a single 26TB drive (over USB? Thunderbolt?): * https://www.techradar.com/news/larger-than-30tb-hard-drives-are-coming-much-sooner-than-expected https://www.techradar.com/news/larger-than-30tb-hard-drives-... Buy multiple drives and a docking station and you can rotate them: * https://www.startech.com/en-us/hdd/docking https://www.startech.com/en-us/hdd/docking ZFS send/recv allows for easy snapshotting and replication, even to the cloud: * https://www.rsync.net/products/zfs.html https://www.rsync.net/products/zfs.html * https://arstechnica.com/information-technology/2015/12/rsync-net-zfs-replication-to-the-cloud-is-finally-here-and-its-fast/ https://arstechnica.com/information-technology/2015/12/rsync...
    • Macha 4y ago
      However, such a drive is getting heavily into diminishing returns territory. e.g. a 20TB drive from Seagate is $500. A 4TB drive is $70, 8TB is $140. Getting the same spend in smaller capacity drives would give you 28TB in the 4TB drives and 24TB/32TB in the 8TB drives (for $80 under/$60 over). Add in a second to rotate and you're spending $1000 in drives, assuming these 26TB drives replace the 20TB drives at a similar price when they trickle down to consumer hands.
      • trollied 4y ago
        You have to factor in the power usage of having multiple drives spinning. Though I’d agree that smaller drives are better when you have a drive failure, as resilvering is quicker.
• throw0101a 4y ago
  22TB of total capacity is interesting because we're now getting >26TB on single drives: * https://www.techradar.com/news/larger-than-30tb-hard-drives-are-coming-much-sooner-than-expected https://www.techradar.com/news/larger-than-30tb-hard-drives-... Crazy.
  • copperfoil 4y ago
    Yes but there’s a price/reliability/performance trade off. Also, with disks that big failures become qualitatively different. For example, when a disk fails in a mirror, the bigger the disk size the higher the chance the 2nd disk will have unreadable blocks.
    • kstenerud 4y ago
      This is what ZFS scrubbing is for. If a drive develops unreadable sectors, ZFS will alert you.
      • giantrobot 4y ago
        If the drives are in a RAID ZFS will not only alert you but fix the corruption from parity on other disks.
• 2OEH8eoCRo0 4y ago
  Why is there a need to proselytize about ZFS in every thread on this topic?
  • dsr_ 4y ago
    It's almost entirely because people really like technology that not only promises to reward you with excellent features and stability, it follows through on those promises.
  • Deritiod 4y ago
    Because it works so well. At least that's why I talk about it.
  • Joel_Mckay 4y ago
    Because people don't know about cephfs yet, and the silliness of degraded raid setups. i.e. trusting a zfs community edition in a production environment can be a painful lesson. ;-)
    • linsomniac 4y ago
      I'm sure it'd be painful, but let's throw "infrequent" onto your description of the lesson. :-) I've run ZFS for home storage and work backups for ~15 years, across Nexenta, ZFS-fuse, FreeBSD, and OpenZFS, backing up hundreds of machines, and have never lost data on one of them.
    • magicalhippo 4y ago
      I know about CephFS, but performance was abysmal compared to ZFS for a home server. On a single box with 4-8 drives I didn't come close to saturating a 10G link, which ZFS managed just fine. It was also very complex to manage compare to ZFS, with many different layers to consider. I'm sure it shines in a data center, for which it has been designed. But unless something radical has changed in the last year, it's not for a budget homelab NAS.
      • Joel_Mckay 4y ago
• planb 4y ago
  Thanks for including energy usage in the article. I carry USB-C SSDs around the house for backups and storage of archived files. Of course this is a bit of a hassle and I played with the idea of either buying or building a NAS. My current standby consumption for all computer stuff (access points, router, switches, a dozen or so microcontrollers and various SmartHome stuff, but not TVs, running computers or gaming consoles) is already above 100w and I would really like to bring this number down. An extra 30-60w makes it really hard to justify the purchase of a NAS ( that I don’t really need). I thought at least the synonogies would use way less power when not in use, so thanks for making me aware of this.
  • memcg 4y ago
    I agree and plan to buy a Kill A Watt P4460 meter. My HPE Gen 9 servers were free, but I still would like to know the operating cost of a single server.
    • bombcar 4y ago
      Some of that off-lease equipment idles pretty low - be sure to check the bios for all “energy saving” options.
  • Spooky23 4y ago
    Check out used Linux thin clients on eBay. They are in a raspberry pi price point but more performant and unusual. Energy use is very low.
  • mtlynch 4y ago
    Thanks for reading! Yeah, I've never thought much about power consumption, but I've done a few write-ups of previous builds, and I received a lot of questions about power draw, so I decided to measure it on this one. I was surprised at how much power the system consumed, and it will be something I think about more up-front on future builds.
  • KaiserPro 4y ago
    There are number of ways around this. If you are not after speed, then you can do a redundant array of cheap nodes. Instead of using raid, just shove in an 8-12tb disk in a number of thin clients. The key is that they spend most of the time turned off.
• farmerstan 4y ago
  I think using 1 disk redundancy is a mistake. It’s not only physical failure you’re worried about, it’s an error upon rebuild when you lose a drive. Bit rot on your remaining drives can occur which wouldn’t be detected until rebuild time when you lose a drive, and that could cause you to lose your entire volume. Bit rot can be checked for but you can’t always be sure and with larger and larger sets of data it gets slower to do. I use raid 6 and also backup my data externally to another nas as well as backup to a static usb drive. Backup requires multiple different types since failures are so catastrophic and can occur in ways you don’t expect.
  • bobcostas55 4y ago
    Doesn't ZFS have a mechanism for periodically checking for and correcting bit rot?
    • loeg 4y ago
      Yes.
  • kalleboo 4y ago
    > Bit rot on your remaining drives can occur which wouldn’t be detected until rebuild time ZFS can perform periodic scrubs to detect and repair bit rot, and I'm pretty sure TrueNAS is configured to do this by default
• NicoJuicy 4y ago
  > I chose raidz1. With only a handful of disks, the odds of two drives failing simultaneously is fairly low. Only if you buy different hard drives or at least from different production batches. I had a lot of trouble on the same premise and I won't make that mistake again. Edit: He mentioned it though ( a bit later in the article) > The problem is that disks aren’t statistically independent. If one disk fails, its neighbor has a substantially higher risk of dying. This is especially true if the disks are the same model, from the same manufacturing batch, and processed the same workloads. Given this, I did what I could to reduce the risk of concurrent disk failures. > I chose two different models of disk from two different manufacturers. To reduce the chances of getting disks from the same manufacturing batch, I bought them from different vendors. I can’t say how much this matters, but it didn’t increase costs significantly, so why not?
  • kalleboo 4y ago
    > I had a lot of trouble on the same premise and I won't make that mistake again. Please elaborate, I'd love to hear your story! I hear a lot of advice around raid/z-levels and it often seems backed up by shaky math that doesn't seem to be backed up by reality (like the blog posts that claim that a rebuild of an array of 8 TB drives will absolutely have hard read errors, no exceptions, and yet monthly ZFS scrubs pass with flying colors?)
    • NicoJuicy 4y ago
      I decided on Raid 5 and 2 drives broke within the week. Not much to tell otherwise though. I was really annoyed with it at the time.
• Mister_Snuggles 4y ago
  I've gone the TrueNAS route, but I'm running it on a QNAP TS-451. I'm running TrueNAS off of a USB stick hanging off the back, so I didn't have do to anything with the hardware and reverting back to QTS is just a matter of setting the boot order in the BIOS. I really like seeing other people's builds, but I know that building my own computer isn't something I want to do. I was happy to see the comparison between the DIY model and the roughly-equivalent commercial units. I'll likely buy another QNAP (to run TrueNAS on) when the time comes, and the comparison tells me that I won't get screwed too badly by doing so.
  • mtlynch 4y ago
    Thanks for reading! >I've gone the TrueNAS route, but I'm running it on a QNAP TS-451. I'm running TrueNAS off of a USB stick hanging off the back Oh, I didn't realize that QNAP allows that. Synology makes it pretty hard to boot any other OS, and I assumed the other vendors were similar. I'll keep that in mind for my next build because I do have fun building servers, but I also really appreciate systems like Synology and QNAP where the hardware and case is optimized for the NAS use-case.
• Deritiod 4y ago
  I don't know Im no longer feel save with >4tb drives and raidz1. I run two offline Nas (I unpower them and do scrabing every month) and have one with raidz2 for all critical things like my photos. To resilver 8tb takes ages and while he wrote his thoughts on it, I was missing the repair risk calculation
• sascha_sl 4y ago
  >If you’re new to the homelab world or have no experience building PCs, I recommend that you don’t build your own NAS. >Before building this system, I had zero experience with ZFS, so I was excited to try it out. Sorry, but this is amusing to me. ZFS on TrueNAS is probably fine, but you're building your production NAS, to replace the Synology device you've become "so dependent on". Don't become dependent on ZFS without knowing the implications! I was facing this choice recently, and I agreed with the other tech savy person in the household that we should just use good old LVM + Btrfs. Not only does it run like a charm, but it also allowed us to switch the LV from single (during the data move) to RAID 1 and eventually to RAID 5/6 with zero issues. It will also be much easier to recover from than ZFS. On another note, it's a bad market to buy NAS drives, especially from Seagate. Seagate Exos drives are at this point in time often cheaper than IronWolf, even non Pro IronWolf. They're slightly more noisy and don't come with the free data recovery, but otherwise they're a straight upgrade over the IronWolf drives.
  • loeg 4y ago
    My tiny NAS is still using ext4 + md raid1. It’s the third incarnation of essentially the same design (previously used raid10 when drives were smaller). When it fills up, I delete some files rather than adding disks.
  • kalleboo 4y ago
    Has someone yet created open source patches for LVM + Btrfs like what Synology does to pierce the layers and use the btrfs checksums as a tie-breaker to tell lvm what disk to trust to repair errors?
    • sascha_sl 4y ago
      I'm not sure. We use the raidintegrity option, which layers dm-integrity on top of the raid. We then run btrfs scrubs periodically to trigger reads on the device. https://man7.org/linux/man-pages/man7/lvmraid.7.html#DATA_INTEGRITY https://man7.org/linux/man-pages/man7/lvmraid.7.html#DATA_IN...
• acheron 4y ago
  I’ve been wanting to build a NAS recently, this looks pretty good. On the other hand, I can’t stand people who say “homelab”. Ugh.
  • pxeger1 4y ago
    Why?
• Jhsto 4y ago
  What is the technical upside of using TrueNAS instead of samba? If you want to optimize for control, it seems a bit weird to me to settle for an "all in one" software stack.
  • Thaxll 4y ago
    Samba config files are painful, it's pretty old school. On top of that you need to setup users/groups etc...
  • Tomdarkness 4y ago
    Not really sure the comparison is valid. TrueNAS uses Samba to provide SMB network shares.
    • Jhsto 4y ago
      I see, so I assume the upside is that it's a time saver. Thanks! I personally wen't with samba on Linux and with btrfs. I was wondering if there's something non-obvious in TrueNAS that I'm missing out on. And to my account, I think my upsides are that: - ability to choose the kernel - no need for SSD for base OS since running off of RAM is rather easy on Linux - samba can run in a container thus a bit more control security-wise - server may run something else as well Of course, this comes with a lot more technical hurdles. More like a side-project than utility really. That's why I was wondering does TrueNAS provide non-obvious upsides that would be lacking in self-rolled one.
      • kalleboo 4y ago
        There are two flavors of TrueNAS - Core and Scale. Core is basically a FreeBSD distro and Scale is basically a Linux distro. They're both a base OS with the typical packages anyone would need for a NAS, with sane defaults + a user-friendly web-based management system. The upsides are that it's plug-and-play for anyone who doesn't want to research all the options available and figure out the various pitfalls on their own. > no need for SSD for base OS since running off of RAM is rather easy on Linux I don't understand this sentence. You're running off a RAM disk with no boot drive? What if you have a power outage? > samba can run in a container thus a bit more control security-wise Core supports FreeBSD jails and Scale supports Docker so you could run samba in a container on either if you're willing to do set it up yourself. > server may run something else as well As before, both have jail/container functionality. I haven't used Scale myself but Core comes with a bunch of "click to install" jail options for stuff like Plex, ZoneMinder, etc. Our machine also runs a Windows VM (ew) and a Wordpress install in a Jail
• mtlynch 4y ago
  Author here. I consider myself an intermediate homelabber and a TrueNAS beginner. I just built my first NAS server, so wrote this to capture everything I wish I'd known at the start. I hope it's helpful for anyone else thinking about building their first NAS server. Any questions or feedback about the post are more than welcome.
  • srinathkrishna 4y ago
    Just wanted to share my appreciation for not just this post but all your work in recent times! Been following your trail since your post about Google promos and the set of useful projects you've been working on since then.
    • mtlynch 4y ago
      Thanks so much for the kind words and for following along!
  • dhzhzjsbevs 4y ago
    Did similar recently. Some suggestions for anyone else looking to do the same: i3 runs a bit cooler than ryzen, still 8 threads. 8tb WD blues (they're SMR at 8 and up). You can find Atx boards with 8 sata ports and dual nvme slots for caching / fast pools.
    • giantrobot 4y ago
      I'd be really careful about SMRs in a RAID. You can end up with no end of performance issues. It's all the downsides of a single SMR drive multiplied by however many drives are in the pools.
      • bcook 4y ago
        I think @dhzhzjsbevs meant that 8TB and higher is CMR. A quick google search seems to support that.
• kalleboo 4y ago
  > And if you’re dumb like me, and you’ve used a Synology-proprietary storage format, you can’t access your data without another Synology system I wonder what he means by this. If he's referring to SHR, then it's just standard mdraid and Synology themselves have instructions on how to mount the volume in Ubuntu https://kb.synology.com/en-us/DSM/tutorial/How_can_I_recover_data_from_my_DiskStation_using_a_PC https://kb.synology.com/en-us/DSM/tutorial/How_can_I_recover... edit: He later mentions encrypted volumes but those are also just using standard encryptfs https://www.impedancemismatch.io/posts/decrypt-synology-backups/ https://www.impedancemismatch.io/posts/decrypt-synology-back... This is one of the reasons I feel comfortable recommending Synology devices - there's not a lot of lock-in
  • mtlynch 4y ago
    Oh, cool! I was referring to SHR. I thought it was a proprietary format and didn't realize you could access it from non-Synology systems. I've updated the post: https://github.com/mtlynch/mtlynch.io/pull/920 https://github.com/mtlynch/mtlynch.io/pull/920
    • AviationAtom 4y ago
      It's pretty cool in that it's a mostly "COTS" implemention. LVM and MD, IIRC.
• CraigJPerry 4y ago
  Back around 2004 there was a technology called VRRP then OpenBSD got a similar thing called CARP - a quick google suggests these still exist but i never see mention of them in my filter bubble for some reason. I was obsessed with making an instant failover cluster. I never managed to get it working exactly how i wanted and it relied on two old UPSs with dead batteries to operate as STONITH devices (they had well supported rs232 interfaces). I sometimes think about investigating that idea again but maybe with raspberry pis and cheap iot plugs.
• kstenerud 4y ago
  I set up my most recent NAS using a TerraMaster unit. It's basically a nifty case (9x5x5 inches) around a low power Intel board with a USB stick for a boot device (which I replaced with a mini 100gb USB SSD). I don't know and don't care about TerraMaster's software (it might be awesome - I have no idea). I just rolled my own NixOS install with ZFS so that I could have a deterministic installation (I've heard good things about the TrueNAS OS as well, but I'm a control freak and like being able to rebuild the entire server with a single command and a config file, so I stick with NixOS). The nice thing is that I essentially got a motherboard, CPU, PSU, and compact case for $350 (for the F2-422). All I had to do was upgrade the RAM (SO-DIMM) and add the drives. I've long since reduced to only two drives for my NAS. At one point I was up to 7 drives before I realized my madness. It's cheap enough to get the storage I need with two mirrored drives, is quieter and uses less energy (I can keep it in the same room), and when I finally outgrow them in 5 years or so, the old drives will be re-purposed as backup via an external USB enclosure I keep around.
  • aborsy 4y ago
    Can you install TrueNAS on it? It’s unclear if one could install TrueNAS on a Synology, QNAP, Teramaster etc. Sometimes hardware is not supported.
    • kstenerud 4y ago
      I don't know, but if I could install NixOS without difficulty, it should be possible. I installed Ubuntu server on it at first and that also worked fine. No tweaking necessary at all. You just flash the standard x64 installer on a USB stick, plug it in, and install like you would on any PC (because it basically is a PC - it even has a working HDMI port). Edit: Looks like someone did a writeup for TrueNAS on a TerraMaster: https://joelduncan.io/freenas-on-terramaster-f2-221/ https://joelduncan.io/freenas-on-terramaster-f2-221/ Also: https://mightygadget.co.uk/how-to-upgrade-the-terramaster-f4-422-to-truenas/ https://mightygadget.co.uk/how-to-upgrade-the-terramaster-f4...
      • aborsy 4y ago
• aborsy 4y ago
  I wonder if it’s possible to back up ZFS (eg, a ZFS server) to btrfs (eg, a synology nas)? I mean backing up the file system (as with ZFS send) not scanning all files (using rsync of restic).
  • dsr_ 4y ago
    The target of a zfs send can be a plain file on another filesystem- btrfs, ext4, xfs, exfat, whatever.
    • aborsy 4y ago
      Yes, but it’s usually not recommended, because the receiver side doesn’t verify that the data is identical to that at the sender side, and a small error could corrupt the file system. ZFS send and receive is a good way to do it, but there is no ZFS send and btrfs receive!
• fuzzy2 4y ago
  I have to say, I'm put off by the power benchmark. I have a way older system with way more stuff (6 3.5 inch HDD, 2 2.5 inch HDD, 2.5 inch SSD, 8-port SAS controller, Intel NIC card) and it idles (all drives spun down) at ~30 watts. When I first bought the system over 10 years ago, ZFS on Linux wasn't really a thing, so I used FreeBSD. I later switched and with the switch came substantial power savings.
  • mtlynch 4y ago
    Oh, that's interesting. TrueNAS is available on Debian now, so I wonder if there would be a big drop in power consumption. Lawrence Systems just ran benchmarks[0] between TrueNAS Core (FreeBSD) and TrueNAS Scale (Debian), but they didn't include power consumption, unfortunately. [0] https://www.youtube.com/watch?v=BoiHHnBDg0E https://www.youtube.com/watch?v=BoiHHnBDg0E
• mmastrac 4y ago
  > I purchased the same model of disk from two different vendors to decrease the chances of getting two disks from the same manufacturing batch. I prefer mixing brands/models instead. Two vendors _might_ get you a different batch, but you could be choosing a bad model. I ended up building mine from three different WD models and two Seagate ones. I'm paranoid and run with two spares.
• sandreas 4y ago
  Building your own NAS Server by hand may be a nice project, but if you would like to get something up and running quickly, you should consider prebuild servers like Dell T* series or HP Microserver. It is real server hardware, supporting ECC RAM and by far less work to build, often providing (semi-)professional remote management. If you plan to build a budget NAS and enough room is not a problem, I personally would recommend to get an old and used Dell T20 Xeon E3-1225v3 with min. 16GB ECC DDR3 RAM, 2x10TB Seagate Exos ZFS RAID and a bootable USB Stick with TrueNAS or if you prefer Linux, TrueNAS Scale / OpenMediaVault. If room IS a problem, you could get a HP Microserver Gen8 or higher with a Xeon the config above. - Server Cost: 150 Bucks - Total Cost: 650 Bucks (150 for server, 500 for HDD) - Power Consumption: 33W Idle, 60W Heavy File Transfer - Silent enough without modding the fans - Ethernet-Transfer-Speed: 110MB/s on 30% System Load I do not own a 10Gbit ethernet card, but I'm pretty sure, transfer speeds with 10Gbit would be acceptable, too.
  • deleted 4y ago
    [deleted]
  • marius_k 4y ago
    what would you recommend for power efficient hdds (when idle)? I have recently built a truenas box with 2x4TB ssd's. But I think I will want to expand it. Currently it runs at 14W idle. If I add 2hdd I expect it to increase to 40W(?). How can I optimize this?
    • sandreas 4y ago
      To answer your question: You could buy WD Red, to save a little noise and power. BUT: You should always consider, that optimizing Power Consumption would mean to buy new hardware. I hardly ever do this. Buying new hardware in many cases is vastly more expensive than keeping your old hw, even if it consumes more power. I'll try to give you an example: - My Dell T20 consumes 33W Idle - There is HW out there, that consumes 14W - diff (19W) - Let's say my Dell costs 150$ vs 600$ (for new hardware) - diff: 450$ - How long can I run my Dell T20 for 450$, without investing in new hardware Calculation: - Dell additional power per year: 0,019kW * 24h * 365 = 166kWh - Estimated cost per year (in germany): 166kWh * 0,30$ = 49,80$ - Means I can run it about 9 years to the break even point Advice: Buy a newer cheap used server consuming 12W in 5 years, and you save a ton of money. AND buying used hardware is better for the environment...
• jazzythom 4y ago
  • the_only_law 4y ago
    > who needs to store 32tb of data? People hosting large private media libraries, I assume. Think people eschew music/movie streaming services and instead download FLACs/videos and stream those over their local network.
    • jazzythom 4y ago
      • jazzythom 4y ago
• benlivengood 4y ago
  About 7 years ago there was an Amazon sale ($300) on Lenovo TS140 towers with the low-powered Xeon chip and ECC RAM and 4 drive bays. Ever since I've been unable to find a similar price point for the same quality, but wanted a backup server. I recently got a Raspberry Pi 4 (8GB model) and external USB hard drive (8TB) mirrored with a s3backer volume on backblaze B2 for about $300 total, and as a backup server it's fast enough (performance limited by the Internet speed to B2) and probably idles at 10W-15W. One of the nice benefits of ZFS native encryption + s3backer is that if I had a total outage locally and needed to recover some files quickly I could mount the s3backer-based zpool from any machine, decrypt the dataset, and pull the individual files out of a filesystem. It's also a weird situation with cloud providers that convenient network-attached block storage is ~10X the price of object storage at the moment but performance can be similar using s3backer.
  • AviationAtom 4y ago
    Appreciate the insights on your S3 backup solution. I will mention that I am one of those folks with a TS140. Love that it's a power sipper. I maxed out the processor and memory, as well as loading it up with two 10 TB rust disks and two 512 GB SSDs.
  • lkxijlewlf 4y ago
    My TS140 is still serving us well. It was (and still is) a heck of a deal.
• DeathArrow 4y ago
  I just use pCloud mounted as a network drive and throw everything I am not currently working on on it. With 10Gbps I have at home, it works wonders. Plus, the storage is unlimited. Plus, it is more resistant to failures and disaster than anything home made. Plus, I don't have to store and take care of another noisy box in my home.
  • red0point 4y ago
    Where did you find the unlimited storage offering? I couldn't find it on their website.
    • aborsy 4y ago
      Lifetime plan on their website.
      • red0point 4y ago
        No that can't be it, it's limited to 2TB. Could you post the link? Thanks!
        • aborsy 4y ago
          Sorry I misread it!
    • DeathArrow 4y ago
      It's 2Tb plan, but unlimited in the sense I can grow it to whatever I need. I got 8Tb with 75% off on Black Friday.
  • mtlynch 4y ago
    Nice, that's a cool setup! What area of the world do you live where you get 10 Gbps to the Internet? Can you reliably get 10 Gbps transfers to pCloud? I got 1 Gbps fiber in the last year, but it's more like 700-800 Mbps in practice. I consider myself lucky to even get that, as my experience before that has always been 100-200 Mbps even on a "1 Gbps" plan. I'm super jealous of people who get a full 10 Gbps Internet connection.
• lazzlazzlazz 4y ago
  My conclusion from this was that the Synology is actually excellent value, and a newer one would likely have been superior on all dimensions (including time spent).
  • ulnarkressty 4y ago
    Right, apart from the entertainment/hobby value I'm not sure I understand these guides. It might be cheaper to build but in the end what you pay for is the software and not spending your time to configure it. At some point I wanted to go the TrueNAS / FreeNAS / OwnCloud etc. route but after seeing the pages upon pages of troubleshooting and lost data horror stories I stuck with a commercial solution.
  • aborsy 4y ago
    It’s hard to beat synology: small form factor, low power, quiet, excellent DSM software, web interface for file browsing, expandable array, a lot of apps including mobile apps for photo backup and backup apps, etc. But Synology doesn’t use ZFS, which is a better filesystem than btrfs. In particular ZFS offers native encryption (instead of the clunky ecryptfs in synology), and allows ZFS send from Linux servers.
• idatum 4y ago
  I use 2 ZFS mirrored 4TB drives mounted on a USB C dual bay device (iDsonix brand) for backing up my ZFS pools. I have a simple script that imports the backup pool and sends snapshots from my main ZFS pools to the backup pool. My question: How do you safely store your physical backup drives/devices? I have a fireproof box, but I don't think it was made for safely storing electronics in the event of a fire.
  • Sylveste 4y ago
    The same solution only powered by POE and buried in a pelican case filled with dessicant sacks in your back garden
    • idatum 4y ago
      :-) To be clear I meant the drives backing up the NAS, not the actual NAS. I think backing up online ultimately is the safest choice, and it takes getting comfortable doing that and being okay with paying a fee. This is for data that I can't lose, like family photos, etc. I started looking into using rclone directly from my FreeBSD NAS device. rclone seems to support many providers.
• termios 4y ago
  this costs as much as every PC i've ever owned put together! my budget homelab is 100% recycled (ewaste): - dual core pc (free) - hard drives (free) - ram (free) - lcd monitor (free) - mdadm + ext4
  • diffeomorphism 4y ago
    What is the power consumption? At the beginning of the year 0.35€/kWh was a good estimate. An extra 10W for one year then costs about 30€. I get the "recycled" motivation, but at that point you might be wasting lots of electricity (and as a result also money).
• js2 4y ago
  New drives are around $16-$20/TB depending on if you catch a sale and are willing to shuck. You can pick up used SAS drives in 3-4TB capacity for around $5/TB. I'm a crazy person, so I built this to hold 11 3TB SAS drives: https://imgur.com/a/JqbBN1p https://imgur.com/a/JqbBN1p https://pcpartpicker.com/list/dLhNvf https://pcpartpicker.com/list/dLhNvf I used a Supermicro MB and ECC RAM. It's not much more expensive and it's nice having IPMI. I personally think it's crazy to forego ECC. The SAS controller, expander, and drives were used, everything else was new. Prices have gone up. The new parts were $638 at the time. The drives were ~$20/ea. The HBA and expander were ~$85 for both. After the fans, cables, extra drive cage and brackets, total cost was around $1K. This hardware is all supported out-of-the-box by TrueNAS. I haven't done the math to figure out when the cost of running this will exceed having purchased higher capacity drives. This is what a typical used SAS drive looks like. 30K hours hours but very few on/off cycles. Zero defect list: https://pastebin.com/WcUYX4JR https://pastebin.com/WcUYX4JR The failed SMART test turned out to be a firmware issue. I had to update the firmware on all the drives. That was a bit of an adventure: https://pastebin.com/j3AGX0xN https://pastebin.com/j3AGX0xN A few drives arrived with a non-zero defect list or otherwise failed burn-in. I contacted the seller on eBay and they sent me replacements w/o any fuss. I'm not necessarily recommending used SAS drives, but I'm not recommended against them either. I will recommend the serverbuilds forum for generally good advice on all this. I think this post got me started: https://forums.serverbuilds.net/t/another-nas-killer-4-0-build-complete-node-804-edition/2386 https://forums.serverbuilds.net/t/another-nas-killer-4-0-bui... The current NAS killer version is 5.0: https://forums.serverbuilds.net/c/builds/18 https://forums.serverbuilds.net/c/builds/18
  • gjsman-1000 4y ago
    The catch with shucking drives from cheap NAS systems like from WD is that those drives have a strong likelihood of being SMR drives instead of CMR. Basically, they'll work fine until something goes wrong requiring, say, a RAID rebuild, and then you'll be out weeks while they rebuild begging that they don't fail in the process because the random write performance is abominable.
    • Nextgrid 4y ago
• krnlpnc 4y ago
  Is anyone running a similar scale setup with cloud storage and rclone? I’ve been considering retiring my 16TB NAS
• xedarius 4y ago
  I have pretty much this existing setup. One thing I'd add is that it's quite noisy. If you have somewhere you can put it, and your house is all cat 6'ed up then great. But if like me you have it in the same room as you, you wil notice it. And it's not the pc, the fractal case has very quite 120mm fans, it's the HDDs.
  • kettleballroll 4y ago
    Are you me? I have almost the exact same build as discussed in the post, and I am super annoyed with how loud the disks are. I have a cronjob that puts them to sleep every night(the NAS is in my bedroom)... For some weird reason they never stop spinning otherwise.
    • Laforet 4y ago
      It really depends on the drives you have. When I upgraded from 3TB to 4TB and 6TB models, I was very much taken aback by how much louder these drives are despite being in the same WD Red CMR product line. Some of their more recent SMR drives have really weird acoustics that always makes me think they are about to fail, but in reality thats just now they should sounds like, yikes.
    • xedarius 4y ago
      Made me chuckle. Until SSD are within the reach of a reasonable price range we're out of luck. I have resorted to leaving my NAS off and turning it on when I need it. For reference here's the drives I have. I don't think they're by any means the loudest and nor are they the quietest. The perception of noise can be a lot to do with surroundings, if you have wooden floor instead of carpet and a whole number of factors. For me the constant whiring is too much. https://www.amazon.co.uk/gp/product/B07H289S79/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1 https://www.amazon.co.uk/gp/product/B07H289S79/ref=ppx_yo_dt...
• layer8 4y ago
  > With only a handful of disks, the odds of two drives failing simultaneously is fairly low. The problem is when the second drive fails while you’re recomputing the parity after having replaced the first faulty drive, a process which may stress the discs more/differently than regular operation, and also tends to take some time. Raidz 2 (or Raid 6) helps to provide some redundancy during that process. Otherwise you don’t have any until the Raid has been rebuilt.
• AviationAtom 4y ago
  This article seems right up my alley, so here are some thoughts: - ZFS is pretty amazing in it's abilities, with it ushering in the age of software RAID over hardware RAID - ZFS shouldn't be limited to FreeBSD. The Linux port has come quite a long way. I'd advise you to use PPA over repo though, as many key features are missing from the version on repos. - TrueNAS is more targeted towards enterprise applications. If you want good utility as a home user then give Proxmox or the like a look. Then you can make it into more than just a NAS (if you're open to it). - If you want to make things even more simple then consider something like UnRAID. - ZFS' snapshotting can really shine on a virtualization server application, with the ability to revert KVM VMs to a previous state in a matter of seconds. Lookup Jim Salter's (great dude) Sanoid project to see a prime example. - I don't recall why, but I've heard that RAIDZ should be avoided, in favor of stripped mirrors.
  • sngz 4y ago
    > If you want good utility as a home user then give Proxmox or the like a look. Isn't proxmox just virtualization? didn't know it can be used as a NAS too.
    • thefunnyman 4y ago
      IMO NAS setup is much more straightforward on truenas. It’s doable in Proxmox inside a VM or LXC, but true as exposes this all directly via a nice UI. I personally use Proxmox with a simple Debian NFS VM, but for less technical users that just want stuff to work I tend to recommend they stick with truenas.
      • sngz 4y ago
        how well does it handle running the NFS VM and using the NAS to serve files to jellyfin / plex running in a VM?
    • AviationAtom 4y ago
      As thefunnyman pointed out, TrueNAS will make for a more user friendly sharing experience, but you could also just load a container up with Samba on Proxmox, to give you the NAS feature you seek. The heavy lifting of software RAID/ZFS would then be handled by Proxmox.
• zamalek 4y ago
  > Power usage A 500W PSU won't necessarily draw more than a 250W PSU, that is merely its maximum sustained load (what the rest of the system asks for) rating. The Bronze 80+ rating is likely part of the problem here, that indicates what the power draw from the wall is compared to what is being provided to your system. Titanium 80+ would net you about 10% reduction in wall power usage. Keep in mind that manufacturers play fast and loose with the certification process and a consumer unit may not actually be what it says on the box, you need to rely on quantitative reviews. Other than that, spend some time in the firmware settings. Powtop also does a great job at shaving off some watts.
  • mrb 4y ago
    "A 500W PSU won't necessarily draw more than a 250W PSU" Mostly true, but not exactly. Most computer PSUs are more efficient when operating around 50% of their rated load. So if a computer consumes 125W internally, a 250W PSU would translate to lower power consumption measured at the wall than a 500W PSU, typically by about 2-5%. For example see the chart https://www.sunpower-uk.com/files/2014/07/What-is-Effciency.jpg https://www.sunpower-uk.com/files/2014/07/What-is-Effciency.... (115 VAC input) : 88% efficiency at 25% load, vs 90.5% efficiency at 50% load. In practice if the consumption is 125W at the PSU's DC output, this translates respectively to 142W vs 138W measured at the wall. This 2-5% difference may not seem much, but it's similar to upgrading 1 or 2 levels in the 80 PLUS ratings (Bronze, to Silver, to Gold, to Platinum, to Titanium).
  • KennyBlanken 4y ago
    Switch-mode PSUs are very inefficient at the low end of their duty cycle. A 250W 80-bronze PSU for a 60W load will be operating at 25% capacity and 82% efficiency or better. A 500W 80-titanium PSU at 60W will be at around 12% and 90% efficiency or better. So, an 8% difference in minimum required efficiency...for a huge increase in cost. It's much better to buy a high "tier" PSU (for reliability and safety), sized so that it spends most of its time at or above 20% duty cycle (which in OP's case would indeed be 250W.) 80-gold is very common in the marketplace and where most people should probably be buying.
    • 4y ago
• louwrentius 4y ago
  > While I obviously don’t want my server to corrupt my data in RAM, I’ve also been using computers for the past 30 years without ECC RAM, and I’ve never noticed data corruption. You never noticed also exactly because you can’t know about data corruption if you don’t run with ECC memory.
  • Jedd 4y ago
    Precisely. My trust in a non-ECC machine is much lower. I've had slow on-disk data corruption that I've tracked down to failing memory in retail-grade hardware, and it's been in a collection of large media files being sync'ed between machines. Those corruptions tend to be un-noticed at the time, and are effectively impossible (without original sources) to resolve later on, but happily also tend to have minimal impact. Either way, I've been using computers longer than TFA and wouldn't build a home lab box without ECC.
    • louwrentius 4y ago
      I should write yet another blog article why ECC memory matters…
• pronoiac 4y ago
  I'm fairly happy with my 4-bay Synology NAS. When I last looked at ZFS, it seemed that piecemeal upgrades - like, upgrade a 4TB drive to 8TB, and get more available space - wouldn't work in ZFS, but it would in SHR, at least if you had more than 2 drives. Having scheduled checks is a good idea: I have weekly short SMART tests, monthly long SMART tests, and quarterly data scrubs. The TinyPilot device looks nifty - it's a Raspberry Pi as a remote KVM switch. I stumbled on that last night as I was banging my head against a familial tech support issue.
  • mrb 4y ago
    Oh, my, you are right, the TinyPilot seems awesome! I see it was developed by the author of this ZFS NAS server blog post. I just ordered one to play with :)
    • pronoiac 4y ago
      Note that The TinyPilot hit the front page with its own post: https://news.ycombinator.com/item?id=31549368 https://news.ycombinator.com/item?id=31549368
• simonjgreen 4y ago
  With over 25 years of large scale *nix sysadmin experience: please please please don't fall in to the trap of thinking RAID5/Z is a good idea. It almost never is. The number 1 trap you fall in to is during rebuild after a failed drive. In order to rebuild every byte on every other drive has to be read. On massive arrays this process invariably throws up additional errors, however this time you might not have the parity data to recover it. This process continues in a snowballing situation. This problem is exacerbated by using unsuitable drives. This author seems to have chosen well, but many choose to select drives for capacity over reliability in a quest for the most TB usable possible. A few years ago there was also the scandal of the WD Red drives that were totally unsuitable for RAID usage. And to make matters worse there is the performance impact. Writing consists of 4 operations: read, read parity, write, write partity. That gives a /4 penalty on the sum of your arrays drives IOPS. RAID6/Z2 gives you slight relief from the above risk, however at the increased cost of an additional performance hit (a /6 penalty) If going RAID(Z), it is generally considered best practice to go for a model that includes a mirror. There are decisions to be made whether you stripe mirrors or mirror a stripe. Personally my preference for reducing complexity and improving quick rebuild is to stripe across mirrors. So that is RAID10. You pair your drives up in mirrors, and then you stripe across those pairs. The capacity penalty is 50%. The performance penalty is close to zero. The author also chose to skip a write buffer (ZIL) drive. This, imo, is a mistake. They are a trivial cost to add (you only require a capacity that gives you the maximum amount of data you can write to your array in 15 seconds (tunable)) and they offer a tremendous advantage. As well as gaining the benefit of SSD IOPS for your writes you also save wear on your data array by coalescing writes in to a larger chunk and buy yourself some security against power cuts etc as faster IOPS give you a reduced likelihood of coinciding with an environmental issue. And if you are especially worried you can add them as a mirrored pair. You can also add SSDs as a cache (L2ARC) drive (I think the author missed this in their article) to speed up reads. In the case of the authors use case this would really help with things like media catalogs etc as well as buffering ahead when streaming media. The ARC in ZFS always happens, and the L1 is in RAM, but a L2ARC is very beneficial. The author did comment on RAM for the ARC and sizing this. ZFS will basically use whatever you give it in this regard. The really heavy use case is if you turn on deduplication but that is an expensive and often unnecessary feature. (An example good use case is a VDI server) Last tip for ZFS: turn on compression. On a modern CPU it's practically free.
• rkagerer 4y ago
  Although I understand, this made me sad: I ultimately decided against ECC RAM Also unsurprised to see load-related issues stemming from his embedded Realtec hardware.
• diekhans 4y ago
  Not used ECC is not a good tradeoff in my experience. The only ZFS corruption I have experienced was direct attached ZFS on a Mac with memory errors undetected by Apple's diagnostics.
• tristor 4y ago
  I would advise using ZFS on Linux over ZFS on FreeBSD. You may find this somewhat surprising if you know my post history being a major FreeBSD advocate, but I have run into a somewhat surprising and persistent (and known, but not to me when I started building) issue with FreeBSD's USB Mass Storage support. This issue does not happen on Linux. This is among several issues I noted which affected my ability to make a budget-friendly homelab NAS. Since you are using an M.2 drive rather than a USB drive for your boot drive, you are not affected by the issue that affected me. But I've reached a point where I would not trust FreeBSD to not have weird and esoteric hardware issues that could affect performance or reliability for storage. I'd recommend using ZFS on Linux (Note, I still use FreeBSD as my primary OS for my personal laptop).
• barbazoo 4y ago
  This is really cool. I've been tinkering, trying to get away from Dropbox and repurposed an old server to SMB share a disk that I occasionally rsync with another disk via ssh. I feel like it's not sufficient to protect against errors. What's a reliable, easy to maintain NAS solution for that purpose, Synology?
  • NegativeLatency 4y ago
    I have a similar setup to yours, but with more disks in the machine and a hot swap bay for offline backups. Did the price comparison for sonology a few years ago and felt it just made more sense to build my own. It’s just the current LTS Ubuntu release and it runs plex, pihole, file sharing, cups print server and some other stuff
• philjohn 4y ago
  Thought I'd chime in here with my low-cost NAS/backup server/home server. It's running in a 2U case I got from servercase UK that takes 6 hard drives, it's running: - Core i3 9100T (35w TDP, configurable down to 25W) - Asrock Rack WS246I (mini itx workstation board, no need for an HBA as there are 8 SATA ports on board, 4 standard and another 4 from the OCuLink) - 32GB ECC DDR4 (2 16 GB sticks) - Solarflare 7 series 10Gb SFP+ NIC (second hand, from ebay) - 6 ironwolf 4TB NAS drives Total cost was just a shade under 1000 GBP and it's racked up with my networking gear in the garage.
• herpderperator 4y ago
  > I chose raidz1. With only a handful of disks, the odds of two drives failing simultaneously is fairly low. Is this how the math works? Does having more drives mean the individual drives themselves are more likely to fail? Is running 4 drives safer than 100?
  • branon 4y ago
    Depends on how you look at it I suppose. The lifespan of a singular disk is likely rather long, but put a dozen of them in the same place and you'll see a failure or two every few years. Of course, we know that having a larger sample size and seeing more failures doesn't _actually_ mean that groups of disks are less reliable, but it could seem that way if you don't think too hard about it.
  • ScottEvtuch 4y ago
    The number of parity drives is often fixed, so the odds of the number of failures being higher than the number of parity drives goes up as you increase drive count.
• UmbertoNoEco 4y ago
  > Based on Backblaze’s stats, high-quality disk drives fail at 0.5-4% per year. A 4% risk per year is a 2% chance in any given week. Two simultaneous failures would happen once every 48 years, so I should be fine, right? Either I misunderstood or there are some typos but this math seems all kind of wrong. A 4% risk per year (assuming failure risk is independent of disk age) is less than 0.1% by week. A 2% risk per week would be a 65% risk per year!! 2 simultaneous failures at the same week for just 2 disks (again with the huge assumption of age-independent risk) would in the order of magnitude of less than 1:10^6 , so more than 20k years(31.2 k years tbc) Of course you either change your drives every few years so the age-independent AFR still holds or you have to model the probability of failure using some exponential distribution like Poisson's. Exercise for the reader to estimate the numbers in that case.
• lvl102 4y ago
  I went down this rabbit hole about a decade ago. Spent a lot of time and money on a home lab. While it’s cool, the payoff is just not there. I switched to Google/AWS a few years ago and never looked back.
• lousken 4y ago
  raidz1 - depending of the type of data, for more important stuff than movies and audio I wouldn't use it it is ok for like an offsite backup that you'll touch maybe once in years, if it blows up one day, just upload a new backup
• hamandcheese 4y ago
  > ZFS doesn’t let you add a new drive to an existing pool, but that feature is under active development. This is not true at all. You can’t add new drives to an existing vdev, but you are free to add new vdevs to an existing pool whenever you want.
• walrus01 4y ago
  I would recommend anyone building a home NAS like this in 2022 to look into buying some slightly older 10GbE network interfaces on ebay (an intel X520-DA2 with 2 x 10Gbps SFP+ ports can be found for $55) as a PCI-E card. It's not hard to exceed the transfer ability of an ordinary 1000BaseT port to a home switch these days. And if you have just a few powerful workstation desktop PCs it's also worth it to connect them at 10GbE to a new switch. here's a fairly typical one. these have excellent freebsd and linux kernel driver support. https://www.ebay.com/itm/265713815725?epid=1537630441&hash=item3dddc710ad%3Ag%3AI9MAAOSweDZijPwO&LH_BIN=1 https://www.ebay.com/itm/265713815725?epid=1537630441&hash=i...
  • semi-extrinsic 4y ago
    Or go a little further and spend ~$200 on a used passive FDR Infiniband switch, $50 per used dual port FDR IB NIC, $40 for a 10 meter fibre optic cable including transceivers at each end. Then run IP over IB on each host and you have a 56 Gbit network that all your applications will just see as another network interface on each host.
    • walrus01 4y ago
      for home use I'd highly recommend sticking with just 10GbE because you're not locking yourself into a dead-end solution of used weird previous gen infiniband stuff. if you get a $200 switch with a few 10GbE interfaces in it you can easily expand things in the future by trunking vlans to another newer 10GbE capable switch, or connecting to a switch that has multi-gig copper ports for access ports to 2.5/5GBaseT capable desktop PCs and laptops, etc. $40 for a 10 meter fiber optic cable is a high price when you can buy LC-LC UPC 9/125 duplex 2 meter cables for $3.50 to $4.70 a piece (or a few cents more for additional meters) and connect them between $20 transceivers. no matter what route someone goes with would recommend buying $30-40 of basic fiber connector cleaning supplies. https://www.fs.com/products/40192.html?attribute=193&id=302696 https://www.fs.com/products/40192.html?attribute=193&id=3026... if one wants to buy used weird previous gen dead-end stuff there are also tons of very cheap 40GbE mellanox ethernet adapters on ebay with the QSFP to go with them, and if you have a place to put a switch that doesn't matter if it's noisy like in a wiring closet somewhere, cheap 1U switches with 40GbE ethernet ports on them that can also be used as individual 10GbE when broken out.
• Vaslo 4y ago
  I have tried a few different OSes but my favorite by far is unRaid. It’s really easy to setup and maintain and it gave me a lot of really good experience with server maintenance and the whole container ecosystem. I bought a 24 drive server chassis and am slowly filling it up. Up to 80 TB now and I only have to have one extra drive for local backup (I also backup to another box that I do periodically).
• wojciii 4y ago
  Good article. He choose a fractal design case which I really like (the company not the specific model). I had all kinds of thermal problems with a too small case that I used for my truenas build. It would turn off without any trace in server logs (I have real server HW and therefore expected something in logs since there is a whole separate computer for this). I changed the case from a NAS case to another fractal desfrign case with lots of space for drives and heatsink. All thermal issues disappeared. I just wanted to warn anyone who is building to take this seriously. Some HW drives generate a lot of heat.
• Teknoman117 4y ago
  I chose the same case for my NAS. Main thing I did different was rather than buying a consumer board, I bought a mini-ITX Xeon-D board from supermicro which had integrated dual 10G NICs, 6x SATA, and an ASPEED IPMI for remote management. Was $400 for that board a few years ago (soldered CPU).
• mrb 4y ago
  I am sure the author will appreciate ditching the proprietary Synology to go instead with a custom ZFS server, as the reliability, recoverability, and feature set of ZFS is quite frankly hard to beat. I have been using ZFS to build my custom NASs for the last... checks notes 17 years. I started back when ZFS was only available on Solaris/OpenSolaris. My builds usually have between 5 and 7 drives (raidz2). However I do not recommend his choice of 4 x 8TB drives in a raidz1. Financially and technically it doesn't make sense. He spent $733 for 24TB usable ($30.5/TB) He should have bought fewer, larger drives. For example 14TB drives sell for $240. So a config with 3 x 14TB in a raidz1 would total $720 for 28TB usable ($25.7/TB). Smaller costs, more storage, one less drive (= increased reliability)! It's win-win-win. Especially given his goal and hope is in a couple years to be able to add an extra drive and reshape the raidz1 to gain usable space, then a 14TB drive then will be significantly cheaper per TB than an 8TB drive (today they are about the same cost per TB). Actually, with only 8.5TB of data to store presently, if I were him I would probably go one step further and go with a simple zfs mirror of 2 x 18TB drives. At $320 per drive that's only $640 total for 18TB usable ($35.6/TB). It's a slightly higher cost per TB (+17%), but the reliability is much improved as we have only 2 drives instead of 4, so totally worth it in my eyes. And bonus: in a few years he can swap them out with 2 bigger-capacity drives, and ZFS already supports resizing mirrors.
  • octopoc 4y ago
    > For example 14TB drives sell for $240 Where? Also, is it worthwhile to buy hard drives explicitly for NAS when you're using ZFS? For example, Seagate has the IronWolf product line explicitly for NAS and cost more.
    • gen220 4y ago
      https://diskprices.com/?locale=us&condition=new&capacity=14-&disk_types=external_hdd,external_hdd25,internal_hdd,internal_hdd25,internal_sshd,internal_sas,external_ssd,internal_ssd,m2_ssd,m2_nvme,u2 https://diskprices.com/?locale=us&condition=new&capacity=14-...
    • mrb 4y ago
      I looked at a popular line (Seagate X16) here: https://www.amazon.com/dp/B07T63FDJQ https://www.amazon.com/dp/B07T63FDJQ - $237.99 right now Drives branded for NAS applications differ slightly from mainstream drives. For example Seagate claims the IronWolf is "designed to reduce vibration, accelerate error recovery and control power consumption" which essentially means the drive head actuators will be operated more gently (reduced vibration) which slightly increases latency and slightly reduces power consumption, and also the firmware is configured so that it does fewer retries on I/O errors, so the disk commands time out more quickly in order to pass the error more quickly to the RAID/ZFS layer (why wait a minute of hardware retries when the RAID can just rebuild the sector from parity or mirror disks.) IMHO for home use, none of this is important. Vibration is only an issue in flimsy chassis, or extreme situations like dozens of disks packed tightly together, or extreme noise as found in a dense data center (see the video of a Sun employee shouting at a server). And whether you have to wait a few seconds vs a few minutes for an I/O operation to timeout when a disk starts failing is completely unimportant in a non-business critical environment like a a home NAS.
• RektBoy 4y ago
  Isn't it illegal to RIP dvds, in your country? Just curious, because I live in country where it's "legal" to download movies, etc., for own use.
• bsder 4y ago
  Unfortunately, he punted on ECC. Is there a pointer to someone who does this but actually goes through the ECC grief? It's really hard to chop through all the ECC "marketing" aka lies from the different motherboard manufacturers. What's a cost effective CPU/mobo/ECC for NAS?
• kennywinker 4y ago
  If your budget server starts with a first step of buying new hardware, I'm going to ignore your advice. A 500W psu? nope. Buy a used thinkcentre with a 250W CPU for $50-$100 and save the planet more of this e-waste.
• RobLach 4y ago
  If you're going for budget, decommissioned data center rack servers work well. I purchased a dual xeon (24 cores total) with 64gb of memory, 12 3.5" bays, dual power supplies, for about $250 from a liquidator. Filling it with HDDs was pricey but you can expand as you need it to spread the expenditure.
  • mobilio 4y ago
    Same here - 32 cores/64G ram/4 3.5" bays, dual power 460w for $300. Results: - idle - 55W - full usage - 200W Not bad for 10 years old server.
    • 6ak74rfy 4y ago
      (Cross-posting from another comment of mine.) I am a newbie with server builds. Can you share an Ebay or Amazon link that can be a good/cheap starting point? Also, will these servers contain everything I need to get up and running (with the exception of hard drives, of course)?
      • mobilio 4y ago
        Here is it: https://www.ebay.de/sch/i.html?_from=R40&_trksid=p2334524.m570.l1313&_nkw=proliant+g8&_sacat=0&LH_TitleDesc=0&_odkw=proliant&_osacat=0 https://www.ebay.de/sch/i.html?_from=R40&_trksid=p2334524.m5... Just check how many RAM they have, how many disk slots and what is their connector (SATA/SAS).
      • RobLach 4y ago
        Some datacenter world lingo also helps: With drive bays you'll see: SFF (small form factors, ie 2.5" drives) and LFF (3.5") Usually written like "12 LFF" for say a server that can hold 12 3.5" drives. Look out for if it comes with drive trays for the bays or not. They're usually surprisingly cheap because there's a bazillion of them out there but it could be $5-$20 each. Then there's SAS / Raid controller stuff you need to look out for in regards to compatibility with your configuration. For example if you're going to run TrueNAS (or whatever it's called nowadays) you'll want to search "XXXXX controller TrueNas compatibility" to stumble upon a thread or some hardware list to hear what people have to say. There actually aren't too many models, or they're clones of each other with a Dell or HP branding label on it, so you're bound to find some stuff, for mainstream stuff sold on ebay anyway. The hardware will typically be at least power-on tested, which for this kind of equipment is usually good enough, but beyond that your experience will vary. There will be typical gremlin stuff like an unseated cable which will throw errors and have you wondering if it's a bad drive, or bad install, or whatever... but I think if you're building your own and bringing older gear back to life, you won't be able to avoid that unless you're buying a pre-packaged NAS box.
• walterbell 4y ago
  Are there current JBOD products in 1U short-depth (15" for wall mounted rack) form factor, e.g. 4 x 3.5" hotswap drive bays with a mini-SAS connection to the unit? This would be useful as a backup device, or low-power NAS when connected to a Linux thin-client with LSI HBA. There were some 1U products which included RAID support, priced around $500, which is a bit much for 1U chassis + SATA/SAS backplane + Pico power supply. 1U chassis with ~11" depth (seems to be a telco standard?) start around $100. StarTech 1U JBOD discontinued, https://www.startech.com/en-us/hdd/sat35401u https://www.startech.com/en-us/hdd/sat35401u Silverstone RS431 JBOD unavailable, https://www.silverstonetek.com/product.php?pid=482&area=en https://www.silverstonetek.com/product.php?pid=482&area=en iStarUSA upcoming (shipping in June) 1U JBOD is $400, http://www.scsi4me.com/istarusa-m-140ss-jb-1u-3-5-4-bay-trayless-jbod-storage-rackmount-chassis.html http://www.scsi4me.com/istarusa-m-140ss-jb-1u-3-5-4-bay-tray... For ~$600, QNAP has an Arm-based 1U short-depth NAS with 2x10GbE and 2x2.5GbE networking, plus dual M.2 NVME slots. Maybe Armbian will run on that SoC, it's part of a supported family. https://www.qnap.com/en-us/product/ts-435xeu https://www.qnap.com/en-us/product/ts-435xeu $100 ODROID M1 SBC has an M.2 NVME slot with 4x PCIe lanes. In theory, this could be bridged to a PCI slot + LSI HBA, within a small case, as a DIY low-power NAS.
• vermaden 4y ago
  Did something similar but with a lot less power consumption and price: - https://vermaden.wordpress.com/2019/04/03/silent-fanless-freebsd-server-redundant-backup/ https://vermaden.wordpress.com/2019/04/03/silent-fanless-fre...
  • tarasglek 4y ago
    These types of builds are a lot more interesting as you imposed some constraints in terms of noise and cost, resulting system is a lot more practical
• gourneau 4y ago
  Cool post thanks for sharing! I wanted to throw out another option, older enterprise server hardware is surprisingly cheap on ebay. https://labgopher.com/ https://labgopher.com/ has a nice page to help you find good deals (I am not affiliated with them). One warning these servers are outrageously loud and power hungry. Also, in the bay area I like the local vendor https://unixsurplus.com/ https://unixsurplus.com/
• trashcorp 4y ago
  https://trash.solar/budget-homelab-server-but-budget/ https://trash.solar/budget-homelab-server-but-budget/
• drewg123 4y ago
  If I was building the same system, I'd use 8GB of RAM and use the NVME drive for an L2ARC. I'd put the root fs into the main zfs dataset. The machine will be faster (~128GB of cache), cheaper and draw less power due to less RAM. I'm not sure if TruNas supports this kind of config, or if they enforce booting from a separate drive. But FreeBSD itself is perfectly happy to boot from ZFS.
• keyle 4y ago
  Sorry but I can't take anyone seriously who buys Seagate as a long term storage solution.
• natmaka 4y ago
  Packing a CPU along with many hard drives into a small box implies cooling, and therefore noise. A large-diameter and rather slow fan fixes this, bumping up the volume. This is often-neglected while designing.
• mlatu 4y ago
  heh, got the same case for my nas, still havent gotten an optical drive though.. but i dont see me buying one either... guess ive got a lot of coasters now
• 8fingerlouie 4y ago
  Oh i've been down this road before, and i ended up with a new Synology, though i keep my lab part on a separate machine and use the Synology for storage only :) Benchmarks are nice, but you're comparing an old NAS to a brand new one with significantly more computer power, and had you compared to a new Synology, you'd probably arrive at the same findings. Parts of you power consumption could very likely be from your choice of hard drives. The 8TB Seagate Ironwolf has an average power consumption of 8.8W [1], where a WD Red 4TB requires at most 4.8W [2]. With 4 drives, that's an additional 16W of power consumption right there. Another thing you could try is to enable powerd and measure if it has any effect on the power consumption. In case it does, you can enable it in TrueNAS Core by adding it as a tunable via the web UI. System -> Tunables -> Add Type = rc.conf Variable = powerd_enable Value = yes FreeBSD Wiki also has some Tuning Parameters [3] [1]: https://www.seagate.com/www-content/datasheets/pdfs/ironwolf-16tb-DS1904-13-1905US-en_US.pdf https://www.seagate.com/www-content/datasheets/pdfs/ironwolf... [2]: https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/internal-drives/wd-red-plus-hdd/product-brief-western-digital-wd-red-plus-hdd.pdf https://documents.westerndigital.com/content/dam/doc-library... [3]: https://wiki.freebsd.org/TuningPowerConsumption https://wiki.freebsd.org/TuningPowerConsumption
• krageon 4y ago
  > budget > 1200+ USD I'm not sure what sort of timeline the author lives in, but that is not budget in mine.
• suavsu 4y ago
  [dead]
• Cyder 4y ago
  The bottleneck on most home labs is the lan. A new spindled hard drive running ext3/4 or ntfs can easily saturate a 1 gig nic. That makes the complications of running an NAS on raid or zfs not worth the time and effort-- unless you like playing with file systems or flooding your lan. I built my own nas a few years back with a multi disk zfs pool and ubuntu running mellanox 10g nics. Fast as it was, the complexity of it all, especially zfs having its own way of doing everything, made it a time hog. Just wasn't worth it. Now I run my nas in a Debian Mint VM with consumer grade 2.5G nic adapters. Simplicity and ease of recovery are my priorities.