29 ms·

My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

419 points by louwrentius 2y ago

n3storm 2y ago
After 20 years using ext3 and ext4 I only lost data when ffing around with parted and did.
- rollcat 2y ago
  There's an infinite amount of ways you can lose data. Here's one of my recent stories: https://www.rollc.at/posts/2022-05-02-disaster-recovery/ https://www.rollc.at/posts/2022-05-02-disaster-recovery/ Just among my "personal" stuff, over the last 12 years I've completely lost 4 hard drives due to age/physical failure. ZFS made it a no-deal twice, and aided with recovery once (once I've dd'd what was left of one drive, zvol snapshots made "risky" experimentation cheap&easy).
  - n3storm 2y ago
    Thanks for the link. I just realised I wrote down "did" instead "dd" :D I used photorec and other utilities with others disks, also FAT.
Tepix 2y ago
Having 24 drives probably offers some performance advantages, but if you don‘t require them, having a 6-bay NAS with 18TB disks instead woukld offer a ton of advantages in terms of power usage, noise, space required, cost and reliability.
- louwrentius 2y ago
  I agree, that’s what I would do today if I wanted the same amount of space, as I states in the article.
- foepys 2y ago
  I'd want more redundancy in that case. With such large HDDs zfs resilver could kill another disk and then you would lose your data.
- prepend 2y ago
  18TB drives didn’t exist back when this setup was designed. Of course a 3 bay with 128TB drives would also be superior, but this comment only makes sense in a few years.
prepend 2y ago
I wish he had talked more about his movie collection. I’m interested in the methods of selecting initial items as well as ones that survive in the collection for 10+ years.
- not_the_fda 2y ago
  My server isn't nearly as big as his, but my collection is mostly Criterion https://www.criterion.com/closet-picks https://www.criterion.com/closet-picks
- lvl155 2y ago
  Off topic but why do people run Plex and movies locally in 2024? Is “iTunes” that bad?
  - prepend 2y ago
    I don’t know about other people but I run Plex because it lets me run my home movie collection through multiple clients (Apple TV, browser, phones, etc). iTunes works great with content bought from Apple, but is useless when playing your own media files from other sources. I just want every Disney movie available so my kids can watch without bothering me.
bevenhall 2y ago
I wouldn't call a single user computer a server if you can "Turn the server off when you're not using it." Not really a challenge or achievement.
lawaaiig 2y ago
Regarding the intermittent power cutoffs during boot it should be noted the drives pull power from the 5V rail on startup: comparable drives typically draw up to 1.2A. Combined with the maximum load of 25A on the 5V rail (Seasonic Platinum 860W), it's likely you'll experience power failures during boot if staggered spinup is not used.
383toast 2y ago
Can someone explain why a single geolocated node makes sense for data storage? If there's a house fire for example, wouldn't all the data be lost?
- Fastjur 2y ago
  I’m guessing that the 71 TiB is mostly used for media, as in, plex/jellyfin media, which is sad to loose but not unrecoverable. How would one ever store that much of personal data? I hope they have an off site backup for the all important unrecoverable data like family photos and whatnot.
  - leptons 2y ago
    I have about about 80TB (my wife's data and mine) backed up to LTO5 tape. It's pretty cheap to get refurb tape drives on ebay. I pay about $5.00/TB for tape storage, not including the ~$200 for the LTO drive and an HBA card, so it was pretty economical.
    - Fastjur 2y ago
      Wow that is surprisingly cheap actually.
      - leptons 2y ago
        I get email alerts from ebay for anything related to LTO-5, and I only buy the cheap used tapes. They are still fine, most of them are very low use, the tapes actually have a chip in them that stores usage data like a car's odometer, so you can know how much a tape has been used. So far I trust tapes more than I'd trust a refurb hard drive for backups. And I also really like that the tapes have a write-protect notch on them, so once I write my backup data, there's no risk of having a tape accidentally get erased, unlike if I plugged in a hard drive and maybe there's some ransomware virus that automatically fucks with any hard drive drive that gets plugged in. It's just one less thing to worry about.
Saris 2y ago
I'm curious what's your use case for 71TB of data where you can also shut it down most of the time? My NAS is basically constantly in use, between video footage being dumped and then pulled for editing, uploading and editing photos, keeping my devices in sync, media streaming in the evening, and backups from my other devices at night..
drzzhan 2y ago
It's always nice to know that people can store their data for so long. In my research lab, we still only use separate external HDD drives due to budget reasons. Last year 4 (over 8) drives failed and we lost the data. I guess we mainly work with public data so it is not a big deal. But, it is a dream of mine to research without such worries. I do keep backups for my stuff though, but only me in my lab.
MrYHdP8nPLiG 2y ago
[flagged]
ggm 2y ago
There have been drives where power cycling was hazardous. So, whilst I agree to the model, it shouldn't be assumed this is always good, all the time, for all people. Some SSD need to be powered periodically. The duty cycle for a NAS probably meets that burden. Probably good, definitely cheaper power costs. Those extra grease on the axle drives were a blip in time. I wonder if backblaze do a drive on-off lifetime stats model? I think they are in the always on problem space.
- bluedino 2y ago
  Long ago I had a client who could have been an episode of "IT Nightmares". They used internal 3.5" hard drives along with USB docks to backup a couple Synology devices...It seemed like 1/10 times when you put a drive back in the dock to restore a file or make another backup, the drive wouldn't power back up.
- louwrentius 2y ago
  > There have been drives where power cycling was hazardous. I know about this story from 30+ years ago. It may have been true then. It may be even true now. Yet, in my case, I don't power cycle these drives often. At most a few times a month. I can't say or prove it's a huge risk. I only believe it's not. I have accepted this risk for over 15+ years. Update: remember that hard drives have an option to spin down when idle. So hard drives can handle many spinups a day.
  - neilv 2y ago
    In the early '90s, some Quantum 105S hard drives had a "stiction" problem, and were shipped with Sun SPARCstations. IME at the time, power off a bunch of workstations, such as for building electrical work, and probably at least one of them wouldn't spin back up the next business day. Pulling the drive sled, and administering percussive maintenance against the desktop, could work. https://sunmanagers.org/1992/0383.html https://sunmanagers.org/1992/0383.html
    - ghaff 2y ago
      Stiction was definitely a thing back in that general period when you'd sometimes knock a drive to get it back to working again.
russfink 2y ago
What does one do with all this storage?
- flounder3 2y ago
  This is a drop in the bucket for photographers, videographers, and general backups of RAW / high resolution videos from mobile devices. 80TB [usable] was "just enough" for my household in 2016.
  - patchymcnoodles 2y ago
    Exactly that. I'm not even shooting in ProRes or similar "raw" video. But one video project easily takes 3TB. And I'm not even a professional.
    - jiggawatts 2y ago
      Holy cow, what are you shooting with!? I have a Nikon Z8 that can output up to 8.3K @ 60 fps raw video, and my biggest project is just 1 TB! Most are on the order of 20 GB, if that.
      - patchymcnoodles 2y ago
        I use a Sony a1, my videos are 4k 100fps. But on the last project I also had an Insta 360 x4 shooting B-Roll. So on some days that adds up a lot.
- andrelaszlo 2y ago
  Especially since it's mostly turned off and it seems like the author is the only user
- rhcom2 2y ago
  For me I have about 35TB and growing in Unraid for Plex/Torrents/Backups/Docker
- tbrownaw 2y ago
2OEH8eoCRo0 2y ago
> The 4 TB HGST drives have roughly 6000 hours on them after ten years. So they mostly sit idle? Mine are about ~4 years old with ~35,000 hours.
- louwrentius 2y ago
  To quote myself from a few lines below that fragment: > My NAS is turned off by default. I only turn it on (remotely) when I need to use it.
  - markoman 2y ago
    I loved the idea around the 10Gb NICs but wondered what model switch are you connecting to? And what throughput is this topology delivering?
    - louwrentius 2y ago
      I don't have a 10Gb switch. I connect this server directly to two other machines as they all have 2 x 10Gbit. This NAS can saturate 10Gbit but the other side can't, so I'm stuck at 500-700 MB/s, I haven't measured it in a while.
fiddlerwoaroof 2y ago
This sounds to me like it's just a matter of luck and not really a model to be imitated.
- louwrentius 2y ago
  It's likely that you are right and that I misjudged the likelihood of this result being special. You can still imitate the model to save money on power, but your drives may not last longer, no evidence for that indeed.
turnsout 2y ago
I’ve heard the exact opposite advice (keep the drives running to reduce wear from power cycling). Not sure what to believe, but I like having my ZFS NAS running so it can regularly run scrubs and check the data. FWIW, I’ve run my 4 drive system for 10 years with 2 drive failures in that time, but they were not enterprise grade drives (WD Green).
- louwrentius 2y ago
  Hard drives are often configured to spin down when idle for a certain time. This can cause many spinups and spindowns per day. So I don't buy this at all. But I don't have supporting evidence that back up this notion.
  - Wowfunhappy 2y ago
    > Hard drives are often configured to spin down when idle for a certain time. This can cause many spinups and spindowns per day. I was under the impression that this was, in fact, known to reduce drive longevity! It is done anyway in order to save power, but the reliability tradeoff is known. No idea where I read that though, I thought it was "common knowledge" so maybe I'm wrong.
    - tekknik 2y ago
      It does, and primarily increases the load / unload cycle count. some WD drives are only rated at a few hundred thousand load / unload cycles. it’s best to buy drives that can handle being on always and leave them on.
  - turnsout 2y ago
    I think NAS systems in particular are often configured to prevent drives from spinning down
  - max-ibel 2y ago
    There seems to be a huge difference between spin-down while NAS is up vs shutting the whole NAS down and restart. When I start my NAS, it takes a bunch of time to be back up: it seems to do a lot of checking on/ syncing off the drives and puts a fair amount of load on them (same is true for CPU as well, just look at your CPU load right after startup). OTOH, when the NAS spins up a single disk again, I haven't notice any additional load. Presumably, the read operation just waits until the disk is ready.
mvanbaak 2y ago
the 'secret' is not that you turn them off. it's simply luck. I have 4TB HGST drives running 24/7 for over a decade. ok, not 24 but 8, and also 0 failures. But I'm also lucky, like you. Some of the people I know have several RMAs with the same drives so there's that. My main question is: What is it that takes 71TB but can be turned off most of the time? Is this the server you store backups?
- ryanjshaw 2y ago
  > What is it that takes 71TB but can be turned off most of the time? Still waiting for somebody to explain this to me as well.
  - leptons 2y ago
    I have a 22TB RAID10 system out in my detached garage that works as an "off-site" backup server for all my other systems. It stays off most of the time. It's on when I'm backing up data to it, or if it's running backups to LTO tape. Or it's on when I'm out in the garage doing whatever project, I use it to play music and look up stuff on the web. Otherwise it's off, most of the time.
- louwrentius 2y ago
  It can be luck, but with 24 drives, it feels very lucky. Somebody with proper statistics knowledge can probably calculate the risk with a guestimated 1% yearly failure rate how likely it would be to have all 24 drives remaining. And remember, my previous NAS with 20 drives also didn't have any failures. So N=44, how lucky must I be? It's for residential usage, and if I need some data, I often just copy it over 10Gbit to a system that uses much less power and this NAS is then turned off again.
  - the_gorilla 2y ago
    We don't really have to guess. Backblaze posted their stats for 4 TB HGST drives for 2024, and of their 10,000 drives, 5 failed. If OP's 2014 4 TB HGST drives are anything like this, then this is just snake oil and magic rituals and it doesn't really matter what you do.
deleted 2y ago
[deleted]
naming_the_user 2y ago
For what it's worth this isn't that uncommon. Most drives fail in the first few years, if you get through that then annualized failure rates are about 1-2%. I've had the (small) SSD in a NAS fail before any of the drives due to TBW.
rkagerer 2y ago
I have a similar-sized array which I also only power on nightly to receive backups, or occasionally when I need access to it for a week or two at a time. It's a whitebox RAID6 running NTFS (tried ReFS, didn't like it), and has been around for 12+ years, although I've upgraded the drives a couple times (2TB --> 4TB --> 16TB) - the older Areca RAID controllers make it super simple to do this. Tools like Hard Disk Sentinel are awesome as well, to help catch drives before they fail. I have an additional, smaller array that runs 24x7, which has been through similar upgrade cycles, plus a handful of clients with whitebox storage arrays that have lasted over a decade. Usually the client ones are more abused (poor temperature control when they delay fixing their serveroom A/C for months but keep cramming in new heat-generating equipment, UPS batteries not replaced diligently after staff turnover, etc...). Do I notice a difference in drive lifespan between the ones that are mostly-off vs. the ones that are always-on? Hard to say. It's too small a sample size and possibly too much variance in 'abuse' between them. But definitely seen a failure rate differential between the ones that have been maintained and kept cool, vs. allowed to get hotter than is healthy. I can attest those 4TB HGST drives mentioned in the article were tanks. Anecdotally, they're the most reliable ones I've ever owned. And I have a more reasonable sample size there as I was buying dozens at a time for various clients back in the day.
- louwrentius 2y ago
  I bought the HGSTs specifically because they showed good stats in those Backblaze drive stats that they just started to publish back then.
lostmsu 2y ago
I have a mini PC + 4x external HDDs (I always bought used) on Windows 10 with ReFS since probably 2016 (recently upgraded to Win 11), maybe earlier. I don't bother powering off. The only time I had problems is when I tried to add a 5th disk using a USB hub, which caused drives attached to the hub get disconnected randomly under load. This actually happened with 3 different hubs, so I since stopped trying to expand that monstrosity and just replace drives with larger ones instead. Don't use hubs for storage, majority of them are shitty. Currently ~64TiB (less with redundancy). Same as OP. No data loss, no broken drives. A couple of years ago I also added an off-site 46TiB system with similar software, but a regular ATX with 3 or 4 internal drives because the spiderweb of mini PC + dangling USBs + power supplies for HDDs is too annoying. I do weekly scrubs. Some notes: https://lostmsu.github.io/ReFS/ https://lostmsu.github.io/ReFS/
anjel 2y ago
Ca't help but wonder how much electricity would have been consumed if you had left it on 24/7 for ten years...
- deleted 2y ago
  [deleted]
- louwrentius 2y ago
  On the original blog it states that the machine used 200W idle. Thats 4,8 KWh a day. 17520 KWh over 10 years? At around 0.30 euro per KWh that's 5K+ if I'm not mistaken.
- nine_k 2y ago
  Why wonder, let's approximate. A typical 7200 rpm disk consumes about 5W when idle. For 24 drives, it's 120W. Rather substantial, but not an electric kettle level. At $0.25 / kWh, it's $0.72 / day, or about $22 / mo, or slightly more than $260 / year. But this is only the disks; the CPU + mobo can easily consume half as much on average, so it would be more like $30-35 / mo. And if you have electricity at a lower price, the numbers change accordingly. This is why my ancient NAS uses 5400 RPM disks, and a future upgrade could use even slower disks if these were available. The reading bandwidth is multiplied by the number of disks involved.
louwrentius 2y ago
Because a lot of people pointed out that it's not that unlikely that all 24 drives survive without failure over 10 years, given the low failure rate reported by Backblaze Drive Stats Reports, I've also updated the article with that notion.
Jedd 2y ago
Surprised to not find 'ecc' on that page. I know it's not a guarantee of no-corruption, and ZFS without ECC is probably no more dangerous than any other file system without ECC, but if data corruption is a major concern for you, and you're building out a pretty hefty system like this, I can't imagine not using ECC. Slow on-disk data corruption resulting from gradual and near-silent RAM failures may be like doing regular 3-2-1 backups -- you either mitigate against the problem because you've been stung previously, or you're in that blissful pre-sting phase of your life. EDIT: I found TFA's link to the original build out - and happily they are in fact running a Xeon with ECC. Surprisingly it's a 16GB box (I thought ZFS was much hungrier on the RAM : disk ratio.) Obviously it hasn't helped for physical disk failures, but the success of the storage array owes a lot to this component.
- louwrentius 2y ago
  The system is using ECC and I specifically - unrelated to ZFS - wanted to use ECC memory to reduce risk of data/fs corruption. I've also added 'ecc' to the original blog post to clarify. Edit: ZFS for home usage doesn't need a ton of RAM as far as I've learned. There is the 1 GB of RAM per 1TB of storage rule of thumb, but that was for a specific context. Maybe the ill-fated data deduplication feature, or was it just to sustain performance?
  - rincebrain 2y ago
    It was a handwavey rule of estimation for dedup, handwavey because dedup scales on number of records, which is going to vary wildly by recordsize.
    - InvaderFizz 2y ago
      Additionally unless it's changed in the last six years, you should pretend ZFS dedupe doesn't exist.
      - rincebrain 2y ago
        Not in a stable release yet, but check out https://github.com/openzfs/zfs/discussions/15896 https://github.com/openzfs/zfs/discussions/15896 if you have a need for that.
throw0101c 2y ago
> This NAS is very quiet for a NAS (video with audio). Big (large radius) fans can move a lot of air even at low RPM. And be much more energy efficient. Oxide Computer, in one of their presentations, talks about using 80mm fans, as they are quiet and (more importantly) don't use much power. They observed, in other servers, as much as 25% of the power went just to powering the fans, versus the ~1% of theirs: * https://www.youtube.com/shorts/hTJYY_Y1H9Q https://www.youtube.com/shorts/hTJYY_Y1H9Q * https://www.youtube.com/watch?v=4vVXClXVuzE https://www.youtube.com/watch?v=4vVXClXVuzE
- chiph 2y ago
  My Synology uses two 120mm fans and you can barely hear them (it's on the desk next to me). I'm sold on the idea of moving more volume at less speed. (which I understand can't happen in a 1U or 2U chassis)
- sss111 2y ago
  just curious, are you associated with them, as these are very obscure youtube videos :D Love it though, even the reduction in fan noise is amazing. I wonder why nobody had thought of it before, it seems so simple.
  - throw0101c 2y ago
    > just curious, are you associated with them, as these are very obscure youtube videos :D Unassociated, but tech-y videos are often recommended to me, and these videos got pushed to me. (I have viewed other, unrelated Tech Day videos, so probably why I got that short. Also an old Solaris admin, so aware of Cantril, especially his rants.) > Love it though, even the reduction in fan noise is amazing. I wonder why nobody had thought of it before, it seems so simple. Depends on the size of the server: can't really expand fans with 1U or even 2U pizza boxes. And for general purpose servers, I'm not sure how many 4U+ systems are purchased—perhaps some more now that perhaps GPUs cards may be a popular add-on. For a while chassis systems (e.g., HP c7000) were popular, but I'm not sure how they are nowadays.
    - 2y ago
MIzOjy4EhuJX 2y ago
[flagged]
bearjaws 2y ago
I feel like 10 years is when my drives started failing the most. I run a 8x8tb array zraid2 redundancy, initially it was a 8x2tb array but drives started failing once every 4 months, after 3 drives failed I upgraded the remaining ones. Only downside to hosting your own is power consumption. OS upgrades have been surprisingly easy.
leighleighleigh 2y ago
Regarding the custom PID controller script: I could have sworn the Linux kernel had a generic PID controller available as a module, which you could setup via the device tree, but I can't seem to find it! (grepping for 'PID' doesn't provide very helpful results lol). I think it was used on nVidia Tegra systems, maybe? I'd be interested to find it again, if anyone knows. :)
- ewalk153 2y ago
  Maybe related to this? https://github.com/torvalds/linux/blob/master/tools/thermal/tmon/pid.c https://github.com/torvalds/linux/blob/master/tools/thermal/...
rnxrx 2y ago
In my experience the environment where the drives are running makes a huge difference in longevity. There's a ton more variability in residential contexts than in data center (or even office) space. Potential temperature and humidity variability is a notable challenge but what surprised me was the marked effect of even small amounts of dust. Many years ago I was running an 8x500G array in an old Dell server in my basement. The drives were all factory-new Seagates - 7200RPM and may have been the "enterprise" versions (i.e. not cheap). Over 5 years I ended up averaging a drive failure every 6 months. I ran with 2 parity drives, kept spares around and RMA'd the drives as they broke. I moved houses and ended up with a room dedicated to lab stuff. With the same setup I ended up going another 5 years without a single failure. It wasn't a surprise that the new environment was better, but it was surprising how much better a cleaner, more stable environment ended up being.
- mapt 2y ago
  "Do you think that's air you're breathing?" This is no longer much of an issue with sealed, helium filled drives, if it ever was.
- stavros 2y ago
  How does dust affect things? The drives are airtight.
  - kenhwang 2y ago
    They're airtight now (at the high end or enterprise level). They weren't airtight not very long ago and had filters to regulate the air exchange.
    - Kirby64 2y ago
      They're not airtight in the true sense (besides the helium filled ones nowadays), but every drive made in the past... 30? 40 years is airtight in the sense that no dust can ever get into the drive. There's a breather hole somewhere (with a big warning to not cover it!) to equalize pressure, and a filter that doesn't allow essentially any particles in.
8n4vidtmkvmk 2y ago
I run a similar but less sophisticated setup. About 18 TiB now, and I run it 16 hours a day. I let it sleep 8 hours per night so that it's well rested in the morning. I just do this on a cron because I'm not clever enough to SSH into a turned off (and unplugged!) machine. 4 drives: 42k hours (4.7 years), 27k hours (3 years), 15k hours (1.6 years), and the last drive I don't know because apparently it isn't SMART. 0 errors according to scrub process. ... but I guess I can't claim 0 HDD failures. There has been 1 or 2, but not for years now. Knock on wood. No data loss because of mirroring. I just can't lose 2 in a pair. (Never run RAID5 BTW, lost my whole rack doing that)
- BenjiWiebe 2y ago
  Looks like you're quite clever actually, if you can get cron to run on a powered off unplugged machine. I think I'm missing something.
  - 8n4vidtmkvmk 2y ago
    Just power, not unplugged. It's simply 0 2 * * * /usr/sbin/rtcwake -m off -s 28800 # off from 2am to 10am "and unplugged" was referring to OP's setup, not mine
  - alanfranz 2y ago
    Some bios and firmware support turning on at a certain time. Maybe cron was a way to simplify.
  - bigiain 2y ago
    I use a wifi controlled powerpoint to power up and down a pair of raid1 backup drives. A weekly cronjob on another (always on) machine does some simple tests (md5 checksums of "canary files" on a few machines on the network) then powers up and mounts the drives, runs an incremental backup, waits for it to finish, then unmounts and powers them back down. (There's also a double-check cronjob that runs 3 hours later that confirms they are powered down, and alerts me if they aren't. My incrementals rarely take more than an hour.)
lvl155 2y ago
I have a similar approach but I don’t use ZFS. It’s a bit superfluous especially if you’re using your storage periodically (turn on and off). I use redundant NVMEs in two stages and periodically save important data into multiple HDDs (cold storage). Worth noting, it’s important to prune your data. I also do not backup photos and videos locally. It’s a major headache and they just take up a crap ton of space when Amazon Prime will give you photo storage for free. Anecdotally, only drives that failed on me were enterprise-grade HDDs. And they all failed within a year and in an always-on system. I also think RAIDs are over-utilized and frankly a big money pit outside of enterprise-level environments.
ed_mercer 2y ago
> Losing the system due to power shenanigans is a risk I accept. A UPS provides more than just that, it delivers constant energy without fluctuations and thus makes your hardware last longer.
- vunderba 2y ago
  Yeah, this definitely caused me to raise an eyebrow. UPS covers brown outs and obviously the occasional temporary power outage. All those drives spinning at full speed suddenly coming to a grinding halt as the power is suddenly cut, and you're quibbling over a paltry additional 10 watts? I can only assume that the data is not that important.
  - flemhans 2y ago
    As part of resilience testing I've been turning off our 24 drive backup drive array daily for two years, by flicking the wall switch. So far nothing happened.
helf 2y ago
[dead]
snvzz 2y ago
>but for residential usage, it's totally reasonable to accept the risk. Polite disagree. Data integrity is the natural expectation humans have from computers, and thus we should stick to filesystems with data checksums such as ZFS, as well as ECC memory.
- rollcat 2y ago
  > we should stick to filesystems with data checksums such as ZFS, as well as ECC memory. While I don't disagree with this statement, consider the reality: - APFS has metadata checksums, but no data checksums. WTF Apple? - Very few Linux distributions ship zfs.ko (&spl.ko); those that do, theoretically face a legal risk (any kernel contributor could sue them for breaching the GPL); rebuilding the driver from source is awkward (even with e.g. DKMS), pulls more power, takes time, and may randomly leave your system unbootable (YES it happened to me once). - Linux itself explicitly treats ZFS as unsupported; loading the module taints the kernel. - FreeBSD is great, and is actually making great progress catching up with Linux on the desktop. Still, it is a catch-up game. I also don't want to install a system that needs to install another guest system to actually run the programs I need. - There are no practical alternatives to ZFS that even come close; sibling comment complains about btrfs data loss. I never had the guts to try btrfs in production after all the horror stories I've heard over the decade+. - ECC memory on laptops is practically unheard of, save for a couple niche Thinkpad models; and comes with large premiums on desktops. What are the practical choices for people who do not want to cosplay as sysadmins?
  - louwrentius 2y ago
    I feel that on HN people tend to be a bit pedantic about topics like data integrity, and in business settings I actually agree with them. But for residential use, risks are just different and as you point out, you have no options except to only use a desktop workstation with ECC. People like/need laptops so that’s not realistic for most people. Just run Linux/freebsd with ZFS isn’t reasonable advice to me. What I feel most strongly about is that it’s all about circumstances, context and risk evaluation. And I see so much blanket absolutist statements that don’t think about the reality of life and people’s circumstances.
why_only_15 2y ago
I'm confused about optimizing 7 watts as important -- rough numbers, 7 watts is 61 kWh/y. If you assume US-average prices of $0.16/kWh that's about $10/year. edit: looks like for the netherlands (where he lives) this is more significant -- $0.50/kWh is the average price, so ~$32/year
- yumraj 2y ago
  CA is average ~$0.5/kWh. Where in US matters a lot. I think OR is pretty cheap.
- louwrentius 2y ago
  Althoug outside the scope of the artice, I try to keep a small electricity usage footprint. My lowest is 85Kwh in an apartment a moth but that was because of perfect solar weather. I average around 130 KWH a month now.
deleted 2y ago
[deleted]
throwaway984393 2y ago
[dead]
ztravis 2y ago
Around 12 years ago I helped design and set up a 48-drive, 9U, ~120TB NAS in the Chenbro RM91250 chassis (still going strong! but plenty of drive failures along the way...). This looks like it's probably the 24-drive/4U entry in the same line (or similar). IIRC the fans were very noisy in their original hot-swappable mounts but replacing them with fixed (screw) mounts made a big difference. I can't tell from the picture if this has hot-swappable fans, though - I think I remember ours having purple plastic hardware.
hi_hi 2y ago
I've had the exact same NAS for over 15 years. It's had 5 hard drives replaced, 2 new enclosures and 1 new power supply, but it's still as good as new...
matheusmoreira 2y ago
> It's possible to create the same amount of redundant storage space with only 6-8 hard drives with RAIDZ2 (RAID 6) redundancy. I've given up on striped RAID. Residential use requires easy expandability to keep costs down. Expanding an existing parity stripe RAID setup involves failing every drive and slowly replacing them one by one with bigger capacity drives while the whole array is in a degraded state and incurring heavy I/O load. It's easier and safer to build a new one and move the data over. So you pretty much need to buy the entire thing up front which is expensive. Btrfs has a flexible allocator which makes expansion easier but btrfs just isn't trustworthy. I spent years waiting for RAID-Z expansion only for it to end up being a suboptimal solution that leaves the array in some kind of split parity state, old data in one format and new data in another format. It's just so tiresome. Just give up on the "storage efficiency" nonsense. Make a pool of double or triple mirrors instead and call it a day. It's simpler to set up, easier to understand, more performant, allows heterogeneous pools of drives which lowers risk of systemic failure due to bad batches, gradual expansion is not only possible but actually easy and doesn't take literal weeks to do, avoids loading the entire pool during resilvering in case of failures, and it offers so much redundancy the only way you'll lose data is if your house literally burns down. https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/ https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs...
- louwrentius 2y ago
  I dislike that article/advice because it’s dishonest / downplaying a limitation of ZFS and advocating that people should spend a lot more money, that may likely not be necessary at all.
  - matheusmoreira 2y ago
    What limitation is it downplaying? If there are hidden downsides to the proposed solution, I would certainly like to know. Compared to a RAID setup, this requires a lot less money. It's really good for residential use.
orbital-decay 2y ago
Do you have a drive rotation schedule? 24 drives. Same model. Likely the same batch. Similar wear. Imagine most of them failing at the same time, and the rest failing as you're rebuilding it due to the increased load, because they're already almost at the same point. Reliable storage is tricky.
- londons_explore 2y ago
  Software bugs might cause that (eg. drive fails after exactly 1 billion IOPS due to some counter overflowing). But hardware wear probably won't be as consistent.
  - lazide 2y ago
    That depends entirely on how good their Q&A and manufacturing quality is - the better it is, the more likely eh? Especially in an array where it’s possible every drive operation will be identical between 2 or 3 different drives.
- deleted 2y ago
  [deleted]
- Tinned_Tuna 2y ago
  I've seen this happen to a friend. Back in the noughties they built a home NAS similar to the one in the article, using fewer (smaller) drives. It was in RAID5 configuration. It lasted until one drive died and a second followed it during the rebuild. Granted, it wasn't using ZFS, there was no regular scrubbing, 00s drive failure rates were probably different, and they didn't power it down when not using it. The point is the correlated failure, not the precise cause. Usual disclaimers, n=1, rando on the internet, etc.
  - layer8 2y ago
    This is the reason why I would always use RAID 6. A second drive failing during rebuild is significantly likely.
tedk-42 2y ago
Really a non article as it feels like an edge case for usage. It's not on 24/7. No mention of I/O metrics or data stored. For all we know, OP is storing their photos and videos and never actually need to have 80% of the drives actually on and connected.
- louwrentius 2y ago
  It’s 80% full. I linked to the original article about the system for perf stats (sequential)
fulafel 2y ago
Regular reminder: RAID (and ZFS) don't replace backups. It's an availability solution to reduce downtime in event of disk failure. Many things can go wrong with your files and filesystem besides disk failure, eg user error, userspace software/script bugs, driver or FS or hardware bugs, ransomware, etc) The article mentions backups near the end saying eg "most of the data is not important" and the "most important" data is backed up. Feeling lucky I guess.
- lonjil 2y ago
  ZFS can help you with backups and data integrity beyond what RAID provides, though. For example, I back up to another machine using zfs's snapshot sending feature. Fast and convenient. I scrub my machine and the backup machine every week, so if any data has become damaged beyond repair on my machine, I know pretty quickly. Same with the backup machine. And because of the regular integrity checking on my machine, it's very unlikely that I accidentally back up damaged data. And finally, frequent snapshots are a great way to recover from software and some user errors. Of course, there are still dangers, but ZFS without backup is a big improvement over RAID, and ZFS with backups is a big improvement over most backup strategies.
- louwrentius 2y ago
  I’m well aware of the risks, I just accept it. You shouldn’t ever do what I do if you really care about tour data.
yread 2y ago
Nowadays you could almost fit all that on a single 61TB SSD and not bother with 24 disks
- tmikaeld 2y ago
  And loose all of it when it fails.
ffsm8 2y ago
> Losing the system due to power shenanigans is a risk I accept. There is another (very rare) failure an ups protects against, and that's imbalance in the electricity. You can get a spike (up or down, both can be destructive) if there is construction in your area and something happens with the electricity, or lightning hits a pylon close enough to your house. First job I worked at had multiple servers die like that, roughly 10 yrs ago. it's the only time I've ever heard of such an issue however To my understanding, an ups protects from such spikes as well, as it will die before letting your servers get damaged
- deltarholamda 2y ago
  This depends very much on the type of UPS. Big, high dollar UPSes will convert the AC to DC and back to AC, which gives amazing pure sine wave power. The $99 850VA APC you get from Office Depot does not do this. It switches from AC to battery very quickly, but it doesn't really do power conditioning. If you can afford the good ones, they genuinely improve reliability of your hardware over the long term. Clean power is great.
- int0x29 2y ago
  Isn't this what a surge protector is for?
  - bayindirh 2y ago
    Yes. In most cases, assuming you live in a 220V country, a surge protector will absorb the upwards spike, and the voltage range (a universal PSU can go as low as 107V) will handle the brownout voltage dip.
  - Kerb_ 2y ago
    Pretty sure surge protectors are less effective against dips than they are spikes
  - acstapleton 2y ago
    Nothing is really going to protect you from a direct lightning strike. Lightning strikes are on the order of millions of volts and thousands of amps. It will arc between circuits that are close enough and it will raise the ground voltage by thousands of volts too. You basically need a lighting rod buried deep into the earth to prevent it hitting your house directly and then you’re still probably going to deal with fried electronics (but your house will survive). Surge protectors are for faulty power supplies and much milder transient events on the grid and maybe a lightning strike a mile or so away.
sneak 2y ago
My home NAS is about 200TB, runs 24/7, is very loud and power inefficient, does a full scrub every Sunday, and also hasn’t had any drive failures. It’s only been 4 or 5 years, however.
tie-in 2y ago
We've been using a multi-TB PostgreSQL database on ZFS for quite a few years in production and have encountered zero problems so far, including no bit flips. In case anyone is interested, our experience is documented here: https://lackofimagination.org/2022/04/our-experience-with-postgresql-on-zfs/ https://lackofimagination.org/2022/04/our-experience-with-po...
lifeisstillgood 2y ago
My takeaway is that there is a difference between residential and industrial usage, just as there is a difference between residential car ownership and 24/7 taxi / industrial use And that no matter how amazing the industrial revolution has been, we can build reliability at the residential level but not the industrial level. And certainly at the price points. The whole “At FAANG scale” is a misnomer - we aren’t supposed to use residential quality (possibly the only quality) at that scale - maybe we are supposed to park our cars in our garages and drive them on a Sunday Maybe we should keep our servers at home, just like we keep our insurance documents and our notebooks
- bofadeez 2y ago
  I might be interested in buying storage at 1/10 of the price if the only tradeoff was a 5 minute wait to power on a hard drive.
tobiasbischoff 2y ago
Let me tell you powering these drives on and off is far more dangerous then just keeping them running. 10 years is well in the MTBF of these enterprise drives. (I worked for 10 years as enterprise storage technician, i saw a lot if sh*).
IPqLanQvqw7x 2y ago
[flagged]
manuel_w 2y ago
Discussions on checksumming filesystems usually revolve around ZFS and BTRFS, but has someone any experience with bcachefs? It's upstreamed in the linux kernel, I learned, and is supposed to have full checksumming. The author also seems to take filesystem responsibility seriously. Is anyone using it around here? https://bcachefs.org/ https://bcachefs.org/
- rollcat 2y ago
  Can't comment on bcachefs (I think it's still early), but I've been running with bcache in production on one "canary" machine for years, and it's been rock-solid.
- ffsm8 2y ago
  I tried it out on my homelab server right after the merge into the Linux kernel. Took roughly one week for the whole raid to stop mounting because of the journal (8hdd, 2 ssd write cache, 2 nvme read cache). The author responded on Reddit within a day, I tried his fix, (which meant compiling the Linux kernel and booting from that), but his fix didn't resolve the issue. He sadly didn't respond after that, so I wiped and switched back to a plain mdadmin raid after a few days of waiting. I had everything important backed up, obviously (though I did lose some unimportant data), but it did remind me that bleeding edge is indeed ... Unstable The setup process and features are fantastic however, simply being able to add a disk and flag it as read/write cache feels great. I'm certain I'll give it another try in a few years, after it had some time in the oven.
  - iforgotpassword 2y ago
    New filesystems seems to have a chicken and egg problem really. It's not like switching from Nvidia's proprietary drivers to nouveau and then back if it turns out they don't work that well. Switching filesystems, especially in larger raid setups where you desperately need more testing and real world usage feedback, is pretty involved, and even if you have everything backed up it's pretty time consuming restoring everything should things go haywire. And even if you have the time and patience to be one of these early adopters, debugging any issues encountered might also be difficult, as ideally you want to give the devs full access to your filesystem for debugging and attempted fixes, which is obviously not always feasible. So anything beyond the most trivial setups and usage patterns gets a miniscule amount of testing. In an ideal world, you'd nail your FS design first try, make no mistakes during implementation and call it a day. I'd like to live in an ideal world.
wazoox 2y ago
I currently support many NAS servers in the 50TB - 2PB range, many of them being 10, 12, and up to 15 years old for some of them. Most of them still run with their original power supplies, motherboard and most of their original (HGST -- now WD -- UlstraStar) drives, though of course a few drives have failed for some of them (but not all). 2, 4, 8TB HGST UltraStar disks are particularly reliable. All of my desktop PCs currently hosts mirrors of 2009 vintage, 2 TB drives that I got when they're put out of service. I have heaps of spare, good 2 TB drives (and a few hundreds still running in production after all these years). For some reason 14TB drives seem to have a much higher failure rate than Helium drives of all sizes. On a fleet of only about 40 14 TB drives, I had more failures than on a fleet of over 1000 12 and 16 TB.

419 points by louwrentius 2y ago

n3storm 2y ago

After 20 years using ext3 and ext4 I only lost data when ffing around with parted and did.

rollcat 2y ago
There's an infinite amount of ways you can lose data. Here's one of my recent stories: https://www.rollc.at/posts/2022-05-02-disaster-recovery/ https://www.rollc.at/posts/2022-05-02-disaster-recovery/ Just among my "personal" stuff, over the last 12 years I've completely lost 4 hard drives due to age/physical failure. ZFS made it a no-deal twice, and aided with recovery once (once I've dd'd what was left of one drive, zvol snapshots made "risky" experimentation cheap&easy).
- n3storm 2y ago
  Thanks for the link. I just realised I wrote down "did" instead "dd" :D I used photorec and other utilities with others disks, also FAT.

Tepix 2y ago

Having 24 drives probably offers some performance advantages, but if you don‘t require them, having a 6-bay NAS with 18TB disks instead woukld offer a ton of advantages in terms of power usage, noise, space required, cost and reliability.

louwrentius 2y ago
I agree, that’s what I would do today if I wanted the same amount of space, as I states in the article.
foepys 2y ago
I'd want more redundancy in that case. With such large HDDs zfs resilver could kill another disk and then you would lose your data.
prepend 2y ago
18TB drives didn’t exist back when this setup was designed. Of course a 3 bay with 128TB drives would also be superior, but this comment only makes sense in a few years.

prepend 2y ago

I wish he had talked more about his movie collection. I’m interested in the methods of selecting initial items as well as ones that survive in the collection for 10+ years.

not_the_fda 2y ago
My server isn't nearly as big as his, but my collection is mostly Criterion https://www.criterion.com/closet-picks https://www.criterion.com/closet-picks
lvl155 2y ago
Off topic but why do people run Plex and movies locally in 2024? Is “iTunes” that bad?
- prepend 2y ago
  I don’t know about other people but I run Plex because it lets me run my home movie collection through multiple clients (Apple TV, browser, phones, etc). iTunes works great with content bought from Apple, but is useless when playing your own media files from other sources. I just want every Disney movie available so my kids can watch without bothering me.

bevenhall 2y ago

I wouldn't call a single user computer a server if you can "Turn the server off when you're not using it." Not really a challenge or achievement.

lawaaiig 2y ago

Regarding the intermittent power cutoffs during boot it should be noted the drives pull power from the 5V rail on startup: comparable drives typically draw up to 1.2A. Combined with the maximum load of 25A on the 5V rail (Seasonic Platinum 860W), it's likely you'll experience power failures during boot if staggered spinup is not used.

383toast 2y ago

Can someone explain why a single geolocated node makes sense for data storage? If there's a house fire for example, wouldn't all the data be lost?

Fastjur 2y ago
I’m guessing that the 71 TiB is mostly used for media, as in, plex/jellyfin media, which is sad to loose but not unrecoverable. How would one ever store that much of personal data? I hope they have an off site backup for the all important unrecoverable data like family photos and whatnot.
- leptons 2y ago
  I have about about 80TB (my wife's data and mine) backed up to LTO5 tape. It's pretty cheap to get refurb tape drives on ebay. I pay about $5.00/TB for tape storage, not including the ~$200 for the LTO drive and an HBA card, so it was pretty economical.
  - Fastjur 2y ago
    Wow that is surprisingly cheap actually.
    - leptons 2y ago
      I get email alerts from ebay for anything related to LTO-5, and I only buy the cheap used tapes. They are still fine, most of them are very low use, the tapes actually have a chip in them that stores usage data like a car's odometer, so you can know how much a tape has been used. So far I trust tapes more than I'd trust a refurb hard drive for backups. And I also really like that the tapes have a write-protect notch on them, so once I write my backup data, there's no risk of having a tape accidentally get erased, unlike if I plugged in a hard drive and maybe there's some ransomware virus that automatically fucks with any hard drive drive that gets plugged in. It's just one less thing to worry about.

Saris 2y ago

I'm curious what's your use case for 71TB of data where you can also shut it down most of the time? My NAS is basically constantly in use, between video footage being dumped and then pulled for editing, uploading and editing photos, keeping my devices in sync, media streaming in the evening, and backups from my other devices at night..

drzzhan 2y ago

It's always nice to know that people can store their data for so long. In my research lab, we still only use separate external HDD drives due to budget reasons. Last year 4 (over 8) drives failed and we lost the data. I guess we mainly work with public data so it is not a big deal. But, it is a dream of mine to research without such worries. I do keep backups for my stuff though, but only me in my lab.

MrYHdP8nPLiG 2y ago

[flagged]

ggm 2y ago

There have been drives where power cycling was hazardous. So, whilst I agree to the model, it shouldn't be assumed this is always good, all the time, for all people. Some SSD need to be powered periodically. The duty cycle for a NAS probably meets that burden. Probably good, definitely cheaper power costs. Those extra grease on the axle drives were a blip in time. I wonder if backblaze do a drive on-off lifetime stats model? I think they are in the always on problem space.

bluedino 2y ago
Long ago I had a client who could have been an episode of "IT Nightmares". They used internal 3.5" hard drives along with USB docks to backup a couple Synology devices...It seemed like 1/10 times when you put a drive back in the dock to restore a file or make another backup, the drive wouldn't power back up.
louwrentius 2y ago
> There have been drives where power cycling was hazardous. I know about this story from 30+ years ago. It may have been true then. It may be even true now. Yet, in my case, I don't power cycle these drives often. At most a few times a month. I can't say or prove it's a huge risk. I only believe it's not. I have accepted this risk for over 15+ years. Update: remember that hard drives have an option to spin down when idle. So hard drives can handle many spinups a day.
- neilv 2y ago
  In the early '90s, some Quantum 105S hard drives had a "stiction" problem, and were shipped with Sun SPARCstations. IME at the time, power off a bunch of workstations, such as for building electrical work, and probably at least one of them wouldn't spin back up the next business day. Pulling the drive sled, and administering percussive maintenance against the desktop, could work. https://sunmanagers.org/1992/0383.html https://sunmanagers.org/1992/0383.html
  - ghaff 2y ago
    Stiction was definitely a thing back in that general period when you'd sometimes knock a drive to get it back to working again.

russfink 2y ago

What does one do with all this storage?

flounder3 2y ago
This is a drop in the bucket for photographers, videographers, and general backups of RAW / high resolution videos from mobile devices. 80TB [usable] was "just enough" for my household in 2016.
- patchymcnoodles 2y ago
  Exactly that. I'm not even shooting in ProRes or similar "raw" video. But one video project easily takes 3TB. And I'm not even a professional.
  - jiggawatts 2y ago
    Holy cow, what are you shooting with!? I have a Nikon Z8 that can output up to 8.3K @ 60 fps raw video, and my biggest project is just 1 TB! Most are on the order of 20 GB, if that.
    - patchymcnoodles 2y ago
      I use a Sony a1, my videos are 4k 100fps. But on the last project I also had an Insta 360 x4 shooting B-Roll. So on some days that adds up a lot.
andrelaszlo 2y ago
Especially since it's mostly turned off and it seems like the author is the only user
rhcom2 2y ago
For me I have about 35TB and growing in Unraid for Plex/Torrents/Backups/Docker
tbrownaw 2y ago

2OEH8eoCRo0 2y ago

> The 4 TB HGST drives have roughly 6000 hours on them after ten years. So they mostly sit idle? Mine are about ~4 years old with ~35,000 hours.

louwrentius 2y ago
To quote myself from a few lines below that fragment: > My NAS is turned off by default. I only turn it on (remotely) when I need to use it.
- markoman 2y ago
  I loved the idea around the 10Gb NICs but wondered what model switch are you connecting to? And what throughput is this topology delivering?
  - louwrentius 2y ago
    I don't have a 10Gb switch. I connect this server directly to two other machines as they all have 2 x 10Gbit. This NAS can saturate 10Gbit but the other side can't, so I'm stuck at 500-700 MB/s, I haven't measured it in a while.

fiddlerwoaroof 2y ago

This sounds to me like it's just a matter of luck and not really a model to be imitated.

louwrentius 2y ago
It's likely that you are right and that I misjudged the likelihood of this result being special. You can still imitate the model to save money on power, but your drives may not last longer, no evidence for that indeed.

turnsout 2y ago

I’ve heard the exact opposite advice (keep the drives running to reduce wear from power cycling). Not sure what to believe, but I like having my ZFS NAS running so it can regularly run scrubs and check the data. FWIW, I’ve run my 4 drive system for 10 years with 2 drive failures in that time, but they were not enterprise grade drives (WD Green).

louwrentius 2y ago
Hard drives are often configured to spin down when idle for a certain time. This can cause many spinups and spindowns per day. So I don't buy this at all. But I don't have supporting evidence that back up this notion.
- Wowfunhappy 2y ago
  > Hard drives are often configured to spin down when idle for a certain time. This can cause many spinups and spindowns per day. I was under the impression that this was, in fact, known to reduce drive longevity! It is done anyway in order to save power, but the reliability tradeoff is known. No idea where I read that though, I thought it was "common knowledge" so maybe I'm wrong.
  - tekknik 2y ago
    It does, and primarily increases the load / unload cycle count. some WD drives are only rated at a few hundred thousand load / unload cycles. it’s best to buy drives that can handle being on always and leave them on.
- turnsout 2y ago
  I think NAS systems in particular are often configured to prevent drives from spinning down
- max-ibel 2y ago
  There seems to be a huge difference between spin-down while NAS is up vs shutting the whole NAS down and restart. When I start my NAS, it takes a bunch of time to be back up: it seems to do a lot of checking on/ syncing off the drives and puts a fair amount of load on them (same is true for CPU as well, just look at your CPU load right after startup). OTOH, when the NAS spins up a single disk again, I haven't notice any additional load. Presumably, the read operation just waits until the disk is ready.

mvanbaak 2y ago

the 'secret' is not that you turn them off. it's simply luck. I have 4TB HGST drives running 24/7 for over a decade. ok, not 24 but 8, and also 0 failures. But I'm also lucky, like you. Some of the people I know have several RMAs with the same drives so there's that. My main question is: What is it that takes 71TB but can be turned off most of the time? Is this the server you store backups?

ryanjshaw 2y ago
> What is it that takes 71TB but can be turned off most of the time? Still waiting for somebody to explain this to me as well.
- leptons 2y ago
  I have a 22TB RAID10 system out in my detached garage that works as an "off-site" backup server for all my other systems. It stays off most of the time. It's on when I'm backing up data to it, or if it's running backups to LTO tape. Or it's on when I'm out in the garage doing whatever project, I use it to play music and look up stuff on the web. Otherwise it's off, most of the time.
louwrentius 2y ago
It can be luck, but with 24 drives, it feels very lucky. Somebody with proper statistics knowledge can probably calculate the risk with a guestimated 1% yearly failure rate how likely it would be to have all 24 drives remaining. And remember, my previous NAS with 20 drives also didn't have any failures. So N=44, how lucky must I be? It's for residential usage, and if I need some data, I often just copy it over 10Gbit to a system that uses much less power and this NAS is then turned off again.
- the_gorilla 2y ago
  We don't really have to guess. Backblaze posted their stats for 4 TB HGST drives for 2024, and of their 10,000 drives, 5 failed. If OP's 2014 4 TB HGST drives are anything like this, then this is just snake oil and magic rituals and it doesn't really matter what you do.

deleted 2y ago

[deleted]

naming_the_user 2y ago

For what it's worth this isn't that uncommon. Most drives fail in the first few years, if you get through that then annualized failure rates are about 1-2%. I've had the (small) SSD in a NAS fail before any of the drives due to TBW.

rkagerer 2y ago

I have a similar-sized array which I also only power on nightly to receive backups, or occasionally when I need access to it for a week or two at a time. It's a whitebox RAID6 running NTFS (tried ReFS, didn't like it), and has been around for 12+ years, although I've upgraded the drives a couple times (2TB --> 4TB --> 16TB) - the older Areca RAID controllers make it super simple to do this. Tools like Hard Disk Sentinel are awesome as well, to help catch drives before they fail. I have an additional, smaller array that runs 24x7, which has been through similar upgrade cycles, plus a handful of clients with whitebox storage arrays that have lasted over a decade. Usually the client ones are more abused (poor temperature control when they delay fixing their serveroom A/C for months but keep cramming in new heat-generating equipment, UPS batteries not replaced diligently after staff turnover, etc...). Do I notice a difference in drive lifespan between the ones that are mostly-off vs. the ones that are always-on? Hard to say. It's too small a sample size and possibly too much variance in 'abuse' between them. But definitely seen a failure rate differential between the ones that have been maintained and kept cool, vs. allowed to get hotter than is healthy. I can attest those 4TB HGST drives mentioned in the article were tanks. Anecdotally, they're the most reliable ones I've ever owned. And I have a more reasonable sample size there as I was buying dozens at a time for various clients back in the day.

louwrentius 2y ago
I bought the HGSTs specifically because they showed good stats in those Backblaze drive stats that they just started to publish back then.

lostmsu 2y ago

I have a mini PC + 4x external HDDs (I always bought used) on Windows 10 with ReFS since probably 2016 (recently upgraded to Win 11), maybe earlier. I don't bother powering off. The only time I had problems is when I tried to add a 5th disk using a USB hub, which caused drives attached to the hub get disconnected randomly under load. This actually happened with 3 different hubs, so I since stopped trying to expand that monstrosity and just replace drives with larger ones instead. Don't use hubs for storage, majority of them are shitty. Currently ~64TiB (less with redundancy). Same as OP. No data loss, no broken drives. A couple of years ago I also added an off-site 46TiB system with similar software, but a regular ATX with 3 or 4 internal drives because the spiderweb of mini PC + dangling USBs + power supplies for HDDs is too annoying. I do weekly scrubs. Some notes: https://lostmsu.github.io/ReFS/ https://lostmsu.github.io/ReFS/

anjel 2y ago

Ca't help but wonder how much electricity would have been consumed if you had left it on 24/7 for ten years...

deleted 2y ago
[deleted]
louwrentius 2y ago
On the original blog it states that the machine used 200W idle. Thats 4,8 KWh a day. 17520 KWh over 10 years? At around 0.30 euro per KWh that's 5K+ if I'm not mistaken.
nine_k 2y ago
Why wonder, let's approximate. A typical 7200 rpm disk consumes about 5W when idle. For 24 drives, it's 120W. Rather substantial, but not an electric kettle level. At $0.25 / kWh, it's $0.72 / day, or about $22 / mo, or slightly more than $260 / year. But this is only the disks; the CPU + mobo can easily consume half as much on average, so it would be more like $30-35 / mo. And if you have electricity at a lower price, the numbers change accordingly. This is why my ancient NAS uses 5400 RPM disks, and a future upgrade could use even slower disks if these were available. The reading bandwidth is multiplied by the number of disks involved.

louwrentius 2y ago

Because a lot of people pointed out that it's not that unlikely that all 24 drives survive without failure over 10 years, given the low failure rate reported by Backblaze Drive Stats Reports, I've also updated the article with that notion.

Jedd 2y ago

Surprised to not find 'ecc' on that page. I know it's not a guarantee of no-corruption, and ZFS without ECC is probably no more dangerous than any other file system without ECC, but if data corruption is a major concern for you, and you're building out a pretty hefty system like this, I can't imagine not using ECC. Slow on-disk data corruption resulting from gradual and near-silent RAM failures may be like doing regular 3-2-1 backups -- you either mitigate against the problem because you've been stung previously, or you're in that blissful pre-sting phase of your life. EDIT: I found TFA's link to the original build out - and happily they are in fact running a Xeon with ECC. Surprisingly it's a 16GB box (I thought ZFS was much hungrier on the RAM : disk ratio.) Obviously it hasn't helped for physical disk failures, but the success of the storage array owes a lot to this component.

louwrentius 2y ago
The system is using ECC and I specifically - unrelated to ZFS - wanted to use ECC memory to reduce risk of data/fs corruption. I've also added 'ecc' to the original blog post to clarify. Edit: ZFS for home usage doesn't need a ton of RAM as far as I've learned. There is the 1 GB of RAM per 1TB of storage rule of thumb, but that was for a specific context. Maybe the ill-fated data deduplication feature, or was it just to sustain performance?
- rincebrain 2y ago
  It was a handwavey rule of estimation for dedup, handwavey because dedup scales on number of records, which is going to vary wildly by recordsize.
  - InvaderFizz 2y ago
    Additionally unless it's changed in the last six years, you should pretend ZFS dedupe doesn't exist.
    - rincebrain 2y ago
      Not in a stable release yet, but check out https://github.com/openzfs/zfs/discussions/15896 https://github.com/openzfs/zfs/discussions/15896 if you have a need for that.

throw0101c 2y ago

> This NAS is very quiet for a NAS (video with audio). Big (large radius) fans can move a lot of air even at low RPM. And be much more energy efficient. Oxide Computer, in one of their presentations, talks about using 80mm fans, as they are quiet and (more importantly) don't use much power. They observed, in other servers, as much as 25% of the power went just to powering the fans, versus the ~1% of theirs: * https://www.youtube.com/shorts/hTJYY_Y1H9Q https://www.youtube.com/shorts/hTJYY_Y1H9Q * https://www.youtube.com/watch?v=4vVXClXVuzE https://www.youtube.com/watch?v=4vVXClXVuzE

chiph 2y ago
My Synology uses two 120mm fans and you can barely hear them (it's on the desk next to me). I'm sold on the idea of moving more volume at less speed. (which I understand can't happen in a 1U or 2U chassis)
sss111 2y ago
just curious, are you associated with them, as these are very obscure youtube videos :D Love it though, even the reduction in fan noise is amazing. I wonder why nobody had thought of it before, it seems so simple.
- throw0101c 2y ago
  > just curious, are you associated with them, as these are very obscure youtube videos :D Unassociated, but tech-y videos are often recommended to me, and these videos got pushed to me. (I have viewed other, unrelated Tech Day videos, so probably why I got that short. Also an old Solaris admin, so aware of Cantril, especially his rants.) > Love it though, even the reduction in fan noise is amazing. I wonder why nobody had thought of it before, it seems so simple. Depends on the size of the server: can't really expand fans with 1U or even 2U pizza boxes. And for general purpose servers, I'm not sure how many 4U+ systems are purchased—perhaps some more now that perhaps GPUs cards may be a popular add-on. For a while chassis systems (e.g., HP c7000) were popular, but I'm not sure how they are nowadays.
  - 2y ago

MIzOjy4EhuJX 2y ago

[flagged]

bearjaws 2y ago

I feel like 10 years is when my drives started failing the most. I run a 8x8tb array zraid2 redundancy, initially it was a 8x2tb array but drives started failing once every 4 months, after 3 drives failed I upgraded the remaining ones. Only downside to hosting your own is power consumption. OS upgrades have been surprisingly easy.

leighleighleigh 2y ago

Regarding the custom PID controller script: I could have sworn the Linux kernel had a generic PID controller available as a module, which you could setup via the device tree, but I can't seem to find it! (grepping for 'PID' doesn't provide very helpful results lol). I think it was used on nVidia Tegra systems, maybe? I'd be interested to find it again, if anyone knows. :)

ewalk153 2y ago
Maybe related to this? https://github.com/torvalds/linux/blob/master/tools/thermal/tmon/pid.c https://github.com/torvalds/linux/blob/master/tools/thermal/...

rnxrx 2y ago

In my experience the environment where the drives are running makes a huge difference in longevity. There's a ton more variability in residential contexts than in data center (or even office) space. Potential temperature and humidity variability is a notable challenge but what surprised me was the marked effect of even small amounts of dust. Many years ago I was running an 8x500G array in an old Dell server in my basement. The drives were all factory-new Seagates - 7200RPM and may have been the "enterprise" versions (i.e. not cheap). Over 5 years I ended up averaging a drive failure every 6 months. I ran with 2 parity drives, kept spares around and RMA'd the drives as they broke. I moved houses and ended up with a room dedicated to lab stuff. With the same setup I ended up going another 5 years without a single failure. It wasn't a surprise that the new environment was better, but it was surprising how much better a cleaner, more stable environment ended up being.

mapt 2y ago
"Do you think that's air you're breathing?" This is no longer much of an issue with sealed, helium filled drives, if it ever was.
stavros 2y ago
How does dust affect things? The drives are airtight.
- kenhwang 2y ago
  They're airtight now (at the high end or enterprise level). They weren't airtight not very long ago and had filters to regulate the air exchange.
  - Kirby64 2y ago
    They're not airtight in the true sense (besides the helium filled ones nowadays), but every drive made in the past... 30? 40 years is airtight in the sense that no dust can ever get into the drive. There's a breather hole somewhere (with a big warning to not cover it!) to equalize pressure, and a filter that doesn't allow essentially any particles in.

8n4vidtmkvmk 2y ago

I run a similar but less sophisticated setup. About 18 TiB now, and I run it 16 hours a day. I let it sleep 8 hours per night so that it's well rested in the morning. I just do this on a cron because I'm not clever enough to SSH into a turned off (and unplugged!) machine. 4 drives: 42k hours (4.7 years), 27k hours (3 years), 15k hours (1.6 years), and the last drive I don't know because apparently it isn't SMART. 0 errors according to scrub process. ... but I guess I can't claim 0 HDD failures. There has been 1 or 2, but not for years now. Knock on wood. No data loss because of mirroring. I just can't lose 2 in a pair. (Never run RAID5 BTW, lost my whole rack doing that)

BenjiWiebe 2y ago
Looks like you're quite clever actually, if you can get cron to run on a powered off unplugged machine. I think I'm missing something.
- 8n4vidtmkvmk 2y ago
  Just power, not unplugged. It's simply 0 2 * * * /usr/sbin/rtcwake -m off -s 28800 # off from 2am to 10am "and unplugged" was referring to OP's setup, not mine
- alanfranz 2y ago
  Some bios and firmware support turning on at a certain time. Maybe cron was a way to simplify.
- bigiain 2y ago
  I use a wifi controlled powerpoint to power up and down a pair of raid1 backup drives. A weekly cronjob on another (always on) machine does some simple tests (md5 checksums of "canary files" on a few machines on the network) then powers up and mounts the drives, runs an incremental backup, waits for it to finish, then unmounts and powers them back down. (There's also a double-check cronjob that runs 3 hours later that confirms they are powered down, and alerts me if they aren't. My incrementals rarely take more than an hour.)

lvl155 2y ago

I have a similar approach but I don’t use ZFS. It’s a bit superfluous especially if you’re using your storage periodically (turn on and off). I use redundant NVMEs in two stages and periodically save important data into multiple HDDs (cold storage). Worth noting, it’s important to prune your data. I also do not backup photos and videos locally. It’s a major headache and they just take up a crap ton of space when Amazon Prime will give you photo storage for free. Anecdotally, only drives that failed on me were enterprise-grade HDDs. And they all failed within a year and in an always-on system. I also think RAIDs are over-utilized and frankly a big money pit outside of enterprise-level environments.

ed_mercer 2y ago

> Losing the system due to power shenanigans is a risk I accept. A UPS provides more than just that, it delivers constant energy without fluctuations and thus makes your hardware last longer.

vunderba 2y ago
Yeah, this definitely caused me to raise an eyebrow. UPS covers brown outs and obviously the occasional temporary power outage. All those drives spinning at full speed suddenly coming to a grinding halt as the power is suddenly cut, and you're quibbling over a paltry additional 10 watts? I can only assume that the data is not that important.
- flemhans 2y ago
  As part of resilience testing I've been turning off our 24 drive backup drive array daily for two years, by flicking the wall switch. So far nothing happened.

helf 2y ago

[dead]

snvzz 2y ago

>but for residential usage, it's totally reasonable to accept the risk. Polite disagree. Data integrity is the natural expectation humans have from computers, and thus we should stick to filesystems with data checksums such as ZFS, as well as ECC memory.

rollcat 2y ago
> we should stick to filesystems with data checksums such as ZFS, as well as ECC memory. While I don't disagree with this statement, consider the reality: - APFS has metadata checksums, but no data checksums. WTF Apple? - Very few Linux distributions ship zfs.ko (&spl.ko); those that do, theoretically face a legal risk (any kernel contributor could sue them for breaching the GPL); rebuilding the driver from source is awkward (even with e.g. DKMS), pulls more power, takes time, and may randomly leave your system unbootable (YES it happened to me once). - Linux itself explicitly treats ZFS as unsupported; loading the module taints the kernel. - FreeBSD is great, and is actually making great progress catching up with Linux on the desktop. Still, it is a catch-up game. I also don't want to install a system that needs to install another guest system to actually run the programs I need. - There are no practical alternatives to ZFS that even come close; sibling comment complains about btrfs data loss. I never had the guts to try btrfs in production after all the horror stories I've heard over the decade+. - ECC memory on laptops is practically unheard of, save for a couple niche Thinkpad models; and comes with large premiums on desktops. What are the practical choices for people who do not want to cosplay as sysadmins?
- louwrentius 2y ago
  I feel that on HN people tend to be a bit pedantic about topics like data integrity, and in business settings I actually agree with them. But for residential use, risks are just different and as you point out, you have no options except to only use a desktop workstation with ECC. People like/need laptops so that’s not realistic for most people. Just run Linux/freebsd with ZFS isn’t reasonable advice to me. What I feel most strongly about is that it’s all about circumstances, context and risk evaluation. And I see so much blanket absolutist statements that don’t think about the reality of life and people’s circumstances.

why_only_15 2y ago

I'm confused about optimizing 7 watts as important -- rough numbers, 7 watts is 61 kWh/y. If you assume US-average prices of $0.16/kWh that's about $10/year. edit: looks like for the netherlands (where he lives) this is more significant -- $0.50/kWh is the average price, so ~$32/year

yumraj 2y ago
CA is average ~$0.5/kWh. Where in US matters a lot. I think OR is pretty cheap.
louwrentius 2y ago
Althoug outside the scope of the artice, I try to keep a small electricity usage footprint. My lowest is 85Kwh in an apartment a moth but that was because of perfect solar weather. I average around 130 KWH a month now.

deleted 2y ago

[deleted]

throwaway984393 2y ago

[dead]

ztravis 2y ago

Around 12 years ago I helped design and set up a 48-drive, 9U, ~120TB NAS in the Chenbro RM91250 chassis (still going strong! but plenty of drive failures along the way...). This looks like it's probably the 24-drive/4U entry in the same line (or similar). IIRC the fans were very noisy in their original hot-swappable mounts but replacing them with fixed (screw) mounts made a big difference. I can't tell from the picture if this has hot-swappable fans, though - I think I remember ours having purple plastic hardware.

hi_hi 2y ago

I've had the exact same NAS for over 15 years. It's had 5 hard drives replaced, 2 new enclosures and 1 new power supply, but it's still as good as new...

matheusmoreira 2y ago

> It's possible to create the same amount of redundant storage space with only 6-8 hard drives with RAIDZ2 (RAID 6) redundancy. I've given up on striped RAID. Residential use requires easy expandability to keep costs down. Expanding an existing parity stripe RAID setup involves failing every drive and slowly replacing them one by one with bigger capacity drives while the whole array is in a degraded state and incurring heavy I/O load. It's easier and safer to build a new one and move the data over. So you pretty much need to buy the entire thing up front which is expensive. Btrfs has a flexible allocator which makes expansion easier but btrfs just isn't trustworthy. I spent years waiting for RAID-Z expansion only for it to end up being a suboptimal solution that leaves the array in some kind of split parity state, old data in one format and new data in another format. It's just so tiresome. Just give up on the "storage efficiency" nonsense. Make a pool of double or triple mirrors instead and call it a day. It's simpler to set up, easier to understand, more performant, allows heterogeneous pools of drives which lowers risk of systemic failure due to bad batches, gradual expansion is not only possible but actually easy and doesn't take literal weeks to do, avoids loading the entire pool during resilvering in case of failures, and it offers so much redundancy the only way you'll lose data is if your house literally burns down. https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/ https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs...

louwrentius 2y ago
I dislike that article/advice because it’s dishonest / downplaying a limitation of ZFS and advocating that people should spend a lot more money, that may likely not be necessary at all.
- matheusmoreira 2y ago
  What limitation is it downplaying? If there are hidden downsides to the proposed solution, I would certainly like to know. Compared to a RAID setup, this requires a lot less money. It's really good for residential use.

orbital-decay 2y ago

Do you have a drive rotation schedule? 24 drives. Same model. Likely the same batch. Similar wear. Imagine most of them failing at the same time, and the rest failing as you're rebuilding it due to the increased load, because they're already almost at the same point. Reliable storage is tricky.

londons_explore 2y ago
Software bugs might cause that (eg. drive fails after exactly 1 billion IOPS due to some counter overflowing). But hardware wear probably won't be as consistent.
- lazide 2y ago
  That depends entirely on how good their Q&A and manufacturing quality is - the better it is, the more likely eh? Especially in an array where it’s possible every drive operation will be identical between 2 or 3 different drives.
deleted 2y ago
[deleted]
Tinned_Tuna 2y ago
I've seen this happen to a friend. Back in the noughties they built a home NAS similar to the one in the article, using fewer (smaller) drives. It was in RAID5 configuration. It lasted until one drive died and a second followed it during the rebuild. Granted, it wasn't using ZFS, there was no regular scrubbing, 00s drive failure rates were probably different, and they didn't power it down when not using it. The point is the correlated failure, not the precise cause. Usual disclaimers, n=1, rando on the internet, etc.
- layer8 2y ago
  This is the reason why I would always use RAID 6. A second drive failing during rebuild is significantly likely.

tedk-42 2y ago

Really a non article as it feels like an edge case for usage. It's not on 24/7. No mention of I/O metrics or data stored. For all we know, OP is storing their photos and videos and never actually need to have 80% of the drives actually on and connected.

louwrentius 2y ago
It’s 80% full. I linked to the original article about the system for perf stats (sequential)

fulafel 2y ago

Regular reminder: RAID (and ZFS) don't replace backups. It's an availability solution to reduce downtime in event of disk failure. Many things can go wrong with your files and filesystem besides disk failure, eg user error, userspace software/script bugs, driver or FS or hardware bugs, ransomware, etc) The article mentions backups near the end saying eg "most of the data is not important" and the "most important" data is backed up. Feeling lucky I guess.

lonjil 2y ago
ZFS can help you with backups and data integrity beyond what RAID provides, though. For example, I back up to another machine using zfs's snapshot sending feature. Fast and convenient. I scrub my machine and the backup machine every week, so if any data has become damaged beyond repair on my machine, I know pretty quickly. Same with the backup machine. And because of the regular integrity checking on my machine, it's very unlikely that I accidentally back up damaged data. And finally, frequent snapshots are a great way to recover from software and some user errors. Of course, there are still dangers, but ZFS without backup is a big improvement over RAID, and ZFS with backups is a big improvement over most backup strategies.
louwrentius 2y ago
I’m well aware of the risks, I just accept it. You shouldn’t ever do what I do if you really care about tour data.

yread 2y ago

Nowadays you could almost fit all that on a single 61TB SSD and not bother with 24 disks

tmikaeld 2y ago
And loose all of it when it fails.

ffsm8 2y ago

> Losing the system due to power shenanigans is a risk I accept. There is another (very rare) failure an ups protects against, and that's imbalance in the electricity. You can get a spike (up or down, both can be destructive) if there is construction in your area and something happens with the electricity, or lightning hits a pylon close enough to your house. First job I worked at had multiple servers die like that, roughly 10 yrs ago. it's the only time I've ever heard of such an issue however To my understanding, an ups protects from such spikes as well, as it will die before letting your servers get damaged

deltarholamda 2y ago
This depends very much on the type of UPS. Big, high dollar UPSes will convert the AC to DC and back to AC, which gives amazing pure sine wave power. The $99 850VA APC you get from Office Depot does not do this. It switches from AC to battery very quickly, but it doesn't really do power conditioning. If you can afford the good ones, they genuinely improve reliability of your hardware over the long term. Clean power is great.
int0x29 2y ago
Isn't this what a surge protector is for?
- bayindirh 2y ago
  Yes. In most cases, assuming you live in a 220V country, a surge protector will absorb the upwards spike, and the voltage range (a universal PSU can go as low as 107V) will handle the brownout voltage dip.
- Kerb_ 2y ago
  Pretty sure surge protectors are less effective against dips than they are spikes
- acstapleton 2y ago
  Nothing is really going to protect you from a direct lightning strike. Lightning strikes are on the order of millions of volts and thousands of amps. It will arc between circuits that are close enough and it will raise the ground voltage by thousands of volts too. You basically need a lighting rod buried deep into the earth to prevent it hitting your house directly and then you’re still probably going to deal with fried electronics (but your house will survive). Surge protectors are for faulty power supplies and much milder transient events on the grid and maybe a lightning strike a mile or so away.

sneak 2y ago

My home NAS is about 200TB, runs 24/7, is very loud and power inefficient, does a full scrub every Sunday, and also hasn’t had any drive failures. It’s only been 4 or 5 years, however.

tie-in 2y ago

We've been using a multi-TB PostgreSQL database on ZFS for quite a few years in production and have encountered zero problems so far, including no bit flips. In case anyone is interested, our experience is documented here: https://lackofimagination.org/2022/04/our-experience-with-postgresql-on-zfs/ https://lackofimagination.org/2022/04/our-experience-with-po...

lifeisstillgood 2y ago

My takeaway is that there is a difference between residential and industrial usage, just as there is a difference between residential car ownership and 24/7 taxi / industrial use And that no matter how amazing the industrial revolution has been, we can build reliability at the residential level but not the industrial level. And certainly at the price points. The whole “At FAANG scale” is a misnomer - we aren’t supposed to use residential quality (possibly the only quality) at that scale - maybe we are supposed to park our cars in our garages and drive them on a Sunday Maybe we should keep our servers at home, just like we keep our insurance documents and our notebooks

bofadeez 2y ago
I might be interested in buying storage at 1/10 of the price if the only tradeoff was a 5 minute wait to power on a hard drive.

tobiasbischoff 2y ago

Let me tell you powering these drives on and off is far more dangerous then just keeping them running. 10 years is well in the MTBF of these enterprise drives. (I worked for 10 years as enterprise storage technician, i saw a lot if sh*).

IPqLanQvqw7x 2y ago

[flagged]

manuel_w 2y ago

Discussions on checksumming filesystems usually revolve around ZFS and BTRFS, but has someone any experience with bcachefs? It's upstreamed in the linux kernel, I learned, and is supposed to have full checksumming. The author also seems to take filesystem responsibility seriously. Is anyone using it around here? https://bcachefs.org/ https://bcachefs.org/

rollcat 2y ago
Can't comment on bcachefs (I think it's still early), but I've been running with bcache in production on one "canary" machine for years, and it's been rock-solid.
ffsm8 2y ago
I tried it out on my homelab server right after the merge into the Linux kernel. Took roughly one week for the whole raid to stop mounting because of the journal (8hdd, 2 ssd write cache, 2 nvme read cache). The author responded on Reddit within a day, I tried his fix, (which meant compiling the Linux kernel and booting from that), but his fix didn't resolve the issue. He sadly didn't respond after that, so I wiped and switched back to a plain mdadmin raid after a few days of waiting. I had everything important backed up, obviously (though I did lose some unimportant data), but it did remind me that bleeding edge is indeed ... Unstable The setup process and features are fantastic however, simply being able to add a disk and flag it as read/write cache feels great. I'm certain I'll give it another try in a few years, after it had some time in the oven.
- iforgotpassword 2y ago
  New filesystems seems to have a chicken and egg problem really. It's not like switching from Nvidia's proprietary drivers to nouveau and then back if it turns out they don't work that well. Switching filesystems, especially in larger raid setups where you desperately need more testing and real world usage feedback, is pretty involved, and even if you have everything backed up it's pretty time consuming restoring everything should things go haywire. And even if you have the time and patience to be one of these early adopters, debugging any issues encountered might also be difficult, as ideally you want to give the devs full access to your filesystem for debugging and attempted fixes, which is obviously not always feasible. So anything beyond the most trivial setups and usage patterns gets a miniscule amount of testing. In an ideal world, you'd nail your FS design first try, make no mistakes during implementation and call it a day. I'd like to live in an ideal world.

wazoox 2y ago

I currently support many NAS servers in the 50TB - 2PB range, many of them being 10, 12, and up to 15 years old for some of them. Most of them still run with their original power supplies, motherboard and most of their original (HGST -- now WD -- UlstraStar) drives, though of course a few drives have failed for some of them (but not all). 2, 4, 8TB HGST UltraStar disks are particularly reliable. All of my desktop PCs currently hosts mirrors of 2009 vintage, 2 TB drives that I got when they're put out of service. I have heaps of spare, good 2 TB drives (and a few hundreds still running in production after all these years). For some reason 14TB drives seem to have a much higher failure rate than Helium drives of all sizes. On a fleet of only about 40 14 TB drives, I had more failures than on a fleet of over 1000 12 and 16 TB.