5 ms·

Another thing is that unless you have a very specific need for SSDs (such as heavily random access focused workloads, very tight space constraints, or working i

by ndiddy 1y ago

Another thing is that unless you have a very specific need for SSDs (such as heavily random access focused workloads, very tight space constraints, or working in a bumpy environment), mechanical hard drives are still way more cost effective for storing lots of data than NVMe. You can get a manufacturer refurbished 12TB hard drive with a multi-year warranty for ~$120, while even an 8TB NVMe drive goes for at least $500. Of course for general-purpose internal drives, NVMe is a far better experience than a mechanical HDD, but my NAS with 6 hard drives in RAIDz2 still gets bottlenecked by my 2.5GBit LAN, not the speeds of the drives.

acranox 1y ago
Don’t forget about power. If you’re trying to build a low power NAS, those hdds idle around 5w each, while the ssd is closer to 5mw. Once you’ve got a few disks, the HDDs can account for half the power or more. The cost penalty for 2TB or 4TB ssds is still big, but not as bad as at the 8TB level.
- markhahn 1y ago
  such power claims are problematic - you're not letting the HDs spin down, for instance, and not crediting the fact that an SSD may easily dissipate more power than an HD under load. (in this thread, the host and network are slow, so it's not relevant that SSDs are far faster when active.)
  - philjohn 1y ago
    There's a lot of "never let your drive spin down! They need to be running 24/7 or they'll die in no time at all!" voices in the various homelab communities sadly. Even the lower tier IronWolf drives from Seagate specify 600k load/unload cycles (not spin down, granted, but gives an idea of the longevity).
    - sandreas 1y ago
      Is there any (semi-)scientific proof to that (serious question)? I did search a lot to this topic but found nothing...
      - espadrine 1y ago
        Here is someone that had significant corruption until they stopped: https://www.xda-developers.com/why-not-to-spin-down-nas-hard-drives/ https://www.xda-developers.com/why-not-to-spin-down-nas-hard... There are many similar articles.
        philjohn 1y ago
        I wonder if they were just hit with the bathtub curve? Or perhaps the fact that my IronWolf drives are 5400rpm rather than 7200rpm means they're still going strong after 4 years with no issues spinning down after 20 minutes. Or maybe I'm just insanely lucky? Before I moved to my desktop machine being 100% SSD I used hard drives for close to 30 years and never had a drive go bad. I did tend to use drives for a max of 3-5 years though before upgrading for more space.
        deleted 1y ago
        [deleted]
        billfor 1y ago
        I wonder if it has to do with the type of HDD. The red NAS drives may not like to be spun down as much. I spin down my drives and have not had a problem except for one drive, after 10 years continuous running, but I use consumer desktop drives which probably expect to be cycled a lot more than a NAS.
  - 1over137 1y ago
    Letting hdds spin down is generally not advisable in a NAS, unless you access it really rarely perhaps.
    - sandreas 1y ago
      Is there any (semi-)scientific proof to that (serious question)? I did search a lot to this topic but found nothing... (see above, same question)
      - gosub100 1y ago
        It's probably decades old anecdata from people who re commissioned old drives that were on the shelf for many years. The theory is that the grease on the spindle dries up and seizes up the platters.
    - Dr4kn 1y ago
      Spin down isn't as problematic today. It really depends on your setup and usage. If the stuff you access often can be cashed to SSDs you rarely access it. Depending on your file system and operating system only drives that are in use can be spun up. If you have multiple drive arrays with media some of it won't be accessed as often. In an enterprise setting it generally doesn't make sense. For a home environment disks you generally don't access the data that often. Automatic downloads and seeding change that.
  - olavgg 1y ago
    I experimented with spindowns, but the fact is, many applications needs to write to disk several times per minute. Because of this I only use SSD's now. Archived files are moved to the Cloud. I think Google Disk is one of the best alternatives out there, as it has true data streaming built in the MacOS or Windows clients. It feels like an external hard drive.
  - sixothree 1y ago
    I've put all of my surveillance cameras on one volume in _hopes_ that I can let my other volumes spin down. But nope. They spend the vast majority of their day spinning.
    - sandreas 1y ago
      Did you consider ZFS with L2ARC? The extra caching device might make this possible...
      - dsr_ 1y ago
        That's not how L2ARC works. It's not how the ZIL SLOG works, either. If a read request can be filled by the OS cache, it will be. Then it will be filled by the ARC, if possible. Then it will be filled by the L2ARC, if it exists. Then it will be filled by the on-disk cache, if possible; finally, it will be filled by a read. An async write will eventually be flushed to a disk write, possibly after seconds of realtime. The ack is sent after the write is complete... which may be while the drive has it in a cache but hasn't actually written it yet. A sync write will be written to the ZIL SLOG, if it exists, while it is being written to the disk. It will be acknowledged as soon as the ZIL finishes the write. If the SLOG does not exist, the ack comes when the disk reports the write complete.
      - sixothree 1y ago
        BTW what I mean is that even with my attempts to limit activity, there seems to be enough network activity to wake these drives pretty much continuously.
- j45 1y ago
  5w to mw is huge, but at the same time the total cost of ownership over 3/5 years depending on the cost of the hardware may not pay off in a 5w spread, especially with SSD premiums. When it comes to self hosted servers for example, using tiny computers as servers often gets you massive power savings that do make a difference compared to buying off-lease rack mount servers that can idle in the hundreds of watts.
throw0101d 1y ago
> […] mechanical hard drives are still way more cost effective for storing lots of data than NVMe. Linux ISOs?
ThatMedicIsASpy 1y ago
Low power, low noise, low profile system, LOW ENTRY COST. I can easily get a beelink me mini or two and build a NAS + offsite storage. Two 1TB SSDs for a mirror are around 100€, two new 1TB HDDs are around 80€. You are thinking in dimensions normal people have no need for. Just the numbers alone speaks volumes, 12TB, 6 hdds, 8TB NVMes, 2.5GB LAN.
kllrnohj 1y ago
It depends on what you consider "lots" of data. For >20tb yes absolutely obviously by a landslide. But if you just want self-hosted Google Drive or Dropbox you're in the 1-4TB range where mechanical drives are a very bad value as they have a pretty significant price floor. WD Blue 1tb hdd is $40 while WD Blue 1tb nvme is $60. The HDD still has a strict price advantage, but the nvme drive uses way less power, is more reliable, doesn't have spinup time (consumer usage is very infrequently accessed, keeping the mechanical drives spinning continuously gets into that awkward zone of worthwhile) And these prices are getting low enough, especially with this NUC-based solutions, to actually be price competitive with the low tiers of drive & dropbox while also being something you actually own and control. Dropbox still charges $120/yr for the entry level plan of just 2TB after all. 3x WD Blue NVMEs + an N150 and you're at break-even in 3 years or less
- gknoy 1y ago
  I appreciate you laying it out like that. I've seen these NVME NAS things mentioned and had been thinking that the reliability of SSDs was so much worse than HDDs.
  - kllrnohj 1y ago
    SSDs are just limited write cycles whereas HDDs literally spin themselves to death. In a simple consumer NAS usage, like if this was just photo backup, that basically means SSDs will last forever. Meanwhile those HDDs start hitting borrowed time at 5-8 years, regardless of write cycles.
    - dotancohen 1y ago
      I have had two Sandisk 2.5 inch SSDS just suddenly fail. No warning that I could discern, and no way to recover afterwards. Both were while running Debian variants as a / partition, luckily I keep /home on a separate partition. Any idea what that failure mode could have been? It worries me tremendously to keep data on an SSD now.
      - 1y ago

by ndiddy 1y ago

acranox 1y ago

Don’t forget about power. If you’re trying to build a low power NAS, those hdds idle around 5w each, while the ssd is closer to 5mw. Once you’ve got a few disks, the HDDs can account for half the power or more. The cost penalty for 2TB or 4TB ssds is still big, but not as bad as at the 8TB level.

markhahn 1y ago
such power claims are problematic - you're not letting the HDs spin down, for instance, and not crediting the fact that an SSD may easily dissipate more power than an HD under load. (in this thread, the host and network are slow, so it's not relevant that SSDs are far faster when active.)
- philjohn 1y ago
  There's a lot of "never let your drive spin down! They need to be running 24/7 or they'll die in no time at all!" voices in the various homelab communities sadly. Even the lower tier IronWolf drives from Seagate specify 600k load/unload cycles (not spin down, granted, but gives an idea of the longevity).
  - sandreas 1y ago
    Is there any (semi-)scientific proof to that (serious question)? I did search a lot to this topic but found nothing...
    - espadrine 1y ago
      Here is someone that had significant corruption until they stopped: https://www.xda-developers.com/why-not-to-spin-down-nas-hard-drives/ https://www.xda-developers.com/why-not-to-spin-down-nas-hard... There are many similar articles.
      - philjohn 1y ago
        I wonder if they were just hit with the bathtub curve? Or perhaps the fact that my IronWolf drives are 5400rpm rather than 7200rpm means they're still going strong after 4 years with no issues spinning down after 20 minutes. Or maybe I'm just insanely lucky? Before I moved to my desktop machine being 100% SSD I used hard drives for close to 30 years and never had a drive go bad. I did tend to use drives for a max of 3-5 years though before upgrading for more space.
        deleted 1y ago
        [deleted]
      - billfor 1y ago
        I wonder if it has to do with the type of HDD. The red NAS drives may not like to be spun down as much. I spin down my drives and have not had a problem except for one drive, after 10 years continuous running, but I use consumer desktop drives which probably expect to be cycled a lot more than a NAS.
- 1over137 1y ago
  Letting hdds spin down is generally not advisable in a NAS, unless you access it really rarely perhaps.
  - sandreas 1y ago
    Is there any (semi-)scientific proof to that (serious question)? I did search a lot to this topic but found nothing... (see above, same question)
    - gosub100 1y ago
      It's probably decades old anecdata from people who re commissioned old drives that were on the shelf for many years. The theory is that the grease on the spindle dries up and seizes up the platters.
  - Dr4kn 1y ago
    Spin down isn't as problematic today. It really depends on your setup and usage. If the stuff you access often can be cashed to SSDs you rarely access it. Depending on your file system and operating system only drives that are in use can be spun up. If you have multiple drive arrays with media some of it won't be accessed as often. In an enterprise setting it generally doesn't make sense. For a home environment disks you generally don't access the data that often. Automatic downloads and seeding change that.
- olavgg 1y ago
  I experimented with spindowns, but the fact is, many applications needs to write to disk several times per minute. Because of this I only use SSD's now. Archived files are moved to the Cloud. I think Google Disk is one of the best alternatives out there, as it has true data streaming built in the MacOS or Windows clients. It feels like an external hard drive.
- sixothree 1y ago
  I've put all of my surveillance cameras on one volume in _hopes_ that I can let my other volumes spin down. But nope. They spend the vast majority of their day spinning.
  - sandreas 1y ago
    Did you consider ZFS with L2ARC? The extra caching device might make this possible...
    - dsr_ 1y ago
      That's not how L2ARC works. It's not how the ZIL SLOG works, either. If a read request can be filled by the OS cache, it will be. Then it will be filled by the ARC, if possible. Then it will be filled by the L2ARC, if it exists. Then it will be filled by the on-disk cache, if possible; finally, it will be filled by a read. An async write will eventually be flushed to a disk write, possibly after seconds of realtime. The ack is sent after the write is complete... which may be while the drive has it in a cache but hasn't actually written it yet. A sync write will be written to the ZIL SLOG, if it exists, while it is being written to the disk. It will be acknowledged as soon as the ZIL finishes the write. If the SLOG does not exist, the ack comes when the disk reports the write complete.
    - sixothree 1y ago
      BTW what I mean is that even with my attempts to limit activity, there seems to be enough network activity to wake these drives pretty much continuously.
j45 1y ago
5w to mw is huge, but at the same time the total cost of ownership over 3/5 years depending on the cost of the hardware may not pay off in a 5w spread, especially with SSD premiums. When it comes to self hosted servers for example, using tiny computers as servers often gets you massive power savings that do make a difference compared to buying off-lease rack mount servers that can idle in the hundreds of watts.

throw0101d 1y ago

> […] mechanical hard drives are still way more cost effective for storing lots of data than NVMe. Linux ISOs?

ThatMedicIsASpy 1y ago

Low power, low noise, low profile system, LOW ENTRY COST. I can easily get a beelink me mini or two and build a NAS + offsite storage. Two 1TB SSDs for a mirror are around 100€, two new 1TB HDDs are around 80€. You are thinking in dimensions normal people have no need for. Just the numbers alone speaks volumes, 12TB, 6 hdds, 8TB NVMes, 2.5GB LAN.

kllrnohj 1y ago

It depends on what you consider "lots" of data. For >20tb yes absolutely obviously by a landslide. But if you just want self-hosted Google Drive or Dropbox you're in the 1-4TB range where mechanical drives are a very bad value as they have a pretty significant price floor. WD Blue 1tb hdd is $40 while WD Blue 1tb nvme is $60. The HDD still has a strict price advantage, but the nvme drive uses way less power, is more reliable, doesn't have spinup time (consumer usage is very infrequently accessed, keeping the mechanical drives spinning continuously gets into that awkward zone of worthwhile) And these prices are getting low enough, especially with this NUC-based solutions, to actually be price competitive with the low tiers of drive & dropbox while also being something you actually own and control. Dropbox still charges $120/yr for the entry level plan of just 2TB after all. 3x WD Blue NVMEs + an N150 and you're at break-even in 3 years or less

gknoy 1y ago
I appreciate you laying it out like that. I've seen these NVME NAS things mentioned and had been thinking that the reliability of SSDs was so much worse than HDDs.
- kllrnohj 1y ago
  SSDs are just limited write cycles whereas HDDs literally spin themselves to death. In a simple consumer NAS usage, like if this was just photo backup, that basically means SSDs will last forever. Meanwhile those HDDs start hitting borrowed time at 5-8 years, regardless of write cycles.
  - dotancohen 1y ago
    I have had two Sandisk 2.5 inch SSDS just suddenly fail. No warning that I could discern, and no way to recover afterwards. Both were while running Debian variants as a / partition, luckily I keep /home on a separate partition. Any idea what that failure mode could have been? It worries me tremendously to keep data on an SSD now.
    - 1y ago