3 ms·

Or perhaps as time passes, a greater portion of their storage is rarely accessed archives so more disks in % are sitting doing nothing.

by cm2187 6y ago

codezero 6y ago
I’m just assuming that folks doing archival storage aren’t using these kinds of spiny disks as it would be super expensive compared to other mediums, right? I do think access patterns in general should contribute to the numbers so that kind of thing can be determined.
- cm2187 6y ago
  But even if it is hot storage, do you touch all your files every day? It is bound that you accumulate over time more files that you never access.
  - codezero 6y ago
    Yeah hard to say, I assume the diversity of their customers and normal distribution should statistic those patterns out but I have no clue :) I also wonder if failures are relating to physical location on disk vs other things like controller or mechanical failures. You may not be reading old files but you’re reading files, or not, so still depends also on over all utilization.
- freeone3000 6y ago
  Compared to what, exactly? Tape is cheaper per GB, but the drives and libraries tip that over the other way. Blu-Ray discs are now more expensive per GB than hard drives, thanks to SMR and He offerings. Also note that Backblaze does backups -- by definition, these are infrequently accessed, usually write-once-read-never. I've personally been a customer for three years and performed a restore exactly once.
  - guenthert 6y ago
    Despite claims to the contrary, tape isn't dead just yet. They are still considerably cheaper than drives. An LTO-8 tape (12TB uncompressed capacity) can be had for about $100, while a 12TB HDD goes for some $300. Tape drives/libraries are quite expensive though, but that just shifts the break-even point out. For the largest sites, its still economical. Not sure, if backblaze is big enough (I'm sure they did their numbers). backglacier anyone?
    - birdman3131 6y ago
      I bought a pair of 12tb drives for $199 the other day and they often go cheaper. Now admittedly if you shuck externals you loose warranty but we are keeping them in the enclosures as these are for backups and thus the ease of taking them off site is great for us.
      - R0b0t1 6y ago
        They claim you lose the warranty but are wrong, they still have to prove you damaged it. Federal law: https://en.wikipedia.org/wiki/Magnuson%E2%80%93Moss_Warranty_Act https://en.wikipedia.org/wiki/Magnuson%E2%80%93Moss_Warranty...
        PhantomGremlin 6y ago
        Are you asking for service for the drive itself or for the drive within an enclosure? Let's say you buy a Ford Explorer SUV. You remove the engine and use it somewhere else for a few years. If that engine breaks within the warranty period of the SUV, can you take just the engine to your local Ford dealer and ask them to fix it? Probably not. Arguably, you can put the engine back into the SUV and take the entire car to a Ford dealer for repair. But would that constitute fraud on your part?
    - StillBored 6y ago
      And a number of the library vendors' libraries last for decades with only drive/tape swaps along the way. The SL8500 is on its second decade of sales for example. Usually what kills them is the vendor deciding not to release firmware updates to support the newer drives. The stock half inch cartridge form factor dates from 1984 with DLT & 3480. Given there have been libraries with grippers capable of moving a wide assortment of DLT/LTO/TXX/etc cartridges at the same time. Its doubtful if that will change anytime in the future. So if you buy one of the big libraries today it will likely last another decade or two, maybe three. There aren't many pieces of IT technology you can utilize that long.
hinkley 6y ago
If they’ve hit on a different access pattern that is more gentle, that might be something useful for posterity and I hope they dig into that possibility. There’s also just the possibility that failure rates are bimodal and so they’ve hit the valley of stability. Are they tracking wall clock time or activity for their failure data?
- brianwski 6y ago
  Disclaimer: I work at Backblaze. > If they've hit on a different access pattern that is more gentle, that might be something useful for posterity and I hope they dig into that possibility. Internally at Backblaze, we're WAY more likely to be spending time trying to figure out why drives (or something else like power supplies or power strips or any other pain point) is failing at a higher rate, than looking into why something is going well. I'm totally serious, if something is going "the same as always or getting better" it just isn't going to get much of any attention. You have to understand that with these stats, we're just reporting on what happened in our datacenter - the outcome of our operations. We don't really have much time to do more research and there isn't much more info than what you have. And if we stumbled upon something useful we would most likely blog about it. :-) So we read all of YOUR comments looking for the insightful gems. We're all in this together desperate for the same information.
  - hinkley 6y ago
    Seems to me that every drive failure causes read/write amplification, so a small decrease in failure rates would compound. Have you folks done any other work to reduce write amplification this year?
ddorian43 6y ago
The bottleneck in HDD in this scenario is bandwidth. What you do is split & spread files as much as possible, so your HDD are all serving the same amount of bandwidth. A disk doing nothing is wasted potential bandwidth (unless it's turned off).
- cm2187 6y ago
  But do they actively move around files to spread bandwidth after the initial write? If they don't, and if I am right that older files tend to be rarely accessed, I would expect entire disks to become unaccessed over time.
  - jeffbee 6y ago
    If they allow that to happen, they are leaving a ton of money on the table. It's typical in the industry to move hot and cold files around to take advantage of the IOPS you already paid for. See, for example, pages 22-23 of http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Keynote.pdf http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Ke...

Or perhaps as time passes, a greater portion of their storage is rarely accessed archives so more disks in % are sitting doing nothing.

by cm2187 6y ago

Or perhaps as time passes, a greater portion of their storage is rarely accessed archives so more disks in % are sitting doing nothing.

codezero 6y ago

I’m just assuming that folks doing archival storage aren’t using these kinds of spiny disks as it would be super expensive compared to other mediums, right? I do think access patterns in general should contribute to the numbers so that kind of thing can be determined.

cm2187 6y ago
But even if it is hot storage, do you touch all your files every day? It is bound that you accumulate over time more files that you never access.
- codezero 6y ago
  Yeah hard to say, I assume the diversity of their customers and normal distribution should statistic those patterns out but I have no clue :) I also wonder if failures are relating to physical location on disk vs other things like controller or mechanical failures. You may not be reading old files but you’re reading files, or not, so still depends also on over all utilization.
freeone3000 6y ago
Compared to what, exactly? Tape is cheaper per GB, but the drives and libraries tip that over the other way. Blu-Ray discs are now more expensive per GB than hard drives, thanks to SMR and He offerings. Also note that Backblaze does backups -- by definition, these are infrequently accessed, usually write-once-read-never. I've personally been a customer for three years and performed a restore exactly once.
- guenthert 6y ago
  Despite claims to the contrary, tape isn't dead just yet. They are still considerably cheaper than drives. An LTO-8 tape (12TB uncompressed capacity) can be had for about $100, while a 12TB HDD goes for some $300. Tape drives/libraries are quite expensive though, but that just shifts the break-even point out. For the largest sites, its still economical. Not sure, if backblaze is big enough (I'm sure they did their numbers). backglacier anyone?
  - birdman3131 6y ago
    I bought a pair of 12tb drives for $199 the other day and they often go cheaper. Now admittedly if you shuck externals you loose warranty but we are keeping them in the enclosures as these are for backups and thus the ease of taking them off site is great for us.
    - R0b0t1 6y ago
      They claim you lose the warranty but are wrong, they still have to prove you damaged it. Federal law: https://en.wikipedia.org/wiki/Magnuson%E2%80%93Moss_Warranty_Act https://en.wikipedia.org/wiki/Magnuson%E2%80%93Moss_Warranty...
      - PhantomGremlin 6y ago
        Are you asking for service for the drive itself or for the drive within an enclosure? Let's say you buy a Ford Explorer SUV. You remove the engine and use it somewhere else for a few years. If that engine breaks within the warranty period of the SUV, can you take just the engine to your local Ford dealer and ask them to fix it? Probably not. Arguably, you can put the engine back into the SUV and take the entire car to a Ford dealer for repair. But would that constitute fraud on your part?
  - StillBored 6y ago
    And a number of the library vendors' libraries last for decades with only drive/tape swaps along the way. The SL8500 is on its second decade of sales for example. Usually what kills them is the vendor deciding not to release firmware updates to support the newer drives. The stock half inch cartridge form factor dates from 1984 with DLT & 3480. Given there have been libraries with grippers capable of moving a wide assortment of DLT/LTO/TXX/etc cartridges at the same time. Its doubtful if that will change anytime in the future. So if you buy one of the big libraries today it will likely last another decade or two, maybe three. There aren't many pieces of IT technology you can utilize that long.

hinkley 6y ago

If they’ve hit on a different access pattern that is more gentle, that might be something useful for posterity and I hope they dig into that possibility. There’s also just the possibility that failure rates are bimodal and so they’ve hit the valley of stability. Are they tracking wall clock time or activity for their failure data?

brianwski 6y ago
Disclaimer: I work at Backblaze. > If they've hit on a different access pattern that is more gentle, that might be something useful for posterity and I hope they dig into that possibility. Internally at Backblaze, we're WAY more likely to be spending time trying to figure out why drives (or something else like power supplies or power strips or any other pain point) is failing at a higher rate, than looking into why something is going well. I'm totally serious, if something is going "the same as always or getting better" it just isn't going to get much of any attention. You have to understand that with these stats, we're just reporting on what happened in our datacenter - the outcome of our operations. We don't really have much time to do more research and there isn't much more info than what you have. And if we stumbled upon something useful we would most likely blog about it. :-) So we read all of YOUR comments looking for the insightful gems. We're all in this together desperate for the same information.
- hinkley 6y ago
  Seems to me that every drive failure causes read/write amplification, so a small decrease in failure rates would compound. Have you folks done any other work to reduce write amplification this year?

ddorian43 6y ago

The bottleneck in HDD in this scenario is bandwidth. What you do is split & spread files as much as possible, so your HDD are all serving the same amount of bandwidth. A disk doing nothing is wasted potential bandwidth (unless it's turned off).

cm2187 6y ago
But do they actively move around files to spread bandwidth after the initial write? If they don't, and if I am right that older files tend to be rarely accessed, I would expect entire disks to become unaccessed over time.
- jeffbee 6y ago
  If they allow that to happen, they are leaving a ton of money on the table. It's typical in the industry to move hot and cold files around to take advantage of the IOPS you already paid for. See, for example, pages 22-23 of http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Keynote.pdf http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Ke...