7 ms·

12 years of Backblaze data center storage drives, visualized

147 points by bddicken 2y ago

deleted 2y ago
[deleted]
RedShift1 2y ago
What exactly is this visualizing? Does each dot represent a chunk of data?
- Piraty 2y ago
  > 1 small node -> 100 drives
- halper 2y ago
  1 small node is a 100 drives. So the small circles represent 100 drives each, I think. Not sure what they … do, though.
  - RedShift1 2y ago
    Maybe the amount of drives they purchased?
    - florbo 2y ago
      I'm assuming it's the acquisition and removal of drives over time.
sebmellen 2y ago
Doesn’t appear to work on mobile safari
- jrimbault 2y ago
  Or firefox and chromium ubuntu desktop
  - alwyn 2y ago
    Works fine for me on Firefox Linux. Interestingly took a lot longer to load in Chromium and Brave but they all work.
  - patchtopic 2y ago
    or Chrome, Brave on Linux ?
    - Tijdreiziger 2y ago
      Doesn't appear to work in Firefox or Chrome on Android either.
      - fluidcruft 2y ago
        Works fine for me in Firefox Android.
        kiwijamo 2y ago
        Ditto worked fine and was quite quick to load.
- codecraze 2y ago
  It works on my iphone with safari
- mvanbaak 2y ago
  It does, it just takes some time to download all the data
NKosmatos 2y ago
Nice one, we can see the (logical) shift to bigger drives. One small comment if I may, after 2020-2021 it gets really crowded with the dots and the number of drives leading to a loss of overall picture ;-)
- bddicken 2y ago
  Thats good feedback. It definitely gets crowded, and sluggish due to the number of dom nodes. I didn't spend much time optimizing performance.
joshka 2y ago
there's a few places where all the dots seem to drop (guessing there's some discontinuities in the data?) There's also a few places where there's duplicate labels (e.g. Hitachi 3TB) Would be great to group by manufacturer somehow (e.g. color) and make the size more prominent. Very cool visualization regardless.
- Damogran6 2y ago
  I took that to be massive decomming of drives
  - bddicken 2y ago
    Yeah, when drives fall off the bottom that represents decommissioning.
    - joshka 2y ago
      No, it's an obvious data error. The same drives reappear in future months. Pause and look at 2013-08-01 through 2013-11-01, or 2015-09-01 through 2016-01-01 to confirm this.
  - deleted 2y ago
    [deleted]
amelius 2y ago
Why doesn't it show a pile of failed drives?
- FlyingAvatar 2y ago
  Yeah, I think it would be a cooler visualization if the drives were in a line instead of a circle and new drives are added on to the right. The failed drives pile up on the bottom. Still fun to watch as it is, though.
- toss1 2y ago
  YES And I'd want to see failed drives somehow organized by TimeInService and maybe origin... We of course expect their drive usage to grow, but what would be surprising (& provide more info) is how the drives fail or age-out. None of us without huge data centers can get that kind of info
  - bobbob1921 2y ago
    Exactly, I’m. I was hoping to see all time drive failure data as well.
- bddicken 2y ago
  I'd like to do another vis that includes failed drives. Please, keep helping me brainstorm.
sega_sai 2y ago
It would be nice if it was just a conventional graph...
- tills13 2y ago
  Maybe like a voroni diagram with area correlating to count would still be cool and less distracting.
- deleted 2y ago
  [deleted]
- ziddoap 2y ago
  Data from https://www.backblaze.com/cloud-storage/resources/hard-drive-test-data https://www.backblaze.com/cloud-storage/resources/hard-drive... if you want to make one.
- dylan604 2y ago
  [flagged]
  - deleted 2y ago
    [deleted]
- bddicken 2y ago
  [OP] Backblaze writes about these data sets, and includes some more conventional graphs. For example: https://www.backblaze.com/blog/backblaze-drive-stats-for-2024/ https://www.backblaze.com/blog/backblaze-drive-stats-for-202... I wanted to do something more fun!
vladde 2y ago
The data takes a while to load – a dozen megabytes of data or so. After a while the visualization loaded. (It loaded about 0.3MB/s for me)
nrh 2y ago
needs the ball sizes to represent the storage capacity!
- pinoy420 2y ago
  That’s not true. Don’t shame me like that
- bddicken 2y ago
  Great suggestion.
xbmcuser 2y ago
I personally thought that ssd density and cost sizes would have crossed hdd prices by now. And we would start seeing them in these stats from blackblaze but hdd manufacturers seem to have stayed ahead of them so far.
- datadrivenangel 2y ago
  Density wins but not cost. Maybe in another 5-10 years we'll get to parity.
  - Dylan16807 2y ago
    A pile of microsd cards has been more dense than a hard drive for 20 years, basically as soon as the format existed and stabilized. But at that point you were paying 100x as much. I hope we reach parity. Right now prices have gone up since 2023, and flash is about 3x as expensive as hard drives.
pcurve 2y ago
As a data guy, I'm not sure how useful this chart is. But it sure is fun to look at. I enjoyed it ;-) Not sure why it would intermittently redraw the whole scene though.. could be a Chrome thing.
- pmontra 2y ago
  It's empty in Firefox Android. Play button, slider and nothing else.
  - bddicken 2y ago
    [OP] Interesting! I just tested it on Android/FF and it works. Could be a version-specific thing. Could also be just taking awhile to load (it has to download a 32MB json file).
    - pmontra 2y ago
      Thanks. I waited and the animation finally started. Too bad that the browser has no feedback for unfinished loading like this one. And I didn't understand what all those dots falling down from above mean, but that's another story. If they wanted to show how many drives there are if each model they could have made the circles larger.
  - mlry 2y ago
    I had to allow d3.js in NoScript for Fennec. Had the same intermittant redraw of the entire screen mentioned above.
BadJo0Jo0 2y ago
Fun to look at! Since I also had a mini project, that utilized this data. Sadly, haven't maintained it in awhile. It's a Show HN on my profile if you're curious. I hope you had a better time with ingesting the data than I did :)
- bddicken 2y ago
  I actually built this primarily with chatGPT o1. One of the things LLMS are really good at is writing scripts for processing and pairing down data. I wanna do a blog post talking about how did some of this, maybe coming up!
fencepost 2y ago
Probably wouldn't be that difficult to organize down the y axis based on drive capacity, and the amount of pointless jostling around of small nodes makes it noticeably bog down as the years go by. Interesting visualization though.
atYevP 2y ago
Yev from Backblaze here -> Y'all this is one of our favorite things and I've spread it internally. So cool <3
- bddicken 2y ago
  OP here - thank you! I'd actually like to more in the future with your data sets. It's so cool that you publish this.
  - atYevP 2y ago
    Love that! I think I re-shared your tweet about it a few days ago! I've been kinda staring at it on a loop on my monitor, fun to watch the balls bouncing around :D

147 points by bddicken 2y ago

deleted 2y ago

[deleted]

RedShift1 2y ago

What exactly is this visualizing? Does each dot represent a chunk of data?

Piraty 2y ago
> 1 small node -> 100 drives
halper 2y ago
1 small node is a 100 drives. So the small circles represent 100 drives each, I think. Not sure what they … do, though.
- RedShift1 2y ago
  Maybe the amount of drives they purchased?
  - florbo 2y ago
    I'm assuming it's the acquisition and removal of drives over time.

sebmellen 2y ago

Doesn’t appear to work on mobile safari

jrimbault 2y ago
Or firefox and chromium ubuntu desktop
- alwyn 2y ago
  Works fine for me on Firefox Linux. Interestingly took a lot longer to load in Chromium and Brave but they all work.
- patchtopic 2y ago
  or Chrome, Brave on Linux ?
  - Tijdreiziger 2y ago
    Doesn't appear to work in Firefox or Chrome on Android either.
    - fluidcruft 2y ago
      Works fine for me in Firefox Android.
      - kiwijamo 2y ago
        Ditto worked fine and was quite quick to load.
codecraze 2y ago
It works on my iphone with safari
mvanbaak 2y ago
It does, it just takes some time to download all the data

NKosmatos 2y ago

Nice one, we can see the (logical) shift to bigger drives. One small comment if I may, after 2020-2021 it gets really crowded with the dots and the number of drives leading to a loss of overall picture ;-)

bddicken 2y ago
Thats good feedback. It definitely gets crowded, and sluggish due to the number of dom nodes. I didn't spend much time optimizing performance.

joshka 2y ago

there's a few places where all the dots seem to drop (guessing there's some discontinuities in the data?) There's also a few places where there's duplicate labels (e.g. Hitachi 3TB) Would be great to group by manufacturer somehow (e.g. color) and make the size more prominent. Very cool visualization regardless.

Damogran6 2y ago
I took that to be massive decomming of drives
- bddicken 2y ago
  Yeah, when drives fall off the bottom that represents decommissioning.
  - joshka 2y ago
    No, it's an obvious data error. The same drives reappear in future months. Pause and look at 2013-08-01 through 2013-11-01, or 2015-09-01 through 2016-01-01 to confirm this.
- deleted 2y ago
  [deleted]

amelius 2y ago

Why doesn't it show a pile of failed drives?

FlyingAvatar 2y ago
Yeah, I think it would be a cooler visualization if the drives were in a line instead of a circle and new drives are added on to the right. The failed drives pile up on the bottom. Still fun to watch as it is, though.
toss1 2y ago
YES And I'd want to see failed drives somehow organized by TimeInService and maybe origin... We of course expect their drive usage to grow, but what would be surprising (& provide more info) is how the drives fail or age-out. None of us without huge data centers can get that kind of info
- bobbob1921 2y ago
  Exactly, I’m. I was hoping to see all time drive failure data as well.
bddicken 2y ago
I'd like to do another vis that includes failed drives. Please, keep helping me brainstorm.

sega_sai 2y ago

It would be nice if it was just a conventional graph...

tills13 2y ago
Maybe like a voroni diagram with area correlating to count would still be cool and less distracting.
deleted 2y ago
[deleted]
ziddoap 2y ago
Data from https://www.backblaze.com/cloud-storage/resources/hard-drive-test-data https://www.backblaze.com/cloud-storage/resources/hard-drive... if you want to make one.
dylan604 2y ago
[flagged]
- deleted 2y ago
  [deleted]
bddicken 2y ago
[OP] Backblaze writes about these data sets, and includes some more conventional graphs. For example: https://www.backblaze.com/blog/backblaze-drive-stats-for-2024/ https://www.backblaze.com/blog/backblaze-drive-stats-for-202... I wanted to do something more fun!

vladde 2y ago

The data takes a while to load – a dozen megabytes of data or so. After a while the visualization loaded. (It loaded about 0.3MB/s for me)

nrh 2y ago

needs the ball sizes to represent the storage capacity!

pinoy420 2y ago
That’s not true. Don’t shame me like that
bddicken 2y ago
Great suggestion.

xbmcuser 2y ago

I personally thought that ssd density and cost sizes would have crossed hdd prices by now. And we would start seeing them in these stats from blackblaze but hdd manufacturers seem to have stayed ahead of them so far.

datadrivenangel 2y ago
Density wins but not cost. Maybe in another 5-10 years we'll get to parity.
- Dylan16807 2y ago
  A pile of microsd cards has been more dense than a hard drive for 20 years, basically as soon as the format existed and stabilized. But at that point you were paying 100x as much. I hope we reach parity. Right now prices have gone up since 2023, and flash is about 3x as expensive as hard drives.

pcurve 2y ago

As a data guy, I'm not sure how useful this chart is. But it sure is fun to look at. I enjoyed it ;-) Not sure why it would intermittently redraw the whole scene though.. could be a Chrome thing.

pmontra 2y ago
It's empty in Firefox Android. Play button, slider and nothing else.
- bddicken 2y ago
  [OP] Interesting! I just tested it on Android/FF and it works. Could be a version-specific thing. Could also be just taking awhile to load (it has to download a 32MB json file).
  - pmontra 2y ago
    Thanks. I waited and the animation finally started. Too bad that the browser has no feedback for unfinished loading like this one. And I didn't understand what all those dots falling down from above mean, but that's another story. If they wanted to show how many drives there are if each model they could have made the circles larger.
- mlry 2y ago
  I had to allow d3.js in NoScript for Fennec. Had the same intermittant redraw of the entire screen mentioned above.

BadJo0Jo0 2y ago

Fun to look at! Since I also had a mini project, that utilized this data. Sadly, haven't maintained it in awhile. It's a Show HN on my profile if you're curious. I hope you had a better time with ingesting the data than I did :)

bddicken 2y ago
I actually built this primarily with chatGPT o1. One of the things LLMS are really good at is writing scripts for processing and pairing down data. I wanna do a blog post talking about how did some of this, maybe coming up!

fencepost 2y ago

Probably wouldn't be that difficult to organize down the y axis based on drive capacity, and the amount of pointless jostling around of small nodes makes it noticeably bog down as the years go by. Interesting visualization though.

atYevP 2y ago

Yev from Backblaze here -> Y'all this is one of our favorite things and I've spread it internally. So cool <3

bddicken 2y ago
OP here - thank you! I'd actually like to more in the future with your data sets. It's so cool that you publish this.
- atYevP 2y ago
  Love that! I think I re-shared your tweet about it a few days ago! I've been kinda staring at it on a loop on my monitor, fun to watch the balls bouncing around :D