11 ms·

Wayback Machine director outlines the scale of everyone's favorite archive

246 points by jor-el 8y ago

ScottBurson 8y ago
Odd this story doesn't mention Brewster Kahle, who AFAIK came up with the idea of archiving the Web. I heard people joke about it in its early years, too. I'm glad he took it seriously.
- gkoberger 8y ago
  He may not be in the article, but he did keynote the Internet Archive’s Annual Bash last week! So he's not forgotten.
homero 8y ago
How do they afford all those disks?
- Gustomaximus 8y ago
  I've donated token amounts on a few occasions. I feel if you have the means its the right thing to do when a project like this does something useful for you. I suspect the bulk comes from a few generous individuals / corporations as this is not the most known platform. Found this with a quick search: https://archive.org/about/credits.php https://archive.org/about/credits.php
- mekarpeles 8y ago
  - foundations + government grants https://archive.org/post/2776/who-funds-this https://archive.org/post/2776/who-funds-this - donations (in-cash / in-kind) https://archive.org/about/credits.php https://archive.org/about/credits.php - digitization services https://archive.org/scanning https://archive.org/scanning - web archiving services https://archive-it.org https://archive-it.org src: I work @ Internet Archive on https://openlibrary.org https://openlibrary.org
  - PM_ME_YOUR_CAT 8y ago
    This is pretty great actually. I see that ads are none of your points: does Wayback Machine either avoid them totally or is it a negligible amount?
    - aaroninsf 8y ago
      No ads full stop!
- Dylan16807 8y ago
  A completely new set of hard drives for 44PB of raw storage would cost 1-1.5 million, and they're only growing by 9% (18%?) per year. It's not all that much, and it's not even close to their biggest budget entry.
  - GordonS 8y ago
    Don't forget that they will need geo-redundancy built in to prevent data loss, and they likely have some kind of offline archive too. Then there is the running cost to power all those drives. Also, when you have that many drives, you're going to be constantly replacing dead ones. I have a feeling this is an expensive operation!
    - Dylan16807 8y ago
      > Don't forget that they will need geo-redundancy built in to prevent data loss This is already factoring in 2x storage, so you just put each half in a different location. > and they likely have some kind of offline archive too. Okay so 2 million. > Then there is the running cost to power all those drives. Yes, that's one of many things that adds on top of the raw drive cost. > Also, when you have that many drives, you're going to be constantly replacing dead ones. It's not actually that bad! If you cram ten thousand drives into a few racks, your drive failures per week will be in the single digits. You could have someone come in once a month. > I have a feeling this is an expensive operation! It is, but my core point is that it would be expensive even if the drives were free. You need a certain amount of server horsepower and the physical resources to support it. This implies that you're not getting to stick 60 drives in a single chassis, so you need more labor as well. There was an estimate last year of $1500 for IA to store and serve a terabyte of data indefinitely. I'd love to see a breakdown of that number, how much goes to day 1 hardware, how much goes to year 1 hosting, how long that hardware is expected to last. But even the day 1 cost is definitely a lot more than $50 for 2TB worth of hard drive.
      - GordonS 8y ago
        > This is already factoring in 2x storage, so you just put each half in a different location. My bad, I missed that! > It is, but my core point is that it would be expensive even if the drives were free. On reflection, and after reading your reply, I agree :) > There was an estimate last year of $1500 for IA to store and serve a terabyte of data indefinitely. I assume that's $1500 per year?
codeulike 8y ago
TLDR: 22 petabytes, stored redundantly as 44 petabytes, and a warehouse of physical media (books, vinyl records) that grows by a 'shipping container' every two weeks. edit: Also this article mentions a bunch of things I didn't know about (e.g. playable 80s video game archive) so its worth reading.
- exikyut 8y ago
  Actually approx 100 PB of raw capacity thanks to redundancy, I was told once. Unsure if this is on RAID or ZFS.
  - ddorian43 8y ago
    I was expecting some kind of reed-solomon encoding.
    - patrickg_zill 8y ago
      Zfs does have Reed-Solomon in the double and triple parity modes of operation.
      - romed 8y ago
        I guess I would expect something more like massively distributed erasure encodings rather than ZFS directly. For example something like Dropbox’s magic pocket but taken to extremes with, say, 100 data stripes and 20 parity stripes, all on different machines.
        klodolph 8y ago
        One of the known problems with these large encodings is that the cost of reconstructing lost data increases as the size of the data increases. If you use Reed-Solomon (100,20), then you only have 20% overhead and have a vanishingly small probability of losing data, but if you lose a single 10TB disk, you need to do 1PB of I/O to rebuild it! Even forgetting I/O for a moment, you might be churning through a bunch of CPU time just to rebuild a single block of data. Of course, you don't need to rebuild immediately. You're effectively working with (100,19), and you can put off the reconstruction as long as you like, maybe you don't reconstruct until someone wants to read the data or until enough other disks fail, and you can prioritize the I/O as low as you like. But in practice, super large encodings become more and more expensive as the size increases.
AdmiralAsshat 8y ago
Wonder what the operating budget of the organization is. I donate $50 a year, because it's pretty much the only site I can check every month or so and reliably expect that it will be better since the last time I visited.
- gojomo 8y ago
  As a charitable non-profit, a lot of financial info is available via the public-record "Form 990" filed with the IRS. See for example: https://projects.propublica.org/nonprofits/organizations/943242767 https://projects.propublica.org/nonprofits/organizations/943... Looking at the form for ty2016, the most recent available, the Internet Archive's expenditures that year were about $16.4 million.
Sargos 8y ago
The Internet Archive is definitely up there with Wikipedia as one of mankind's greatest treasures. It really should be globally funded and endowed for the foreseeable future.

246 points by jor-el 8y ago

Odd this story doesn't mention Brewster Kahle, who AFAIK came up with the idea of archiving the Web. I heard people joke about it in its early years, too. I'm glad he took it seriously.

gkoberger 8y ago
He may not be in the article, but he did keynote the Internet Archive’s Annual Bash last week! So he's not forgotten.

homero 8y ago

How do they afford all those disks?

Gustomaximus 8y ago
I've donated token amounts on a few occasions. I feel if you have the means its the right thing to do when a project like this does something useful for you. I suspect the bulk comes from a few generous individuals / corporations as this is not the most known platform. Found this with a quick search: https://archive.org/about/credits.php https://archive.org/about/credits.php
mekarpeles 8y ago
- foundations + government grants https://archive.org/post/2776/who-funds-this https://archive.org/post/2776/who-funds-this - donations (in-cash / in-kind) https://archive.org/about/credits.php https://archive.org/about/credits.php - digitization services https://archive.org/scanning https://archive.org/scanning - web archiving services https://archive-it.org https://archive-it.org src: I work @ Internet Archive on https://openlibrary.org https://openlibrary.org
- PM_ME_YOUR_CAT 8y ago
  This is pretty great actually. I see that ads are none of your points: does Wayback Machine either avoid them totally or is it a negligible amount?
  - aaroninsf 8y ago
    No ads full stop!
Dylan16807 8y ago
A completely new set of hard drives for 44PB of raw storage would cost 1-1.5 million, and they're only growing by 9% (18%?) per year. It's not all that much, and it's not even close to their biggest budget entry.
- GordonS 8y ago
  Don't forget that they will need geo-redundancy built in to prevent data loss, and they likely have some kind of offline archive too. Then there is the running cost to power all those drives. Also, when you have that many drives, you're going to be constantly replacing dead ones. I have a feeling this is an expensive operation!
  - Dylan16807 8y ago
    > Don't forget that they will need geo-redundancy built in to prevent data loss This is already factoring in 2x storage, so you just put each half in a different location. > and they likely have some kind of offline archive too. Okay so 2 million. > Then there is the running cost to power all those drives. Yes, that's one of many things that adds on top of the raw drive cost. > Also, when you have that many drives, you're going to be constantly replacing dead ones. It's not actually that bad! If you cram ten thousand drives into a few racks, your drive failures per week will be in the single digits. You could have someone come in once a month. > I have a feeling this is an expensive operation! It is, but my core point is that it would be expensive even if the drives were free. You need a certain amount of server horsepower and the physical resources to support it. This implies that you're not getting to stick 60 drives in a single chassis, so you need more labor as well. There was an estimate last year of $1500 for IA to store and serve a terabyte of data indefinitely. I'd love to see a breakdown of that number, how much goes to day 1 hardware, how much goes to year 1 hosting, how long that hardware is expected to last. But even the day 1 cost is definitely a lot more than $50 for 2TB worth of hard drive.
    - GordonS 8y ago
      > This is already factoring in 2x storage, so you just put each half in a different location. My bad, I missed that! > It is, but my core point is that it would be expensive even if the drives were free. On reflection, and after reading your reply, I agree :) > There was an estimate last year of $1500 for IA to store and serve a terabyte of data indefinitely. I assume that's $1500 per year?

codeulike 8y ago

TLDR: 22 petabytes, stored redundantly as 44 petabytes, and a warehouse of physical media (books, vinyl records) that grows by a 'shipping container' every two weeks. edit: Also this article mentions a bunch of things I didn't know about (e.g. playable 80s video game archive) so its worth reading.

exikyut 8y ago
Actually approx 100 PB of raw capacity thanks to redundancy, I was told once. Unsure if this is on RAID or ZFS.
- ddorian43 8y ago
  I was expecting some kind of reed-solomon encoding.
  - patrickg_zill 8y ago
    Zfs does have Reed-Solomon in the double and triple parity modes of operation.
    - romed 8y ago
      I guess I would expect something more like massively distributed erasure encodings rather than ZFS directly. For example something like Dropbox’s magic pocket but taken to extremes with, say, 100 data stripes and 20 parity stripes, all on different machines.
      - klodolph 8y ago
        One of the known problems with these large encodings is that the cost of reconstructing lost data increases as the size of the data increases. If you use Reed-Solomon (100,20), then you only have 20% overhead and have a vanishingly small probability of losing data, but if you lose a single 10TB disk, you need to do 1PB of I/O to rebuild it! Even forgetting I/O for a moment, you might be churning through a bunch of CPU time just to rebuild a single block of data. Of course, you don't need to rebuild immediately. You're effectively working with (100,19), and you can put off the reconstruction as long as you like, maybe you don't reconstruct until someone wants to read the data or until enough other disks fail, and you can prioritize the I/O as low as you like. But in practice, super large encodings become more and more expensive as the size increases.

AdmiralAsshat 8y ago

Wonder what the operating budget of the organization is. I donate $50 a year, because it's pretty much the only site I can check every month or so and reliably expect that it will be better since the last time I visited.

gojomo 8y ago
As a charitable non-profit, a lot of financial info is available via the public-record "Form 990" filed with the IRS. See for example: https://projects.propublica.org/nonprofits/organizations/943242767 https://projects.propublica.org/nonprofits/organizations/943... Looking at the form for ty2016, the most recent available, the Internet Archive's expenditures that year were about $16.4 million.

Sargos 8y ago

The Internet Archive is definitely up there with Wikipedia as one of mankind's greatest treasures. It really should be globally funded and endowed for the foreseeable future.