4 ms·

"Let's Say You Wanted to Back Up The Internet Archive" by Jason Scott https://www.reddit.com/r/DataHoarder/comments/h02jl4/lets_say_you_wanted_to_back_up_the_in

"Let's Say You Wanted to Back Up The Internet Archive" by Jason Scott https://www.reddit.com/r/DataHoarder/comments/h02jl4/lets_say_you_wanted_to_back_up_the_internet/ https://www.reddit.com/r/DataHoarder/comments/h02jl4/lets_sa...

fouc 6y ago
Very interesting discussion here. Apparently can get 16TB drives at $100 each on enterprise bulk pricing. An entire backup would be 3,125 drives for $312,500. But apparently the wayback machine itself is only about 2 Petabytes.. so if you don't need the collections perhaps only 125 drives needed, or $12,500.
- jonah-archive 6y ago
  Disclaimer: I run the infrastructure/ops team at the Internet Archive. Unfortunately none of those numbers are really even close to correct (the discussion is always fun, but the folks in r/datahoarder are often not correctly informed. textfiles has more patience for it than I do). It would probably cost around 1.5M in drives, even at reasonable current enterprise volume pricing, to back up the 60+ PB of unique data in the Internet Archive (plus, as someone does note in that thread, the cost of running them -- even if it were a static backup to cold disks, you still need chassis to run them in for the backup process, space and infra for them, electricity, people, &c). I don't know offhand how much space the contents of the Wayback currently take up, but it's definitely an order of magnitude more than that number as well.
  - DaiPlusPlus 6y ago
    Does that 1.5m figure include redundancy (e.g. RAID)? What about if the data is compressed (either trivial gzipping each "file" the archive has, or using a compression window that spans multiple files)? I imagine the HTML/plaintext content of the Internet Archive would compress very well.
    - jonah-archive 6y ago
      No, that's for a single raw copy (it could go lower based on implementation -- the sweet spot on $/bit pricing is around 8TB/disk right now, but that would actually be more expensive for us in total cost because of the increased infra/space/power necessary to run them). We could probably get a relatively trivial 20-30% savings on space for Wayback contents via compression, maybe more with work (various projects are underway to do this), but much of the rest of the contents are difficult to compress, or already compressed (music, imagery for books, video, software archives, etc). We have also historically been very reluctant to deduplicate heavily, though we are experimenting with it for certain types of content -- one principle of operation is that as an archive of last resort, we're unable to have a true deaccession plan as some other archives have. A compromise we make is that our hard drives are "landfill-ready" -- that is, the contents of a drive (assuming you can read the filesystem) are inherently meaningful, content is housed with its metadata, and so forth. This produces some unusual restrictions on how far we can take compression and certain types of bitwise redundancy.
      - jakeogh 6y ago
        By unique data, that excluding generated data? Can you please estimate the space for just the wayback machine? It's the actual target. What's the wayback machine with and without images? Is it possible that we could distribute the ASCII/Unicode content now?
        sp332 6y ago
        It's the actual target. It may be your target, but I would not be so dismissive of the other data in the Archive. The Software Library, the tens of millions of scanned books, the music, etc etc. On top of that, the raw scrape data driving the Wayback Machine is not currently made available to download from the archive. It's stored in WARC files, which would include both the images and text of all scrapes and would not be trivial to disentangle.
        jakeogh 6y ago
  - schoen 6y ago
    Doesn't Jason Scott work for the Archive? Couldn't he just ask you for better numbers or something? :-)
    - jonah-archive 6y ago
      Oh, Jason's numbers are fine. He's not the one asserting that you can buy storage for <1¢/GB (and he correctly notes that the Wayback takes up >20PB).
      - jakeogh 6y ago
        What would a representative file size historgam look like?
        sp332 6y ago
        Here's some very detailed data from 2016 https://archive.org/details/ia_census_201604 https://archive.org/details/ia_census_201604 and Jason has asked for a new census to be made now. https://www.reddit.com/r/DataHoarder/comments/h02jl4/lets_say_you_wanted_to_back_up_the_internet/ftk4h7y/ https://www.reddit.com/r/DataHoarder/comments/h02jl4/lets_sa...
  - bits_n_bytes 6y ago
    Hi Jonah! Can you ping me by email?
  - Causality1 6y ago
    I'd love to know what the internal atmosphere was like when the IA did this. Who should I hold a grudge against for pulling possibly the stupidest move in the history of digital intellectual property? This is the kind of radical demonstration I expect out of some fly-by-night startup, not a twenty-four year old nonprofit with less annual revenue than the lawyers who're suing them.
  - andromeduck 6y ago
    Google offers archival storage at $14.4 /TB/a so about $1mm/a + access costs.
walrus01 6y ago
On the scale of what a bored multi-billionaire tech philanthropist could choose to fund on a whim, 50PB of storage is a tiny amount. People spend far more than that on their second yacht. As an extremely rough calculation, I've seen figures that it would be about 1.2 million dollars of hard drives. Built into 4U size, 36-drive servers with RAIDZ2 arrays, it could be $8 to $10 million of hardware. And then you need a place to put it and keep it powered up and running, let's say that's a million dollars a year for datacenter space. That's a tiny amount of money compared to some of the other nonprofit initiatives various billionaires have funded.

fouc 6y ago

Very interesting discussion here. Apparently can get 16TB drives at $100 each on enterprise bulk pricing. An entire backup would be 3,125 drives for $312,500. But apparently the wayback machine itself is only about 2 Petabytes.. so if you don't need the collections perhaps only 125 drives needed, or $12,500.

jonah-archive 6y ago
Disclaimer: I run the infrastructure/ops team at the Internet Archive. Unfortunately none of those numbers are really even close to correct (the discussion is always fun, but the folks in r/datahoarder are often not correctly informed. textfiles has more patience for it than I do). It would probably cost around 1.5M in drives, even at reasonable current enterprise volume pricing, to back up the 60+ PB of unique data in the Internet Archive (plus, as someone does note in that thread, the cost of running them -- even if it were a static backup to cold disks, you still need chassis to run them in for the backup process, space and infra for them, electricity, people, &c). I don't know offhand how much space the contents of the Wayback currently take up, but it's definitely an order of magnitude more than that number as well.
- DaiPlusPlus 6y ago
  Does that 1.5m figure include redundancy (e.g. RAID)? What about if the data is compressed (either trivial gzipping each "file" the archive has, or using a compression window that spans multiple files)? I imagine the HTML/plaintext content of the Internet Archive would compress very well.
  - jonah-archive 6y ago
    No, that's for a single raw copy (it could go lower based on implementation -- the sweet spot on $/bit pricing is around 8TB/disk right now, but that would actually be more expensive for us in total cost because of the increased infra/space/power necessary to run them). We could probably get a relatively trivial 20-30% savings on space for Wayback contents via compression, maybe more with work (various projects are underway to do this), but much of the rest of the contents are difficult to compress, or already compressed (music, imagery for books, video, software archives, etc). We have also historically been very reluctant to deduplicate heavily, though we are experimenting with it for certain types of content -- one principle of operation is that as an archive of last resort, we're unable to have a true deaccession plan as some other archives have. A compromise we make is that our hard drives are "landfill-ready" -- that is, the contents of a drive (assuming you can read the filesystem) are inherently meaningful, content is housed with its metadata, and so forth. This produces some unusual restrictions on how far we can take compression and certain types of bitwise redundancy.
    - jakeogh 6y ago
      By unique data, that excluding generated data? Can you please estimate the space for just the wayback machine? It's the actual target. What's the wayback machine with and without images? Is it possible that we could distribute the ASCII/Unicode content now?
      - sp332 6y ago
        It's the actual target. It may be your target, but I would not be so dismissive of the other data in the Archive. The Software Library, the tens of millions of scanned books, the music, etc etc. On top of that, the raw scrape data driving the Wayback Machine is not currently made available to download from the archive. It's stored in WARC files, which would include both the images and text of all scrapes and would not be trivial to disentangle.
        jakeogh 6y ago
- schoen 6y ago
  Doesn't Jason Scott work for the Archive? Couldn't he just ask you for better numbers or something? :-)
  - jonah-archive 6y ago
    Oh, Jason's numbers are fine. He's not the one asserting that you can buy storage for <1¢/GB (and he correctly notes that the Wayback takes up >20PB).
    - jakeogh 6y ago
      What would a representative file size historgam look like?
      - sp332 6y ago
        Here's some very detailed data from 2016 https://archive.org/details/ia_census_201604 https://archive.org/details/ia_census_201604 and Jason has asked for a new census to be made now. https://www.reddit.com/r/DataHoarder/comments/h02jl4/lets_say_you_wanted_to_back_up_the_internet/ftk4h7y/ https://www.reddit.com/r/DataHoarder/comments/h02jl4/lets_sa...
- bits_n_bytes 6y ago
  Hi Jonah! Can you ping me by email?
- Causality1 6y ago
  I'd love to know what the internal atmosphere was like when the IA did this. Who should I hold a grudge against for pulling possibly the stupidest move in the history of digital intellectual property? This is the kind of radical demonstration I expect out of some fly-by-night startup, not a twenty-four year old nonprofit with less annual revenue than the lawyers who're suing them.
- andromeduck 6y ago
  Google offers archival storage at $14.4 /TB/a so about $1mm/a + access costs.

walrus01 6y ago

On the scale of what a bored multi-billionaire tech philanthropist could choose to fund on a whim, 50PB of storage is a tiny amount. People spend far more than that on their second yacht. As an extremely rough calculation, I've seen figures that it would be about 1.2 million dollars of hard drives. Built into 4U size, 36-drive servers with RAIDZ2 arrays, it could be $8 to $10 million of hardware. And then you need a place to put it and keep it powered up and running, let's say that's a million dollars a year for datacenter space. That's a tiny amount of money compared to some of the other nonprofit initiatives various billionaires have funded.