4 ms·

There needs to be a global effort to backup the Internet Archive at this point.

esskay 2y ago
Just need to find someone with ~220pb of storage and the ability to increase that by approximately 50% annually forever more.
- sidewndr46 2y ago
  Whenever you have that much data stored how do you actually know the data is still there and can be retrieved? Even if you have absolutely insane connectivity to it at some point don't you run out of time to check it? Apparent 200 PiB at 1 GiB per second would take about 58254 hours to retrieve.
  - danielheath 2y ago
    It's not like it's all coming from one disk, or going to one single CPU. 20TB drives with 500mb/s sequential read are available today. Reading the whole disk takes about half a day. If your storage pod has 12 of those, even a $50 n100 CPU can run xxHash at 6gb / s (could probably even manage MurmurHash).
  - tap-snap-or-nap 2y ago
    I won't even pretend to know how to begin with this type of project.
    - InDubioProRubio 2y ago
      Crawlers with jobs, building searchable indexes? Similar to youtube. Down at the source its blobs, but above it all floats a layers of tags, metrics and searchable text. That is what the searches run against and the preferences algo builds its lineup against?
  - deleted 2y ago
    [deleted]
- adastra22 2y ago
  That's only about 38 racks of storage, at a cost of ~$3.5M for the hard drives (redundancy included). Not that big, in the grand scheme of things.
  - djhn 2y ago
    That actually sounds remarkably accessible. Considering how much of a donation you need to make for naming rights to a rural university professorship/library building, surely this would appeal to some freshly minted startup decamillionaire with a slight peterthielite anti-establishment bent?
    - adastra22 2y ago
      Actually 15 racks if you’re using backblaze storage pods. Which now that I think about it, is about how many racks I saw in the various rooms of the church. [I just happened to be at IA headquarters last weekend.] The storage pods hardware itself would be another $1m, and then let’s assume other $0.5M for various things I’m not considering (network equipment, power transformers, etc.). Still just $5m for the base hardware to store that info. Yeah, pretty affordable.
      - rthnbgrredf 2y ago
        Well buying 220pb of storage space is really not the problem nowadays, at least from a cost perspective. But you need to maintain all that stuff. What happens when a disk goes broke, what if a network switch goes broke, how do you update your software at scale and so on. I think it would be best to put it on AWS S3 Glacier Deep Archive for about 2.5 million dollar per year.
        adastra22 2y ago
        2.5 million per year is about 10x what the worst case ongoing costs would be.
        2y ago
BSDobelix 2y ago
There is, at least with book's etc: https://annas-archive.org/torrents https://annas-archive.org/torrents

Just need to find someone with ~220pb of storage and the ability to increase that by approximately 50% annually forever more.

sidewndr46 2y ago
Whenever you have that much data stored how do you actually know the data is still there and can be retrieved? Even if you have absolutely insane connectivity to it at some point don't you run out of time to check it? Apparent 200 PiB at 1 GiB per second would take about 58254 hours to retrieve.
- danielheath 2y ago
  It's not like it's all coming from one disk, or going to one single CPU. 20TB drives with 500mb/s sequential read are available today. Reading the whole disk takes about half a day. If your storage pod has 12 of those, even a $50 n100 CPU can run xxHash at 6gb / s (could probably even manage MurmurHash).
- tap-snap-or-nap 2y ago
  I won't even pretend to know how to begin with this type of project.
  - InDubioProRubio 2y ago
    Crawlers with jobs, building searchable indexes? Similar to youtube. Down at the source its blobs, but above it all floats a layers of tags, metrics and searchable text. That is what the searches run against and the preferences algo builds its lineup against?
- deleted 2y ago
  [deleted]
adastra22 2y ago
That's only about 38 racks of storage, at a cost of ~$3.5M for the hard drives (redundancy included). Not that big, in the grand scheme of things.
- djhn 2y ago
  That actually sounds remarkably accessible. Considering how much of a donation you need to make for naming rights to a rural university professorship/library building, surely this would appeal to some freshly minted startup decamillionaire with a slight peterthielite anti-establishment bent?
  - adastra22 2y ago
    Actually 15 racks if you’re using backblaze storage pods. Which now that I think about it, is about how many racks I saw in the various rooms of the church. [I just happened to be at IA headquarters last weekend.] The storage pods hardware itself would be another $1m, and then let’s assume other $0.5M for various things I’m not considering (network equipment, power transformers, etc.). Still just $5m for the base hardware to store that info. Yeah, pretty affordable.
    - rthnbgrredf 2y ago
      Well buying 220pb of storage space is really not the problem nowadays, at least from a cost perspective. But you need to maintain all that stuff. What happens when a disk goes broke, what if a network switch goes broke, how do you update your software at scale and so on. I think it would be best to put it on AWS S3 Glacier Deep Archive for about 2.5 million dollar per year.
      - adastra22 2y ago
        2.5 million per year is about 10x what the worst case ongoing costs would be.
        2y ago

BSDobelix 2y ago

There is, at least with book's etc: https://annas-archive.org/torrents https://annas-archive.org/torrents