5 ms·

We need to put Internet Archive on IPFS or something similar, finding a single location for 50PB+ archive is going to be difficult but one can easily find 50000

We need to put Internet Archive on IPFS or something similar, finding a single location for 50PB+ archive is going to be difficult but one can easily find 50000 people with 1TB of free space and a stable server+internet connection.

toomuchtodo 6y ago
https://decentralizedweb.net/ https://decentralizedweb.net/
progval 6y ago
https://www.archiveteam.org/index.php?title=INTERNETARCHIVE.BAK https://www.archiveteam.org/index.php?title=INTERNETARCHIVE....
- toomuchtodo 6y ago
  > IA.BAK has been broken and unmaintained since about December 2016. The above status page is not accurate as of December 2019.
generationP 6y ago
Is there a way to cut the archive down to a manageable yet relevant subset based on one's interests? Say, "I have 500GB free space and I care about mathematics, what do I take from the archive"? Because 50000TB is too much for crowdsourcing, particularly if it's supposed to be crowd-served rather than just crowd-hoarded. I would guess a high percentage of that is video, which perhaps should never have been part of the archive to begin with due to its signal-to-noise rate.
- dannyw 6y ago
  In 30 years with our 2^largenumber drives, we’d be regretting we didn’t add chive more video.
  - boogies 6y ago
    But will we have "2^largenumber drives"? Aren't SSDs already beginning to have problems with bit flipping?
    - boogies 6y ago
      Addendum: QLC is one way some SSDs are made slightly denser¹ that comes with a cost: decreased reliability and extended sustained write performance (the amount of data you can write quickly in one go is limited, and decreases as the drive fills up). https://www.howtogeek.com/428869/ssds-are-getting-denser-and-slower-thanks-to-qlc-flash/ https://www.howtogeek.com/428869/ssds-are-getting-denser-and... [1]: 25% denser than TLC
- nikisweeting 6y ago
  Yes, start building your own local archive with something like ArchiveBox.io or Webrecorder.io.
  - generationP 6y ago
    I don't want to make my new archive (I mean, I don't mind, but that's not my point). I want to crowd-host a piece of the OG Archive.
    - nikisweeting 6y ago
      Yeah, I do agree that crowd-hosting a piece of Archive.org on a distributed filesystem would be better. I say exactly that towards the end of my "Intro to the Internet Archiving world" talk here too: https://github.com/pirate/internet-archiving-talk https://github.com/pirate/internet-archiving-talk I think they've thought about IPFS and BitTorrent in the past, but 50PB is a lot of data and they already have a few redundant backups around the world that are hard enough to manage as-is. Hopefully the latest round of lawsuits and public support pushes them more in the direction of letting the public help them with hosting.
q3k 6y ago
> one can easily find 50000 people with 1TB of free space and a stable server+internet connection. That's not how this works. There's currently no truly decentralized storage that could handle the IA workload. IPFS certainly doesn't cut it. People have tried. It didn't work. Not to mention I'd take a single, stable, commercial Internet link over 50k consumer Internet links where you can't even tell if the other side is temporarily shut down, the network got partitioned/congested or the hard drive got wiped.

toomuchtodo 6y ago

https://decentralizedweb.net/ https://decentralizedweb.net/

progval 6y ago

https://www.archiveteam.org/index.php?title=INTERNETARCHIVE.BAK https://www.archiveteam.org/index.php?title=INTERNETARCHIVE....

toomuchtodo 6y ago
> IA.BAK has been broken and unmaintained since about December 2016. The above status page is not accurate as of December 2019.

generationP 6y ago

Is there a way to cut the archive down to a manageable yet relevant subset based on one's interests? Say, "I have 500GB free space and I care about mathematics, what do I take from the archive"? Because 50000TB is too much for crowdsourcing, particularly if it's supposed to be crowd-served rather than just crowd-hoarded. I would guess a high percentage of that is video, which perhaps should never have been part of the archive to begin with due to its signal-to-noise rate.

dannyw 6y ago
In 30 years with our 2^largenumber drives, we’d be regretting we didn’t add chive more video.
- boogies 6y ago
  But will we have "2^largenumber drives"? Aren't SSDs already beginning to have problems with bit flipping?
  - boogies 6y ago
    Addendum: QLC is one way some SSDs are made slightly denser¹ that comes with a cost: decreased reliability and extended sustained write performance (the amount of data you can write quickly in one go is limited, and decreases as the drive fills up). https://www.howtogeek.com/428869/ssds-are-getting-denser-and-slower-thanks-to-qlc-flash/ https://www.howtogeek.com/428869/ssds-are-getting-denser-and... [1]: 25% denser than TLC
nikisweeting 6y ago
Yes, start building your own local archive with something like ArchiveBox.io or Webrecorder.io.
- generationP 6y ago
  I don't want to make my new archive (I mean, I don't mind, but that's not my point). I want to crowd-host a piece of the OG Archive.
  - nikisweeting 6y ago
    Yeah, I do agree that crowd-hosting a piece of Archive.org on a distributed filesystem would be better. I say exactly that towards the end of my "Intro to the Internet Archiving world" talk here too: https://github.com/pirate/internet-archiving-talk https://github.com/pirate/internet-archiving-talk I think they've thought about IPFS and BitTorrent in the past, but 50PB is a lot of data and they already have a few redundant backups around the world that are hard enough to manage as-is. Hopefully the latest round of lawsuits and public support pushes them more in the direction of letting the public help them with hosting.

q3k 6y ago

> one can easily find 50000 people with 1TB of free space and a stable server+internet connection. That's not how this works. There's currently no truly decentralized storage that could handle the IA workload. IPFS certainly doesn't cut it. People have tried. It didn't work. Not to mention I'd take a single, stable, commercial Internet link over 50k consumer Internet links where you can't even tell if the other side is temporarily shut down, the network got partitioned/congested or the hard drive got wiped.