5 ms·

Is there a way to cut the archive down to a manageable yet relevant subset based on one's interests? Say, "I have 500GB free space and I care about mathematics,

Is there a way to cut the archive down to a manageable yet relevant subset based on one's interests? Say, "I have 500GB free space and I care about mathematics, what do I take from the archive"? Because 50000TB is too much for crowdsourcing, particularly if it's supposed to be crowd-served rather than just crowd-hoarded. I would guess a high percentage of that is video, which perhaps should never have been part of the archive to begin with due to its signal-to-noise rate.

dannyw 6y ago
In 30 years with our 2^largenumber drives, we’d be regretting we didn’t add chive more video.
- boogies 6y ago
  But will we have "2^largenumber drives"? Aren't SSDs already beginning to have problems with bit flipping?
  - boogies 6y ago
    Addendum: QLC is one way some SSDs are made slightly denser¹ that comes with a cost: decreased reliability and extended sustained write performance (the amount of data you can write quickly in one go is limited, and decreases as the drive fills up). https://www.howtogeek.com/428869/ssds-are-getting-denser-and-slower-thanks-to-qlc-flash/ https://www.howtogeek.com/428869/ssds-are-getting-denser-and... [1]: 25% denser than TLC
nikisweeting 6y ago
Yes, start building your own local archive with something like ArchiveBox.io or Webrecorder.io.
- generationP 6y ago
  I don't want to make my new archive (I mean, I don't mind, but that's not my point). I want to crowd-host a piece of the OG Archive.
  - nikisweeting 6y ago
    Yeah, I do agree that crowd-hosting a piece of Archive.org on a distributed filesystem would be better. I say exactly that towards the end of my "Intro to the Internet Archiving world" talk here too: https://github.com/pirate/internet-archiving-talk https://github.com/pirate/internet-archiving-talk I think they've thought about IPFS and BitTorrent in the past, but 50PB is a lot of data and they already have a few redundant backups around the world that are hard enough to manage as-is. Hopefully the latest round of lawsuits and public support pushes them more in the direction of letting the public help them with hosting.

dannyw 6y ago

In 30 years with our 2^largenumber drives, we’d be regretting we didn’t add chive more video.

boogies 6y ago
But will we have "2^largenumber drives"? Aren't SSDs already beginning to have problems with bit flipping?
- boogies 6y ago
  Addendum: QLC is one way some SSDs are made slightly denser¹ that comes with a cost: decreased reliability and extended sustained write performance (the amount of data you can write quickly in one go is limited, and decreases as the drive fills up). https://www.howtogeek.com/428869/ssds-are-getting-denser-and-slower-thanks-to-qlc-flash/ https://www.howtogeek.com/428869/ssds-are-getting-denser-and... [1]: 25% denser than TLC

nikisweeting 6y ago

Yes, start building your own local archive with something like ArchiveBox.io or Webrecorder.io.

generationP 6y ago
I don't want to make my new archive (I mean, I don't mind, but that's not my point). I want to crowd-host a piece of the OG Archive.
- nikisweeting 6y ago
  Yeah, I do agree that crowd-hosting a piece of Archive.org on a distributed filesystem would be better. I say exactly that towards the end of my "Intro to the Internet Archiving world" talk here too: https://github.com/pirate/internet-archiving-talk https://github.com/pirate/internet-archiving-talk I think they've thought about IPFS and BitTorrent in the past, but 50PB is a lot of data and they already have a few redundant backups around the world that are hard enough to manage as-is. Hopefully the latest round of lawsuits and public support pushes them more in the direction of letting the public help them with hosting.