4 ms·

Something I wish we could have is some kind of peer mirror of archive.org. The main IA web application gets angry pretty quickly if you're trying to click throu

by arjie 1y ago

Something I wish we could have is some kind of peer mirror of archive.org. The main IA web application gets angry pretty quickly if you're trying to click through a few different dates. If there were some kind of way to slowly mirror (torrent-style) and offer pages as a peer from archive.org that would be neat. It would be cool to show up as an alternative source for the data and the archive.org app could fetch it out of there on a user's choice and validate the checksum if required. In the end, I've ended up just keeping my own ArchiveBox and it's an all right experience. In the end, it's only useful for things I know I wanted to archive. For almost everything I go to the IA - which has so much.

zapataband2 1y ago
Is there such thing as "versioned" torrents? Assuming you have the right PGP key you could mix bittorrent and packaging systems to get an update-able distribution
- hsbauauvhabzb 1y ago
  A Torrent would probably suffocate under the small file distribution. I’m not sure how the romset torrents work but I thought they were versioned. But torrent is probably the wrong tech. I’m sure there would be many players willing to host a few TB or more each, which could be fronted via something so it’s transparent to the user. But a better option might be a subscription model, anything else will be slammed by crawlers.
- pabs3 1y ago
  Couple of BEPs related to updating torrents: https://www.bittorrent.org/beps/bep_0039.html https://www.bittorrent.org/beps/bep_0039.html https://www.bittorrent.org/beps/bep_0046.html https://www.bittorrent.org/beps/bep_0046.html
- throawayonthe 1y ago
  trere is the bittorrent v2 standard: https://blog.libtorrent.org/2020/09/bittorrent-v2/ https://blog.libtorrent.org/2020/09/bittorrent-v2/ but unfortunately most foss torrent clients do not support it, partly because at release libtorrent 2.0.x had poor io performance in some cases so torrent clients reverted to the 1.2.x branch
- pronoiac 1y ago
  I think SciOp is doing something in that area, with a catalog site and webseeds. https://sciop.net/ https://sciop.net/
renegat0x0 1y ago
- I can confirm that the web archive can be really slow - I think I have seen that AI scrapers create bottleneck in the bandwidth - To some digital archives you need to create scientific accounts (I think Common Crawl works like that) - Data quite easily can be very big. The goal is to store many things. We not only store Internet, but with additional dimension of time - Since there is a lot of data, it is difficult to navigate it, search it, so it easily can become unusable - For example that is why I created my own meta data link, I needed some information about domains Link: https://github.com/rumca-js/Internet-Places-Database https://github.com/rumca-js/Internet-Places-Database
pronoiac 1y ago
The Archive Team - not part of the Internet Archive - worked on a distributed backup of a portion of the Internet Archive - https://wiki.archiveteam.org/index.php/INTERNETARCHIVE.BAK https://wiki.archiveteam.org/index.php/INTERNETARCHIVE.BAK It's been dormant / on hiatus for a few years now.
- smallerize 1y ago
  That can only cover other collections though, because the WARC files from the Wayback Machine web scrapes are not public.
giancarlostoro 1y ago
I do wonder why IA does not maintain a IPFS instance, or if they do, why they're not more popular? There's tons of IPFS mirror services out there that operate at reasonable speeds. One issue I've run into with IA is old enough websites that there's JS or CSS that just wont render, what I'm not sure about is, can we retroactively fix such things? Would be nice to be able to un-ruin the code somehow if they exported everything possible at the time. Edit: Would be really neat if you could click on a domain while on IA, and a desktop client downloads as many WAR files in a slower priority download queue, as many as you're interested in, with higher priority pages first, and then you can view it fully offline.
- stavros 1y ago
  Because nobody pins on IPFS. It's basically http with extra steps, at this point.
- TechSquidTV 1y ago
  They do torrents. I was looking into this recently as well, considering building an Activity Pub alternative to IA. I came to what I assume is the same conclusion that IA came to. No one uses IPFS. For the average user, it is significantly more difficult to get started. For the experienced user, the ecosystem of tools around IPFS is extremely small. All in all, IPFS offers very little benefit over torrents in practice and has a much smaller user pool.
  - outside1234 1y ago
    IPFS is a great idea poorly executed. Content addressable storage is a great idea, but it is so difficult to use in practice for real world scaled scenarios (larger than one hard disk drive).
  - kevincox 1y ago
    The problems with the torrents is that they can be updated if the file changes (sometimes small metadata changes) and now your seeders can't be found. Maybe if they also kept a list of old hashes so that you could at least manually try to recover data from the older torrent?
stavros 1y ago
I have a design for a system where you can "donate" your disk space to a provider. Basically, you run the client, you say you want to make 1TB available to archive.org, and their server can push the rarest content to your computer. It's based on torrents, and you can easily make a content delivery system on top of this (so people can fetch data from this network). I emailed a few archiving teams but nobody seemed interested, so I never made it.
- deleted 1y ago
  [deleted]
- toomuchtodo 1y ago
  It's a hard problem to solve, because its easy to temporarily donate resources to archiving ops via the ArchiveTeam warrior, but a long term commitment to run persistent compute and storage to mirror a chunk of the internet archive. It's why I think Filecoin isn't going to work either; very little overlap between the people who feel its important to keep these archives alive versus people who would run distributed storage to collect financial compensation for doing so. Easier to send fiat to IA for them to invest (~$2/GB) and to pay to keep the disks spinning somewhere safe across the world. (ia volunteer, no affiliation otherwise)
  - stavros 1y ago
    The system I have in mind is strictly volunteer-run, and it automatically balances the files so that it minimises rare copies. You're right, though, long-term commitment is rare from volunteers. That's why the idea is to make short-term commitment so easy that you have a good enough pool of short-termers that it works out in the aggregate.
    - toomuchtodo 1y ago
      Appreciate your work on this.
      - stavros 1y ago
uses 1y ago
Yeah, I did a scraping project a while back where I wanted to look back at historical snapshots. Getting the info out of Internet Archive was surprisingly difficult. I ended up using https://pypi.org/project/pywaybackup/ https://pypi.org/project/pywaybackup/, which helped quite a bit.
deleted 1y ago
[deleted]
abustamam 1y ago
That kinda sounds like ipfs https://ipfs.tech/ https://ipfs.tech/

by arjie 1y ago

zapataband2 1y ago

Is there such thing as "versioned" torrents? Assuming you have the right PGP key you could mix bittorrent and packaging systems to get an update-able distribution

hsbauauvhabzb 1y ago
A Torrent would probably suffocate under the small file distribution. I’m not sure how the romset torrents work but I thought they were versioned. But torrent is probably the wrong tech. I’m sure there would be many players willing to host a few TB or more each, which could be fronted via something so it’s transparent to the user. But a better option might be a subscription model, anything else will be slammed by crawlers.
pabs3 1y ago
Couple of BEPs related to updating torrents: https://www.bittorrent.org/beps/bep_0039.html https://www.bittorrent.org/beps/bep_0039.html https://www.bittorrent.org/beps/bep_0046.html https://www.bittorrent.org/beps/bep_0046.html
throawayonthe 1y ago
trere is the bittorrent v2 standard: https://blog.libtorrent.org/2020/09/bittorrent-v2/ https://blog.libtorrent.org/2020/09/bittorrent-v2/ but unfortunately most foss torrent clients do not support it, partly because at release libtorrent 2.0.x had poor io performance in some cases so torrent clients reverted to the 1.2.x branch
pronoiac 1y ago
I think SciOp is doing something in that area, with a catalog site and webseeds. https://sciop.net/ https://sciop.net/

renegat0x0 1y ago

- I can confirm that the web archive can be really slow - I think I have seen that AI scrapers create bottleneck in the bandwidth - To some digital archives you need to create scientific accounts (I think Common Crawl works like that) - Data quite easily can be very big. The goal is to store many things. We not only store Internet, but with additional dimension of time - Since there is a lot of data, it is difficult to navigate it, search it, so it easily can become unusable - For example that is why I created my own meta data link, I needed some information about domains Link: https://github.com/rumca-js/Internet-Places-Database https://github.com/rumca-js/Internet-Places-Database

pronoiac 1y ago

The Archive Team - not part of the Internet Archive - worked on a distributed backup of a portion of the Internet Archive - https://wiki.archiveteam.org/index.php/INTERNETARCHIVE.BAK https://wiki.archiveteam.org/index.php/INTERNETARCHIVE.BAK It's been dormant / on hiatus for a few years now.

smallerize 1y ago
That can only cover other collections though, because the WARC files from the Wayback Machine web scrapes are not public.

giancarlostoro 1y ago

I do wonder why IA does not maintain a IPFS instance, or if they do, why they're not more popular? There's tons of IPFS mirror services out there that operate at reasonable speeds. One issue I've run into with IA is old enough websites that there's JS or CSS that just wont render, what I'm not sure about is, can we retroactively fix such things? Would be nice to be able to un-ruin the code somehow if they exported everything possible at the time. Edit: Would be really neat if you could click on a domain while on IA, and a desktop client downloads as many WAR files in a slower priority download queue, as many as you're interested in, with higher priority pages first, and then you can view it fully offline.

stavros 1y ago
Because nobody pins on IPFS. It's basically http with extra steps, at this point.
TechSquidTV 1y ago
They do torrents. I was looking into this recently as well, considering building an Activity Pub alternative to IA. I came to what I assume is the same conclusion that IA came to. No one uses IPFS. For the average user, it is significantly more difficult to get started. For the experienced user, the ecosystem of tools around IPFS is extremely small. All in all, IPFS offers very little benefit over torrents in practice and has a much smaller user pool.
- outside1234 1y ago
  IPFS is a great idea poorly executed. Content addressable storage is a great idea, but it is so difficult to use in practice for real world scaled scenarios (larger than one hard disk drive).
- kevincox 1y ago
  The problems with the torrents is that they can be updated if the file changes (sometimes small metadata changes) and now your seeders can't be found. Maybe if they also kept a list of old hashes so that you could at least manually try to recover data from the older torrent?

stavros 1y ago

I have a design for a system where you can "donate" your disk space to a provider. Basically, you run the client, you say you want to make 1TB available to archive.org, and their server can push the rarest content to your computer. It's based on torrents, and you can easily make a content delivery system on top of this (so people can fetch data from this network). I emailed a few archiving teams but nobody seemed interested, so I never made it.

deleted 1y ago
[deleted]
toomuchtodo 1y ago
It's a hard problem to solve, because its easy to temporarily donate resources to archiving ops via the ArchiveTeam warrior, but a long term commitment to run persistent compute and storage to mirror a chunk of the internet archive. It's why I think Filecoin isn't going to work either; very little overlap between the people who feel its important to keep these archives alive versus people who would run distributed storage to collect financial compensation for doing so. Easier to send fiat to IA for them to invest (~$2/GB) and to pay to keep the disks spinning somewhere safe across the world. (ia volunteer, no affiliation otherwise)
- stavros 1y ago
  The system I have in mind is strictly volunteer-run, and it automatically balances the files so that it minimises rare copies. You're right, though, long-term commitment is rare from volunteers. That's why the idea is to make short-term commitment so easy that you have a good enough pool of short-termers that it works out in the aggregate.
  - toomuchtodo 1y ago
    Appreciate your work on this.
    - stavros 1y ago

uses 1y ago

Yeah, I did a scraping project a while back where I wanted to look back at historical snapshots. Getting the info out of Internet Archive was surprisingly difficult. I ended up using https://pypi.org/project/pywaybackup/ https://pypi.org/project/pywaybackup/, which helped quite a bit.

deleted 1y ago

[deleted]

abustamam 1y ago

That kinda sounds like ipfs https://ipfs.tech/ https://ipfs.tech/