4 ms·
Is arxiv.org mirrored on archive.org? The latter has its Wayback Machine of course, but that might not necessarily follow .ps and .pdf links.
by zozbot234 7y ago
Is arxiv.org mirrored on archive.org? The latter has its Wayback Machine of course, but that might not necessarily follow .ps and .pdf links.
- bordercases 7y agoWayback Machine crawls files as well as pages.
- kragen 7y agoArXiv has historically been extremely hostile to crawlers, which is one of the reasons for its low costs.
- Dylan16807 7y agoBah. If they average thousands of downloads per file, there's plenty of room in there for some crawlers. And the total data set is less than a terabyte; seed a torrent somewhere for $20. The user-pays S3 bucket also exists as a good thing but S3 is much more expensive than data needs to be.
- kragen 7y agoThey started in 1991, when a terabyte was an unimaginably huge quantity of data and it was common for anonymous FTP servers like xxx.lanl.gov to request that you not connect until after business hours to avoid interfering with the main purpose of the machines. When I joined the internet in 1992, our 7.5-MHz VAX had a 56-kbps frame-relay link to New Mexico Technet (TECNET on our DECNET), which I think may also have provided LANL's rather beefier internet connection. They started providing WWW access in 1993, before Apache added preforking to NCSA HTTPD, and in fact I think before NCSA HTTPD itself. This means that initially every new HTTP request involved forking a new child process from the HTTP server, which took a few hundred milliseconds. This is the context in which the arXiv's hostile stance toward spidering was established. I agree that it would be an extremely valuable course of action to seed a series of torrents, since a single torrent wouldn't work; it would have to be replaced every time a new paper was uploaded, fragmenting the swarm enough to render it useless. Also, they could surely use Fastly and permit spidering.
- Dylan16807 7y ago> They started in 1991, when a terabyte was an unimaginably huge quantity of data The collection was a lot smaller then. The whole thing has fit on one hard drive for a long time. And as far as interpreting my post as criticism goes, apply it to the last ten years only. > This is the context in which the arXiv's hostile stance toward spidering was established. They should have reconsidered it at some point. > series of torrents Sure. A torrent for each 500MB chunk they already collate, or yearly, or both.
- murkt 7y agoarXiv has bulk dumps that anyone can S3. They aren't updated every day, not even every month, but still they have it. I used it myself.
- Dylan16807 7y agohttps://news.ycombinator.com/item?id=22115894 https://news.ycombinator.com/item?id=22115894
- mkl 7y agoThey do provide bulk access [1], so I'm sure there will be backups. The numbers on that page are quite out of date, so I'm not sure it's current. [1] https://arxiv.org/help/bulk_data_s3 https://arxiv.org/help/bulk_data_s3