4 ms·
I do like the Internet Archive but I wish I could retrieve large segments from it myself. It has very aggressive 429s. It would be a nice higher donor tier feat
by arjie 1mo ago
I do like the Internet Archive but I wish I could retrieve large segments from it myself. It has very aggressive 429s. It would be a nice higher donor tier feature. Maybe it’s not possible to do so and retain 501c3 status? Regardless I have had to start performing personal datahoarding. C’est la vie.
- eru 1mo agoI wonder if they could sell you access to a bittorrent tracker? Officially it would be independent of the donations, just an extra service they sell?
- queenkjuul 1mo agoThey have torrent downloads for a lot of content, I've downloaded almost a terabyte from them via torrent, they're just not bundled, as in you would need to retrieve each torrent file for each item individually, which might still get you a 429 if you automated it idk. I've never actually got a 429 using a browser though personally, they do throttle aggressively though
- smolder 1mo agoAnyone with a lot of data has to carefully apply limits to not get bankrupted by frontier AI people with residential proxies scraping the shit out of everything.
- yreg 1mo agoWhy sell? I thought the idea of donating to them is to make the information available for everyone for free.
- eru 1mo agoBandwidth costs money, and one could sell premium access to fund one's more public minded services.
- thechairman12 1mo agoThey posted on X about supporting this when someone was spamming them with requests
- anilakar 1mo agoI was looking for some CNATRA (US naval aviation) unclassified, non-distribution-controlled training manuals and noticed that the newest ones had been deindexed when they were put behind authentication. Stop hiding public data and I might reconsider.
- Drupon 1mo agoYeah they are more than happy to allow my VPN traffic to access these donation pages, but the second I try to access anything, it's a blanket 429 to anything I try to go to. Why would I donate to something I can't access?
- jcrawfordor 1mo agoYesterday I discovered that WayBackMachine appears to be serving 429s to all Chrome-based browsers, across all network connections. Using Firefox (or a firefox UA) fixes it. I assume this is a temporary situation, or even by mistake, but underscores that archive.org is getting creaky and hard to actually use.