4 ms·
I really appreciate the Archive team's efforts to make the Wayback Machine more responsive, and have donated a few times to support them. Unfortunately, the re
by timpera 11d ago
I really appreciate the Archive team's efforts to make the Wayback Machine more responsive, and have donated a few times to support them.
Unfortunately, the restrictions have been way too strict for the last few months: from my residential IP, simply moving the mouse on the calendar for a specific URL is enough to get stuck on 429 error messages for a while; and from corporate ISPs (for example, on airport WiFi), you often can't access the WM at all. I hope they'll find a way to relax those.
- mitxela 10d agoI wonder if your browser is prefetching every link you move the mouse over.
- timpera 10d agoI don't think so, but moving the mouse over any date in the calendar makes a request for the list of snapshots taken on that day.
- account42 10d agoI have had similar experiences just opening archived pages with a couple of embedded images. It's at a level where just using the site normally is painful.
- mrweasel 10d agoGenerally speaking I feel like detecting the bots might be a lost cause. For someone like the Internet Archive I don't know how to deal with it, for smaller sites, cache everything, static pages whenever possible. Sadly I see rate-limiting usage in general becoming a thing. With residential proxies and more sophisticated bots either pretending to be Chrome or directly piloting Chrome, it's going to become impossible to tell a real user from a bot. Only solution is to pretend that everyone is a bot and design for it.
- markalby 10d agowe’ve been using Datadome at work for this because it’s nice to get someone else to think about the constant bot cat and mouse, and we all benefit from rules and fixes created from other client data. Not an ad- that service is eye watering expensive but I think it makes sense as something to offload.
- mrweasel 10d agoWe use spur.us which provides us with information on IPs they believe is running proxies. It's a nice service, in terms of pricing it's completely reasonable, for our use case. I can't imagine how much work goes into compiling their data.
- arbol 10d agoHow much is eye watering expensive roughly?
- keywordbarrage 5d ago[flagged]