4 ms·
Generally speaking I feel like detecting the bots might be a lost cause. For someone like the Internet Archive I don't know how to deal with it, for smaller sit
by mrweasel 10d ago
Generally speaking I feel like detecting the bots might be a lost cause. For someone like the Internet Archive I don't know how to deal with it, for smaller sites, cache everything, static pages whenever possible.
Sadly I see rate-limiting usage in general becoming a thing. With residential proxies and more sophisticated bots either pretending to be Chrome or directly piloting Chrome, it's going to become impossible to tell a real user from a bot. Only solution is to pretend that everyone is a bot and design for it.
- markalby 10d agowe’ve been using Datadome at work for this because it’s nice to get someone else to think about the constant bot cat and mouse, and we all benefit from rules and fixes created from other client data. Not an ad- that service is eye watering expensive but I think it makes sense as something to offload.
- mrweasel 10d agoWe use spur.us which provides us with information on IPs they believe is running proxies. It's a nice service, in terms of pricing it's completely reasonable, for our use case. I can't imagine how much work goes into compiling their data.
- arbol 10d agoHow much is eye watering expensive roughly?
- keywordbarrage 5d ago[flagged]