4 ms·
Just put in a multi second delay to _all_ requests to that API, humans wait but this costs bots and slows the aggregate progress. Add HTTP headers documenting
by buro9 3y ago
Just put in a multi second delay to _all_ requests to that API, humans wait but this costs bots and slows the aggregate progress.
Add HTTP headers documenting the API you want them to use as a bot author will look and wonder why it's performing so badly.
The difference between human speed and computer speed is so noticeable that it can be leveraged... No high bar or complex adversarial solution needed, and the side benefit is that if a bot does persist it spreads the load.
- mordae 3y agoNot sure why downvoted without much comments. I can see a possible problem in that the bots would just establish multiple parallel connections and wait.
- kevincox 3y agoThis will harm the experience for the intended user and will barely affect the bots because they will just make more requests in parallel. The user is going to be much more speed sensitive than a scraper.
- nijave 3y agoA delay would tie up TCP connections/sockets on the bots but it would add trivially compute cost. Couldn't you just increase the parallelism? I think a lot of these schemes rely on solving a CPU or memory based computation with JavaScript
- jeroenhd 3y agoThe bots send out half a million requests, who cares if they need to wait ten seconds per request. You're adding a total of ten seconds to hours of scraping.