3 ms·
There are two solutions 1) Keep a list of all scrapping tools, versions, user-agent string, and which headers opt-out of each one and how can user misconfigure
by gus_massa 4y ago
There are two solutions
1) Keep a list of all scrapping tools, versions, user-agent string, and which headers opt-out of each one and how can user misconfigure them to avoid the opt-out.
2) Add a general rate limit per ip, Bonus points for a te4mporal ban in case they don't understand the polite version.