4 ms·
Fair enough, though it's open source right, so if it didn't have the setting it could be trivially changed anyway. In either case, it's the user ignoring the ex
by version_five 3y ago
Fair enough, though it's open source right, so if it didn't have the setting it could be trivially changed anyway. In either case, it's the user ignoring the expressed will of the site as opposed to the maintainers of the tool
- gus_massa 3y agoThere are two solutions 1) Keep a list of all scrapping tools, versions, user-agent string, and which headers opt-out of each one and how can user misconfigure them to avoid the opt-out. 2) Add a general rate limit per ip, Bonus points for a te4mporal ban in case they don't understand the polite version.