6 ms·
The tool has instructions for how to opt out. They could also ban the user agent. This is just someone venting about something that's currently popular.
by version_five 3y ago
The tool has instructions for how to opt out. They could also ban the user agent. This is just someone venting about something that's currently popular.
- rolph 3y agothe tool also has settings that ignore opt out. >> Websites can pass the http headers X-Robots-Tag: noai, X-Robots-Tag: noindex , X-Robots-Tag: noimageai and X-Robots-Tag: noimageindex By default img2dataset will ignore images with such headers.<< >>To disable this behavior and download all images, you may pass --disallowed_header_directives '[]'<<
- version_five 3y agoFair enough, though it's open source right, so if it didn't have the setting it could be trivially changed anyway. In either case, it's the user ignoring the expressed will of the site as opposed to the maintainers of the tool
- gus_massa 3y agoThere are two solutions 1) Keep a list of all scrapping tools, versions, user-agent string, and which headers opt-out of each one and how can user misconfigure them to avoid the opt-out. 2) Add a general rate limit per ip, Bonus points for a te4mporal ban in case they don't understand the polite version.
- adultSwim 3y agoThe tool came up with it's own solution to an already solved problem. It should respect robots.txt