3 ms·
This is a scraping tool that can download a list of URLs very fast. It's not an AI tool per se but the advertised use is to download corpora for AI training. Tr
by wmf 4y ago
This is a scraping tool that can download a list of URLs very fast. It's not an AI tool per se but the advertised use is to download corpora for AI training. Traditionally, polite Web spiders tried not to hit any one server too hard but when your goal is to download a bunch of data from a small number of sites fast you're inevitably going to be putting real load on each site. I'm not sure there's a solution that can satisfy everyone.
- rolph 4y agoask first, or respect robots.txt no semantic malarchy
- gus_massa 4y agoI guess the "solution" is that sites detect abuse and ban the ip for 24 hours. When enough users of this tool get banned, they will fill the github issues tracker with complain and the maintainers will have to learn how to write a polite spider. We have the inverse problem in the university. We are all (many thousands) behind a proxy, so some popular sites detect the avalanche of request and add a captcha or other anti abuse tool to everyone.
- version_five 4y agoThe tool has instructions for how to opt out. They could also ban the user agent. This is just someone venting about something that's currently popular.
- rolph 4y agothe tool also has settings that ignore opt out. >> Websites can pass the http headers X-Robots-Tag: noai, X-Robots-Tag: noindex , X-Robots-Tag: noimageai and X-Robots-Tag: noimageindex By default img2dataset will ignore images with such headers.<< >>To disable this behavior and download all images, you may pass --disallowed_header_directives '[]'<<
- version_five 4y agoFair enough, though it's open source right, so if it didn't have the setting it could be trivially changed anyway. In either case, it's the user ignoring the expressed will of the site as opposed to the maintainers of the tool
- gus_massa 4y agoThere are two solutions 1) Keep a list of all scrapping tools, versions, user-agent string, and which headers opt-out of each one and how can user misconfigure them to avoid the opt-out. 2) Add a general rate limit per ip, Bonus points for a te4mporal ban in case they don't understand the polite version.
- adultSwim 4y agoThe tool came up with it's own solution to an already solved problem. It should respect robots.txt
- rolph 4y agoit seems like what would be reqired is to whitelist, known acceptable user agents, then review requests from green agents and whiten them based on X criteria, otherwise leave green agents green and put them in a hole.
- yarg 4y agoI once accidentally got my company's office blocked from w3.org automatically downloading the schemas for some XML documents. What then happens is all requests to w3.org then redirect to a page explaining how anyone automatically downloading those schemas from those URLs became part of an accidental DDOS attack.