8 ms·
Please make this tool “opt-in” by default
- rolph 3y agorom1504 /img2dataset https://github.com/rom1504/img2dataset https://github.com/rom1504/img2dataset https://github.com/rom1504/img2dataset#opt-out-directives https://github.com/rom1504/img2dataset#opt-out-directives
- deleted 3y ago[deleted]
- cs702 3y agoThis exchange first made me laugh, and then it made me think: edent: OK, so every time someone builds a tool, I have to specifically opt out of it. Is that what you're saying? rom1504: Yes, for example if you want to ban the tool "Google Chrome" from accessing your website, you can do so.
- wmf 3y agoThis is a scraping tool that can download a list of URLs very fast. It's not an AI tool per se but the advertised use is to download corpora for AI training. Traditionally, polite Web spiders tried not to hit any one server too hard but when your goal is to download a bunch of data from a small number of sites fast you're inevitably going to be putting real load on each site. I'm not sure there's a solution that can satisfy everyone.
- rolph 3y agoask first, or respect robots.txt no semantic malarchy
- gus_massa 3y agoI guess the "solution" is that sites detect abuse and ban the ip for 24 hours. When enough users of this tool get banned, they will fill the github issues tracker with complain and the maintainers will have to learn how to write a polite spider. We have the inverse problem in the university. We are all (many thousands) behind a proxy, so some popular sites detect the avalanche of request and add a captcha or other anti abuse tool to everyone.
- version_five 3y agoThe tool has instructions for how to opt out. They could also ban the user agent. This is just someone venting about something that's currently popular.
- rolph 3y agothe tool also has settings that ignore opt out. >> Websites can pass the http headers X-Robots-Tag: noai, X-Robots-Tag: noindex , X-Robots-Tag: noimageai and X-Robots-Tag: noimageindex By default img2dataset will ignore images with such headers.<< >>To disable this behavior and download all images, you may pass --disallowed_header_directives '[]'<<
- version_five 3y agoFair enough, though it's open source right, so if it didn't have the setting it could be trivially changed anyway. In either case, it's the user ignoring the expressed will of the site as opposed to the maintainers of the tool
- gus_massa 3y agoThere are two solutions 1) Keep a list of all scrapping tools, versions, user-agent string, and which headers opt-out of each one and how can user misconfigure them to avoid the opt-out. 2) Add a general rate limit per ip, Bonus points for a te4mporal ban in case they don't understand the polite version.
- adultSwim 3y agoThe tool came up with it's own solution to an already solved problem. It should respect robots.txt
- rolph 3y agoit seems like what would be reqired is to whitelist, known acceptable user agents, then review requests from green agents and whiten them based on X criteria, otherwise leave green agents green and put them in a hole.
- iudqnolq 3y agoThis tool seems designed to download from many domains at once based on the readme section about optimizing DNS resolution. Considering that wouldn't a simple optimization be to shard by domain instead of arbitrarily?
- version_five 3y agoIronically, the reason this tool is popular is because copyright prevents sharing of the datasets, and so all organizations like LAION can do is share URLs. The fuss people are kicking up about scrapers is nothing compared to what they'd do if their copyrighted images were in a public ML dataset. If they (or legislators) just agreed that being part of training data was fair use, it would save a lot of servers a lot of bandwidth, and make it more convenient for people working with these datasets. It would be win/win, but there's a prisoners dilemma + misplaced greed that means it will never happen like that.
- xg15 3y ago> being part of training data was fair use It's not though, and never has been. Why should it be? Why should people contribute to making their own jobs obsolete and not even be compensated for it? > It would be win/win In what way would this be a win for artists?
- scarface74 3y ago> Why should people contribute to making their own jobs obsolete and not even be compensated for it? Haven’t technologist been saying for years about blue collar workers whose jobs have been made obsolete by technology that they should learn a new skill? Now that it’s hitting white collar workers, we are singing a different tune. And most of the time especially here on HN most people want an expansive definition of fair use as long as it is directed toward using work created by the big media companies.
- Riverheart 3y agoWell there are only two types of collars, blue and white. Is there another shore to swim towards? Pretty typical human behavior.
- flangola7 3y agoAll of OpenAI's data collection is done under fair use
- 3y ago
- deanCommie 3y agoWhile this may seem quaint, let's review the github package's README: ``` Opt-out directives Websites can pass the http headers `X-Robots-Tag: noai`, `X-Robots-Tag: noindex` , `X-Robots-Tag: noimageai` and `X-Robots-Tag: noimageindex` By default img2dataset will ignore images with such headers. To disable this behavior and download all images, you may pass --disallowed_header_directives '[]' See [AI use impact](https://github.com/rom1504/img2dataset#ai-use-impact https://github.com/rom1504/img2dataset#ai-use-impact) to understand better why you may decide to enable or disable this feature. ``` This does not pass the minimum thresholds of ethics for me. It may be legal, but it's not ethical. It's good that the default behaviour respects the headers. It's fine that there is an opt-out (if you didn't offer one, someone would just clone your code and offer it). But if the most important information you provide is "here's how to use my tool unethically" rather than "here's why the default ethical use is important, though if you really have a good use case, here you go", then you're complicit in unethical use of this tool.
- idontreadtheop 3y agoThe idea that you should be able to publish information and then control how people use it is indeed popular, but claiming that failing to embrace this idea is inherently unethical … well, I’d say “citation needed” at a minimum.
- deanCommie 3y agoEthics are subjective, and debatable. I'm happy to discuss further, but "citation needed" makes it sound like you think I must prove to you with some facts of the validity of the position. That's not the conversation we need to have. I don't think you can take a general summary of this situation, round off edges and nuance, and then say that I made the argument that this DIFFERENT scenario is "inherently unethical". Ethics are nuanced and highly specific to the context, and the framing. A single word, phrasing, emphasis (or lack thereof) can change the scenario from an ethical to an unethical one, and vice versa. For example, here is how I would phrase the exact same content in a way that i believe WOULD be ethical: ``` //First sentence unchanged Websites can pass the http headers `X-Robots-Tag: noai`, `X-Robots-Tag: noindex` , `X-Robots-Tag: noimageai` and `X-Robots-Tag: noimageindex` By default img2dataset will ignore images with such headers. //Then pull up the policy link first To understand why image creators and artists may choose to specify such headers for their images, and choose to actively not consent to their images being collected, see [AI use impact](https://github.com/rom1504/img2dataset#ai-use-impact https://github.com/rom1504/img2dataset#ai-use-impact). AI training would not be possible without the contribution of artists, and it is the recommendation of this tool's authors that you should respect their communicated wishes. However, if you have a legitimate reason to bypass these headers, to disable this behavior and download all images, you may pass --disallowed_header_directives '[]' ```
- kazinator 3y agoI would say, if you're not happy with your site being scraped, use tools like captchas, rate limiting and quotas per IP address range or whatever.
- vivegi 3y agoI wonder what happens when sites start using techniques like Zip/Compression bombs to defend against impolite mass downloaders. Particularly, in this case of downloading images without honoring robots.txt, a PNG/JPEG/ZIP bomb [1] might be a countermeasure in this arms race. [1]: https://www.bamsoftware.com/hacks/deflate.html https://www.bamsoftware.com/hacks/deflate.html
- eqvinox 3y agoPNG bombs are easy (since it's zlib/deflate), but I'm not aware of JPEG bombs (NB: not JPEG XL) — do you know any?
- vivegi 3y agoShould have read JPEG XL.
- curiousbean 3y agoI appreciate the idea, but it's also a border line crime if you end up bringing down the requester (not that I would shed any tears).
- vivegi 3y agoIf one posts a section in their web page stating: WARNING This PNG image <link> is an example of a compressed image file that might expand to a huge size upon decompression. PLEASE DO NOT DOWNLOAD IT. We are not responsible for consequences if you do not follow this instruction. and someone does that anyways, they are to blame, aren't they? Especially downloaders who claim they are entitled to download it and hosters must use their proprietary opt-out scheme (like the ones on the original linked article) should also learn that they should follow the hoster's instruction and NOT download the zip bomb. (And the hoster also lists that specific file in robots.txt)
- curiousbean 3y agoI am not a lawyer, but I don't think it would work.
- eqvinox 3y ago(from that project's README:) > AI art is democratizing art and letting hundred of millions of people express themselves through art. Making art much more prevalent and unique My contemporary art major friend blows a vein every time they read something like this. It's clear what it tries to say, but that's not art. It's fancy pictures. Which isn't to say it's somehow wrong, or doesn't allow people to express themselves — but if your creative input is limited to some AI prompt, then the art, if anything, is the prompt itself. (And, just to be clear, the definition of "art" here isn't "you have to paint something yourself". It's "you have to perform a creative action yourself". It's fine if that action is just instructions — that's why the prompt could be art.)
- OscarCunningham 3y agoBy analogy, would the art of a photograph consist only of the location and orientation of the camera at the time it was taken?
- eqvinox 3y agoIn a way, yes. But both for a photograph as well as AI-generated art, you'd want (or even need) to show you can in fact execute it. That's where the actual result comes together. (There's also slightly distinct terms to throw around here, art, artwork, creative spark, …) The problem with most "AI art" is that it has absolutely no visible creative spark. It's entirely possible to create art with AI, just — millions of random images or text aren't it. (btw: I'm trying to explain the understanding of "art" as it is [fuzzily] agreed upon by people active in the field. I may be getting things wrong, or explaining them badly. Of note, however, is that this isn't "just my personal opinion", I'm trying to convey an existing consensus of that field.)
- Am4TIfIsER0ppos 3y ago> you have to perform a creative action yourself The artist's unmade bed says hello!
- shadowgovt 3y agoThere are two creative actions related to AI art engines: 1) Selecting the prompt 2) Selecting the output
- Spivak 3y agoI'm not really sure what the hope here is for the people in the thread. Okay so the author makes it opt-in by default. The first user downloads the tool, tries to run it, goes "oh, well that's useless" forks the repo deletes the header check and runs it. Now to block them you have to employ "big league" blocking techniques instead of setting a header. Having it be opt-out actually makes it so server operators have the ability to turn it off because a few people doing it will go unnoticed. It's like how Microsoft killed DNT by making it the default. Sites can be like "sure, we can support DNT for the users that are really privacy conscious" but once it's every user the rubber meets the road of ad revenue.
- curiousbean 3y agoI offer a suggestion. Detect the scraper's IP/fingerprint and return deplorable porn instead of the original image.
- kungpao42 3y agoWhat user-agent does the tool use by default?
- blindpainter 3y agoThis is set of lawsuits waiting to happen. Bypassing or ignoring common practices like robots.txt put this on par with theft. Practices like this are a big part of the reason the EU and California adopted GDPR. The creator is clearly unethical and too lazy to ask for permission.