23 ms·

Extracting training data from diffusion models

• Archelaos 4y ago
  When I experiment with Stable Diffusion, I quite often come across blurred "Getty Images" labels.
  • distantsounds 4y ago
    don't worry, all the VC founders will exit with their startup funds long before the lawsuits come in!
  • nine_k 4y ago
    What if the prompt explicitly asks for an image from an image bank? For an image with a particular kind of watermark? I wonder if the dataset was marked up for that. Also intrtesting: a prompt asking for a well-known image, without a watermark.
  • astrange 4y ago
    If Getty Images has a problem with that, they should try having a working robots.txt. Instead they have one that's seemingly been copied from a porn site. https://www.gettyimages.com/robots.txt https://www.gettyimages.com/robots.txt
    • distantsounds 4y ago
      Ah yes, because robots.txt is definitely a valid deterrent for preventing people from stealing your images and using them for their own purpose!
      • aqme28 4y ago
        A watermark is the other good way around it, and it looks like that is working.
      • 2OEH8eoCRo0 4y ago
        These image sites might have a strong case. AI art can be considered a competitor and their copyrighted material was scraped and used to harm their business.
    • madeofpalk 4y ago
      Do these image sets not contain images found only via robots.txt excluded urls?
      • astrange 4y ago
        LAION respects robots.txt exclusions. Just think of it as Google Image Search.
    • BaseballPhysics 4y ago
      Unfortunately copyright law doesn't contain a "you didn't tell me not to take your images" affirmative defense.
    • fleddr 4y ago
      What would a Getty "working" robots.txt look like? For their images to not be permitted to be indexed? Being able to find their images is the entire point of the service. As is the case with almost every content creator/platform ever. Signaling that you allow indexing is no opt-in for any use or abuse.
    • dahart 4y ago
      This could be an attempt to keep logs and scrapes clear of explicit material, especially if Getty offers upload features to paying customers (I don’t know, but I did run a whiteboard site where one of the features allowed people to upload images, and saw first hand what some people do.) This doesn’t look anything like porn hub’s robots.txt…
• ericwallace_ucb 4y ago
  The paper shows that Stable Diffusion and Google's Imagen regenerate individual images from their training sets. They show it is very rare, but can be found reliably.
  • pavlov 4y ago
    Seems relevant to the Getty Images lawsuit against Stable Diffusion.
    • sdenton4 4y ago
      Eh... The mp3 decoder can generate copyrighted music if you feed it the right inputs... Likewise, in this work they prime the pump by using exact training prompts of highly duplicated training images. And then you have to generate 500 images from that prompt to find 10 duplications. You've really gotta want to find the duplicates, which indicates that these are going to be extremely rare in practice, and even more rare once the training data is hardened against the attack by deduplication.
      • BaseballPhysics 4y ago
        > Eh... The mp3 decoder can generate copyrighted music if you feed it the right inputs... Good analogy! An MP3 decoder takes an input and produces an output. If the output is copyrighted material, it's well understood that the inputs is simply a transformed version of that same copyrighted material and is similarly copyrighted. The SD model is very much analogous. The prompt causes the algorithm to extract some output from the input model. If the output is copyrighted material then similarly the input model must carry a transformed version of that same copyrighted material and is therefore also subject to copyright. Right? By the way, I pose this, but I highly doubt this is actually how the courts will rule. I think they'll find the model itself is fine, that the training is subject to a fair use defense, but that the outputs may be subject to copyright if there's substantial similarly to an existing work in the training set.
    • PartiallyTyped 4y ago
      The probability that synthesis of similar enough training samples given that the dataset does not contain duplicates is astronomically small, in this work they purposefully manipulated the dataset, used specific prompts and gave all sorts of advantages to an adversary, and created many images per prompt to be able to find such cases.
• larsiusprime 4y ago
  Some interesting commentary by AI expert Alex J. Champandard: https://twitter.com/alexjc/status/1620466058565132288 https://twitter.com/alexjc/status/1620466058565132288
  • polygamous_bat 4y ago
    I am highly sceptical of the "Expertise" of the "expert" you cite, as a current Ph.D student in a relevant field. The "hot take" also seems highly technically inaccurate, presenting it as a "lossy database" when it's very clear that the average or even more than the 90% percent of the images could not be reproduced. In which way is this a "lossy db"? That, combined with some snark and sound bites, thoroughly turned me off from continuing to read the thread. I am generally concerned, these days, because it seems like the days when I could just keep my head down and do science is over, and now I have to also defend myself against opportunist hype-masters who probably were jumping on the crypto bandwagon six months ago and now are self proclaimed AI experts.
    • haswell 4y ago
      > I am highly sceptical of the "Expertise" of the "expert" you cite, as a current Ph.D student in a relevant field. Will you expand on why? This is a big accusation to just throw out without exposition, your disagreement with the notion of a "lossy db" notwithstanding. You seem to be projecting quite a lot with this comment, re: snark. A few moments searching quickly show Alex isn't some crypto bandwagoner, and the implication that he is is distinctly unhelpful when trying to judge the value of each participant's comments in this conversation, here and on Twitter.
• jxy 4y ago
  I don't understand what is so surprising here. The training of the model consists of adding noise to training samples and denoising the resulting random samples to reproduce training samples. If you have one training sample, you can find the optimized random sample that reproduce the training sample.
  • chickenpotpie 4y ago
    Research isn't always about finding surprising things. Some of the most important research ever conducted is proving things that seem intuitive, especially when it proves the opposite.
  • sdenton4 4y ago
    An alternative framing if the article is that reproving of training examples is /rare/. It works for highly replicated images and, in larger models trained for longer periods of time, really unusual training examples. Both of these failure modes have pretty obvious mitigations: deduplicate the training data, and trim the outliers.
  • yorwba 4y ago
    After adding noise to a training sample, there's still some information about the original contained in the noisy image, so the model only needs to fix it up a little with the help of the prompt. But when you start from 100% noise, all of the information needs to come from the model and the prompt. Seen from another angle, if you take a random value for each pixel, you're unlikely to generate anything resembling a picture, let alone any given training picture you're trying to reproduce. That there's an input that makes the model output a training sample doesn't necessarily mean that it's easy to find. But the paper shows that you can find several by guessing randomly.
  • bfgoodrich 4y ago
    [dead]
• saurabh20n 4y ago
  The last author's tweet thread and replies have some interesting tidbits: https://twitter.com/Eric_Wallace_/status/1620449934863642624 https://twitter.com/Eric_Wallace_/status/1620449934863642624 * "We propose to extract memorized images by generating many times with the same prompt and flagging cases where many of the generations are the same." * "- Diffusion models memorize more than GANs - Outlier images are memorized more - Existing privacy-preserving methods largely fail" * "Stable Diffusion is small relative to its training set (2GB of weights and many TB of data). So, while memorization is rare by design, future (larger) diffusion models will memorize more." * "It only memorizes a very small subset of the images that it trains on." * "our goal is to show that models can output training images when generating in the same fashion that normal users do."
  • freilanzer 4y ago
    > * "It only memorizes a very small subset of the images that it trains on." An interesting question here would be: why does it memorise these images over others? Can the other images still be synthesised with loss via a suitable prompt? If so, are the memorised images important for this? Can this set be reduced further?
• jjcon 4y ago
  Is there any reason we shouldn’t view diffusion models as any other tool? I can infringe copyright with photoshop too… even accidentally. If I generate original work with either that seems like fair game. I imagine with the right prompt one could coax out a copywritten image even if it hadn’t ever seen it before
  • yunwal 4y ago
    There's still valid reason to be concerned here. For example, this implies building a diffusion model based on private data can leak it. If I can generate a whole bunch of prompts like "MRI Joe Biden brain tumor" and 1 out of a million times I get a consistent result, that's unacceptable. Github Copilot could be leaking private code as well.
    • zamadatix 4y ago
      It seems the only way they can get an extraction is if a highly duplicated (vs the average) training image is used which is counter to a privacy concern.
      • yunwal 4y ago
        This thread shows that there are outliers that are non-duplicated in the training set that still show up in results. https://twitter.com/alexjc/status/1620466058565132288 https://twitter.com/alexjc/status/1620466058565132288 Specifically, this post confirms cases of a single image in the training set being "memorized" https://twitter.com/Eric_Wallace_/status/1620475626611421186?s=20&t=7c_Atr8sQJSxNr9glANXAg https://twitter.com/Eric_Wallace_/status/1620475626611421186...
        • zamadatix 4y ago
          Ah I see the section on this in the paper now, the 2nd half of 7.1 on page 14.
• mxwsn 4y ago
  Their extraction: (1) assumes the attacker knows the caption for some training images, and (2) primarily works on images duplicated 100x-3000x in the training dataset. Their attack does not succeed for any singleton images. Deduplicating can be challenging on internet-scale datasets, but their work as presented does not appear to be a major concern for releasing diffusion models trained on other smaller datasets. On memorization - I suspect this is a great thing for downstream performance, and a positive indicator that diffusion models are actually better generative models than prior methods (VAEs, GANs, etc). This mirrors the finding that feedforward neural networks can memorize randomly labeled data very well. Intuitively it feels like memorization is a quantifiable behavior that is a foundational activity in information processing - it is one type of optimal usage of observed data - that superpowers downstream performance.
  • PartiallyTyped 4y ago
    > are actually better generative models than prior methods (VAEs, GANs, etc). Diffusion models are VAEs and follow the same variational framework. You could imagine that VAEs are diffusion models with a single step in the forward and backward processes ;). They actually optimize the same VLB objective, but with diffusion models the objective is a trajectory instead of a single step, however, when training we are optimizing single step transitions. This is possible because the objective ends up being a sum of logarithms, thus there is no dependence between terms. In practice we solve a simplified objective which looks a lot like as we do with standard AutoEncoders ;) The key component that differentiates the two is in what we expect of the underlying neural network. It is far easier to parameterize small changes than large ones, with VAEs you ask the decoder to produce a large change in the latent variable, whereas with diffusion, we generally split them into 4000 smaller changes assuming you are using the DDPM approach and not the DDIM one. Because we are improving with very small steps, we avoid the blurriness of VAEs, and we don't go out of distribution when sampling random noise. VAEs are often difficult to synthesize because even with KLD in the objective, the encoder produces a low variance distribution, and so when we sample noise from a high(er) variance gaussian, we are out of distribution rather quickly.
    • mxwsn 4y ago
• Imnimo 4y ago
  Only 109 retrievable images out of the 350,000 most-duplicated is fewer than I expected. Maybe it's just the stringent definition of retrieval, but I would have expected many famous works of art like the Mona Lisa and Girl with a Pearl Earring to be readily extractable. Maybe these just aren't quite pixel-perfect enough?
• yorwba 4y ago
  I wonder whether the "data dimension" from https://transformer-circuits.pub/2023/toy-double-descent/index.html#comment-mnist https://transformer-circuits.pub/2023/toy-double-descent/ind... could be used to identify the model parameters involved in memorization and remove them without having to retrain from scratch on a cleaned-up dataset.
• mshake2 4y ago
  I expect to see this paper in many lawsuits soon as evidence of copyright infringement.
• Lerc 4y ago
  Figure 2 doesn't fill me with confidence as to the ability to detect similar images. The best example is the bottom right match which hits because the collar is in the same position and a bunch of white being in the same place outweighs a lot more meaningful data. This probably means there are far more matches to be found that would be considered clearly copies to humans. SSIM might be a bit heavy for the task but a simple comparison of the gradients from neighboring pixels might match quite a lot more.
• GaggiX 4y ago
  So wait they only found 109 matches after generating 175 milion images using the prompt from the most duplicated samples from the dataset and SD v1.4? Also almost all of them have more than 300 copies in the dataset, so with a model with the same size and trained on a dedup dataset like SD 2.0/2.1 there will be almost no matches, even after generating 175 mln images and knowing the prompts used in the dataset. Finally Google at el need to explain how an attacker that want to extract images from a trained model somehow has the prompts for the top X duplicated images in the dataset but not the images themselves, and thus will going to spent an incredible amount of money to generate something like 175 mln samples and test them together to find the matches. Edit: I also want to add that google seems to try really hard to show themselves as the good guys by not releasing its models because it's not safe enough, but in this paper they used an incredible amount of computation and show me otherwise.
  • babel_ 4y ago
    The argument here is more that this attack serves to just "find" these, and while it uses significant compute to do so, it's also not using many of the complex and more efficient extraction approaches that have been used on GANs and such in prior research. They're using a simple method, and exploring what it does; most of their 175 million images comes from effectively "retrying" each prompt 500 times, which could easily be cut down when others are trying attacks, since they're usually much more targetted than this at what they're trying to extract. Also, "almost no matches" is still problematic, because they still occur, and since they don't seem prevalent, they're unlikely to have been accounted for in any other way (SD 2 being finally based on a deduplicated dataset). So, in a sense, they're "still in there", it's just harder to find by a casual or simple approach as above. Again, one of the central themes is that diffusion seems to be less resistant to attacks than, say, GANs, and that may mean more efficient and complex extraction attacks may translate over even more effectively.
    • GaggiX 4y ago
      >it's also not using many of the complex and more efficient extraction approaches that have been used on GANs and such in prior research Some links? >most of their 175 million images comes from effectively "retrying" each prompt 500 times Prompts from the most duplicated samples in the dataset, a really important aspect if you actually want to used this method in the wild, this is also one of the reasons why I said that this attack seems so implausible. >they're usually much more targetted than this Even if you target some images you would still need an absurd amount of luck if with the most duplicated sample you only get 109, we can be generous and think that with the whole dataset will have something like 200 matches, the probability of finding an image with a direct attack is still less than a million (even if you know the prompt) and we're not talking about a model trained on a deduped dataset.
• 6gvONxR4sf7o 4y ago
  It's work like this that makes me frustrated at the popular discourse around generative models (especially here). There's a ton we don't know about these models, and yet you get tons of people arguing that these models absolutely don't memorize, or that they learn like we do and so their learning should be treated like ours (legally and ethically). Then you get work like this showing that yes they actually do some memorization and regurgitation. There's still so much we don't know here. My fear is that when things like this come up for lawsuits, overconfident experts are going to talk out of their asses about how these models do or don't work, and that's going to determine how automation affects our society. On a technical level, I'd love to see a patch-wise version of this investigation. This shows whole images being regurgitated near-exactly rarely. I expect that small part-of-the-image patches are regurgitated even more often. But is it simple stuff like edges being regurgitated or are larger parts regurgitated frequently too? Given the architectures generally used, I'd guess that it's significant.
  • deleted 4y ago
    [deleted]
  • cma 4y ago
    Artists who copy another's style well surely memorize many of their works too.
    • motoxpro 4y ago
      I can memorize your patented design too. Doesn't make it less illegal for me recreate it from memory.
      • kurthr 4y ago
        It does if it's over 20 years from the application date. It does if you match on a design published before the application date. It does if you change the design just enough not to read onto a single claim. Honestly, training a generative model on the patent database could be very useful for inventing and invalidating patents. I wouldn't be surprised if examiners are using it in 5+ years. Since they've got required response count and time, I wouldn't be surprised, if they started using it just to speed up their own work.
• diimdeep 4y ago
  Already there are hundreds 'fine-tuned' or merged models, made with base models from Stable Diffusion and easy to use inference and training tools like this[2] I wonder whether extraction attacks easier if you have many ancestral models? [2] https://github.com/AUTOMATIC1111/stable-diffusion-webui#stable-diffusion-web-ui https://github.com/AUTOMATIC1111/stable-diffusion-webui#stab...
• rvz 4y ago
  Also another previous study that has Stable Diffusion (SD) emitting images in the training set [0] It is now clear that SD is treading on thin ice: training on watermarked and copyrighted images without their author's permission, then attempting to commercialize it even when the model emits images that resemble a high similarity of the original training data including watermarks or copyrighted images: (Mickey Mouse, Getty Images watermarks, Bloodborne art cover, etc). This weakens their fair use argument, especially with Getty Images also threatening to sue SD for the same reason. If OpenAI was able to get permission to train on shutterstock images [1], then SD could have done the same, but chose not to. Perhaps SD thought they could get away with it and launch their grift (DreamStudio) on digital images and artists. It turns out that now SD creates an opt-out system afterwards but artists can already find out if their images are in the training set. [2]. [0] https://arxiv.org/pdf/2212.03860.pdf https://arxiv.org/pdf/2212.03860.pdf [1] https://www.prnewswire.com/news-releases/shutterstock-partners-with-openai-and-leads-the-way-to-bring-ai-generated-content-to-all-301658310.html https://www.prnewswire.com/news-releases/shutterstock-partne... [2] https://haveibeentrained.com/ https://haveibeentrained.com/
  • FeepingCreature 4y ago
    Note that the authors of this study believe that deduplication should be effective with SD, where there is a large ratio of dataset size to model size. (See section 7.1.)
  • mensetmanusman 4y ago
    There would have to be a new law, because stable diffusion is a type of search space algorithm. Previous rulings said watermarked images are fair use for search engines.
    • madeofpalk 4y ago
      Couldn't a lawsuit 'simply' declare what Stable Diffusion is doing is not fair use, and thus falls foul of existing laws?
      • 4y ago
• FeepingCreature 4y ago
  100 images out of 350,000 that they looked at were memorized. This seems to mostly happen when an image appears frequently (more than 100 times) in the training data, and/or the dataset is small relative to the model.
  • deleted 4y ago
    [deleted]
  • londons_explore 4y ago
    Removing duplicates and very near duplicates ought to be one of the first things done to any training dataset... I wonder why this wasn't done? Too computation heavy?
    • Xelynega 4y ago
      If by "computationally heavy" you're talking about the financial overhead of hiring another layer of human filters to filter duplicates out of the dataset.
      • ben_w 4y ago
        That's an unreasonable presumption. Deduplication, to the best of my knowledge, requires every image be compared to every other image. This is necessarily O(n^2) on n images. IIRC the training set is 2.3 billion images, if so that's 0.5 * 5.29e18 comparisons[0], which, if done by humans, would require employing literally all humans for approximately a year even if we compared 12 images per second 24/7. This has to be done computationally, not by humans. [0] half because (a = b) <=> (b = a)
        • raincole 4y ago
          You can roughly classify the images first (we already have very good AI model for this) and use humans as an additional layer. If two images don't get the same top-3 labels from the classifier, the chance that they're duplicates is neglectable.
• danaris 4y ago
  I know it's not cool to say "I told you so", but... This was entirely predictable, and is one prong of the primary arguments that these ML models, trained on datasets including copyrighted images taken without permission, infringe on the copyright of those images' creators. Train the damn things on public domain images and images you have explicit permission for, and you'll be fine. Stop acting like you have a right to just vacuum up every image ever created because it's "AI".
  • astrange 4y ago
    They literally have such a right because there is literally such a right written into German law. And UK law is adding that right even for commercial models.
    • omnicognate 4y ago
      Which laws are you referring to? It's the first I've heard of countries adding explicit rights for training models.
      • astrange 4y ago
        https://www.clarin.eu/content/clic-copyright-exceptions-germany https://www.clarin.eu/content/clic-copyright-exceptions-germ... https://www.linklaters.com/en/insights/blogs/digilinks/2022/july/uk-government-announces-new-copyright-exception-for-commercial-text-and-data-mining https://www.linklaters.com/en/insights/blogs/digilinks/2022/... (Note, it's looking less likely the UK one will actually get extended that way now.)
        • omnicognate 4y ago
          "... for non-commercial purposes only ..." seems an important limitation in the first link.
    • danaris 4y ago
      So as long as you can find (or bribe) one country that will let you skate by, you think it's fair game to just declare that your model's country of residence, then take all the images you want and put out your trained model to the whole world? Somehow, I don't think that one's likely to fly.
• xiphias2 4y ago
  Looks like it was funded by Deepmind for the purpose of fighting more open models on the legal field. I don't think they are just ,,protecting the artists''.
• sschueller 4y ago
  What will happen if SD looses the court case? The cat is out of the bag and the data set can be downloaded by anyone today.
  • sekai 4y ago
    Nothing, no one can stop this now. It's like horses being angry at cars.
    • a13o 4y ago
      Cars don't have tiny horses inside them making them work, so the analogy doesn't apply. These AI models can't do anything without first gorging on existing content. A lot of that existing content is copyrighted, and demonstrably present in the resulting product; so these companies have an uphill battle _not_ to be stopped by the courts.
      • sekai 4y ago
        > A lot of that existing content is copyrighted, and demonstrably present in the resulting product; so these companies have an uphill battle _not_ to be stopped by the courts. These lawsuits will go exactly like in the war against piracy, impossible to win. Cut one head off, ten more appear.
        • a13o 4y ago
          Piracy lawsuits won constantly, enough to push all piracy to the dark web. They even sued little old grandmas and won. No legitimate business today is built on top of piracy. I'm not arguing that generative art will cease to exist, I'm just arguing that the path to legal revenue will involve paying copyright holders; and the real gold in this industry is in licensing training datasets to businesses that just want 'AI Magic' in their product marketing.
        • pessimizer 4y ago
          But they did win the war on piracy. They're still making tons of money, the piracy community is the same size or smaller than it was 10 years ago, and most of the people inside it are either committed cheapskates who would never have been customers, or committed collectors who while pirating are also buying twice as much as their neighbors.
• singularity2001 4y ago
  This is really bad news for the community, especially in the context of the Copilot lawsuit. Soon lawyers will terrorize network creators, startups and users.
  • JustFinishedBSG 4y ago
    Is the bad news that they were right? Because that's only a bad news if you didn't like the truth.
    • deleted 4y ago
      [deleted]
  • phoe-krk 4y ago
    People profitting from techniques meant to infrige copyright will be persecuted for copyright infrigement, "bad" news at 11.
    • scotty79 4y ago
      If you copyrighted a point (2, 1) then is the equation y=3x-5 infringing on that copyright?
      • deleted 4y ago
        [deleted]
      • alkonaut 4y ago
        Yes, if the law considered a point(2,1) to be copyrightable, and no otherwise. But more seriously: I think the question is perhaps a bit vague.
        • scotty79 4y ago
          That's insane. Then by just copyrighting a single point you make infinite number of math functions infringing on your copyright.
• usrbinbash 4y ago
  So some images were overrepresented in the dataset, and subsequently, the network overfitted. Known problem, known solution.
  • cool_dude85 4y ago
    The thread specifically says that deduplication, based on their own attempts with smaller models, helps but is not sufficient to prevent this.
  • haswell 4y ago
    The problem and solution are independent of the implications of this finding and how those implications are likely to influence the legal landscape. To me, the implication is that these models cannot be seamlessly exchanged for a human brain when considering their impact and compatibility with current laws.
    • gpderetta 4y ago
      Human brains can produce near duplicates of things they have seen in the past. Of course, like an human, an NN can be used to generate copyright infringing images, but it doesn't follow that any generated image is infringing.
      • haswell 4y ago
        > Human brains can produce near duplicates of things they have seen in the past. Most humans cannot, and this is important, because the fact that most humans cannot arguably played a role in the formulation of all current rules. So can cameras. And there are laws that restrict where cameras can be pointed in places that do not restrict human sight, because the two kinds of "seeing" are distinctly different. Layering AI into the mix doesn't suddenly remove or mitigate the technical realities of the software. > but it doesn't follow that any generated image is infringing. I agree, and this is not my claim.
        • pixl97 4y ago
• bethecloud 4y ago
  Person enters "Ann Graham Lotz", image of Ann Graham Lotz appears. Why does this upset people and google image search doesn't
  • JustFinishedBSG 4y ago
    Because people like to pretend here and on other tech sites that somehow generative models cannot and do not memorize things and therefore those pesky things like copyrights and attribution should not apply to them. Compression is prediction.
  • raincole 4y ago
    Because google image search "cites" the source. It's a big difference. Of course google image search can show a wrong source, but it's still a big difference between "absolutely nothing" and "potentially wrong thing".
    • hiidrew 4y ago
      probably absolutely madness but would be cool if these generative tools could cite the training images based on prompts, and maybe artists could be verified and receive a micropayment for when their image is used as an input for an image
      • raincole 4y ago
        "these" generative tools can't do that. The transformer-based ML algorithm just can't do that. Of course maybe in the future we'll have a better method, but it will be a very different thing from what we have today.
      • eropple 4y ago
        That seems generally antithetical to the goals of the folks making these, which is "don't pay a human for what a computer can do instead".
        • bethecloud 4y ago
• alexb_ 4y ago
  Anybody who knows what the pigeonhole principle is should know that a lot of these fears are complete bunk.
  • cedilla 4y ago
    I know what the pigeonhole principle is. Could you elaborate how it leads to what fears to be bunk how?
    • alexb_ 4y ago
      The biggest fears are "AI is theft, the model just remembers what you give it. The proof is in the fact that you can reproduce what it gets". When in reality, a model that is smaller than the training data literally cannot reproduce all of the images it's trained on.
      • alkonaut 4y ago
        The argument isn't that it can reproduce ALL images, but that it might occasionally reproduce one image. And if that is devastating to e.g. the economic feasibility of a product built on the model - then that's an existential threat to that business model. To make a really dumb example: I can make a black box model that I push in 1000 bestseller novels through, that has the total "storage size" of 1 novel. The pigeonhole principle says my model can't possibly contain all 1000 novels. If you ask it (anything) it will respond with that one novel, verbatim. It never reproduces anything else. Does it now matter whether I trained it on 1000 novels? Does it matter that my "black box" is just that, a literal black cardboard box containing just a copy of a novel?
        • pessimizer 4y ago
          Actually a really great example. Wouldn't be surprised if some enthusiastic future corporate-friendly circuit court ruling accidentally made this a legal copyright-removing machine.
  • tsukikage 4y ago
• Glyptodon 4y ago
  To me this is kind of like being shocked that people who've seen the Starry Night can remember what it looks like.
  • tomasreimers 4y ago
    I think until now this has been a complaint about generative AI applications that has been handwaved away ("what if co-pilot generates GPL code?" -> "Well it will generate a blend of many pieces of code") If there is now evidence that it does memorize and emit training data, that argument had new life. That said, I imagine it's easy enough to add a post-process step to ensure your result isn't a member of the training set.
    • Glyptodon 4y ago
      I don't know. It's not like remembering GPL'd code you've read makes your brain subject to GPL or code you write that uses similar structure or variable names for a loop or something GPL.
  • BaseballPhysics 4y ago
    And the reaction among many, here, seems to be shock or disbelief that someone who's memorized Starry Night and then proceeded to reproduced the work would run afoul of copyright law.
    • Glyptodon 4y ago
      I don't know, people training themselves by trying to emulate well known or successful artists and styles has long been a thing if my memory of Deviant Art 20 years ago is accurate. And most every comic con I've attended is overrun with examples of the same kind of thing even now. Or tattoo artists giving people tattoos of movies characters and scenes. Or the multiple times in school that an art assignment involved more or less copying or reinterpretation of an existing work. That said, with humans it's not like anyone tries to prevent us from having the capability, it's just that usually people know better. Otherwise going to art galleries would be problematic.
• yetanotheruser8 4y ago
  This study was organized by Google (Technically DeepMind). I wouldn't be surprised if Google is wanting the lawsuit to lose. It would block open-source models like these from existing and give them potentially a competitive advantage to be able to afford whatever compliance is mandated. They'd be able to offer services that comply, but open-source models would only have access to lower quality data and would be stunted.
  • random_moonwalk 4y ago
    I think there's a concern also that smaller companies can gain a competitive edge via resistance to reputational damage. As Yann LeCun recently tweeted: 'By releasing public demos that, as impressive & useful as they may be, have major flaws, established companies have less to gain & more to lose than cash-hungry startups. If Google & Meta haven't released chatGPT-like things, it's not because they can't. It's because they won't.' (https://twitter.com/ylecun/status/1617908306420600833 https://twitter.com/ylecun/status/1617908306420600833)
    • sangnoir 4y ago
      To your point on reputation: Didn't Meta recently unpublish a LLM after it was shown to hallucinate wrong answers? Smaller AI companies would have stuck to their guns, but Meta has internal controls to guard against damaging the "brand". IIRC, the tweet announcing the retraction was particularly salty,<speculation> sounded like someone whose hand was forced</speculation>
      • dr_zoidberg 4y ago
        Yes, the Galactica LLM by Meta. Though LeCun isn't an author in the paper, he is "Chief AI Scientist for Facebook AI Research (FAIR)"[0], and he was quite angry about closing the Galactica demo[1]. [0] https://ai.facebook.com/people/yann-lecun/ https://ai.facebook.com/people/yann-lecun/ [1] https://twitter.com/ylecun/status/1593293058174500865 https://twitter.com/ylecun/status/1593293058174500865
• babel_ 4y ago
  The tweet/paper co-author posted the paper (https://arxiv.org/abs/2301.13188 https://arxiv.org/abs/2301.13188) yesterday on HN (https://news.ycombinator.com/item?id=34596187 https://news.ycombinator.com/item?id=34596187) and ironically the top comment there is referencing to this exact tweet thread (which was posted yesterday as well). Evidence for the metacircular evaluation of HN comments? I think the paper is well worth the read, it's not particularly long (much is references and appendices), and nicely written, with at least a quick bit on most things I would think to test as part of something like this. Good stuff.
  • dang 4y ago
    We'll merge the threads. Thanks! (Normally I wouldn't have moved your comment as part of the merge but (a) you said something about the paper, and (b) after name-dropping metacircularity how could I not)
• ornornor 4y ago
  In case you’d rather not suffer twitter’s abysmal UI on mobile web browser: https://nitter.net/Eric_Wallace_/status/1620449934863642624 https://nitter.net/Eric_Wallace_/status/1620449934863642624
• 6gvONxR4sf7o 4y ago
  I’m disappointed in all the anthropomorphizing in this thread. Time and time again, we make analogies for how black box ML algos must work like people, only for researchers to come along and show that they actually just use shortcuts that don’t remotely resemble human learning/thinking. When will we learn to stop being overconfident about how these things work? Just say “we don’t know yet.” Anthropomorphism and overconfidence are dangerous in that we could set the wrong precedents (culturally and legally) for how these are used and how automation affects society.
  • welshwelsh 4y ago
    That only answers half of the question. Maybe ML uses some weird shortcut, but how do we know the human brain doesn't use the same shortcut? If it's possible to use some simple hack to do something, why didn't we evolve to work that way?
    • rileyphone 4y ago
      Brains are radically different from GPUs.
      • celu 4y ago
        The same calculations can be performed by an abacus. What is doing the calculation is irrelevant. The question is what the calculation is
        • vlovich123 4y ago
          That's kind of a philosophical question no? Is being able to model an analog behavior accurately the same thing as the analog behavior itself? While we know from math there's an equivalence / bounded error rate, it certainly seems like emulation in the digital domain is far more power intensive which would indicate that it's not the thing itself. A clearer example is photon collisions. Simulating that behavior on a computer is not the same thing as colliding the photons. Could be wrong though.
• fab1an 4y ago
  The way the author summarizes his own study in this thread borders on misinformation. You could actually take their findings and write the opposite headline, which would more accurately reflect their actual research results: "Critics claim that models such as Stable Diffusion act like modern collage tools, recreating copyrighted and sensitive material. Yet, our new paper shows that this behaviour is exceedingly rare, recreating copies in less than 0,00006% of 175M test cases."
• seydor 4y ago
  Eigenimages
• KHRZ 4y ago
  Copyright holders screeching again? Maybe I should copyright a black image and sue anyone who turns off their screen.
• adrianmonk 4y ago
  Ironically, this almost makes it more human. It's a surprisingly common experience for music students to excitedly tell everyone they know about a new piece of music they've been composing, saying it is probably the best thing they've ever written, and then a friend or teacher has to say, "I don't know how to break it to you, but you've 'composed' the Xth movement of Beethoven's Yth symphony." And sometimes they will say, "I have? I don't think I've ever heard Beethoven's Yth symphony." But of course they have, just without realizing it. It was in the background of some movie they watched or something like that. Unlike humans, I don't think AIs have any belief about whether their work is original or not, but it's the same type of error. And with similar legal consequences: people have been sued for stealing a melody (presumably not always consciously). The difference with AIs is they can produce much more output than humans, and it's muddier what is actually doing the creating (AI authors? users?).
  • unixlikeposting 4y ago
    >Ironically, this almost makes it more human. not really, no. It's more the case that a NN is an encoding of a large dataset space, and if you go far enough away from the average you may find yourself on an island with an individual outlier. This is why you simply get back the training example, it's regurgitating all that it was fed. For example, try to train a model using a single image, with a space as wide as a sole point you will find that all you get back is that point. A human being re-inventing without remembering the source is distinct from remembering the source and returning it when lacking other data.
• deleted 4y ago
  [deleted]
• dang 4y ago
  See also https://twitter.com/Eric_Wallace_/status/1620449934863642624 https://twitter.com/Eric_Wallace_/status/1620449934863642624. (Thanks to all who posted that. We merged the threads now.)