4 ms·

Although the weights aren't available, I wanted to note that the model source itself is actually hosted at https://github.com/openai/guided-diffusion https://gi

by gfaure 5y ago

Although the weights aren't available, I wanted to note that the model source itself is actually hosted at https://github.com/openai/guided-diffusion https://github.com/openai/guided-diffusion.

sillysaurusx 5y ago
Indeed, the weights aren't available, and researchers seem to console themselves by saying "at least we're publishing code." Well, thank you. Code is nice. Y'know what else is nice? Being able to reproduce your claimed results without investing six months of mental effort and 500 GPU hours on a V100. (Sorry, I'll step off the rant now. I just miss the old days, the ye olde long-long ago of "one and a half years." It seemed like every other day there was another neat model release that happened to be world-changing as a matter of course: StyleGAN, BigGAN, GPT-2. Before that there was YOLO and a ton of other neat stuff. Nowadays it's "Oh golly gee, I'm not so sure we should release this panda generator, it might have ethical implications down the line if people start generating pandas in sexy poses." But hopefully it's just a phase or something.)
- austinjp 5y ago
  No need to stop the rant :) I'm curious about the reasons models +/- weights are published less frequently. For starters, is that actually the case? I don't know, genuine question. Second, and I recognise the humorous intent regarding sexy pandas, are ethical concerns actually the reasons some models/weights are not released? Or is it about hoarding a potentially valuable commodity? Or some other reason, or a combination of several?
  - sillysaurusx 5y ago
    is that actually the case? [are model weights being published less frequently?] It sure feels like it. But it could be survivorship bias: https://www.youtube.com/watch?v=_Qd3erAPI9w&t=52s&ab_channel=2veritasium https://www.youtube.com/watch?v=_Qd3erAPI9w&t=52s&ab_channel... or regression to the mean: https://www.youtube.com/watch?v=1tSqSMOyNFE&ab_channel=Veritasium https://www.youtube.com/watch?v=1tSqSMOyNFE&ab_channel=Verit.... It's possible that StyleGAN and GPT-2 just happened to coincide, and both companies just happened to decide to release their models rather than hoard them, and that the usual case is for companies to rarely (if ever) release weights with their research. But the more interesting question is, why wouldn't they? So onward to your second one: are ethical concerns actually the reasons some models/weights are not released? Or is it about hoarding a potentially valuable commodity? Or some other reason, or a combination of several? As with most things, the answer is likely the mundane, dull truth that big companies tend to have lots of inertia and friction preventing such things from happening. Sources of friction seem to include (but are not limited to): - are we legally responsible for what other people do with the model? (no.) - should we restrict people from using the model commercially? (why bother? but sure, nVidia did that for stylegan. It didn't stop Artbreeder from blatantly ignoring and monetizing it anyway. And while that sounds negative, Artbreeder is one of the coolest ML projects ever, as far as I'm concerned, so the world is better off for ignoring your stupid policy.) - What if people start misusing the model? (So what? I feel like I should just get in the face of whoever is asking this question and repeat "So what?" until they have the police escort me off the premises. You can keep asking it to whatever they reply with, and eventually their logic never seems to go anywhere but one big ass-covering circle.) - What if we might look bad because of it? Now that last bullet point deserves some real treatment. In my opinion, the head of Facebook AI has done one of the most damaging things possible to the ML scene by getting everyone riled up about GPT generating "immoral" outputs: https://twitter.com/an_open_mind/status/1285940858290933767 https://twitter.com/an_open_mind/status/1285940858290933767 https://twitter.com/an_open_mind/status/1286034549345071105 https://twitter.com/an_open_mind/status/1286034549345071105 https://twitter.com/an_open_mind/status/1285691693677850627 https://twitter.com/an_open_mind/status/1285691693677850627 Good lord, AIs are generating racist outputs! They're saying that cis white men are intellectually superior to trans people! People are generating child porn in AI dungeon! Someone, do something! ... please. Every researcher I've talked to has rolled their eyes hard at the criticisms raised by Jerome. But since he's the head of Facebook AI, no one dares say so publicly. It may as well be an outsider like me: Jerome, I know your heart is in the right place, and that you believe very strongly that this is an important moral issue. But your moral concerns need to be balanced by the ethical considerations of the massive chilling effect you had by publicly shaming OpenAI so hard that they ended up completely losing their confidence, adding some ridiculous profanity filter to GPT-3 that flags pretty much anything mildly naughty, or the fact that AI dungeon is now in hot water because you've given the world an excuse to be pissed off about a mindless, memory-less AI (https://news.ycombinator.com/item?id=23346972 https://news.ycombinator.com/item?id=23346972) generating text that offends someone, somewhere, for some reason. Well, people are going to be offended. Let them be. I am half Jewish. There is a very realistic chance that GPT-3 was trained on the full text of "Mein Kampf," and I could care less. Even if someone went out of their way to train the most offensive, most threatening language model the world has ever seen, what could they really do with that power? Are they going to hurt you right in the feelings? Is the language model going to argue vigorously for the extermination of all Jews? If it did, who would listen? No no, don't try to say that yes, there's a very real danger that people might generate propaganda and influence others. You know as well as I do that this is extraordinarily difficult in practice for multiple reasons, and that no one has ever seen an interactive, adaptive AI that can dynamically deliver the most persuasive propaganda to Reddit and fool everybody into thinking it's a genuine grassroots movement. And you know as well as I do that that would be roughly equivalent to inventing AGI, and that AGI is still nowhere in sight. It's not even living in the same country. Hell, it's not in our sector of the galaxy. See that star in the sky? It's possible that we're more likely to reach that star before we invent AGI, because nobody knows how to invent AGI yet. Before I get too worked up, I should keep some focus and say constructive things. One. If you, as an ML researcher, find yourself wanting to release your work, but your corporation is putting the red tape around you, push back. They need you more than you need them, even if it doesn't feel like it. T Two. Releasing work is generally beneficial for the world. I can't think of a single model release that has ever harmed the world. Let's wait until one actually does before we freak out about whether it might. Three (https://www.youtube.com/watch?v=jpw2ebhTSKs&ab_channel=TheChalkeaters https://www.youtube.com/watch?v=jpw2ebhTSKs&ab_channel=TheCh...). By releasing your work, you give countless people the opportunity to better themselves. Thanks to GPT-2 1.5B, we were able to vastly extend the capabilities of https://reddit.com/r/subsimulatorgpt2 https://reddit.com/r/subsimulatorgpt2 to include over a hundred subreddits in a single model. This has roughly zero commercial value, yet has improved the lives of countless people who show up to giggle at all the (sometimes horribly offensive) things that robots say to each other. I was proud to be a part of that, and I want to be a part of more. Please release your models. The most recent model release was CLIP, and it's already had a profound impact. Just look at how freaking awesome this is! https://twitter.com/l4rz/status/1367853921427984390 https://twitter.com/l4rz/status/1367853921427984390 They're using CLIP to turn someone into Dracula! That's badass, and it inspires people (like me, at one time) to get into ML and become the researchers of tomorrow (as I try to be now). I remember rooting for OpenAI's Dota 2 bot while they were facing off vs .. OG, I think? It's been several years. I spent five years playing dota-type games. It was one of the most exciting things I'd ever seen. And being able to beat rtz 1v1 mid SF? Not merely beat him, but annihilate him? Holy crap, give me that model. Why are you not releasing that model?! It's so cool! And now nobody gets to play with it ever, and it's locked up in the real AI dungeon: OpenAI's. I hope they reconsider.
    - ctchocula 5y ago
      > Two. Releasing work is generally beneficial for the world. I can't think of a single model release that has ever harmed the world. Let's wait until one actually does before we freak out about whether it might. I like a lot of the idealism in this comment, and learned a few things too, so thanks for that. I'm not an ML researcher, which may cloud my views a bit, but I think you would be hardpressed to think that the release of ML models hasn't harmed the world. Perhaps we haven't reached AGI, but we don't need to for the release of ML models to harm the world. I'll give two examples: 1) image detection - Now that image detection is good enough, surveillance in dictatorships can be run more efficiently and scalably in a way KGB couldn't have dreamt of. Even if the release of a model like YOLO saves an hour's worth of time of an ML researcher working for the dictatorship on their surveillance project, this can cause a lot of harm to the oppressed ppl living in those countries. 2) troll bots based on GPT-2. You gave an example of high-quality, human-quality propaganda, but what troll bots lack in quality can be made up by quantity. If you run a lot of troll bots and can sway the dominant viewpoint on all the forums you want to target (which you can since these bots are infinitely scalable), you've achieved your purpose. I also think you are overestimating the quality needed to influence the average person's worldview. For example, we saw from the Cambridge Analytica news that all they had to do to affect a few swing voters' behavior was target them with a few ads. I personally read through some of the example output from GPT-3 and if I were browsing a forum, I wouldn't be all that confident whether they were a bot or not.
      - anewhnaccount2 5y ago
        The thing is that training the models themselves is well within the resources of even the smaller actors that would want to use them in these way. It's the interested but not-quite-interested-enough or time-poor enthusiasts that are hit the worst.
      - _hyn3 5y ago
        Why do research on something at all, if it could be misused? What if we invent a new sabre blade that's made out of light and can cut through almost anything, including people? Should we not invent it? Would we only give it to a Jedi, who could be trained to never do bad things with it? Why are these researchers doing research on this? Seriously, why? Why do research on something and then not release the model over gasp ethical concerns -- if they had ethical concerns, why did they pursue this avenue of research in the first place? Why advance the state of the art if it's wrong to do so? Not releasing the model for "ethical" concerns is a cop-out. There's probably another reason; what is that reason?
- deleted 5y ago
  [deleted]
- queuebert 5y ago
  Very generous of you to assume their model is reproducible.
- kevinwang 5y ago
  Well, better safe than sorry with releasing a model with potentially harmful effects, right?
  - deleted 5y ago
    [deleted]

by gfaure 5y ago

Although the weights aren't available, I wanted to note that the model source itself is actually hosted at https://github.com/openai/guided-diffusion https://github.com/openai/guided-diffusion.

sillysaurusx 5y ago

Indeed, the weights aren't available, and researchers seem to console themselves by saying "at least we're publishing code." Well, thank you. Code is nice. Y'know what else is nice? Being able to reproduce your claimed results without investing six months of mental effort and 500 GPU hours on a V100. (Sorry, I'll step off the rant now. I just miss the old days, the ye olde long-long ago of "one and a half years." It seemed like every other day there was another neat model release that happened to be world-changing as a matter of course: StyleGAN, BigGAN, GPT-2. Before that there was YOLO and a ton of other neat stuff. Nowadays it's "Oh golly gee, I'm not so sure we should release this panda generator, it might have ethical implications down the line if people start generating pandas in sexy poses." But hopefully it's just a phase or something.)

austinjp 5y ago
No need to stop the rant :) I'm curious about the reasons models +/- weights are published less frequently. For starters, is that actually the case? I don't know, genuine question. Second, and I recognise the humorous intent regarding sexy pandas, are ethical concerns actually the reasons some models/weights are not released? Or is it about hoarding a potentially valuable commodity? Or some other reason, or a combination of several?
- sillysaurusx 5y ago
  is that actually the case? [are model weights being published less frequently?] It sure feels like it. But it could be survivorship bias: https://www.youtube.com/watch?v=_Qd3erAPI9w&t=52s&ab_channel=2veritasium https://www.youtube.com/watch?v=_Qd3erAPI9w&t=52s&ab_channel... or regression to the mean: https://www.youtube.com/watch?v=1tSqSMOyNFE&ab_channel=Veritasium https://www.youtube.com/watch?v=1tSqSMOyNFE&ab_channel=Verit.... It's possible that StyleGAN and GPT-2 just happened to coincide, and both companies just happened to decide to release their models rather than hoard them, and that the usual case is for companies to rarely (if ever) release weights with their research. But the more interesting question is, why wouldn't they? So onward to your second one: are ethical concerns actually the reasons some models/weights are not released? Or is it about hoarding a potentially valuable commodity? Or some other reason, or a combination of several? As with most things, the answer is likely the mundane, dull truth that big companies tend to have lots of inertia and friction preventing such things from happening. Sources of friction seem to include (but are not limited to): - are we legally responsible for what other people do with the model? (no.) - should we restrict people from using the model commercially? (why bother? but sure, nVidia did that for stylegan. It didn't stop Artbreeder from blatantly ignoring and monetizing it anyway. And while that sounds negative, Artbreeder is one of the coolest ML projects ever, as far as I'm concerned, so the world is better off for ignoring your stupid policy.) - What if people start misusing the model? (So what? I feel like I should just get in the face of whoever is asking this question and repeat "So what?" until they have the police escort me off the premises. You can keep asking it to whatever they reply with, and eventually their logic never seems to go anywhere but one big ass-covering circle.) - What if we might look bad because of it? Now that last bullet point deserves some real treatment. In my opinion, the head of Facebook AI has done one of the most damaging things possible to the ML scene by getting everyone riled up about GPT generating "immoral" outputs: https://twitter.com/an_open_mind/status/1285940858290933767 https://twitter.com/an_open_mind/status/1285940858290933767 https://twitter.com/an_open_mind/status/1286034549345071105 https://twitter.com/an_open_mind/status/1286034549345071105 https://twitter.com/an_open_mind/status/1285691693677850627 https://twitter.com/an_open_mind/status/1285691693677850627 Good lord, AIs are generating racist outputs! They're saying that cis white men are intellectually superior to trans people! People are generating child porn in AI dungeon! Someone, do something! ... please. Every researcher I've talked to has rolled their eyes hard at the criticisms raised by Jerome. But since he's the head of Facebook AI, no one dares say so publicly. It may as well be an outsider like me: Jerome, I know your heart is in the right place, and that you believe very strongly that this is an important moral issue. But your moral concerns need to be balanced by the ethical considerations of the massive chilling effect you had by publicly shaming OpenAI so hard that they ended up completely losing their confidence, adding some ridiculous profanity filter to GPT-3 that flags pretty much anything mildly naughty, or the fact that AI dungeon is now in hot water because you've given the world an excuse to be pissed off about a mindless, memory-less AI (https://news.ycombinator.com/item?id=23346972 https://news.ycombinator.com/item?id=23346972) generating text that offends someone, somewhere, for some reason. Well, people are going to be offended. Let them be. I am half Jewish. There is a very realistic chance that GPT-3 was trained on the full text of "Mein Kampf," and I could care less. Even if someone went out of their way to train the most offensive, most threatening language model the world has ever seen, what could they really do with that power? Are they going to hurt you right in the feelings? Is the language model going to argue vigorously for the extermination of all Jews? If it did, who would listen? No no, don't try to say that yes, there's a very real danger that people might generate propaganda and influence others. You know as well as I do that this is extraordinarily difficult in practice for multiple reasons, and that no one has ever seen an interactive, adaptive AI that can dynamically deliver the most persuasive propaganda to Reddit and fool everybody into thinking it's a genuine grassroots movement. And you know as well as I do that that would be roughly equivalent to inventing AGI, and that AGI is still nowhere in sight. It's not even living in the same country. Hell, it's not in our sector of the galaxy. See that star in the sky? It's possible that we're more likely to reach that star before we invent AGI, because nobody knows how to invent AGI yet. Before I get too worked up, I should keep some focus and say constructive things. One. If you, as an ML researcher, find yourself wanting to release your work, but your corporation is putting the red tape around you, push back. They need you more than you need them, even if it doesn't feel like it. T Two. Releasing work is generally beneficial for the world. I can't think of a single model release that has ever harmed the world. Let's wait until one actually does before we freak out about whether it might. Three (https://www.youtube.com/watch?v=jpw2ebhTSKs&ab_channel=TheChalkeaters https://www.youtube.com/watch?v=jpw2ebhTSKs&ab_channel=TheCh...). By releasing your work, you give countless people the opportunity to better themselves. Thanks to GPT-2 1.5B, we were able to vastly extend the capabilities of https://reddit.com/r/subsimulatorgpt2 https://reddit.com/r/subsimulatorgpt2 to include over a hundred subreddits in a single model. This has roughly zero commercial value, yet has improved the lives of countless people who show up to giggle at all the (sometimes horribly offensive) things that robots say to each other. I was proud to be a part of that, and I want to be a part of more. Please release your models. The most recent model release was CLIP, and it's already had a profound impact. Just look at how freaking awesome this is! https://twitter.com/l4rz/status/1367853921427984390 https://twitter.com/l4rz/status/1367853921427984390 They're using CLIP to turn someone into Dracula! That's badass, and it inspires people (like me, at one time) to get into ML and become the researchers of tomorrow (as I try to be now). I remember rooting for OpenAI's Dota 2 bot while they were facing off vs .. OG, I think? It's been several years. I spent five years playing dota-type games. It was one of the most exciting things I'd ever seen. And being able to beat rtz 1v1 mid SF? Not merely beat him, but annihilate him? Holy crap, give me that model. Why are you not releasing that model?! It's so cool! And now nobody gets to play with it ever, and it's locked up in the real AI dungeon: OpenAI's. I hope they reconsider.
  - ctchocula 5y ago
    > Two. Releasing work is generally beneficial for the world. I can't think of a single model release that has ever harmed the world. Let's wait until one actually does before we freak out about whether it might. I like a lot of the idealism in this comment, and learned a few things too, so thanks for that. I'm not an ML researcher, which may cloud my views a bit, but I think you would be hardpressed to think that the release of ML models hasn't harmed the world. Perhaps we haven't reached AGI, but we don't need to for the release of ML models to harm the world. I'll give two examples: 1) image detection - Now that image detection is good enough, surveillance in dictatorships can be run more efficiently and scalably in a way KGB couldn't have dreamt of. Even if the release of a model like YOLO saves an hour's worth of time of an ML researcher working for the dictatorship on their surveillance project, this can cause a lot of harm to the oppressed ppl living in those countries. 2) troll bots based on GPT-2. You gave an example of high-quality, human-quality propaganda, but what troll bots lack in quality can be made up by quantity. If you run a lot of troll bots and can sway the dominant viewpoint on all the forums you want to target (which you can since these bots are infinitely scalable), you've achieved your purpose. I also think you are overestimating the quality needed to influence the average person's worldview. For example, we saw from the Cambridge Analytica news that all they had to do to affect a few swing voters' behavior was target them with a few ads. I personally read through some of the example output from GPT-3 and if I were browsing a forum, I wouldn't be all that confident whether they were a bot or not.
    - anewhnaccount2 5y ago
      The thing is that training the models themselves is well within the resources of even the smaller actors that would want to use them in these way. It's the interested but not-quite-interested-enough or time-poor enthusiasts that are hit the worst.
    - _hyn3 5y ago
      Why do research on something at all, if it could be misused? What if we invent a new sabre blade that's made out of light and can cut through almost anything, including people? Should we not invent it? Would we only give it to a Jedi, who could be trained to never do bad things with it? Why are these researchers doing research on this? Seriously, why? Why do research on something and then not release the model over gasp ethical concerns -- if they had ethical concerns, why did they pursue this avenue of research in the first place? Why advance the state of the art if it's wrong to do so? Not releasing the model for "ethical" concerns is a cop-out. There's probably another reason; what is that reason?
deleted 5y ago
[deleted]
queuebert 5y ago
Very generous of you to assume their model is reproducible.
kevinwang 5y ago
Well, better safe than sorry with releasing a model with potentially harmful effects, right?
- deleted 5y ago
  [deleted]