16 ms·

I really strongly disagree with this. I don't have much time to write, but: 1. Photo manipulation has been extremely destructive in a variety of ways. People "

I really strongly disagree with this. I don't have much time to write, but: 1. Photo manipulation has been extremely destructive in a variety of ways. People "know" that photoshop is a thing, yet fake pictures abound at all levels of publications, and unrealistic standards propagate at full speed nonetheless. The effects are wide spread and insidious. A way to automate the production of median-literacy shitposts on the internet tuned to whatever you want to propagandize would be a devastating blow. People "knowing" that it was possible would do precious little to stop the onslaught, and it all but destroy online discourse in any open channels. I'm certain there would be other consequences that I can't fathom right now because I don't grok the potential scale or nth order effects. 2. There is no fire alarm for AGI[1]. Is this thing the key to really dangerous AI? Will the next thing be? No one knows. It's better to be conservative. [1] https://intelligence.org/2017/10/13/fire-alarm/ https://intelligence.org/2017/10/13/fire-alarm/

hughzhang 8y ago
Thanks for your comments! On 1) photo manipulation. If "dangerous" means that random, relatively harmless fake facts easily spread, I'm inclined to agree. I'm sure I've been tricked by a whole host of random facts which are probably actually BS based on random stuff I've read online. On the other hand, if "dangerous" means that a malicious agent can systematically manipulate the public via propaganda a la 20th century Stalin, I think this is basically impossible. Given widespread knowledge of photoshop, changing someone's mind about something significant with a doctored photo seems difficult (it is hard enough to change someone's mind with the truth!). Doctored photographs aren't completely harmless in that they can reinforce existing held beliefs (confirmation bias) and random fake facts aren't totally harmless either, I just think that the danger here is far below what has been implied in other places. In a nutshell, the effects are indeed wide spread, but perhaps not so insidious. On 2) there being no fire alarm, I'm actually very thankful to OpenAI for raising this discussion. While I disagree with their decision not to open source, this discussion was certainly worth having.
- wallflower 8y ago
  > On 1) photo manipulation. If "dangerous" means that random, relatively harmless fake facts easily spread, I'm inclined to agree. No, I think it is even more insidious what they were referring to. I think they meant that Photoshop has enabled and continues to enable unrealistic physical standards of beauty and BMI (mostly for women) that are transmitted in advertisements and news articles. In other words, Photoshop has enabled an alternative visual reality that is not realistic.
  - hughzhang 8y ago
    But still much less dangerous than stuff like nuclear fallout or orwellian propaganda? This isn't to say that unrealistic beauty standards aren't a problem, just one that isn't worth trying to stop the march of technology for. Also, I think advertisements are more to blame here than photoshop. Do you think the ancient Greeks and Romans look like Michelangelo's David or any of their other statues? They weren't bombarded by ads though.
    - fossuser 8y ago
      Can't disinformation campaigns lead to instability and increase the risk of nuclear weapon use? I think the damage that disinformation can cause to democracy is real and we've already seen some of its effects without the ability to automate these things.
  - Mirioron 8y ago
    Any kind of art that's good enough could also create an alternative visual reality.
- intended 8y ago
  > "dangerous" means that a malicious agent can systematically manipulate the public via propaganda a la 20th century Stalin You don't need total control to succeed. The same way that early VOIP calls didn't need perfect sound to be better than normal dial up. All you need is an economic, efficiency, quality or quantity advantage to create a new weapon/tool. India is facing a spate of lynchings over child kidnappings - driven by whatsapp forwards that combine videos from brutal mexican gang killings of children, pakistani safety warnings among others. I've seen them, they are being disseminated to people who have never had to harden themselves to the internet or behavior on the net. (There are many people who don't have knowledge of photoshop, and getting the knowledge to them is Hard. Even getting the knowledge to them can be used to weaken Facts instead of propaganda.) As a result villagers have lynched strangers, mental handicapped people, widows and anyone they suspect of kidnapping or potentially harming children. And I have NO idea, why or who is cutting up and making these videos to share on whatsapp. Evil manipulators, people who are doing it for laughs, well intentioned but misguided good samaritans? As a result of such forwards, whatsapp rolled out a limit on how many times a message can be forwarded - a feature they have pushed worldwide recently. We don't need perfection to do damage, we need efficiency. We already have other systems which can make up the difference in results.
ForHackernews 8y ago
> A way to automate the production of median-literacy shitposts on the internet tuned to whatever you want to propagandize We already have this: it's called mechanical turk (or the 50 Cent army, if you prefer) and it's happening all over the place. I'm not sure computer-generated posts would make much difference.
- asdf333 8y ago
  it makes it virtually cost less to do... that is a big deal 50 cents is the difference between real mail spam and email spam.
msvbvm60 8y ago
Online discourse is already a net harm to society. Destroying it would be a good thing.
tomp 8y ago
> unrealistic standards propagate at full speed nonetheless "Propagate"? If you mean e.g. unrealistic physical standards, it seems that (at least in the US) in recent years, there has been more and more fat people, and more and more acceptance of "different" bodies (trans, fat, alternative styles - in fashion and society), so if anything, I'd say the reverse...
ryanmarsh 8y ago
Nonproliferation strategies do not work with software. By not open sourcing the model, OpenAI merely slows the proliferation of weaponized language models. Furthermore it insures a temporary period of power imbalance whereby only the most capable can wield these. The result is the rest of the internet is not equally forced to come up with solutions to increase the resiliency and safety of their online communities. The internet will need a “captcha for weaponized language models”. We need a level playing field so Google needs a solution as much as any phpBB.
bitL 8y ago
For 1. you only need a few bored grads at MIT/Stanford/another top school (where did the DeepFake come from?) that think it's a cool idea and no resistance from OpenAI et al. would matter (it should be fairly simple to reverse engineer small model they released and augment it with more data/layers/loss functions). Soon the "next cool chat app" will have a shitpost generator as its main feature, possibly trained on humor/sarcasm to make people ROTFLing, like Snapchat used to blow people away with (whatever) face filters, making this technique extremely popular.
echelon 8y ago
Welcome to the post-information world. We're going to have to get used to this new reality. Just as those living in the era before recorded pictures or words didn't have reliable records to lean on, we'll grow to distrust all of the things that are posted. This will happen naturally as the volume of fakes increases; people will start to develop a doubtfulness around recorded media. I don't think we need to worry about propaganda and fake news if the Facebooks and Instagrams become flooded with fake posts from the people we know. It'll become expected and mundane. This seems good for the development of a skeptical public. When we do need to rely on audiovisual records, such as for court cases, we'll have experts that can investigate and verify.
- entee 8y ago
  Perhaps as a byproduct we'll end up with more value placed on human-curated and generated information. Just because there's a lot of noise in the world doesn't mean there aren't objective truths. If FB and Insta get shunted off as sources of news, maybe the good ones will be all that remains trusted. Dare to dream.
  - komali2 8y ago
    The idea of Reputation Economy, yea? Combined with the need for good technology that "guarantees" that https://www.washingtonpost.com https://www.washingtonpost.com really is the washington post. Reminds me of that theory of the mind stuff around trying to figure out if we're really brains in jars or not - we need to find ways to measure our trust of the source, and the means by which the source is transmitted to us.
- zanny 8y ago
  I wanted to suggest a crypto enclave built into recording devices that could sign recordings with private keys controlled by the manufacturer but even that wouldn't be too hard to circumvent by piping fake images/audio into such an enclave so long as you have hardware access.
  - TomMarius 8y ago
cabalamat 8y ago
> A way to automate the production of median-literacy shitposts on the internet tuned to whatever you want to propagandize would be a devastating blow. It would be Shiri's Scissor[1]. [1]: https://slatestarcodex.com/2018/10/30/sort-by-controversial/ https://slatestarcodex.com/2018/10/30/sort-by-controversial/
- CamperBob2 8y ago
  Awesome, that belongs on /r/nosleep. Um, it is fiction, right?
- intended 8y ago
  Wow. Thanks. Nice read. As a concept it’s intetesting. A statement controversial, such that it activates only after discussing it with someone else. Would Shiri's scissor also be a scissor?
- kregasaurusrex 8y ago
  Wow. This looks it would belong on r/nosleep with how close it comes to crossing the uncanny valley.
Bartweiss 8y ago
> 2. There is no fire alarm for AGI[1]. Is this thing the key to really dangerous AI? Will the next thing be? No one knows. It's better to be conservative. This is certainly a point worth discussing, but it's odd for OpenAI to suddenly stand on principle about this specific result. OpenAI was founded on the idea that public access to AI tools and research would improve and democratize them. Their website and founders make statements like "Are we really willing to let our society be infiltrated by autonomous software and hardware agents whose details of operation are known only to a select few? Of course not." and "...empower as many people as possible to have AI. If everyone has AI powers, then there's not any one person or a small set of individuals who can have AI superpower." Meanwhile, their view on AGI risk seems to be that more access will improve scrutiny and oversight. Since day one, they've taken flak from people like Nick Bostrom and Scott Alexander (https://futureoflife.org/2015/12/17/should-ai-be-open https://futureoflife.org/2015/12/17/should-ai-be-open) who argued that their stance amounted to publicly sharing untested, hazardous tools. And now they balked because... they built an essay generator that's not quite good enough to compete with cheap troll farms that governments can already afford? I don't think they're necessarily wrong to not share this, and I know Sam Altman said a while back that they might not release all their source code. But I don't think this should be looked at in isolation. If the people going to bat for democratized access to AI suddenly decided that the public can't be trusted with access to AI tools, that's worth talking about.
- fossuser 8y ago
  I think they did change their mind on this after considering the AGI risk more and getting a better understanding of it. I'm not sure where I read a discussion about this, but I do think it was something they've talked about before.
  - goatlover 8y ago
    Is OpenAI anywhere near AGI such that it would warrant such concerns?
    - fossuser 8y ago
m3kw9 8y ago
This model will probably be reproduced quite quickly with in 2-3 months if there is an entity that has the resource and enough incentive to do it. They did release the paper and that should be enough to get to even close
m3kw9 8y ago
This is nowhere close to AGI
- CamperBob2 8y ago
  And nowhere close to "Open," either. They need to change their name if they're going to take this stance.
intended 8y ago
I agree with you, and would suggest that language based AI research needs to be slowed down. It allows computation owners to scan and therefore create content faster than humans would naturally. That would allow them to pollute and crowd the conversation streams we have used since we learnt to talk, in a way and scale, no predator or propagandist has before.
- freyir 8y ago
  Who knows, maybe we’d go back to valuing face-to-face communication.
Al-Khwarizmi 8y ago
NLP researcher here. Leaving aside the discussion on whether the model is really dangerous or not, the paper gives enough details that the model can be reasonably reproduced (and even if it couldn't, it's an incremental result built on other research that is public, the novelty is mainly in scale). Any big company, government or lab with enough compute can recreate it (and maybe improve it) in six months, maybe less. Not releasing the model doesn't prevent misuse. It just restricts the set of entities that can afford it: only those with enough resources can build it and misuse. Those with less resources cannot, because training it is very expensive. I personally don't feel very reassured by knowing that only the powerful have access to this. The actors I fear the most are powerful and could train it, anyway. I'd rather it be public so the community can scrutinize it, find out if it's really worth worrying about, and find ways to detect/fight it.
- ace_of_spades 8y ago
  Thanks for laying out your reasoning here :) However, I don’t think you have fully adressed the worrying point. The concern here is that good quality faking will become less expensive, thus, more faking will happen. Making it even just marginally more expensive means that there will be less faking overall (no matter the resources of actors) which is seen as good. Thus, your argument towards more equitable access seems to have to explain, how it provides more value than the harm that may be caused by this dynamic, which would in my opinion require some argument for the potential good that could be done by this technology vs. the harm it could cause - which you haven‘t really done. Just arguing for the „balancing of scales“ doesn‘t go far enough. For example, you can draw an analogy to weapons regulation. Just trying to balance the scales by giving people easy access to weapons has shown to not work too well in practice. The rich still have more money to buy better, bigger weapons or defenses while the poor shoot themselves almost regularly. So, to make a convincing case for open sourcing this stuff you should provide a good argument for why we are not talking about weapons here but positive, productive goods with low potential for abuse.

hughzhang 8y ago

Thanks for your comments! On 1) photo manipulation. If "dangerous" means that random, relatively harmless fake facts easily spread, I'm inclined to agree. I'm sure I've been tricked by a whole host of random facts which are probably actually BS based on random stuff I've read online. On the other hand, if "dangerous" means that a malicious agent can systematically manipulate the public via propaganda a la 20th century Stalin, I think this is basically impossible. Given widespread knowledge of photoshop, changing someone's mind about something significant with a doctored photo seems difficult (it is hard enough to change someone's mind with the truth!). Doctored photographs aren't completely harmless in that they can reinforce existing held beliefs (confirmation bias) and random fake facts aren't totally harmless either, I just think that the danger here is far below what has been implied in other places. In a nutshell, the effects are indeed wide spread, but perhaps not so insidious. On 2) there being no fire alarm, I'm actually very thankful to OpenAI for raising this discussion. While I disagree with their decision not to open source, this discussion was certainly worth having.

wallflower 8y ago
> On 1) photo manipulation. If "dangerous" means that random, relatively harmless fake facts easily spread, I'm inclined to agree. No, I think it is even more insidious what they were referring to. I think they meant that Photoshop has enabled and continues to enable unrealistic physical standards of beauty and BMI (mostly for women) that are transmitted in advertisements and news articles. In other words, Photoshop has enabled an alternative visual reality that is not realistic.
- hughzhang 8y ago
  But still much less dangerous than stuff like nuclear fallout or orwellian propaganda? This isn't to say that unrealistic beauty standards aren't a problem, just one that isn't worth trying to stop the march of technology for. Also, I think advertisements are more to blame here than photoshop. Do you think the ancient Greeks and Romans look like Michelangelo's David or any of their other statues? They weren't bombarded by ads though.
  - fossuser 8y ago
    Can't disinformation campaigns lead to instability and increase the risk of nuclear weapon use? I think the damage that disinformation can cause to democracy is real and we've already seen some of its effects without the ability to automate these things.
- Mirioron 8y ago
  Any kind of art that's good enough could also create an alternative visual reality.
intended 8y ago
> "dangerous" means that a malicious agent can systematically manipulate the public via propaganda a la 20th century Stalin You don't need total control to succeed. The same way that early VOIP calls didn't need perfect sound to be better than normal dial up. All you need is an economic, efficiency, quality or quantity advantage to create a new weapon/tool. India is facing a spate of lynchings over child kidnappings - driven by whatsapp forwards that combine videos from brutal mexican gang killings of children, pakistani safety warnings among others. I've seen them, they are being disseminated to people who have never had to harden themselves to the internet or behavior on the net. (There are many people who don't have knowledge of photoshop, and getting the knowledge to them is Hard. Even getting the knowledge to them can be used to weaken Facts instead of propaganda.) As a result villagers have lynched strangers, mental handicapped people, widows and anyone they suspect of kidnapping or potentially harming children. And I have NO idea, why or who is cutting up and making these videos to share on whatsapp. Evil manipulators, people who are doing it for laughs, well intentioned but misguided good samaritans? As a result of such forwards, whatsapp rolled out a limit on how many times a message can be forwarded - a feature they have pushed worldwide recently. We don't need perfection to do damage, we need efficiency. We already have other systems which can make up the difference in results.

ForHackernews 8y ago

> A way to automate the production of median-literacy shitposts on the internet tuned to whatever you want to propagandize We already have this: it's called mechanical turk (or the 50 Cent army, if you prefer) and it's happening all over the place. I'm not sure computer-generated posts would make much difference.

asdf333 8y ago
it makes it virtually cost less to do... that is a big deal 50 cents is the difference between real mail spam and email spam.

msvbvm60 8y ago

Online discourse is already a net harm to society. Destroying it would be a good thing.

tomp 8y ago

> unrealistic standards propagate at full speed nonetheless "Propagate"? If you mean e.g. unrealistic physical standards, it seems that (at least in the US) in recent years, there has been more and more fat people, and more and more acceptance of "different" bodies (trans, fat, alternative styles - in fashion and society), so if anything, I'd say the reverse...

ryanmarsh 8y ago

Nonproliferation strategies do not work with software. By not open sourcing the model, OpenAI merely slows the proliferation of weaponized language models. Furthermore it insures a temporary period of power imbalance whereby only the most capable can wield these. The result is the rest of the internet is not equally forced to come up with solutions to increase the resiliency and safety of their online communities. The internet will need a “captcha for weaponized language models”. We need a level playing field so Google needs a solution as much as any phpBB.

bitL 8y ago

For 1. you only need a few bored grads at MIT/Stanford/another top school (where did the DeepFake come from?) that think it's a cool idea and no resistance from OpenAI et al. would matter (it should be fairly simple to reverse engineer small model they released and augment it with more data/layers/loss functions). Soon the "next cool chat app" will have a shitpost generator as its main feature, possibly trained on humor/sarcasm to make people ROTFLing, like Snapchat used to blow people away with (whatever) face filters, making this technique extremely popular.

echelon 8y ago

Welcome to the post-information world. We're going to have to get used to this new reality. Just as those living in the era before recorded pictures or words didn't have reliable records to lean on, we'll grow to distrust all of the things that are posted. This will happen naturally as the volume of fakes increases; people will start to develop a doubtfulness around recorded media. I don't think we need to worry about propaganda and fake news if the Facebooks and Instagrams become flooded with fake posts from the people we know. It'll become expected and mundane. This seems good for the development of a skeptical public. When we do need to rely on audiovisual records, such as for court cases, we'll have experts that can investigate and verify.

entee 8y ago
Perhaps as a byproduct we'll end up with more value placed on human-curated and generated information. Just because there's a lot of noise in the world doesn't mean there aren't objective truths. If FB and Insta get shunted off as sources of news, maybe the good ones will be all that remains trusted. Dare to dream.
- komali2 8y ago
  The idea of Reputation Economy, yea? Combined with the need for good technology that "guarantees" that https://www.washingtonpost.com https://www.washingtonpost.com really is the washington post. Reminds me of that theory of the mind stuff around trying to figure out if we're really brains in jars or not - we need to find ways to measure our trust of the source, and the means by which the source is transmitted to us.
zanny 8y ago
I wanted to suggest a crypto enclave built into recording devices that could sign recordings with private keys controlled by the manufacturer but even that wouldn't be too hard to circumvent by piping fake images/audio into such an enclave so long as you have hardware access.
- TomMarius 8y ago

cabalamat 8y ago

> A way to automate the production of median-literacy shitposts on the internet tuned to whatever you want to propagandize would be a devastating blow. It would be Shiri's Scissor[1]. [1]: https://slatestarcodex.com/2018/10/30/sort-by-controversial/ https://slatestarcodex.com/2018/10/30/sort-by-controversial/

CamperBob2 8y ago
Awesome, that belongs on /r/nosleep. Um, it is fiction, right?
intended 8y ago
Wow. Thanks. Nice read. As a concept it’s intetesting. A statement controversial, such that it activates only after discussing it with someone else. Would Shiri's scissor also be a scissor?
kregasaurusrex 8y ago
Wow. This looks it would belong on r/nosleep with how close it comes to crossing the uncanny valley.

Bartweiss 8y ago

> 2. There is no fire alarm for AGI[1]. Is this thing the key to really dangerous AI? Will the next thing be? No one knows. It's better to be conservative. This is certainly a point worth discussing, but it's odd for OpenAI to suddenly stand on principle about this specific result. OpenAI was founded on the idea that public access to AI tools and research would improve and democratize them. Their website and founders make statements like "Are we really willing to let our society be infiltrated by autonomous software and hardware agents whose details of operation are known only to a select few? Of course not." and "...empower as many people as possible to have AI. If everyone has AI powers, then there's not any one person or a small set of individuals who can have AI superpower." Meanwhile, their view on AGI risk seems to be that more access will improve scrutiny and oversight. Since day one, they've taken flak from people like Nick Bostrom and Scott Alexander (https://futureoflife.org/2015/12/17/should-ai-be-open https://futureoflife.org/2015/12/17/should-ai-be-open) who argued that their stance amounted to publicly sharing untested, hazardous tools. And now they balked because... they built an essay generator that's not quite good enough to compete with cheap troll farms that governments can already afford? I don't think they're necessarily wrong to not share this, and I know Sam Altman said a while back that they might not release all their source code. But I don't think this should be looked at in isolation. If the people going to bat for democratized access to AI suddenly decided that the public can't be trusted with access to AI tools, that's worth talking about.

fossuser 8y ago
I think they did change their mind on this after considering the AGI risk more and getting a better understanding of it. I'm not sure where I read a discussion about this, but I do think it was something they've talked about before.
- goatlover 8y ago
  Is OpenAI anywhere near AGI such that it would warrant such concerns?
  - fossuser 8y ago

m3kw9 8y ago

This model will probably be reproduced quite quickly with in 2-3 months if there is an entity that has the resource and enough incentive to do it. They did release the paper and that should be enough to get to even close

m3kw9 8y ago

This is nowhere close to AGI

CamperBob2 8y ago
And nowhere close to "Open," either. They need to change their name if they're going to take this stance.

intended 8y ago

I agree with you, and would suggest that language based AI research needs to be slowed down. It allows computation owners to scan and therefore create content faster than humans would naturally. That would allow them to pollute and crowd the conversation streams we have used since we learnt to talk, in a way and scale, no predator or propagandist has before.

freyir 8y ago
Who knows, maybe we’d go back to valuing face-to-face communication.

Al-Khwarizmi 8y ago

NLP researcher here. Leaving aside the discussion on whether the model is really dangerous or not, the paper gives enough details that the model can be reasonably reproduced (and even if it couldn't, it's an incremental result built on other research that is public, the novelty is mainly in scale). Any big company, government or lab with enough compute can recreate it (and maybe improve it) in six months, maybe less. Not releasing the model doesn't prevent misuse. It just restricts the set of entities that can afford it: only those with enough resources can build it and misuse. Those with less resources cannot, because training it is very expensive. I personally don't feel very reassured by knowing that only the powerful have access to this. The actors I fear the most are powerful and could train it, anyway. I'd rather it be public so the community can scrutinize it, find out if it's really worth worrying about, and find ways to detect/fight it.

ace_of_spades 8y ago
Thanks for laying out your reasoning here :) However, I don’t think you have fully adressed the worrying point. The concern here is that good quality faking will become less expensive, thus, more faking will happen. Making it even just marginally more expensive means that there will be less faking overall (no matter the resources of actors) which is seen as good. Thus, your argument towards more equitable access seems to have to explain, how it provides more value than the harm that may be caused by this dynamic, which would in my opinion require some argument for the potential good that could be done by this technology vs. the harm it could cause - which you haven‘t really done. Just arguing for the „balancing of scales“ doesn‘t go far enough. For example, you can draw an analogy to weapons regulation. Just trying to balance the scales by giving people easy access to weapons has shown to not work too well in practice. The rich still have more money to buy better, bigger weapons or defenses while the poor shoot themselves almost regularly. So, to make a convincing case for open sourcing this stuff you should provide a good argument for why we are not talking about weapons here but positive, productive goods with low potential for abuse.