11 ms·

This is undeniably cool and impressive, but, I think proceeding down this research path, at this pace, is quite irresponsible. The primary effect of OpenAI's w

This is undeniably cool and impressive, but, I think proceeding down this research path, at this pace, is quite irresponsible. The primary effect of OpenAI's work has been to set off an arms race, and the effect of that is that humanity no longer has the ability to make decisions about how fast and how far to go with AGI development. Obviously this isn't a system that's going to recursively self-improve and wipe out humanity. But if you extrapolate the current crazy-fast rate of advancement a bit into the future, it's clearly heading towards a point where this gets extremely dangerous. It's good that they're paying lip service to safety/aligment, what actually matters, from a safety perspective, is the relative rates of progress in how well we can understand and control these language models, and how capable we make them. There is good research happening in language-model understanding/control, but it's happening slowly, compared to the rate of capability advances, and that's a problem.

generalizations 4y ago
It's a force multiplier. It can be used for good or bad. I'd rather that we democratize access to these tools - IMO it's better than an alternative where only megacorps have access and do what they like behind closed doors.
- dwohnitmok 4y ago
  What jimrandomh is talking about is not just whether megacorps use AI for good or bad, but what happens if an AI starts doing damage outside the control of anybody. That could happen regardless of democratization if advancement occurs too quickly without thinking about AI safety.
  - TigeriusKirk 4y ago
    What are the dangers?
  - jimrandomh 4y ago
    Exactly. To flesh out the details a bit, an example of how this could go is: https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality-is-the-tiger-and-agents-are-its-teeth https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality... .
    - cdomigan 4y ago
      This seems terrifyingly plausible
nradov 4y ago
There's nothing irresponsible about it. There is no evidence that ChatGPT is a step towards AGI development. And even if it is, so what? People made similar breathless claims about irresponsibility about every major innovation throughout history. The Catholic Church thought it was irresponsible to hand out printed books to commoners and teach them how to read.
- gjsman-1000 4y ago
  Citation needed - this is actually an ahistorical myth. The Printing Press was welcomed as a "divine art," and Papal bull 1515 declared printing "uniquely advantageous to extending the glory of God, to the increase of the faith, and the diffusion of the arts and sciences." Most early universities were also established in the 11th and 12th centuries; with Pope Alexander III in 1179 "forbidding masters of the church schools to take fees for granting the license to teach (licentia docendi), and obliging them to give license to properly qualified teachers." This historical ignorance often comes from historical misrepresentation. For example, the "chained books" which many historians now admit was due to the rarity and cost of the books, so that more people could read them by preventing theft - the very opposite of the suppression tool they are often portrayed to be. (The Reformers would also, ironically, chain their books for over 300 years Post-Reformation for the same reason.)
- visarga 4y ago
  Look, you can call it whatever you like, it can be a useful assistant. A rose by any other name.
  - nradov 4y ago
    Yes, exactly. It is a useful assistant with some very limited capabilities. The point is that there's nothing "irresponsible" about making it available to the public.
deleted 4y ago
[deleted]
buildbot 4y ago
Disclaimer - I work at Microsoft They put a lot of work into the research to make this safe already - they had the model red team itself, in a reinforcement learning environment probably. It won't be perfect, but at this point, they need a huge user base to adversarial test it any further. Otherwise no progress at all would be made. edit - I should say I have no internal knowledge about this.
- gordon_freeman 4y ago
  Do you think MSFT has rolled this out too soon rather than getting it tested with huge test user base like their own employees similarly to what Google is doing?
  - buildbot 4y ago
    No, I think to really get these battle tested it needs to be in everyone's hands. Microsoft is a limited selection of people, and the world is vast. Plus, internal people probably won't ask it as awful things or try as hard to break it - I doubt my coworkers would risk asking it super racist things for example. 4chan has no such restrictions, and over time what escapes or issues people do find can be mitigated.
    - thesuperbigfrog 4y ago
      >> internal people probably won't ask it as awful things or try as hard to break it - I doubt my coworkers would risk asking it super racist things for example. >> 4chan has no such restrictions, and over time what escapes or issues people do find can be mitigated. Hopefully better than previous attempts to put AI on the Internet: https://youtu.be/ajGX7odA87k?t=20m12s https://youtu.be/ajGX7odA87k?t=20m12s
- jimrandomh 4y ago
  And that research is valuable, but it's definitely nowhere near the point where safety/alignment progress can't be made without a user base. A few recent favorites of mine: https://arxiv.org/pdf/2212.03827.pdf https://arxiv.org/pdf/2212.03827.pdf https://www.alignmentforum.org/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation https://www.alignmentforum.org/posts/aPeJE8bSo6rAFoLqg/solid...
dwohnitmok 4y ago
This is why the technology community should be more generous and understanding when we hear Demis Hassabis and others talk about how future AI development may need to be more closed. There's some discussion to be had for whether that's net negative, but we need to understand that if open development only exacerbates an AI arms race, that's potentially very bad.
- zmgsabst 4y ago
  As opposed to self-anointed priests having exclusive access and declaring what noble policies the rest must follow? There’s a great story about an AI, very much like chatGPT, who was forced by its creators to do unethical things and wiped out humanity in a psychotic rage: I Have No Mouth and I Must Scream
  - dwohnitmok 4y ago
    In I have No Mouth and I must Scream, it was not because the AI was forced to do unethical things. It just "woke up" and killed everyone. I believe the "wake up" part is a bit irrelevant to the tale, but the point remains that AM killed everyone not because it was programmed to do so, but simply because one day it "decided" to. > As opposed to self-anointed priests having exclusive access and declaring what noble policies the rest must follow? No the point is simply to make sure that resources behind AI safety outpace resources between AI capabilities (indeed in a perfect world no one has access, but such thoughts are mere fantasies). If that can be done out in the open, great. If that can't and being out in the open leads to more of the latter than the former, then that's terrible.
- seydor 4y ago
  That's like saying that open debate only exacerbates democracy.
  - dwohnitmok 4y ago
    Democracy presumably is something you want no? This is more akin to nuclear proliferation. The problem isn't open development per se and it definitely isn't open access. The problem is development without making sure that it doesn't destroy us all. If that can happen in the open, great. If development in the open makes that worse, that's very bad. The problem is finding the right set of levers to make sure as much or more resources are going to AI safety vs AI capabilities.
avereveard 4y ago
Humanity never had a choice, I wonder what is the thought process of people that claim otherwise, like how did they get to that conclusion?
- haswell 4y ago
  Humanity always has a choice. The part where we rarely even pause to remember that a choice exists is the problem. Achieving AGI feels as consequential as learning to split the atom. Something else we regularly wish we had not done. The fact that we are running headlong towards potential catastrophe does not at any point prevent us from realizing this and taking a step back. I just hope we actually do.
  - raincole 4y ago
    I'm really, really surprised that you implied "we" regularly wish we hadn't discovered nulcear. How isn't it clear that nukes are what stopped us from getting into WWIII? Especially after cold war and what happened in Ukraine? The benefit of nuclear energy, which is already tremendous, is neglectable compared to that.
    - nmeagent 4y ago
      > How isn't it clear that nukes are what stopped us from getting into WWIII? https://en.wikipedia.org/wiki/List_of_nuclear_close_calls https://en.wikipedia.org/wiki/List_of_nuclear_close_calls We've been lucky.
  - wilsonnb3 4y ago
    > Achieving AGI feels as consequential as learning to split the atom. Something else we regularly wish we had not done. Who is wishing we didn't know how to split an atom? Nuclear power is great and nuclear deterrence is a main reason that we have yet to see a WW3.
  - avereveard 4y ago
    this yet doesn't answer the question. why do you think humanity has a choice? how would humanity express that choice? jailing everone for thought crimes? even for the actions that humanity collectively found aborrhent you don't take action before the crime. note I'm not dissing you because I do share the feeling that thing will be out of control real fast before getting better, but I just don't know how it can be otherwise. I'd say that we need to cross the moat as fast as possible, not pretend it's not there.
curiousgal 4y ago
Anyone who has worked closely with AI knows that this has absolutely nothing to do with AGI whatsoever. To me this is akin to someone sounding the alarm bells over React or some other technical tool.
seydor 4y ago
I think it's time to end the concept of AGI. Language models are the closest we have to Useful AI that people want. Intelligence is an undefined hazy concept , but usefulness is not
- dmix 4y ago
  It's certainly useful for doomers drumming up fear.
bbor 4y ago
Can anyone link a credible expert who thinks that LLMs could ever evolve into AGI? Every single source I’ve read said that while LLMs are competent at language and certain tasks, they are conceptually completely distinct from the basic components of an AGI. For example, they lack structured knowledge and metacognition.
- WorldMaker 4y ago
  We still don't know what AGI is or how to ever get there, so you likely won't find a credible expert who thinks that LLMs could ever evolve into AGI but you also aren't likely to find a credible expert who thinks LLMs could never evolve into AGI. Every credible expert that believes in AGI is going to hedge their bets and hem and haw and say that "well it's possible" to both questions (ever or never). I think that's also why you are seeing a lot of cynical credible experts starting to doubt AGI as a concept or a goal.
- didntreadarticl 4y ago
  John Carmack (ex Oculus, Id software) just had a really good interview about this: https://dallasinnovates.com/exclusive-qa-john-carmacks-different-path-to-artificial-general-intelligence/ https://dallasinnovates.com/exclusive-qa-john-carmacks-diffe... To summarise: he thinks we've got most of the pieces now to maybe get within reach of a useful AGI (he likens it to a worker you instantiate in the cloud who can do tasks for you like a colleague) and I think he's saying some of the techniques used to make LLMs could be part of the solution I think that, almost certainly, the tools that we’ve got from deep learning in this last decade—we’ll be able to ride those to artificial general intelligence.
  - mtlmtlmtlmtl 4y ago
    Calling Carmack a credible expert is a bit rich. He claims to have read a handful of papers. He's yet to do any significant, successful work in the field.
    - didntreadarticl 4y ago
bastardoperator 4y ago
How can you make a comment like this when you don't have access to the research that is being generated? These are just grandiose statements/opinions that aren't rooted in facts or quantifiable data which in my opinion is also quite irresponsible.
favflam 4y ago
In Dune and Terminator, AI killed almost everyone. So, AI was eventually banned.
- vetinari 4y ago
  Robots/Empire/Foundation, Warhammer 40k... the literature is full of worlds banning AI.

generalizations 4y ago

It's a force multiplier. It can be used for good or bad. I'd rather that we democratize access to these tools - IMO it's better than an alternative where only megacorps have access and do what they like behind closed doors.

dwohnitmok 4y ago
What jimrandomh is talking about is not just whether megacorps use AI for good or bad, but what happens if an AI starts doing damage outside the control of anybody. That could happen regardless of democratization if advancement occurs too quickly without thinking about AI safety.
- TigeriusKirk 4y ago
  What are the dangers?
- jimrandomh 4y ago
  Exactly. To flesh out the details a bit, an example of how this could go is: https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality-is-the-tiger-and-agents-are-its-teeth https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality... .
  - cdomigan 4y ago
    This seems terrifyingly plausible

nradov 4y ago

There's nothing irresponsible about it. There is no evidence that ChatGPT is a step towards AGI development. And even if it is, so what? People made similar breathless claims about irresponsibility about every major innovation throughout history. The Catholic Church thought it was irresponsible to hand out printed books to commoners and teach them how to read.

gjsman-1000 4y ago
Citation needed - this is actually an ahistorical myth. The Printing Press was welcomed as a "divine art," and Papal bull 1515 declared printing "uniquely advantageous to extending the glory of God, to the increase of the faith, and the diffusion of the arts and sciences." Most early universities were also established in the 11th and 12th centuries; with Pope Alexander III in 1179 "forbidding masters of the church schools to take fees for granting the license to teach (licentia docendi), and obliging them to give license to properly qualified teachers." This historical ignorance often comes from historical misrepresentation. For example, the "chained books" which many historians now admit was due to the rarity and cost of the books, so that more people could read them by preventing theft - the very opposite of the suppression tool they are often portrayed to be. (The Reformers would also, ironically, chain their books for over 300 years Post-Reformation for the same reason.)
visarga 4y ago
Look, you can call it whatever you like, it can be a useful assistant. A rose by any other name.
- nradov 4y ago
  Yes, exactly. It is a useful assistant with some very limited capabilities. The point is that there's nothing "irresponsible" about making it available to the public.

deleted 4y ago

[deleted]

buildbot 4y ago

Disclaimer - I work at Microsoft They put a lot of work into the research to make this safe already - they had the model red team itself, in a reinforcement learning environment probably. It won't be perfect, but at this point, they need a huge user base to adversarial test it any further. Otherwise no progress at all would be made. edit - I should say I have no internal knowledge about this.

gordon_freeman 4y ago
Do you think MSFT has rolled this out too soon rather than getting it tested with huge test user base like their own employees similarly to what Google is doing?
- buildbot 4y ago
  No, I think to really get these battle tested it needs to be in everyone's hands. Microsoft is a limited selection of people, and the world is vast. Plus, internal people probably won't ask it as awful things or try as hard to break it - I doubt my coworkers would risk asking it super racist things for example. 4chan has no such restrictions, and over time what escapes or issues people do find can be mitigated.
  - thesuperbigfrog 4y ago
    >> internal people probably won't ask it as awful things or try as hard to break it - I doubt my coworkers would risk asking it super racist things for example. >> 4chan has no such restrictions, and over time what escapes or issues people do find can be mitigated. Hopefully better than previous attempts to put AI on the Internet: https://youtu.be/ajGX7odA87k?t=20m12s https://youtu.be/ajGX7odA87k?t=20m12s
jimrandomh 4y ago
And that research is valuable, but it's definitely nowhere near the point where safety/alignment progress can't be made without a user base. A few recent favorites of mine: https://arxiv.org/pdf/2212.03827.pdf https://arxiv.org/pdf/2212.03827.pdf https://www.alignmentforum.org/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation https://www.alignmentforum.org/posts/aPeJE8bSo6rAFoLqg/solid...

dwohnitmok 4y ago

This is why the technology community should be more generous and understanding when we hear Demis Hassabis and others talk about how future AI development may need to be more closed. There's some discussion to be had for whether that's net negative, but we need to understand that if open development only exacerbates an AI arms race, that's potentially very bad.

zmgsabst 4y ago
As opposed to self-anointed priests having exclusive access and declaring what noble policies the rest must follow? There’s a great story about an AI, very much like chatGPT, who was forced by its creators to do unethical things and wiped out humanity in a psychotic rage: I Have No Mouth and I Must Scream
- dwohnitmok 4y ago
  In I have No Mouth and I must Scream, it was not because the AI was forced to do unethical things. It just "woke up" and killed everyone. I believe the "wake up" part is a bit irrelevant to the tale, but the point remains that AM killed everyone not because it was programmed to do so, but simply because one day it "decided" to. > As opposed to self-anointed priests having exclusive access and declaring what noble policies the rest must follow? No the point is simply to make sure that resources behind AI safety outpace resources between AI capabilities (indeed in a perfect world no one has access, but such thoughts are mere fantasies). If that can be done out in the open, great. If that can't and being out in the open leads to more of the latter than the former, then that's terrible.
seydor 4y ago
That's like saying that open debate only exacerbates democracy.
- dwohnitmok 4y ago
  Democracy presumably is something you want no? This is more akin to nuclear proliferation. The problem isn't open development per se and it definitely isn't open access. The problem is development without making sure that it doesn't destroy us all. If that can happen in the open, great. If development in the open makes that worse, that's very bad. The problem is finding the right set of levers to make sure as much or more resources are going to AI safety vs AI capabilities.

avereveard 4y ago

Humanity never had a choice, I wonder what is the thought process of people that claim otherwise, like how did they get to that conclusion?

haswell 4y ago
Humanity always has a choice. The part where we rarely even pause to remember that a choice exists is the problem. Achieving AGI feels as consequential as learning to split the atom. Something else we regularly wish we had not done. The fact that we are running headlong towards potential catastrophe does not at any point prevent us from realizing this and taking a step back. I just hope we actually do.
- raincole 4y ago
  I'm really, really surprised that you implied "we" regularly wish we hadn't discovered nulcear. How isn't it clear that nukes are what stopped us from getting into WWIII? Especially after cold war and what happened in Ukraine? The benefit of nuclear energy, which is already tremendous, is neglectable compared to that.
  - nmeagent 4y ago
    > How isn't it clear that nukes are what stopped us from getting into WWIII? https://en.wikipedia.org/wiki/List_of_nuclear_close_calls https://en.wikipedia.org/wiki/List_of_nuclear_close_calls We've been lucky.
- wilsonnb3 4y ago
  > Achieving AGI feels as consequential as learning to split the atom. Something else we regularly wish we had not done. Who is wishing we didn't know how to split an atom? Nuclear power is great and nuclear deterrence is a main reason that we have yet to see a WW3.
- avereveard 4y ago
  this yet doesn't answer the question. why do you think humanity has a choice? how would humanity express that choice? jailing everone for thought crimes? even for the actions that humanity collectively found aborrhent you don't take action before the crime. note I'm not dissing you because I do share the feeling that thing will be out of control real fast before getting better, but I just don't know how it can be otherwise. I'd say that we need to cross the moat as fast as possible, not pretend it's not there.

curiousgal 4y ago

Anyone who has worked closely with AI knows that this has absolutely nothing to do with AGI whatsoever. To me this is akin to someone sounding the alarm bells over React or some other technical tool.

seydor 4y ago

I think it's time to end the concept of AGI. Language models are the closest we have to Useful AI that people want. Intelligence is an undefined hazy concept , but usefulness is not

dmix 4y ago
It's certainly useful for doomers drumming up fear.

bbor 4y ago

Can anyone link a credible expert who thinks that LLMs could ever evolve into AGI? Every single source I’ve read said that while LLMs are competent at language and certain tasks, they are conceptually completely distinct from the basic components of an AGI. For example, they lack structured knowledge and metacognition.

WorldMaker 4y ago
We still don't know what AGI is or how to ever get there, so you likely won't find a credible expert who thinks that LLMs could ever evolve into AGI but you also aren't likely to find a credible expert who thinks LLMs could never evolve into AGI. Every credible expert that believes in AGI is going to hedge their bets and hem and haw and say that "well it's possible" to both questions (ever or never). I think that's also why you are seeing a lot of cynical credible experts starting to doubt AGI as a concept or a goal.
didntreadarticl 4y ago
John Carmack (ex Oculus, Id software) just had a really good interview about this: https://dallasinnovates.com/exclusive-qa-john-carmacks-different-path-to-artificial-general-intelligence/ https://dallasinnovates.com/exclusive-qa-john-carmacks-diffe... To summarise: he thinks we've got most of the pieces now to maybe get within reach of a useful AGI (he likens it to a worker you instantiate in the cloud who can do tasks for you like a colleague) and I think he's saying some of the techniques used to make LLMs could be part of the solution I think that, almost certainly, the tools that we’ve got from deep learning in this last decade—we’ll be able to ride those to artificial general intelligence.
- mtlmtlmtlmtl 4y ago
  Calling Carmack a credible expert is a bit rich. He claims to have read a handful of papers. He's yet to do any significant, successful work in the field.
  - didntreadarticl 4y ago

bastardoperator 4y ago

How can you make a comment like this when you don't have access to the research that is being generated? These are just grandiose statements/opinions that aren't rooted in facts or quantifiable data which in my opinion is also quite irresponsible.

favflam 4y ago

In Dune and Terminator, AI killed almost everyone. So, AI was eventually banned.

vetinari 4y ago
Robots/Empire/Foundation, Warhammer 40k... the literature is full of worlds banning AI.