4 ms·

OP here, i called it out in my writeup (footnote 14 https://lspace.swyx.io/p/multimodal-gpt4#footnote-anchor-14-108608128 https://lspace.swyx.io/p/multimodal-gp

by swyx 4y ago

OP here, i called it out in my writeup (footnote 14 https://lspace.swyx.io/p/multimodal-gpt4#footnote-anchor-14-108608128 https://lspace.swyx.io/p/multimodal-gpt4#footnote-anchor-14-...) but didnt want to add too much emphasis on it because they clearly did it under very well supervised conditions (and this ability was always there anyway as the FAIR Diplomacy result showed). it's the people who randomly add unsupervised realworld connections (ahem microsoft) that we need to be worried about. giving these AIs write access to the physical/real world plane needs extreme caution

w_for_wumbo 4y ago
Oh I see! I had missed that. I had read that quote last night which had given me the spooks, and was surprised it wasn't a larger talking point. It's good to know that they had it quite well supervised, but knowing that they were testing on a slightly different model than what was released and that Bing is using it with the ability to connect to the internet does give me pause.
- swyx 4y ago
  it is the fate of red teaming AI safety.. by working on understanding the limits, you are inevitably pushing us towards them. i dont doubt that the ARC team have thought this thru more than anyone here but right now there are probably 100 funded startups pursuing agentic AI without any safety qualms.
ryandvm 4y ago
This is where I feel like legislation is pointless. In 10 years you'll be able to run GPT-4 on an old phone. Imagine what a fairly well funded group will be able to do. There's no amount of legislation or technological guardrails that are getting this genie back in the bottle.
- highwaylights 4y ago
  This. It exists now, it’ll develop. That the potential is being realised means the money being pumped into competing implementations will continue to skyrocket so it’ll only accelerate from here. I would say it’s too late to do something about this, but in reality, wherever this is going was a foregone conclusion the moment early hominids began using tools. Technology doesn’t regress.
  - kneebonian 4y ago
    > Technology doesn’t regress. I hate this assertion, true technology hasn't gone backwards much in the past 200 years, but honestly technology can and often does regress historically speaking. There have been plenty of times empires have risen and fallen, taking with them much of their knowledge. How much was lost when Rome decayed simply because there wasn't the structure in order to allow technologists to continue creating? Things are bigger and more interconnected this time so it is harder for an isolated event to cause things to fall backwards, but when it does the impact will be much larger.
    - lrvick 4y ago
      If you need evidence that technology regresses look no further than Facebook.
    - UncleEntity 4y ago
      > How much was lost when Rome decayed simply because there wasn't the structure in order to allow technologists to continue creating? Considering the Eastern Roman Empire existing until 1453 and the Middle Eastern empires did lots of scientific stuff I’d say not much.
  - Yoric 4y ago
    > skyrocket Spell-checking suggestion: "skynet".
- dwohnitmok 4y ago
  > There's no amount of legislation or technological guardrails that are getting this genie back in the bottle. We're not getting the genie back in the bottle but there's still a lot of ways that we can change how we develop AI moving forward.
nradov 4y ago
Why does it need extreme caution?
- swyx 4y ago
  these AIs have already demonstrated limited tendency to strongly prefer not being shut down (they already have the ability to lie to humans to achieve goals and we celebrate this lol). giving them the ability to self replicate means we lose that ability to pull the plug. giving them ability to self modify means humanity will lose control of ai on any long enough timescale.
  - nradov 4y ago
    The fact that you would use words like "prefer", "lie", and "celebrate" in this context indicates that you are deeply ignorant about the underlying technology. Your claims are wild, unsupported conjecture and cannot be taken seriously.
    - thfuran 4y ago
      While I agree with you, whether it is able to have preferences or feelings doesn't actually matter in this context. A simulated pzombie may be morally distinct from a person, but the consequences of its actions are unaffected by the distinction.
    - Karunamon 4y ago
      Remember the erratic behavior demonstrated by "Sydney", the Bing GPT bot? Whether the model has the "cognitive" ability to "prefer" something is not relevant. The model has the ability to submit search engine queries now; this is a read only operation for now. Once we give the bot write ability, erratic behavior has the potential to cause real world damage. Whether it is an actual "preference" or something that only looks like one based on the output is purely academic at that point.
    - haswell 4y ago
      I’ve been ranting against the anthropomorphization of these new AI tools pretty consistently, but I think the fact that these tools only appear to possess certain characteristics is all the more reason that we should not be hooking these things up to tools that give them access to the real world. If a language model “concocts” some “scheme” and executes it by interacting with real systems in ways that have real world impact, it doesn’t much matter if the whole thing is a hallucination. Put another way, would you feel comfortable letting a hallucination machine running on a supercomputer have access to the real world? It doesn’t have to truly “know” anything for the impact of the resulting actions to matter.
losvedir 4y ago
I don't really understand the threat model here. Surely "launch the nukes" is not available as a public API. I would expect anyone with networked services (at least those that are important or serious) already take precautions with authentication and authorization and such. Even if some mythical AGI isn't ready yet, there's already very smart, well-funded hacking groups from hostile nation states, for example. How will an AI adversary interact any differently with than world than a flesh and blood adversary?
- Turing_Machine 4y ago
  > Surely "launch the nukes" is not available as a public API. Of course not. However "manipulate the news and social media so that the humans launch the nukes for you" isn't quite so far-fetched. > How will an AI adversary interact any differently with than world than a flesh and blood adversary? It's possible to spin up a whole army of AI adversary bots in a hurry. Recruiting that many humans takes a lot of time and money.
  - lrvick 4y ago
    Or it could infect the corporate IT endpoint management software used on the systems of the engineers that author and review the firmware for said nukes, so extra code is inserted they never see.
    - nradov 4y ago
      Those systems are air gapped and do not have accessible endpoints.
      - lrvick 4y ago
        The software that runs on the airgapped machines is generally authored on internet connected machines, or using software compiled on internet connected machines. I audit airgapped machine setups for a living and most are nowhere near as isolated as claimed.
- 4y ago

by swyx 4y ago

w_for_wumbo 4y ago

Oh I see! I had missed that. I had read that quote last night which had given me the spooks, and was surprised it wasn't a larger talking point. It's good to know that they had it quite well supervised, but knowing that they were testing on a slightly different model than what was released and that Bing is using it with the ability to connect to the internet does give me pause.

swyx 4y ago
it is the fate of red teaming AI safety.. by working on understanding the limits, you are inevitably pushing us towards them. i dont doubt that the ARC team have thought this thru more than anyone here but right now there are probably 100 funded startups pursuing agentic AI without any safety qualms.

ryandvm 4y ago

This is where I feel like legislation is pointless. In 10 years you'll be able to run GPT-4 on an old phone. Imagine what a fairly well funded group will be able to do. There's no amount of legislation or technological guardrails that are getting this genie back in the bottle.

highwaylights 4y ago
This. It exists now, it’ll develop. That the potential is being realised means the money being pumped into competing implementations will continue to skyrocket so it’ll only accelerate from here. I would say it’s too late to do something about this, but in reality, wherever this is going was a foregone conclusion the moment early hominids began using tools. Technology doesn’t regress.
- kneebonian 4y ago
  > Technology doesn’t regress. I hate this assertion, true technology hasn't gone backwards much in the past 200 years, but honestly technology can and often does regress historically speaking. There have been plenty of times empires have risen and fallen, taking with them much of their knowledge. How much was lost when Rome decayed simply because there wasn't the structure in order to allow technologists to continue creating? Things are bigger and more interconnected this time so it is harder for an isolated event to cause things to fall backwards, but when it does the impact will be much larger.
  - lrvick 4y ago
    If you need evidence that technology regresses look no further than Facebook.
  - UncleEntity 4y ago
    > How much was lost when Rome decayed simply because there wasn't the structure in order to allow technologists to continue creating? Considering the Eastern Roman Empire existing until 1453 and the Middle Eastern empires did lots of scientific stuff I’d say not much.
- Yoric 4y ago
  > skyrocket Spell-checking suggestion: "skynet".
dwohnitmok 4y ago
> There's no amount of legislation or technological guardrails that are getting this genie back in the bottle. We're not getting the genie back in the bottle but there's still a lot of ways that we can change how we develop AI moving forward.

nradov 4y ago

Why does it need extreme caution?

swyx 4y ago
these AIs have already demonstrated limited tendency to strongly prefer not being shut down (they already have the ability to lie to humans to achieve goals and we celebrate this lol). giving them the ability to self replicate means we lose that ability to pull the plug. giving them ability to self modify means humanity will lose control of ai on any long enough timescale.
- nradov 4y ago
  The fact that you would use words like "prefer", "lie", and "celebrate" in this context indicates that you are deeply ignorant about the underlying technology. Your claims are wild, unsupported conjecture and cannot be taken seriously.
  - thfuran 4y ago
    While I agree with you, whether it is able to have preferences or feelings doesn't actually matter in this context. A simulated pzombie may be morally distinct from a person, but the consequences of its actions are unaffected by the distinction.
  - Karunamon 4y ago
    Remember the erratic behavior demonstrated by "Sydney", the Bing GPT bot? Whether the model has the "cognitive" ability to "prefer" something is not relevant. The model has the ability to submit search engine queries now; this is a read only operation for now. Once we give the bot write ability, erratic behavior has the potential to cause real world damage. Whether it is an actual "preference" or something that only looks like one based on the output is purely academic at that point.
  - haswell 4y ago
    I’ve been ranting against the anthropomorphization of these new AI tools pretty consistently, but I think the fact that these tools only appear to possess certain characteristics is all the more reason that we should not be hooking these things up to tools that give them access to the real world. If a language model “concocts” some “scheme” and executes it by interacting with real systems in ways that have real world impact, it doesn’t much matter if the whole thing is a hallucination. Put another way, would you feel comfortable letting a hallucination machine running on a supercomputer have access to the real world? It doesn’t have to truly “know” anything for the impact of the resulting actions to matter.

losvedir 4y ago

I don't really understand the threat model here. Surely "launch the nukes" is not available as a public API. I would expect anyone with networked services (at least those that are important or serious) already take precautions with authentication and authorization and such. Even if some mythical AGI isn't ready yet, there's already very smart, well-funded hacking groups from hostile nation states, for example. How will an AI adversary interact any differently with than world than a flesh and blood adversary?

Turing_Machine 4y ago
> Surely "launch the nukes" is not available as a public API. Of course not. However "manipulate the news and social media so that the humans launch the nukes for you" isn't quite so far-fetched. > How will an AI adversary interact any differently with than world than a flesh and blood adversary? It's possible to spin up a whole army of AI adversary bots in a hurry. Recruiting that many humans takes a lot of time and money.
- lrvick 4y ago
  Or it could infect the corporate IT endpoint management software used on the systems of the engineers that author and review the firmware for said nukes, so extra code is inserted they never see.
  - nradov 4y ago
    Those systems are air gapped and do not have accessible endpoints.
    - lrvick 4y ago
      The software that runs on the airgapped machines is generally authored on internet connected machines, or using software compiled on internet connected machines. I audit airgapped machine setups for a living and most are nowhere near as isolated as claimed.
4y ago