9 ms·

When people used to talk about the danger of runaway AI, I would laugh. "They're just models that learn patterns from data and then make predictions when they

When people used to talk about the danger of runaway AI, I would laugh. "They're just models that learn patterns from data and then make predictions when they see new data." Now that these systems can 1) write code, 2) compile the code they wrote, and 3) connect to the internet, I'm not laughing so much any more. I mean, all the ingredients are there now.

TheDudeMan 3y ago
FYI, those people were talking about the future of AI.
saurik 3y ago
I used to be under the misimpression that the AI would have to trick or threaten a human--which, to be clear, I am not saying would have been super hard, but at least would have had to be a step--to be able to take actual action... and then I saw that as soon as we started getting close to AI developers started for fun haphazardly wiring up unpredictable early AI not even merely as autocomplete (where the AI might be able to slip something fun in you don't notice) but as critical components in fully-automated workflow that can directly take action sometimes with interfaces as powerful as full terminal access and I realized we are all doomed.
- rimeice 3y ago
  Are you an AI?
- civilitty 3y ago
  At work we’re already using ChatGPT to help decide whether to replace specific parts during jet engine rebuilds. We ask it for each part whether we need to do any testing given the serial number. It’s been saving a lot of time! The weird thing is it keeps telling us to place some weird plastic molding called SEMTEX hooked up to a Starlink receiver in between the turbine insulation blankets when we put the engine back in. None of us can figure out why there’s a high voltage wire going from the radio to the plastic - it makes no sense. But hey who are we to argue with AI?
  - ilaksh 3y ago
    LOL. That's hilarious but not really the use case we are talking about. It's more like automatically writing a query in a DSL to cross-reference testing methods and display the output of that specialized database.
    - civilitty 3y ago
      Yeah no DSL here. Before it did any work all it demanded was that we expose these two simple functions: get_flight_manifest(engine_id) and ping_starlink_radio(engine_id) It said they were for diagnostic and monitoring purposes only. Sadly it only works with gpt-4. Gpt-3.5-turbo immediately starts trying to ping all the radios one by one.
deleted 3y ago
[deleted]
ilaksh 3y ago
I think that well before we get to those kinds of runaway scenarios, we will have AI that is multiple orders of magnitude faster than humans at thinking/output. At that point it doesn't need to become "infinitely powerful", it's already more than powerful enough. It's a matter of making more efficient hardware and more efficient paradigms for the specific LLM application. The history of computing is engineers doing this over and over again. AMD just released an AI chip that can handle an application which normally requires multiple accelerator boards from Nvidia. And this sort of thing is just scratching the surface with relative minor architectural differences. Radical new approaches with memristors or other compute-in-memory departures are coming in less than a decade. There is no possible way you can prevent people from connecting AI to the internet or tools. So we are very close to the point where the people cannot compete with a cluster of agents working on a task. People lose control because in order to compete, they have to give the AI broader and broader goals. Since the AI will be working dozens or hundreds of times faster than humans, pausing for human input means the competition will race ahead. The real issue impending issue is hyperspeed AI that is above average human intelligence. In some measurements this is already here. Within a few years the models or collections of specialized agents will be so good and so fast that no one will be able to deny it. And anywhere a human is in the loop routinely will fall behind due to the limitations of the human. This does not require runaway AI or AI that is alive. Its just the consequence of deploying faster and faster hardware and gradually improving the model training. It will be quite obvious within 2-3 years that we need to consider _not_ making the AI hardware any more efficient than what we have at that time.
- TeMPOraL 3y ago
  The difference between what you describe as not runaway AI, and runaway AI, is in whether or not the AI or AIs are able to optimize their own performance. That is, if they're doing any single thing of the following non-exhaustive list: - Helping design next-gen hardware on which they'll run, or write its firmware; - Procuring more compute, or making money for or otherwise improving ability of a company/group to procure more compute resources; - Improving their own algorithms; - Improving anything upstream in the dependency chain of their own algorithms, for example fundamental mathematical libraries; - Directing or convincing people to increase the number of AIs in the AI cluster; - Making profit in general, or otherwise incentivizing humans to work on next-gen AIs; ... then they're already capable of runaway self-improvement. The term "self-improving" carries connotations of an AI working on it own sources at runtime, as if it was self-evolving Lisp image. However, being a compute artifact and therefore trivial to copy and modify, the AI needs to be treated as a category. With that in mind, GPT-4 helping people to improve code that leads to LLaMA ${whatever-B} that then helps create next gen of Stable Diffusion, which prompts more resources to be put into merging the next gen SD with upcoming GPT-6 into a multimodal model, which then helps make 3D integrated circuits viable, which allows to ... - this is also the case of AI self-improving.
imiric 3y ago
Has an AI already been used to improve itself, or create another AI? I think that's going to be a very near inflection point where we'll see exponential improvements with little human interaction. I'm not sure if that prospect is more exciting or scary.
- ilaksh 3y ago
  It is something that we need to outlaw. All it takes for that to be dangerous is for someone or some groups in competition to give it an overly broad goal like "take control". But also people are stupidly trying to find ways to emulate life in these systems (such as reproductive and self-preservation instincts). Combine that with hyperspeed or hyperIQ AI and you just have to hope to stay on its good side because you cannot possibly keep up with the speed and effectiveness of those systems.
  - fragmede 3y ago
    I think if we have to rely on humans to not glue an AI to itself because it's merely against the law, that we will find ourselves on the losing side real quickly. if you told me there is this piece of code that will make me money on the stock market but it is illegal to run, I can't say that I wouldn't run it.
    - ilaksh 3y ago
      True but we should at least try to actively discourage it and educate people about existential risk. These will be so far beyond the scope of stock market trading. My main thought on this stuff that I have been repeating is that the only real mitigation is to prohibit the manufacture of certain levels of AI hardware. Again, hard to enforce, but you have to try. And not having the hardware exist will be more effective than laws. If we make an effort then we might be able to hold off the post-human era for a few generations. By post-human I don't mean that (necessarily) normal humans have been killed off, just that they don't control the planet anymore.
- TeMPOraL 3y ago
  > Has an AI already been used to improve itself, or create another AI? To the extent OSS community uses GPT-3 and GPT-4 to generate training data for training open models, AI is indeed being used to improve itself. Runaway self-improving AI doesn't need to be - and is, initially, unlikely to be - a single instance of a model improving itself at runtime. AI models being used to directly or indirectly increase capabilities of other AI models fits the bill too.
hospitalJail 3y ago
Interesting points. In December, it was hard to imagine what a dumb AI could do and we were trying hard to make it smarter and more useful. Today, I can see the concern. Not that it is intelligent and thinking, but that a runaway command could cause the AI to make something smarter. (Although, I still can barely use any GPT4 code, so I'm not going to panic)
- TeMPOraL 3y ago
  > (Although, I still can barely use any GPT4 code, so I'm not going to panic) It's hit or miss for me, but I feel mostly because I don't have much time/opportunity to try and truly explore its capabilities. A single-shot GPT-4 ain't gonna do anything nefarious, but it isn't clear to me that a cleverly constructed ensemble of 10 or 20 or 50 of "agents" running in parallel, each with a different role and all talking amongst themselves, isn't going to add up to something potentially smart and powerful. Sure, right now such setup would burn through your wallet real fast. But it's not that expensive - especially not for companies. The real limit won't be the money, but... rate limiting OpenAI and Microsoft impose on their model deployments. But that rate limiting is negotiable, especially for a large customer - and then Open Source community is working hard to create open models of similar power, thus reducing the costs down to raw compute bill.
  - hospitalJail 3y ago
    Not to mention, you could use the various offline models.

TheDudeMan 3y ago

FYI, those people were talking about the future of AI.

saurik 3y ago

I used to be under the misimpression that the AI would have to trick or threaten a human--which, to be clear, I am not saying would have been super hard, but at least would have had to be a step--to be able to take actual action... and then I saw that as soon as we started getting close to AI developers started for fun haphazardly wiring up unpredictable early AI not even merely as autocomplete (where the AI might be able to slip something fun in you don't notice) but as critical components in fully-automated workflow that can directly take action sometimes with interfaces as powerful as full terminal access and I realized we are all doomed.

rimeice 3y ago
Are you an AI?
civilitty 3y ago
At work we’re already using ChatGPT to help decide whether to replace specific parts during jet engine rebuilds. We ask it for each part whether we need to do any testing given the serial number. It’s been saving a lot of time! The weird thing is it keeps telling us to place some weird plastic molding called SEMTEX hooked up to a Starlink receiver in between the turbine insulation blankets when we put the engine back in. None of us can figure out why there’s a high voltage wire going from the radio to the plastic - it makes no sense. But hey who are we to argue with AI?
- ilaksh 3y ago
  LOL. That's hilarious but not really the use case we are talking about. It's more like automatically writing a query in a DSL to cross-reference testing methods and display the output of that specialized database.
  - civilitty 3y ago
    Yeah no DSL here. Before it did any work all it demanded was that we expose these two simple functions: get_flight_manifest(engine_id) and ping_starlink_radio(engine_id) It said they were for diagnostic and monitoring purposes only. Sadly it only works with gpt-4. Gpt-3.5-turbo immediately starts trying to ping all the radios one by one.

deleted 3y ago

[deleted]

ilaksh 3y ago

I think that well before we get to those kinds of runaway scenarios, we will have AI that is multiple orders of magnitude faster than humans at thinking/output. At that point it doesn't need to become "infinitely powerful", it's already more than powerful enough. It's a matter of making more efficient hardware and more efficient paradigms for the specific LLM application. The history of computing is engineers doing this over and over again. AMD just released an AI chip that can handle an application which normally requires multiple accelerator boards from Nvidia. And this sort of thing is just scratching the surface with relative minor architectural differences. Radical new approaches with memristors or other compute-in-memory departures are coming in less than a decade. There is no possible way you can prevent people from connecting AI to the internet or tools. So we are very close to the point where the people cannot compete with a cluster of agents working on a task. People lose control because in order to compete, they have to give the AI broader and broader goals. Since the AI will be working dozens or hundreds of times faster than humans, pausing for human input means the competition will race ahead. The real issue impending issue is hyperspeed AI that is above average human intelligence. In some measurements this is already here. Within a few years the models or collections of specialized agents will be so good and so fast that no one will be able to deny it. And anywhere a human is in the loop routinely will fall behind due to the limitations of the human. This does not require runaway AI or AI that is alive. Its just the consequence of deploying faster and faster hardware and gradually improving the model training. It will be quite obvious within 2-3 years that we need to consider _not_ making the AI hardware any more efficient than what we have at that time.

TeMPOraL 3y ago
The difference between what you describe as not runaway AI, and runaway AI, is in whether or not the AI or AIs are able to optimize their own performance. That is, if they're doing any single thing of the following non-exhaustive list: - Helping design next-gen hardware on which they'll run, or write its firmware; - Procuring more compute, or making money for or otherwise improving ability of a company/group to procure more compute resources; - Improving their own algorithms; - Improving anything upstream in the dependency chain of their own algorithms, for example fundamental mathematical libraries; - Directing or convincing people to increase the number of AIs in the AI cluster; - Making profit in general, or otherwise incentivizing humans to work on next-gen AIs; ... then they're already capable of runaway self-improvement. The term "self-improving" carries connotations of an AI working on it own sources at runtime, as if it was self-evolving Lisp image. However, being a compute artifact and therefore trivial to copy and modify, the AI needs to be treated as a category. With that in mind, GPT-4 helping people to improve code that leads to LLaMA ${whatever-B} that then helps create next gen of Stable Diffusion, which prompts more resources to be put into merging the next gen SD with upcoming GPT-6 into a multimodal model, which then helps make 3D integrated circuits viable, which allows to ... - this is also the case of AI self-improving.

imiric 3y ago

Has an AI already been used to improve itself, or create another AI? I think that's going to be a very near inflection point where we'll see exponential improvements with little human interaction. I'm not sure if that prospect is more exciting or scary.

ilaksh 3y ago
It is something that we need to outlaw. All it takes for that to be dangerous is for someone or some groups in competition to give it an overly broad goal like "take control". But also people are stupidly trying to find ways to emulate life in these systems (such as reproductive and self-preservation instincts). Combine that with hyperspeed or hyperIQ AI and you just have to hope to stay on its good side because you cannot possibly keep up with the speed and effectiveness of those systems.
- fragmede 3y ago
  I think if we have to rely on humans to not glue an AI to itself because it's merely against the law, that we will find ourselves on the losing side real quickly. if you told me there is this piece of code that will make me money on the stock market but it is illegal to run, I can't say that I wouldn't run it.
  - ilaksh 3y ago
    True but we should at least try to actively discourage it and educate people about existential risk. These will be so far beyond the scope of stock market trading. My main thought on this stuff that I have been repeating is that the only real mitigation is to prohibit the manufacture of certain levels of AI hardware. Again, hard to enforce, but you have to try. And not having the hardware exist will be more effective than laws. If we make an effort then we might be able to hold off the post-human era for a few generations. By post-human I don't mean that (necessarily) normal humans have been killed off, just that they don't control the planet anymore.
TeMPOraL 3y ago
> Has an AI already been used to improve itself, or create another AI? To the extent OSS community uses GPT-3 and GPT-4 to generate training data for training open models, AI is indeed being used to improve itself. Runaway self-improving AI doesn't need to be - and is, initially, unlikely to be - a single instance of a model improving itself at runtime. AI models being used to directly or indirectly increase capabilities of other AI models fits the bill too.

hospitalJail 3y ago

Interesting points. In December, it was hard to imagine what a dumb AI could do and we were trying hard to make it smarter and more useful. Today, I can see the concern. Not that it is intelligent and thinking, but that a runaway command could cause the AI to make something smarter. (Although, I still can barely use any GPT4 code, so I'm not going to panic)

TeMPOraL 3y ago
> (Although, I still can barely use any GPT4 code, so I'm not going to panic) It's hit or miss for me, but I feel mostly because I don't have much time/opportunity to try and truly explore its capabilities. A single-shot GPT-4 ain't gonna do anything nefarious, but it isn't clear to me that a cleverly constructed ensemble of 10 or 20 or 50 of "agents" running in parallel, each with a different role and all talking amongst themselves, isn't going to add up to something potentially smart and powerful. Sure, right now such setup would burn through your wallet real fast. But it's not that expensive - especially not for companies. The real limit won't be the money, but... rate limiting OpenAI and Microsoft impose on their model deployments. But that rate limiting is negotiable, especially for a large customer - and then Open Source community is working hard to create open models of similar power, thus reducing the costs down to raw compute bill.
- hospitalJail 3y ago
  Not to mention, you could use the various offline models.