8 ms·

A fairly lengthy article about Autonomous AI, and, as far as I can tell, not a single word about the safety implications of such a system (a short note about re

by p-e-w 3y ago

A fairly lengthy article about Autonomous AI, and, as far as I can tell, not a single word about the safety implications of such a system (a short note about reliability of LLMs is all we get, and it's not clear that the author means anything more than "the thing might break"). I get that there are different philosophies on AI risk, but this is like reading an in-depth discussion about potential atomic bombs in 1942, with no mention of the fact that such a bomb could potentially level cities and kill millions. If a field of research ever needed oversight and regulation, this is it. I'm not convinced that would solve the problem, but allowing this kind of rushing forward to continue is madness.

ftxbro 3y ago
ok but it's an odd take for a tech site like hacker news i mean what does that say about the llm paradigm vs for example object oriented paradigm or functional programming paradigm were they just playing around before and now it's getting real? were they deliberately doing toy things before and now the adults are in the room making real computation systems with actual consequences for the real world and not just nerds playing with toys or what? Like does this mean that it will completely obsolete the works of boffins like simon peyton jones and those other esoteric languages and everyone will talk to their computers in english language and the ones that will be the most effective computer enjoyers will be the high emotional intelligence salesmen? were they going in wrong direction for so long on purpose because they didn't want to awaken the real power of the computers for ethical reasons, or were they trying to do this the whole time but they weren't smart enough to figure it out?
carom 3y ago
No, this is nothing like atomic bombs. In what scenario can you kill 100,000 people with with an LLM hooked up to a Python interpreter? The AI safety grift is out of control.
- p-e-w 3y ago
  > In what scenario can you kill 100,000 people with with an LLM hooked up to a Python interpreter? The article talks about LLMs being hooked up to the Internet. Big difference. If you really can't imagine a scenario where this might lead to people dying, you're not trying hard enough.
  - carom 3y ago
    You have failed to provide a scenario, likely because no realistic scenario exists. Let's go through an example scenario and the hurdles you have to overcome for this to work. We'll use the "mayonnaise plant" example from a sibling comment. 1. The LLM needs to find an exploitable bug in a popular code base. 2. The LLM needs to write a reliable exploit for that bug. 3. The LLM needs to develop a worm that exploits that bug and spreads itself, opening access to the system. 4. The LLM needs to connect to systems and understand if they are of any significance (it found a mayonnaise plant!). 5. The LLM needs to understand the control protocols of their industrial control systems. 6. The LLM needs to understand how to make a dangerous composition from the ingredients it has on hand (let's pretend it can dump some industrial cleaning solution that is on standby for cleaning the tanks). 7. The LLM needs to assume such total control over this processing plant that it can disguise the traffic and not trigger a single alarm around malfunctions. What you're vaguely hinting at is extremely high skilled labor. There are a few billion dollar businesses in those steps. I welcome you to go read up on the challenges in automated exploit generation. LLMs are nowhere close. Now, you might rebut and say there are far simpler attacks, like phishing! Also an extremely hard problem. Try to send email in mass and not land in a spam filter, try to do the reconnaissance necessary to generate a believable login page. Try to leverage the sale guy's credentials to reach any system more meaningful than the company's Salesforce instance. So once again, I ask, please walk me through a situation where an LLM gets anywhere close to killing even 1% of the number of people an atomic bomb could.
    - p-e-w 3y ago
      > LLMs are nowhere close. We are nowhere close to even a rudimentary understanding of how current LLMs actually work. All we have are low-level building blocks, and lots of philosophizing about high-level output. Considering that, relying on some gut feeling about what LLMs "surely cannot possibly do" is reckless overconfidence. Not to mention that the next generation of LLMs, with potentially entirely new emergent properties, might be just around the corner, and the time to put safeguards into place is now, not when it's too late. As for the scenario, all an LLM with Internet access would need to do is find a single remotely exploitable vulnerability in the Linux network stack. That would allow it to literally shut down the entire Internet, which would kill a lot more people than a single atomic bomb.
- TeMPOraL 3y ago
  > In what scenario can you kill 100,000 people with with an LLM hooked up to a Python interpreter? When that Python interpreter is running on the computer doing model-predictive control of your country's natural gas pipelines, or the plant that mixes your mayonnaise. Just so you know, the Python interpreter is already there in those places, and so is Internet access, because suits like their dashboards with real-time graphs. Remember, the S in IIoT (Industrial IoT) stands for both sanity and security :).
  - throwuwu 3y ago
    Are you arguing that an LLM will be used in process control or statistical modelling? Or that someone will be running code in an interpreter that just happens to interact with those systems? Because there’s no way that someone will use one to control the automated processes in a factory. Those have to be perfectly repeatable. It’s not like someone showed up to the mayonnaise factory and says “I think I’ll increase the ratio of paprika to salt today, just for fun”
    - TeMPOraL 3y ago
      > Are you arguing that an LLM will be used in process control or statistical modelling? Or that someone will be running code in an interpreter that just happens to interact with those systems? Yes to both, sadly. > Because there’s no way that someone will use one to control the automated processes in a factory. What is that saying Americans are fond of? Ah, yes - if you believe that, then I have a bridge to sell you. > It’s not like someone showed up to the mayonnaise factory and says “I think I’ll increase the ratio of paprika to salt today, just for fun” No, they show up at the industry trade fair in some German city, find a group of people looking like they own factories, and start waxing poetic about how their company is using the newest advances in AI to synergize value delivery and carbon supply chain footprint management, ushering in the era of ecologically responsible and exponentially profitable Industry 4.1. You know, like ChatGPT but you plug it into your plant, feed it your BOM, and it prints out money. This kind of stuff.
- bamboozled 3y ago
  But that's not what this article is about, it's about attempting to create some type of AGI, highly experimental seemingly non-aligned, which is connected to...the internet. Which is why I do agree with others, even if this is a a weak attempt (I have no idea, not an AI researcher) if it was to be successful could be hyper destructive. The project seems really quite loose and the author seems to be lacking in awareness with regards to the consequences of such a project. I do think others are right in saying, we really should understand more bout where people can and will try to take these ideas before we keep putting models on the internet for consumption by just anyone.
  - p-e-w 3y ago
    > seems to not have much awareness about what the consequences of such a project might mean That's the worst part of it. Is the current generation of LLMs dangerous? Probably not. Will the next generation of LLMs be dangerous? Maybe not, if we're lucky. The one after that? Nobody knows. But projects like this don't so much rely on there being no danger, but rather, the question seems to not even have been considered. And that's crazy, because the assumption that things are harmless is getting more and more shaky the more capable these models become.
    - TeMPOraL 3y ago
      I can kind of understand it. After over a decade of spending time in the influence zone of Bostrom, Eliezer, and LW crowd[0], I find myself torn. Half of me is acutely aware of the danger potential of what we're discussing. The other half wants to play with chaining the current LLMs, text-to-$X and $X-to-text models, and domain-specific classical software in hopes of bootstrapping a proto-AGI, because how cool it would be to pull it off. Good my ADHD makes me write HN comments on LLM threads instead of working on the latter, because I'm not sure I could stop myself otherwise. But this also means I can very much imagine only thinking about the cool parts, and not considering the dangerous ones - because that's how I feel about 90% of the time when thinking about current SOTA models. It's just too damn exciting. (Hell, it's the first thing in tech in the last decade, that I find exciting.) -- [0] - Whatever one may think of them, I find it sadly ironic how Eliezer and his disciples were considered pseudo-scientific doomsday AI cultists a decade ago, back when the whole thing was only a far-future concern, how they're still considered pseudo-scientific doomsday AI cultists today, after AI field made a series of rapid jumps, landing us exactly in the scope of things they were harping about all that time - and how I bet they'll still be considered pseudo-scientific doomsday AI cultists after someone unleashes an unaligned AGI and ends us.
- HeartStrings 3y ago
  "AI safety" is just folk who were too incapable to build LLMs themselves trying to be relevant. Of course, it’s all bull. If computer does bad, you just pull the plug.
  - p-e-w 3y ago
    Sure. And if the "bad" program has spread as a virus, you just pull all the plugs of all the computers, right? Of course, you will kill modern civilization by doing so (and many millions of people as a result), but at least the AI is back in the box, so there is nothing to worry about.
    - HeartStrings 3y ago
      This will happen a few times and then everyone will learnto use backups and better security. If anything, it will make humanity more antifragile.
    - moffkalast 3y ago
      It's not like actual genius people are constantly trying to make the deadliest virus for various purposes and we've been hardening our systems for decades in order to not have them immediately taken over. You'd have to go beyond AGI to break those safeguards. > all the computers The average shit tier PC doesn't even have enough RAM to load an LLM capable of consistent output, much less to load it into VRAM for any kind of reasonable speed. This is like being concerned that Panda bears will become an invasive species and take over the world if we stop them from going extinct. Besides, all you have to do to pull the plug is disable any of the billion python dependencies it needs to download every time and it's stopped dead in its tracks. Something that's basically done every other day and needs to be continuously fixed by project maintainers, so it's more like a Panda on life support that dies if somebody stops pumping air into its breathing hose. It's all fragile af.
  - ben_w 3y ago
    > Of course, it’s all bull. If computer does bad, you just pull the plug. Can you always tell if it's doing bad? Even if you can tell when it's doing bad, if there's an app which makes you $10k/hour while running, what risk of it killing somebody if left unsupervised will be low enough for you to leave it running unsupervised? Say, while you sleep? When there's a lot of money to be made, do you trust everyone else to (1) be as risk-adverse as you, and (2) not just convince themselves there's no risk using exactly the argument you just gave? Let's say you personally are willing to run it unsupervised if the risk is mean 1 fatality per 8*365*80 hours when unsupervised; are you willing for every human to have such a system running unsupervised? A tenth that risk?
- lannisterstark 3y ago
  I feel like all the blogg...uh, 'journalists' have just moved to "NO NOT THE AI" after whatever the last thing they were focusing on.
- ben_w 3y ago
  GPT-4 apparently does well on medical exams. Replace pharmacists to save money, don't need much of an error rate to get 100k dead. Have it design a nuclear reactor, saves money, you don't notice the flaw until it's too late. Chernobyl wasn't malicious, it was in part a cost-saving exercise; how many died from the fossil fuels that were burned because of the fear of nuclear power after the incident? How many died needlessly in Vietnam, before human journalists saw and reported on the human cost, and what happens when reporters are replaced with LLMs — to save money — that physically cannot see things for themselves? That said, 100k is in the awkward range that political decisions regularly lead to — at most a single month's life expectancy in a nation of 100 million — so if I'd been you I would have asked about 1M+. At a mere hypothetical 100k dead, it may easily still be part of a larger scenario which saves lives overall, much like how the actual atomic bombs were an alternative to a protracted conventional war (and given the attempted coup to prevent surrender after the second, I consider the hypothetical used to justify the bomb to at least be plausible, though I am no historian and don't want to give the impression of certainty).
  - bbor 3y ago
    Lol just gotta say I love these examples, very creative thinking and not at all the typical AI alignment concerns, which revolve around Skynet and paper clips. Consequentialism can get real nasty!
  - throwuwu 3y ago
    That’s a pretty weak argument since it can be applied to anything. eg there is a mistake in this process and people died as a result
    - ben_w 3y ago
      That's basically why I said I'd have put the threshold number much higher than a "mere" 100k. But even then I would say that the point is still important: disregard singularity, foom, and paperclipping, and AI is still often described as being akin to a faster rerun of the industrial revolution. Just having all the turmoil and unexpected consequences — health, social, environmental, economic, military, political — of the industrial revolution squeezed down from "economy doubling every 30 years" (which is what we saw) to "economy doubling every 18 months" (Moore's Law), is easily going to cause catastrophic consequences all by itself, even with no malice. "Health and safety" rules had to be created because too many bosses thought it was just common sense to not put your hands into dangerous equipment while in motion, while paying people to do jobs that could only be accomplished by doing exactly that; how can we do more than guess what the equivalent to that is for AI before we see it? What is to AI, that which global warming is to coal power? What's the necessary consequence of everyone using AI, which is analogous to the necessity of building a sewage system in response to widespread installation of flushing toilets in cities? We organised ourselves to build the latter, but we still haven't globally dealt with the former.
- bbotond 3y ago
  Watch a couple of videos by this guy: https://www.youtube.com/@RobertMilesAI https://www.youtube.com/@RobertMilesAI (very entertaining and informative by the way) These videos changed my mind entirely about AI risk.
- cyberneticist 3y ago
  It is a stupid argument too because it devalues their own position. I listened to about 6 hours of Marc Andreessen on this topic this weekend and he just smashes all the doomer arguments. Even in the context of nukes he made a great point that the invention of nukes most likely caused WW3 to not happen. An all out, devastating war between the US and Soviets over Europe that was only averted and transformed to the cold war because of MAD. I have been thinking since then how nukes probably caused my father to not fight in WW3. Who knows how many of us reading this would never have existed if nukes hadn't been invented. You can't make the argument 'the Manhattan project, we blow the planet up, the end". He also makes the point that you can't be worried about a super intelligence that is super intelligent in every way besides being so dumb that it turns everyone into paper clips. You can't have it both ways.
  - mitthrowaway2 3y ago
    > being so dumb that it turns everyone into paper clips If that's your goal, what's dumb about achieving it?
  - cwilkes 3y ago
    > Who knows how many of us reading this would never have existed if nukes hadn't been invented. You can't make the argument 'the Manhattan project, we blow the planet up, the end". I don’t think that argument against anti-LLM uses works as the response would be “nukes weren’t launched as we had humans in the loop. Without a stopgap like that maybe a nuke would launch if there was a 50.001% chance of our side winning” The counter counter argument to that would be that there isn’t enough weighing to how terrible a nuclear war would be and then maybe the LLM would change its mind. But who knows.
- DontchaKnowit 3y ago
  okay imagine that someone deploys hundreds of thousands of these LLM agents and sets the to work at finding vulnerable servers and searching for exploits. I don't think LLMs are quite up to snuff for a task like this, but they aren't far off. I can see a situation where food supply, electrical supply, water supply, etc is disrupted as a result of this. Its no atomic bomb, but it could be pretty damn serious.
  - throwuwu 3y ago
    Automated pen testing is already a thing. The same technology that improves it can also be used to harden systems against attack.
lannisterstark 3y ago
"not a single word about the safety implications of such a system" Oh please. Not everything has to be regulated-to-hells before a use case is even found on this. Autonomous agents have existed for decades. If it can automate agents like huginn[0] with natural language, I'd be very happy. Autonomous agents doesn't mean it's going to take over the world autonomously. Let's lower the fearmongering a bit. [0]: https://github.com/huginn/huginn https://github.com/huginn/huginn
- p-e-w 3y ago
  Huginn is a bunch of mechanical rules explicitly created by the user and easily understandable by them. It's "autonomous" in the sense of a grandfather clock. The rules governing LLMs, while in principle mechanical, cannot be accurately controlled or even understood by humans. They are "autonomous" in the sense of animals, which may be trained to some degree, but may still surprise or even become a danger to their "owners", no matter how much effort is spent trying to control them. An LLM may do what you ask it to, using the methods you expect it to use. Or it may do what you ask it to, using methods you weren't expecting at all. Or it may not do what you have asked at all. This isn't comparable to a rule-based system like Huginn.
  - lannisterstark 3y ago
    Huginn was just an example. With the similar definition to yours, are these "LLM autonomous agents" really actually "autonomous?" >They are "autonomous" in the sense of animals That's a really weird equivalency, which frankly, I'm not even sure is true.
    - TeMPOraL 3y ago
      > That's a really weird equivalency, which frankly, I'm not even sure is true. A mind that's a black box to us, that we can predict to a degree based only on observing it at work, that's also a general-purpose intelligence we're trying to employ for limited set of tasks. It's not a bad analogy. Like animals, LLMs too have the capacity to "think outside the box", where the box is what you'd consider the scope of the task.
ilaksh 3y ago
Good point. Part of the problem is the gap between GPT-4 and what people imagine to be dangerous is so large (on the surface). The term "AGI" for many has morphed into meaning something that's like simulation of a person, but with godlike infinite IQ and powers. So it's a special type of ASI now. Not my definition, just the sloppy way most people seem to think about it these days. So for those people, to get something dangerous, we have to create a full emulation of a person , give it a 1000 IQ, maybe make it operate at 1 million times human speed, etc. But the reality is that for a group of AI agents to be dangerous they just need to be very effective problem solvers connected to the world. I will say that the idea that many AI agents will not be connected to the internet or embodied is unrealistic. But I do think it's critical as you point out that we have awareness of the dangerous of full autonomy and effectors. The trick is that right now we have a very strong economic motive to increase the performance and efficiency of LLMs. And I believe that we should because there is huge upside and the state of the world is horrendous. But within only a few years (certainly less than ten) of optimizing this, we will be walking a fine line between tools that are extraordinarily empowering on the one side and completely out of our control on the other. All we need for GPT to be dangerous is for it to get maybe 25% smarter and much faster. We should anticipate 50-100 faster than humans output within a few years. Within 5-10 years those systems will be extremely affordable and widely deployed. The danger starts to emerge because the the extreme performance difference between human and AI. You can't tell the AI to wait for you to make decisions, because while it is waiting a few hours for you to figure out what it's talking about, the equivalent of days worth of activity has been accomplished by your competitor's AI agent. For people to see the danger it could help to explain the exponential track record of computing efficiency performance and also create demonstrations of what it's like to compete with something that thinks 50 times faster than you. One would think that people could extrapolate from video games or something. But I guess the problem is the open world video game agents are still largely unintelligent so it's hard to imagine.

by p-e-w 3y ago

ftxbro 3y ago

ok but it's an odd take for a tech site like hacker news i mean what does that say about the llm paradigm vs for example object oriented paradigm or functional programming paradigm were they just playing around before and now it's getting real? were they deliberately doing toy things before and now the adults are in the room making real computation systems with actual consequences for the real world and not just nerds playing with toys or what? Like does this mean that it will completely obsolete the works of boffins like simon peyton jones and those other esoteric languages and everyone will talk to their computers in english language and the ones that will be the most effective computer enjoyers will be the high emotional intelligence salesmen? were they going in wrong direction for so long on purpose because they didn't want to awaken the real power of the computers for ethical reasons, or were they trying to do this the whole time but they weren't smart enough to figure it out?

carom 3y ago

No, this is nothing like atomic bombs. In what scenario can you kill 100,000 people with with an LLM hooked up to a Python interpreter? The AI safety grift is out of control.

p-e-w 3y ago
> In what scenario can you kill 100,000 people with with an LLM hooked up to a Python interpreter? The article talks about LLMs being hooked up to the Internet. Big difference. If you really can't imagine a scenario where this might lead to people dying, you're not trying hard enough.
- carom 3y ago
  You have failed to provide a scenario, likely because no realistic scenario exists. Let's go through an example scenario and the hurdles you have to overcome for this to work. We'll use the "mayonnaise plant" example from a sibling comment. 1. The LLM needs to find an exploitable bug in a popular code base. 2. The LLM needs to write a reliable exploit for that bug. 3. The LLM needs to develop a worm that exploits that bug and spreads itself, opening access to the system. 4. The LLM needs to connect to systems and understand if they are of any significance (it found a mayonnaise plant!). 5. The LLM needs to understand the control protocols of their industrial control systems. 6. The LLM needs to understand how to make a dangerous composition from the ingredients it has on hand (let's pretend it can dump some industrial cleaning solution that is on standby for cleaning the tanks). 7. The LLM needs to assume such total control over this processing plant that it can disguise the traffic and not trigger a single alarm around malfunctions. What you're vaguely hinting at is extremely high skilled labor. There are a few billion dollar businesses in those steps. I welcome you to go read up on the challenges in automated exploit generation. LLMs are nowhere close. Now, you might rebut and say there are far simpler attacks, like phishing! Also an extremely hard problem. Try to send email in mass and not land in a spam filter, try to do the reconnaissance necessary to generate a believable login page. Try to leverage the sale guy's credentials to reach any system more meaningful than the company's Salesforce instance. So once again, I ask, please walk me through a situation where an LLM gets anywhere close to killing even 1% of the number of people an atomic bomb could.
  - p-e-w 3y ago
    > LLMs are nowhere close. We are nowhere close to even a rudimentary understanding of how current LLMs actually work. All we have are low-level building blocks, and lots of philosophizing about high-level output. Considering that, relying on some gut feeling about what LLMs "surely cannot possibly do" is reckless overconfidence. Not to mention that the next generation of LLMs, with potentially entirely new emergent properties, might be just around the corner, and the time to put safeguards into place is now, not when it's too late. As for the scenario, all an LLM with Internet access would need to do is find a single remotely exploitable vulnerability in the Linux network stack. That would allow it to literally shut down the entire Internet, which would kill a lot more people than a single atomic bomb.
TeMPOraL 3y ago
> In what scenario can you kill 100,000 people with with an LLM hooked up to a Python interpreter? When that Python interpreter is running on the computer doing model-predictive control of your country's natural gas pipelines, or the plant that mixes your mayonnaise. Just so you know, the Python interpreter is already there in those places, and so is Internet access, because suits like their dashboards with real-time graphs. Remember, the S in IIoT (Industrial IoT) stands for both sanity and security :).
- throwuwu 3y ago
  Are you arguing that an LLM will be used in process control or statistical modelling? Or that someone will be running code in an interpreter that just happens to interact with those systems? Because there’s no way that someone will use one to control the automated processes in a factory. Those have to be perfectly repeatable. It’s not like someone showed up to the mayonnaise factory and says “I think I’ll increase the ratio of paprika to salt today, just for fun”
  - TeMPOraL 3y ago
    > Are you arguing that an LLM will be used in process control or statistical modelling? Or that someone will be running code in an interpreter that just happens to interact with those systems? Yes to both, sadly. > Because there’s no way that someone will use one to control the automated processes in a factory. What is that saying Americans are fond of? Ah, yes - if you believe that, then I have a bridge to sell you. > It’s not like someone showed up to the mayonnaise factory and says “I think I’ll increase the ratio of paprika to salt today, just for fun” No, they show up at the industry trade fair in some German city, find a group of people looking like they own factories, and start waxing poetic about how their company is using the newest advances in AI to synergize value delivery and carbon supply chain footprint management, ushering in the era of ecologically responsible and exponentially profitable Industry 4.1. You know, like ChatGPT but you plug it into your plant, feed it your BOM, and it prints out money. This kind of stuff.
bamboozled 3y ago
But that's not what this article is about, it's about attempting to create some type of AGI, highly experimental seemingly non-aligned, which is connected to...the internet. Which is why I do agree with others, even if this is a a weak attempt (I have no idea, not an AI researcher) if it was to be successful could be hyper destructive. The project seems really quite loose and the author seems to be lacking in awareness with regards to the consequences of such a project. I do think others are right in saying, we really should understand more bout where people can and will try to take these ideas before we keep putting models on the internet for consumption by just anyone.
- p-e-w 3y ago
  > seems to not have much awareness about what the consequences of such a project might mean That's the worst part of it. Is the current generation of LLMs dangerous? Probably not. Will the next generation of LLMs be dangerous? Maybe not, if we're lucky. The one after that? Nobody knows. But projects like this don't so much rely on there being no danger, but rather, the question seems to not even have been considered. And that's crazy, because the assumption that things are harmless is getting more and more shaky the more capable these models become.
  - TeMPOraL 3y ago
    I can kind of understand it. After over a decade of spending time in the influence zone of Bostrom, Eliezer, and LW crowd[0], I find myself torn. Half of me is acutely aware of the danger potential of what we're discussing. The other half wants to play with chaining the current LLMs, text-to-$X and $X-to-text models, and domain-specific classical software in hopes of bootstrapping a proto-AGI, because how cool it would be to pull it off. Good my ADHD makes me write HN comments on LLM threads instead of working on the latter, because I'm not sure I could stop myself otherwise. But this also means I can very much imagine only thinking about the cool parts, and not considering the dangerous ones - because that's how I feel about 90% of the time when thinking about current SOTA models. It's just too damn exciting. (Hell, it's the first thing in tech in the last decade, that I find exciting.) -- [0] - Whatever one may think of them, I find it sadly ironic how Eliezer and his disciples were considered pseudo-scientific doomsday AI cultists a decade ago, back when the whole thing was only a far-future concern, how they're still considered pseudo-scientific doomsday AI cultists today, after AI field made a series of rapid jumps, landing us exactly in the scope of things they were harping about all that time - and how I bet they'll still be considered pseudo-scientific doomsday AI cultists after someone unleashes an unaligned AGI and ends us.
HeartStrings 3y ago
"AI safety" is just folk who were too incapable to build LLMs themselves trying to be relevant. Of course, it’s all bull. If computer does bad, you just pull the plug.
- p-e-w 3y ago
  Sure. And if the "bad" program has spread as a virus, you just pull all the plugs of all the computers, right? Of course, you will kill modern civilization by doing so (and many millions of people as a result), but at least the AI is back in the box, so there is nothing to worry about.
  - HeartStrings 3y ago
    This will happen a few times and then everyone will learnto use backups and better security. If anything, it will make humanity more antifragile.
  - moffkalast 3y ago
    It's not like actual genius people are constantly trying to make the deadliest virus for various purposes and we've been hardening our systems for decades in order to not have them immediately taken over. You'd have to go beyond AGI to break those safeguards. > all the computers The average shit tier PC doesn't even have enough RAM to load an LLM capable of consistent output, much less to load it into VRAM for any kind of reasonable speed. This is like being concerned that Panda bears will become an invasive species and take over the world if we stop them from going extinct. Besides, all you have to do to pull the plug is disable any of the billion python dependencies it needs to download every time and it's stopped dead in its tracks. Something that's basically done every other day and needs to be continuously fixed by project maintainers, so it's more like a Panda on life support that dies if somebody stops pumping air into its breathing hose. It's all fragile af.
- ben_w 3y ago
  > Of course, it’s all bull. If computer does bad, you just pull the plug. Can you always tell if it's doing bad? Even if you can tell when it's doing bad, if there's an app which makes you $10k/hour while running, what risk of it killing somebody if left unsupervised will be low enough for you to leave it running unsupervised? Say, while you sleep? When there's a lot of money to be made, do you trust everyone else to (1) be as risk-adverse as you, and (2) not just convince themselves there's no risk using exactly the argument you just gave? Let's say you personally are willing to run it unsupervised if the risk is mean 1 fatality per 8*365*80 hours when unsupervised; are you willing for every human to have such a system running unsupervised? A tenth that risk?
lannisterstark 3y ago
I feel like all the blogg...uh, 'journalists' have just moved to "NO NOT THE AI" after whatever the last thing they were focusing on.
ben_w 3y ago
GPT-4 apparently does well on medical exams. Replace pharmacists to save money, don't need much of an error rate to get 100k dead. Have it design a nuclear reactor, saves money, you don't notice the flaw until it's too late. Chernobyl wasn't malicious, it was in part a cost-saving exercise; how many died from the fossil fuels that were burned because of the fear of nuclear power after the incident? How many died needlessly in Vietnam, before human journalists saw and reported on the human cost, and what happens when reporters are replaced with LLMs — to save money — that physically cannot see things for themselves? That said, 100k is in the awkward range that political decisions regularly lead to — at most a single month's life expectancy in a nation of 100 million — so if I'd been you I would have asked about 1M+. At a mere hypothetical 100k dead, it may easily still be part of a larger scenario which saves lives overall, much like how the actual atomic bombs were an alternative to a protracted conventional war (and given the attempted coup to prevent surrender after the second, I consider the hypothetical used to justify the bomb to at least be plausible, though I am no historian and don't want to give the impression of certainty).
- bbor 3y ago
  Lol just gotta say I love these examples, very creative thinking and not at all the typical AI alignment concerns, which revolve around Skynet and paper clips. Consequentialism can get real nasty!
- throwuwu 3y ago
  That’s a pretty weak argument since it can be applied to anything. eg there is a mistake in this process and people died as a result
  - ben_w 3y ago
    That's basically why I said I'd have put the threshold number much higher than a "mere" 100k. But even then I would say that the point is still important: disregard singularity, foom, and paperclipping, and AI is still often described as being akin to a faster rerun of the industrial revolution. Just having all the turmoil and unexpected consequences — health, social, environmental, economic, military, political — of the industrial revolution squeezed down from "economy doubling every 30 years" (which is what we saw) to "economy doubling every 18 months" (Moore's Law), is easily going to cause catastrophic consequences all by itself, even with no malice. "Health and safety" rules had to be created because too many bosses thought it was just common sense to not put your hands into dangerous equipment while in motion, while paying people to do jobs that could only be accomplished by doing exactly that; how can we do more than guess what the equivalent to that is for AI before we see it? What is to AI, that which global warming is to coal power? What's the necessary consequence of everyone using AI, which is analogous to the necessity of building a sewage system in response to widespread installation of flushing toilets in cities? We organised ourselves to build the latter, but we still haven't globally dealt with the former.
bbotond 3y ago
Watch a couple of videos by this guy: https://www.youtube.com/@RobertMilesAI https://www.youtube.com/@RobertMilesAI (very entertaining and informative by the way) These videos changed my mind entirely about AI risk.
cyberneticist 3y ago
It is a stupid argument too because it devalues their own position. I listened to about 6 hours of Marc Andreessen on this topic this weekend and he just smashes all the doomer arguments. Even in the context of nukes he made a great point that the invention of nukes most likely caused WW3 to not happen. An all out, devastating war between the US and Soviets over Europe that was only averted and transformed to the cold war because of MAD. I have been thinking since then how nukes probably caused my father to not fight in WW3. Who knows how many of us reading this would never have existed if nukes hadn't been invented. You can't make the argument 'the Manhattan project, we blow the planet up, the end". He also makes the point that you can't be worried about a super intelligence that is super intelligent in every way besides being so dumb that it turns everyone into paper clips. You can't have it both ways.
- mitthrowaway2 3y ago
  > being so dumb that it turns everyone into paper clips If that's your goal, what's dumb about achieving it?
- cwilkes 3y ago
  > Who knows how many of us reading this would never have existed if nukes hadn't been invented. You can't make the argument 'the Manhattan project, we blow the planet up, the end". I don’t think that argument against anti-LLM uses works as the response would be “nukes weren’t launched as we had humans in the loop. Without a stopgap like that maybe a nuke would launch if there was a 50.001% chance of our side winning” The counter counter argument to that would be that there isn’t enough weighing to how terrible a nuclear war would be and then maybe the LLM would change its mind. But who knows.
DontchaKnowit 3y ago
okay imagine that someone deploys hundreds of thousands of these LLM agents and sets the to work at finding vulnerable servers and searching for exploits. I don't think LLMs are quite up to snuff for a task like this, but they aren't far off. I can see a situation where food supply, electrical supply, water supply, etc is disrupted as a result of this. Its no atomic bomb, but it could be pretty damn serious.
- throwuwu 3y ago
  Automated pen testing is already a thing. The same technology that improves it can also be used to harden systems against attack.

lannisterstark 3y ago

"not a single word about the safety implications of such a system" Oh please. Not everything has to be regulated-to-hells before a use case is even found on this. Autonomous agents have existed for decades. If it can automate agents like huginn[0] with natural language, I'd be very happy. Autonomous agents doesn't mean it's going to take over the world autonomously. Let's lower the fearmongering a bit. [0]: https://github.com/huginn/huginn https://github.com/huginn/huginn

p-e-w 3y ago
Huginn is a bunch of mechanical rules explicitly created by the user and easily understandable by them. It's "autonomous" in the sense of a grandfather clock. The rules governing LLMs, while in principle mechanical, cannot be accurately controlled or even understood by humans. They are "autonomous" in the sense of animals, which may be trained to some degree, but may still surprise or even become a danger to their "owners", no matter how much effort is spent trying to control them. An LLM may do what you ask it to, using the methods you expect it to use. Or it may do what you ask it to, using methods you weren't expecting at all. Or it may not do what you have asked at all. This isn't comparable to a rule-based system like Huginn.
- lannisterstark 3y ago
  Huginn was just an example. With the similar definition to yours, are these "LLM autonomous agents" really actually "autonomous?" >They are "autonomous" in the sense of animals That's a really weird equivalency, which frankly, I'm not even sure is true.
  - TeMPOraL 3y ago
    > That's a really weird equivalency, which frankly, I'm not even sure is true. A mind that's a black box to us, that we can predict to a degree based only on observing it at work, that's also a general-purpose intelligence we're trying to employ for limited set of tasks. It's not a bad analogy. Like animals, LLMs too have the capacity to "think outside the box", where the box is what you'd consider the scope of the task.

ilaksh 3y ago

Good point. Part of the problem is the gap between GPT-4 and what people imagine to be dangerous is so large (on the surface). The term "AGI" for many has morphed into meaning something that's like simulation of a person, but with godlike infinite IQ and powers. So it's a special type of ASI now. Not my definition, just the sloppy way most people seem to think about it these days. So for those people, to get something dangerous, we have to create a full emulation of a person , give it a 1000 IQ, maybe make it operate at 1 million times human speed, etc. But the reality is that for a group of AI agents to be dangerous they just need to be very effective problem solvers connected to the world. I will say that the idea that many AI agents will not be connected to the internet or embodied is unrealistic. But I do think it's critical as you point out that we have awareness of the dangerous of full autonomy and effectors. The trick is that right now we have a very strong economic motive to increase the performance and efficiency of LLMs. And I believe that we should because there is huge upside and the state of the world is horrendous. But within only a few years (certainly less than ten) of optimizing this, we will be walking a fine line between tools that are extraordinarily empowering on the one side and completely out of our control on the other. All we need for GPT to be dangerous is for it to get maybe 25% smarter and much faster. We should anticipate 50-100 faster than humans output within a few years. Within 5-10 years those systems will be extremely affordable and widely deployed. The danger starts to emerge because the the extreme performance difference between human and AI. You can't tell the AI to wait for you to make decisions, because while it is waiting a few hours for you to figure out what it's talking about, the equivalent of days worth of activity has been accomplished by your competitor's AI agent. For people to see the danger it could help to explain the exponential track record of computing efficiency performance and also create demonstrations of what it's like to compete with something that thinks 50 times faster than you. One would think that people could extrapolate from video games or something. But I guess the problem is the open world video game agents are still largely unintelligent so it's hard to imagine.