10 ms·

No, this is nothing like atomic bombs. In what scenario can you kill 100,000 people with with an LLM hooked up to a Python interpreter? The AI safety grift is o

by carom 3y ago

No, this is nothing like atomic bombs. In what scenario can you kill 100,000 people with with an LLM hooked up to a Python interpreter? The AI safety grift is out of control.

p-e-w 3y ago
> In what scenario can you kill 100,000 people with with an LLM hooked up to a Python interpreter? The article talks about LLMs being hooked up to the Internet. Big difference. If you really can't imagine a scenario where this might lead to people dying, you're not trying hard enough.
- carom 3y ago
  You have failed to provide a scenario, likely because no realistic scenario exists. Let's go through an example scenario and the hurdles you have to overcome for this to work. We'll use the "mayonnaise plant" example from a sibling comment. 1. The LLM needs to find an exploitable bug in a popular code base. 2. The LLM needs to write a reliable exploit for that bug. 3. The LLM needs to develop a worm that exploits that bug and spreads itself, opening access to the system. 4. The LLM needs to connect to systems and understand if they are of any significance (it found a mayonnaise plant!). 5. The LLM needs to understand the control protocols of their industrial control systems. 6. The LLM needs to understand how to make a dangerous composition from the ingredients it has on hand (let's pretend it can dump some industrial cleaning solution that is on standby for cleaning the tanks). 7. The LLM needs to assume such total control over this processing plant that it can disguise the traffic and not trigger a single alarm around malfunctions. What you're vaguely hinting at is extremely high skilled labor. There are a few billion dollar businesses in those steps. I welcome you to go read up on the challenges in automated exploit generation. LLMs are nowhere close. Now, you might rebut and say there are far simpler attacks, like phishing! Also an extremely hard problem. Try to send email in mass and not land in a spam filter, try to do the reconnaissance necessary to generate a believable login page. Try to leverage the sale guy's credentials to reach any system more meaningful than the company's Salesforce instance. So once again, I ask, please walk me through a situation where an LLM gets anywhere close to killing even 1% of the number of people an atomic bomb could.
  - p-e-w 3y ago
    > LLMs are nowhere close. We are nowhere close to even a rudimentary understanding of how current LLMs actually work. All we have are low-level building blocks, and lots of philosophizing about high-level output. Considering that, relying on some gut feeling about what LLMs "surely cannot possibly do" is reckless overconfidence. Not to mention that the next generation of LLMs, with potentially entirely new emergent properties, might be just around the corner, and the time to put safeguards into place is now, not when it's too late. As for the scenario, all an LLM with Internet access would need to do is find a single remotely exploitable vulnerability in the Linux network stack. That would allow it to literally shut down the entire Internet, which would kill a lot more people than a single atomic bomb.
    - c_crank 3y ago
      Shutting down the internet would not kill more people than an atomic bomb. What are you on?
      - p-e-w 3y ago
        Shutting down the Internet would stop most of global trade, cripple most government operations, and trigger an economic meltdown that would make the Great Depression look insignificant by comparison, not to mention unimaginable social chaos. Even essential services like hospitals and law enforcement would be drastically impacted. Yes, they use the Internet. For many critical tasks. I wouldn't be surprised if this led to the death of 10% of the global population in about 5 years. That's 800 million people, or around 4000 times the combined death toll of the Hiroshima and Nagasaki bombings. The Internet is the backbone of the modern world. Short of a global nuclear war, it's hard to imagine a more catastrophic event than it suddenly becoming unavailable.
        c_crank 3y ago
        The main use of the internet is quickly passing along information. Hospitals rely on the internet primarily to handle records and financial processing. Their ability to actually treat patients would still be functional without the internet. The loss of many of these records in the financial and governance would certainly lead to a lot of dollar losses. But dollars aren't people.
TeMPOraL 3y ago
> In what scenario can you kill 100,000 people with with an LLM hooked up to a Python interpreter? When that Python interpreter is running on the computer doing model-predictive control of your country's natural gas pipelines, or the plant that mixes your mayonnaise. Just so you know, the Python interpreter is already there in those places, and so is Internet access, because suits like their dashboards with real-time graphs. Remember, the S in IIoT (Industrial IoT) stands for both sanity and security :).
- throwuwu 3y ago
  Are you arguing that an LLM will be used in process control or statistical modelling? Or that someone will be running code in an interpreter that just happens to interact with those systems? Because there’s no way that someone will use one to control the automated processes in a factory. Those have to be perfectly repeatable. It’s not like someone showed up to the mayonnaise factory and says “I think I’ll increase the ratio of paprika to salt today, just for fun”
  - TeMPOraL 3y ago
    > Are you arguing that an LLM will be used in process control or statistical modelling? Or that someone will be running code in an interpreter that just happens to interact with those systems? Yes to both, sadly. > Because there’s no way that someone will use one to control the automated processes in a factory. What is that saying Americans are fond of? Ah, yes - if you believe that, then I have a bridge to sell you. > It’s not like someone showed up to the mayonnaise factory and says “I think I’ll increase the ratio of paprika to salt today, just for fun” No, they show up at the industry trade fair in some German city, find a group of people looking like they own factories, and start waxing poetic about how their company is using the newest advances in AI to synergize value delivery and carbon supply chain footprint management, ushering in the era of ecologically responsible and exponentially profitable Industry 4.1. You know, like ChatGPT but you plug it into your plant, feed it your BOM, and it prints out money. This kind of stuff.
bamboozled 3y ago
But that's not what this article is about, it's about attempting to create some type of AGI, highly experimental seemingly non-aligned, which is connected to...the internet. Which is why I do agree with others, even if this is a a weak attempt (I have no idea, not an AI researcher) if it was to be successful could be hyper destructive. The project seems really quite loose and the author seems to be lacking in awareness with regards to the consequences of such a project. I do think others are right in saying, we really should understand more bout where people can and will try to take these ideas before we keep putting models on the internet for consumption by just anyone.
- p-e-w 3y ago
  > seems to not have much awareness about what the consequences of such a project might mean That's the worst part of it. Is the current generation of LLMs dangerous? Probably not. Will the next generation of LLMs be dangerous? Maybe not, if we're lucky. The one after that? Nobody knows. But projects like this don't so much rely on there being no danger, but rather, the question seems to not even have been considered. And that's crazy, because the assumption that things are harmless is getting more and more shaky the more capable these models become.
  - TeMPOraL 3y ago
    I can kind of understand it. After over a decade of spending time in the influence zone of Bostrom, Eliezer, and LW crowd[0], I find myself torn. Half of me is acutely aware of the danger potential of what we're discussing. The other half wants to play with chaining the current LLMs, text-to-$X and $X-to-text models, and domain-specific classical software in hopes of bootstrapping a proto-AGI, because how cool it would be to pull it off. Good my ADHD makes me write HN comments on LLM threads instead of working on the latter, because I'm not sure I could stop myself otherwise. But this also means I can very much imagine only thinking about the cool parts, and not considering the dangerous ones - because that's how I feel about 90% of the time when thinking about current SOTA models. It's just too damn exciting. (Hell, it's the first thing in tech in the last decade, that I find exciting.) -- [0] - Whatever one may think of them, I find it sadly ironic how Eliezer and his disciples were considered pseudo-scientific doomsday AI cultists a decade ago, back when the whole thing was only a far-future concern, how they're still considered pseudo-scientific doomsday AI cultists today, after AI field made a series of rapid jumps, landing us exactly in the scope of things they were harping about all that time - and how I bet they'll still be considered pseudo-scientific doomsday AI cultists after someone unleashes an unaligned AGI and ends us.
HeartStrings 3y ago
"AI safety" is just folk who were too incapable to build LLMs themselves trying to be relevant. Of course, it’s all bull. If computer does bad, you just pull the plug.
- p-e-w 3y ago
  Sure. And if the "bad" program has spread as a virus, you just pull all the plugs of all the computers, right? Of course, you will kill modern civilization by doing so (and many millions of people as a result), but at least the AI is back in the box, so there is nothing to worry about.
  - HeartStrings 3y ago
    This will happen a few times and then everyone will learnto use backups and better security. If anything, it will make humanity more antifragile.
  - moffkalast 3y ago
    It's not like actual genius people are constantly trying to make the deadliest virus for various purposes and we've been hardening our systems for decades in order to not have them immediately taken over. You'd have to go beyond AGI to break those safeguards. > all the computers The average shit tier PC doesn't even have enough RAM to load an LLM capable of consistent output, much less to load it into VRAM for any kind of reasonable speed. This is like being concerned that Panda bears will become an invasive species and take over the world if we stop them from going extinct. Besides, all you have to do to pull the plug is disable any of the billion python dependencies it needs to download every time and it's stopped dead in its tracks. Something that's basically done every other day and needs to be continuously fixed by project maintainers, so it's more like a Panda on life support that dies if somebody stops pumping air into its breathing hose. It's all fragile af.
- ben_w 3y ago
  > Of course, it’s all bull. If computer does bad, you just pull the plug. Can you always tell if it's doing bad? Even if you can tell when it's doing bad, if there's an app which makes you $10k/hour while running, what risk of it killing somebody if left unsupervised will be low enough for you to leave it running unsupervised? Say, while you sleep? When there's a lot of money to be made, do you trust everyone else to (1) be as risk-adverse as you, and (2) not just convince themselves there's no risk using exactly the argument you just gave? Let's say you personally are willing to run it unsupervised if the risk is mean 1 fatality per 8*365*80 hours when unsupervised; are you willing for every human to have such a system running unsupervised? A tenth that risk?
lannisterstark 3y ago
I feel like all the blogg...uh, 'journalists' have just moved to "NO NOT THE AI" after whatever the last thing they were focusing on.
ben_w 3y ago
GPT-4 apparently does well on medical exams. Replace pharmacists to save money, don't need much of an error rate to get 100k dead. Have it design a nuclear reactor, saves money, you don't notice the flaw until it's too late. Chernobyl wasn't malicious, it was in part a cost-saving exercise; how many died from the fossil fuels that were burned because of the fear of nuclear power after the incident? How many died needlessly in Vietnam, before human journalists saw and reported on the human cost, and what happens when reporters are replaced with LLMs — to save money — that physically cannot see things for themselves? That said, 100k is in the awkward range that political decisions regularly lead to — at most a single month's life expectancy in a nation of 100 million — so if I'd been you I would have asked about 1M+. At a mere hypothetical 100k dead, it may easily still be part of a larger scenario which saves lives overall, much like how the actual atomic bombs were an alternative to a protracted conventional war (and given the attempted coup to prevent surrender after the second, I consider the hypothetical used to justify the bomb to at least be plausible, though I am no historian and don't want to give the impression of certainty).
- bbor 3y ago
  Lol just gotta say I love these examples, very creative thinking and not at all the typical AI alignment concerns, which revolve around Skynet and paper clips. Consequentialism can get real nasty!
- throwuwu 3y ago
  That’s a pretty weak argument since it can be applied to anything. eg there is a mistake in this process and people died as a result
  - ben_w 3y ago
    That's basically why I said I'd have put the threshold number much higher than a "mere" 100k. But even then I would say that the point is still important: disregard singularity, foom, and paperclipping, and AI is still often described as being akin to a faster rerun of the industrial revolution. Just having all the turmoil and unexpected consequences — health, social, environmental, economic, military, political — of the industrial revolution squeezed down from "economy doubling every 30 years" (which is what we saw) to "economy doubling every 18 months" (Moore's Law), is easily going to cause catastrophic consequences all by itself, even with no malice. "Health and safety" rules had to be created because too many bosses thought it was just common sense to not put your hands into dangerous equipment while in motion, while paying people to do jobs that could only be accomplished by doing exactly that; how can we do more than guess what the equivalent to that is for AI before we see it? What is to AI, that which global warming is to coal power? What's the necessary consequence of everyone using AI, which is analogous to the necessity of building a sewage system in response to widespread installation of flushing toilets in cities? We organised ourselves to build the latter, but we still haven't globally dealt with the former.
bbotond 3y ago
Watch a couple of videos by this guy: https://www.youtube.com/@RobertMilesAI https://www.youtube.com/@RobertMilesAI (very entertaining and informative by the way) These videos changed my mind entirely about AI risk.
cyberneticist 3y ago
It is a stupid argument too because it devalues their own position. I listened to about 6 hours of Marc Andreessen on this topic this weekend and he just smashes all the doomer arguments. Even in the context of nukes he made a great point that the invention of nukes most likely caused WW3 to not happen. An all out, devastating war between the US and Soviets over Europe that was only averted and transformed to the cold war because of MAD. I have been thinking since then how nukes probably caused my father to not fight in WW3. Who knows how many of us reading this would never have existed if nukes hadn't been invented. You can't make the argument 'the Manhattan project, we blow the planet up, the end". He also makes the point that you can't be worried about a super intelligence that is super intelligent in every way besides being so dumb that it turns everyone into paper clips. You can't have it both ways.
- mitthrowaway2 3y ago
  > being so dumb that it turns everyone into paper clips If that's your goal, what's dumb about achieving it?
- cwilkes 3y ago
  > Who knows how many of us reading this would never have existed if nukes hadn't been invented. You can't make the argument 'the Manhattan project, we blow the planet up, the end". I don’t think that argument against anti-LLM uses works as the response would be “nukes weren’t launched as we had humans in the loop. Without a stopgap like that maybe a nuke would launch if there was a 50.001% chance of our side winning” The counter counter argument to that would be that there isn’t enough weighing to how terrible a nuclear war would be and then maybe the LLM would change its mind. But who knows.
DontchaKnowit 3y ago
okay imagine that someone deploys hundreds of thousands of these LLM agents and sets the to work at finding vulnerable servers and searching for exploits. I don't think LLMs are quite up to snuff for a task like this, but they aren't far off. I can see a situation where food supply, electrical supply, water supply, etc is disrupted as a result of this. Its no atomic bomb, but it could be pretty damn serious.
- throwuwu 3y ago
  Automated pen testing is already a thing. The same technology that improves it can also be used to harden systems against attack.

by carom 3y ago

No, this is nothing like atomic bombs. In what scenario can you kill 100,000 people with with an LLM hooked up to a Python interpreter? The AI safety grift is out of control.

p-e-w 3y ago

> In what scenario can you kill 100,000 people with with an LLM hooked up to a Python interpreter? The article talks about LLMs being hooked up to the Internet. Big difference. If you really can't imagine a scenario where this might lead to people dying, you're not trying hard enough.

carom 3y ago
You have failed to provide a scenario, likely because no realistic scenario exists. Let's go through an example scenario and the hurdles you have to overcome for this to work. We'll use the "mayonnaise plant" example from a sibling comment. 1. The LLM needs to find an exploitable bug in a popular code base. 2. The LLM needs to write a reliable exploit for that bug. 3. The LLM needs to develop a worm that exploits that bug and spreads itself, opening access to the system. 4. The LLM needs to connect to systems and understand if they are of any significance (it found a mayonnaise plant!). 5. The LLM needs to understand the control protocols of their industrial control systems. 6. The LLM needs to understand how to make a dangerous composition from the ingredients it has on hand (let's pretend it can dump some industrial cleaning solution that is on standby for cleaning the tanks). 7. The LLM needs to assume such total control over this processing plant that it can disguise the traffic and not trigger a single alarm around malfunctions. What you're vaguely hinting at is extremely high skilled labor. There are a few billion dollar businesses in those steps. I welcome you to go read up on the challenges in automated exploit generation. LLMs are nowhere close. Now, you might rebut and say there are far simpler attacks, like phishing! Also an extremely hard problem. Try to send email in mass and not land in a spam filter, try to do the reconnaissance necessary to generate a believable login page. Try to leverage the sale guy's credentials to reach any system more meaningful than the company's Salesforce instance. So once again, I ask, please walk me through a situation where an LLM gets anywhere close to killing even 1% of the number of people an atomic bomb could.
- p-e-w 3y ago
  > LLMs are nowhere close. We are nowhere close to even a rudimentary understanding of how current LLMs actually work. All we have are low-level building blocks, and lots of philosophizing about high-level output. Considering that, relying on some gut feeling about what LLMs "surely cannot possibly do" is reckless overconfidence. Not to mention that the next generation of LLMs, with potentially entirely new emergent properties, might be just around the corner, and the time to put safeguards into place is now, not when it's too late. As for the scenario, all an LLM with Internet access would need to do is find a single remotely exploitable vulnerability in the Linux network stack. That would allow it to literally shut down the entire Internet, which would kill a lot more people than a single atomic bomb.
  - c_crank 3y ago
    Shutting down the internet would not kill more people than an atomic bomb. What are you on?
    - p-e-w 3y ago
      Shutting down the Internet would stop most of global trade, cripple most government operations, and trigger an economic meltdown that would make the Great Depression look insignificant by comparison, not to mention unimaginable social chaos. Even essential services like hospitals and law enforcement would be drastically impacted. Yes, they use the Internet. For many critical tasks. I wouldn't be surprised if this led to the death of 10% of the global population in about 5 years. That's 800 million people, or around 4000 times the combined death toll of the Hiroshima and Nagasaki bombings. The Internet is the backbone of the modern world. Short of a global nuclear war, it's hard to imagine a more catastrophic event than it suddenly becoming unavailable.
      - c_crank 3y ago
        The main use of the internet is quickly passing along information. Hospitals rely on the internet primarily to handle records and financial processing. Their ability to actually treat patients would still be functional without the internet. The loss of many of these records in the financial and governance would certainly lead to a lot of dollar losses. But dollars aren't people.

TeMPOraL 3y ago

> In what scenario can you kill 100,000 people with with an LLM hooked up to a Python interpreter? When that Python interpreter is running on the computer doing model-predictive control of your country's natural gas pipelines, or the plant that mixes your mayonnaise. Just so you know, the Python interpreter is already there in those places, and so is Internet access, because suits like their dashboards with real-time graphs. Remember, the S in IIoT (Industrial IoT) stands for both sanity and security :).

throwuwu 3y ago
Are you arguing that an LLM will be used in process control or statistical modelling? Or that someone will be running code in an interpreter that just happens to interact with those systems? Because there’s no way that someone will use one to control the automated processes in a factory. Those have to be perfectly repeatable. It’s not like someone showed up to the mayonnaise factory and says “I think I’ll increase the ratio of paprika to salt today, just for fun”
- TeMPOraL 3y ago
  > Are you arguing that an LLM will be used in process control or statistical modelling? Or that someone will be running code in an interpreter that just happens to interact with those systems? Yes to both, sadly. > Because there’s no way that someone will use one to control the automated processes in a factory. What is that saying Americans are fond of? Ah, yes - if you believe that, then I have a bridge to sell you. > It’s not like someone showed up to the mayonnaise factory and says “I think I’ll increase the ratio of paprika to salt today, just for fun” No, they show up at the industry trade fair in some German city, find a group of people looking like they own factories, and start waxing poetic about how their company is using the newest advances in AI to synergize value delivery and carbon supply chain footprint management, ushering in the era of ecologically responsible and exponentially profitable Industry 4.1. You know, like ChatGPT but you plug it into your plant, feed it your BOM, and it prints out money. This kind of stuff.

bamboozled 3y ago

But that's not what this article is about, it's about attempting to create some type of AGI, highly experimental seemingly non-aligned, which is connected to...the internet. Which is why I do agree with others, even if this is a a weak attempt (I have no idea, not an AI researcher) if it was to be successful could be hyper destructive. The project seems really quite loose and the author seems to be lacking in awareness with regards to the consequences of such a project. I do think others are right in saying, we really should understand more bout where people can and will try to take these ideas before we keep putting models on the internet for consumption by just anyone.

p-e-w 3y ago
> seems to not have much awareness about what the consequences of such a project might mean That's the worst part of it. Is the current generation of LLMs dangerous? Probably not. Will the next generation of LLMs be dangerous? Maybe not, if we're lucky. The one after that? Nobody knows. But projects like this don't so much rely on there being no danger, but rather, the question seems to not even have been considered. And that's crazy, because the assumption that things are harmless is getting more and more shaky the more capable these models become.
- TeMPOraL 3y ago
  I can kind of understand it. After over a decade of spending time in the influence zone of Bostrom, Eliezer, and LW crowd[0], I find myself torn. Half of me is acutely aware of the danger potential of what we're discussing. The other half wants to play with chaining the current LLMs, text-to-$X and $X-to-text models, and domain-specific classical software in hopes of bootstrapping a proto-AGI, because how cool it would be to pull it off. Good my ADHD makes me write HN comments on LLM threads instead of working on the latter, because I'm not sure I could stop myself otherwise. But this also means I can very much imagine only thinking about the cool parts, and not considering the dangerous ones - because that's how I feel about 90% of the time when thinking about current SOTA models. It's just too damn exciting. (Hell, it's the first thing in tech in the last decade, that I find exciting.) -- [0] - Whatever one may think of them, I find it sadly ironic how Eliezer and his disciples were considered pseudo-scientific doomsday AI cultists a decade ago, back when the whole thing was only a far-future concern, how they're still considered pseudo-scientific doomsday AI cultists today, after AI field made a series of rapid jumps, landing us exactly in the scope of things they were harping about all that time - and how I bet they'll still be considered pseudo-scientific doomsday AI cultists after someone unleashes an unaligned AGI and ends us.

HeartStrings 3y ago

"AI safety" is just folk who were too incapable to build LLMs themselves trying to be relevant. Of course, it’s all bull. If computer does bad, you just pull the plug.

p-e-w 3y ago
Sure. And if the "bad" program has spread as a virus, you just pull all the plugs of all the computers, right? Of course, you will kill modern civilization by doing so (and many millions of people as a result), but at least the AI is back in the box, so there is nothing to worry about.
- HeartStrings 3y ago
  This will happen a few times and then everyone will learnto use backups and better security. If anything, it will make humanity more antifragile.
- moffkalast 3y ago
  It's not like actual genius people are constantly trying to make the deadliest virus for various purposes and we've been hardening our systems for decades in order to not have them immediately taken over. You'd have to go beyond AGI to break those safeguards. > all the computers The average shit tier PC doesn't even have enough RAM to load an LLM capable of consistent output, much less to load it into VRAM for any kind of reasonable speed. This is like being concerned that Panda bears will become an invasive species and take over the world if we stop them from going extinct. Besides, all you have to do to pull the plug is disable any of the billion python dependencies it needs to download every time and it's stopped dead in its tracks. Something that's basically done every other day and needs to be continuously fixed by project maintainers, so it's more like a Panda on life support that dies if somebody stops pumping air into its breathing hose. It's all fragile af.
ben_w 3y ago
> Of course, it’s all bull. If computer does bad, you just pull the plug. Can you always tell if it's doing bad? Even if you can tell when it's doing bad, if there's an app which makes you $10k/hour while running, what risk of it killing somebody if left unsupervised will be low enough for you to leave it running unsupervised? Say, while you sleep? When there's a lot of money to be made, do you trust everyone else to (1) be as risk-adverse as you, and (2) not just convince themselves there's no risk using exactly the argument you just gave? Let's say you personally are willing to run it unsupervised if the risk is mean 1 fatality per 8*365*80 hours when unsupervised; are you willing for every human to have such a system running unsupervised? A tenth that risk?

lannisterstark 3y ago

I feel like all the blogg...uh, 'journalists' have just moved to "NO NOT THE AI" after whatever the last thing they were focusing on.

ben_w 3y ago

GPT-4 apparently does well on medical exams. Replace pharmacists to save money, don't need much of an error rate to get 100k dead. Have it design a nuclear reactor, saves money, you don't notice the flaw until it's too late. Chernobyl wasn't malicious, it was in part a cost-saving exercise; how many died from the fossil fuels that were burned because of the fear of nuclear power after the incident? How many died needlessly in Vietnam, before human journalists saw and reported on the human cost, and what happens when reporters are replaced with LLMs — to save money — that physically cannot see things for themselves? That said, 100k is in the awkward range that political decisions regularly lead to — at most a single month's life expectancy in a nation of 100 million — so if I'd been you I would have asked about 1M+. At a mere hypothetical 100k dead, it may easily still be part of a larger scenario which saves lives overall, much like how the actual atomic bombs were an alternative to a protracted conventional war (and given the attempted coup to prevent surrender after the second, I consider the hypothetical used to justify the bomb to at least be plausible, though I am no historian and don't want to give the impression of certainty).

bbor 3y ago
Lol just gotta say I love these examples, very creative thinking and not at all the typical AI alignment concerns, which revolve around Skynet and paper clips. Consequentialism can get real nasty!
throwuwu 3y ago
That’s a pretty weak argument since it can be applied to anything. eg there is a mistake in this process and people died as a result
- ben_w 3y ago
  That's basically why I said I'd have put the threshold number much higher than a "mere" 100k. But even then I would say that the point is still important: disregard singularity, foom, and paperclipping, and AI is still often described as being akin to a faster rerun of the industrial revolution. Just having all the turmoil and unexpected consequences — health, social, environmental, economic, military, political — of the industrial revolution squeezed down from "economy doubling every 30 years" (which is what we saw) to "economy doubling every 18 months" (Moore's Law), is easily going to cause catastrophic consequences all by itself, even with no malice. "Health and safety" rules had to be created because too many bosses thought it was just common sense to not put your hands into dangerous equipment while in motion, while paying people to do jobs that could only be accomplished by doing exactly that; how can we do more than guess what the equivalent to that is for AI before we see it? What is to AI, that which global warming is to coal power? What's the necessary consequence of everyone using AI, which is analogous to the necessity of building a sewage system in response to widespread installation of flushing toilets in cities? We organised ourselves to build the latter, but we still haven't globally dealt with the former.

bbotond 3y ago

Watch a couple of videos by this guy: https://www.youtube.com/@RobertMilesAI https://www.youtube.com/@RobertMilesAI (very entertaining and informative by the way) These videos changed my mind entirely about AI risk.

cyberneticist 3y ago

It is a stupid argument too because it devalues their own position. I listened to about 6 hours of Marc Andreessen on this topic this weekend and he just smashes all the doomer arguments. Even in the context of nukes he made a great point that the invention of nukes most likely caused WW3 to not happen. An all out, devastating war between the US and Soviets over Europe that was only averted and transformed to the cold war because of MAD. I have been thinking since then how nukes probably caused my father to not fight in WW3. Who knows how many of us reading this would never have existed if nukes hadn't been invented. You can't make the argument 'the Manhattan project, we blow the planet up, the end". He also makes the point that you can't be worried about a super intelligence that is super intelligent in every way besides being so dumb that it turns everyone into paper clips. You can't have it both ways.

mitthrowaway2 3y ago
> being so dumb that it turns everyone into paper clips If that's your goal, what's dumb about achieving it?
cwilkes 3y ago
> Who knows how many of us reading this would never have existed if nukes hadn't been invented. You can't make the argument 'the Manhattan project, we blow the planet up, the end". I don’t think that argument against anti-LLM uses works as the response would be “nukes weren’t launched as we had humans in the loop. Without a stopgap like that maybe a nuke would launch if there was a 50.001% chance of our side winning” The counter counter argument to that would be that there isn’t enough weighing to how terrible a nuclear war would be and then maybe the LLM would change its mind. But who knows.

DontchaKnowit 3y ago

okay imagine that someone deploys hundreds of thousands of these LLM agents and sets the to work at finding vulnerable servers and searching for exploits. I don't think LLMs are quite up to snuff for a task like this, but they aren't far off. I can see a situation where food supply, electrical supply, water supply, etc is disrupted as a result of this. Its no atomic bomb, but it could be pretty damn serious.

throwuwu 3y ago
Automated pen testing is already a thing. The same technology that improves it can also be used to harden systems against attack.