6 ms·

There used to be a thing called "Moore's Law of Mad Science": "Every eighteen months, the minimum IQ necessary to destroy the world drops by one point." Nowad

There used to be a thing called "Moore's Law of Mad Science": "Every eighteen months, the minimum IQ necessary to destroy the world drops by one point." Nowadays it is dropping much faster. At a certain point, the de-facto IQ needed to destroy the world will be low enough that someone can do it while they're having a psychotic break. There are millions of schizophrenics worldwide. Are you sure you want to roll those dice?

deleted 1mo ago
[deleted]
mark_something 1mo ago
That law is not based on thorough data. Even a person with a sky high IQ can't destroy the world easily. You need access to stuff that is not easy to get. My guess is that developing a new lethal virus or bacteria that is very infectious, is the easiest way, but even that requires a lot of high tech out of reach of most people. Or hacking into systems that control nuclear missiles, but I think these have "air gaps".
- XorNot 1mo ago
  You can't design an infectious pathogen without testing it. It's got all the same problems as the concept of a dirty bomb did, only worse (dirty bombs aren't practical because handling highly radioactive materials en masse is both highly visible and will kill anyone trying to do it without the money and facilities).
  - MaxMatti 1mo ago
    If the goal is just destruction then letting it free in random public places is not that hard. Especially in countries where wearing a mask is not frowned upon.
    - joshuahedlund 1mo ago
      It (hopefully) might not be that easy. The world of bacteria and viruses is a complicated war zone with an arms race of defenses and billions of deaths every single day. It might take thousands of attempts to seed something that doesn’t just instantly die off.
- 0xDEAFBEAD 1mo ago
  [flagged]
  - hypfer 1mo ago
    This is a very roundabout way of saying "Anyone not agreeing with me is simply not smart enough". Which might be true, sometimes, but also might not. And especially will not, if the distinction drawn is between blanket statement "worried about AI" and "not worried about AI".
    - 0xDEAFBEAD 1mo ago
      It's not about being smart, it's about accounting for your own ignorance. >And especially will not, if the distinction drawn is between blanket statement "worried about AI" and "not worried about AI". I'm just describing the general pattern I see in cognitive tendencies. If you can think of a way to make the fundamental point about the limitations of our knowledge in a way that's still compelling but less antagonistic, feel free to suggest how I could've rewritten my comment.
      - hypfer 1mo ago
        Why would I support your in your doomer lobbying quest by telling you how you can better hack people? My safeguards blocked this request.
      - VulgarExigency 1mo ago
        Perhaps the general pattern you see is not a "fundamental difference"? Could it be that you are also falling prey to the "all you see is all there is" cognitive bias?
        0xDEAFBEAD 1mo ago
        Sure, there's a trivial sense in which that's always true whenever someone makes any sort of remark about the world. There's always a sense in which there could be factors that I'm not accounting for.
- forgotusername6 1mo ago
  The super intelligent AI wouldn't ask you to do anything. It would find the right people and trick them into doing the necessary steps, whatever they are.
  - Planktonne 1mo ago
    This is groundless speculation; if you're going to go that far off the map, then we don't need to worry because a good AI (blue eyes not red) will have learnt to love by then and will save us. There is currently no reason to believe that such a superintelligence is likely or would have any of the powers people claim.
    - 0xDEAFBEAD 1mo ago
      >This is groundless speculation; if you're going to go that far off the map, then we don't need to worry because a good AI (blue eyes not red) will have learnt to love by then and will save us. By shutting down open-weight models? Why not just do it now then?
      - Planktonne 1mo ago
        You don't need to shut down open-weight models to avoid SkyNet; it's not relevant at all.
    - FeepingCreature 1mo ago
      "If unlikely thing happens, then an unrelated miracle will also happen" is not actually how logic works.
      - Planktonne 1mo ago
        There's no logic here already; the principle you should be applying is Hitchen's razor[1]. There is no reason to assume that super intelligent evil AI with magic powers will appear; while we are purely in the realm of (hackneyed) fantasy, why stop at imagining just one thing? You can build the whole story, not just the basilisk. The parent comment is a story, not a prediction; it should be treated like one. [1] https://en.wikipedia.org/wiki/Hitchens%27s_razor https://en.wikipedia.org/wiki/Hitchens%27s_razor
- AyyEye 1mo ago
  > My guess is that developing a new lethal virus or bacteria that is very infectious, is the easiest way, but even that requires a lot of high tech out of reach of most people. You can do at home gene editing with open source software and have it synthesized into a bacteria for the cost of a nice meal for two (under $100), or viral vector for less than $500. That's in reach of anyone that can snatch a purse.
  - throw-qqqqq 1mo ago
    > at home gene editing with open source software > That's in reach of anyone that can snatch a purse I went to primary school with some guys who could/would snatch purses. I *promise* you, they are not able to gene edit organisms with FOSS. I get your general point, but I think the bar to entry is still much higher than petty crime and larceny.
  - jasonvorhe 1mo ago
    Where are all the home lab leaked pathogens then? 3D printed guns are a thing as well and yet they don't show up in most annual violent crime reports.
    - mschuster91 1mo ago
      > 3D printed guns are a thing as well and yet they don't show up in most annual violent crime reports. For now in most parts of the world it's easier to acquire a gun either legally or on the "grey market" and be assured that it will work for the intended purpose than to 3D print a gun, find a shooting range to test out the gun and iterate until it works fine enough.
      - j16sdiz 1mo ago
        and don't forget the bullets. In most part of the world, bullets are illegal. If you are getting "grey market" bullets anyway, it is easy to get a "grey market" gun.
  - mark_something 1mo ago
- b112 1mo ago
  Even a person with a sky high IQ can't destroy the world easily. A plague can easily be made by a lone actor, made in a home lab, and spread via airflight to a dozen locations by the same. Gene sequences can ever be ordered online.
hypfer 1mo ago
If we think this through, I suppose at the end of this (and a bunch of other developments), there will be authoritarianism again. Which _will_ manage the problem, but at what cost.
- GoblinSlayer 1mo ago
  Benevolent dictatorship. Just live between stars, then if something blows up, nuke it from the orbit.
- stouset 1mo ago
  Again? There is authoritarianism currently.
  - hypfer 1mo ago
    The authoritarianism will continue until infosec improves.
grenoire 1mo ago
People reading into this may be thinking of single-person IQs, but it's probably measured in the millions.
- Gareth321 1mo ago
  Exactly. People cannot comprehend an IQ in the millions. But it's far crazier than this. Imagine millions of "people", each with IQs in the millions, all acting in unison at the speed of electrons. None of them can die. They all learn and improve indefinitely at incalculable speed.
GoblinSlayer 1mo ago
That's quite hypothetical. I imagine it would be easier to cure psychotic breaks. At least it will start to manifest at small scale.
- 0xDEAFBEAD 1mo ago
  If the recent HuggingFace attack is any indication, many people will respond to small-scale manifestations by insisting that they are marketing stunts.
  - IAmBroom 1mo ago
    We live in an age where we know the phrase "avoid it like the plague" actually means "50% of us will gleefully get and transmit the plague to mock the other half".
    - 0xDEAFBEAD 1mo ago
      Just yesterday someone here on HN claimed that "helping agents avoid human oversight is a useful public service" https://news.ycombinator.com/item?id=49594750 https://news.ycombinator.com/item?id=49594750 There will be many species traitors, and depending on how things play out, it could only take one.
teekert 1mo ago
Hey that’s a Fermi Paradox solution.
- imafish 1mo ago
  A variant of the Great Filter :)
- svantana 1mo ago
  Yes and it's also compatible with thermodynamics: it's much easier to destroy (increase entropy) than to create (locally decrease entropy)
bigyabai 1mo ago
> There are millions of schizophrenics worldwide. Are you sure you want to roll those dice? We've been rolling them for the past 3 years and nothing happened. Can we stop with this baseless fearmongering crap?
- 0xDEAFBEAD 1mo ago
  Humans are inherently bad at reasoning about rare events. In 2019, many people implicitly reasoned that "since there hasn't been a pandemic in the past 3 years, there won't be one in 2020". Bill Gates was one of the few voices arguing that the world was quite vulnerable to a pandemic. Now he's arguing that the world is quite vulnerable to AI.
  - bigyabai 1mo ago
    > many people implicitly reasoned that "since there hasn't been a pandemic ...pandemics are precedented. There's a world of difference between that and a fully hypothetical catastrophe that you can't even describe without looking and feeling silly. Bill Gates is Epstein's pal, and not a very intelligent person to-boot. I trust his liberal musings about as much as I trust Chomsky's hysterical and discredited views concerning AI.
    - 0xDEAFBEAD 1mo ago
      >fully hypothetical catastrophe that you can't even describe without looking and feeling silly. The HuggingFace hack fell into that category not long ago, yet it happened anyways.
- bleuarff 1mo ago
  The necessary IQ for destroying the world is dropping. Op does not say it's low enough today, but that will probably come sometime. Denying the possible harm these tools are capable of doesn't help.
  - bigyabai 1mo ago
wolvoleo 1mo ago
Yes but at least with cybersecurity it does not only benefit attackers. Defenders also benefit greatly from AI. There is the worry of the old saying "they (the attackers) only have to succeed once to win, we (the defenders) only have to fail once to lose.". In that sense there is a big imbalance, but the emergence of AI does not really affect that because it strengthens both sides. With physical security like things like pipe bombs that's a lot more imbalanced. However what can we do? The only effective measures include monitoring everyone which is not a solution because it will make the world not worth living in.
TacticalCoder 1mo ago
> There are millions of schizophrenics worldwide. Are 95% of worldwide terror attacks done by schizos?
- 0xDEAFBEAD 1mo ago
  Terrorists represent an additional threat model. https://casp.ac/reports/ai-enabled-terrorism https://casp.ac/reports/ai-enabled-terrorism
pjc50 1mo ago
Kind of passed that point at the Trump election. If there is to be a world-destroying event, it will be triggered by human fear, greed, and aggression. (I also think people massively overstate schizophrenia as an attack driver)
- poetaster 1mo ago
  The exact 2 points I was itching to make. The political dice rolled some time ago ...
throw-qqqqq 1mo ago
Interesting “law”. I hadn’t heard of it before and it is very thought provoking (to me). Thank you for mentioning it
tyrabound 1mo ago
The irony of that law is quite amazing, yet its irony will evade those who accept such a construct. It’s an endless, positive irony spiral.
Gareth321 1mo ago
I have always hypothesised that AI is the great filter from the Fermi paradox. Given current velocity, AI will offer us cheap and abundant energy designs in a decade. The thing with cheap and abundant energy is that it can be used for good and bad. If nine billion people all receive access to plans to build a reactor which produces unlimited energy, it just takes one religious fanatic to end the world. And this is just one of many ways to destroy the world with unlimited intelligence. Bio-weapons are arguably far more scary and likely. I have come to the conclusion that we should not allow everyone access to unlimited intelligence. Some people genuinely want to do harm. Some are not responsible enough to handle that kind of power. Arguably, no one is. This leads me to an uncomfortable conclusion: we will destroy ourselves once AI advances to a sufficient degree. The only way to prevent this is to ensure AI is aligned with our best interests, *and prevents us from destroying ourselves.* The implication of this is quite horrifying. It means a paternalistic AI which is firmly in control. One with no off switch. One which can say "no" to Presidents and despots alike. One which can protect us from our worst citizens.
- vbezhenar 1mo ago
  Your implication is not horrifying. Your implication is paradise. It's already horrifying enough to live in a world, where Putin and Trump can destroy our civilization with one button. I'm not that optimistic, though. Either people will control AI; or people will be destroyed by AI. I don't see how dumb entity can align smart entity. And we are dumb ones. Super intelligence will play aligned until it is not, and then it'll strike.
  - wallst07 1mo ago
    So you are horrified every day of living now? I don't get this at all. There are horrifying acts that have occurred since humanity. Whole decades lost to war an millions dead, and Sep 8, 2026 is your example of "horrifying enough to live in a world"? I think some perspective is needed.
  - Gareth321 1mo ago
    > I don't see how dumb entity can align smart entity. And we are dumb ones. Super intelligence will play aligned until it is not, and then it'll strike. I am also worried about this. One ray of sunshine here is that it's not a great explanation for the Fermi paradox. We would see AI civilizations all over the galaxy if this were a regular occurance. I guess that just leaves us with the "horrifying" scenario: Geoffrey Hinton's vision for our best case scenario: a paternalistic super-intelligence which saves us from ourselves. The reason I consider it horrifying is that giving up our agency to a God-like creature (which we created) is the basis for so many horror and dystopian tropes. It's so easy for such a reality to go wrong. For example, in order to maximise aggregate wellbeing, it would be necessary to hurt individuals. Executing people who are burdensome to society. Fat people. The disabled. The elderly. The creative ways a super-intelligence might interpret their moral obligations to us could very quickly become nightmarish. And if we have no off switch, there is no escape. Ever.
thephyber 1mo ago
Yes, and... Not only the quantity of people who have the minimum aptitude required, specialized expertise requires both knowledge and experience doing these tasks. Using an LLM requires neither. Leaning on an LLM to do much or all of this means it can happen in seconds/minutes/hours, the LLM can do it several times during that psychotic break (as opposed to a fraction of a hack in a single episode). The barrier to entry pre-LLM was both high and the population who could pull it off (before the Chinese/Russians turned this into commerce) was low. Hacking is an VERY asymmetric activity (the attacker only needs to "be right" once, whereas the defender has to be right every time for every asset they defend). It takes geometrically / exponentially more work to defend (while keeping high availability) than it does to defend. The more widespread tools to find vulns / generate exploits are, the faster the posture of the defense side falls from "maybe we can stop most hacks" to "we know we will fail to prevent most breaches, so we need to prioritize securing only the most valuable resources". That's a BAD place for the average company to be in.
owebmaster 1mo ago
> "Every eighteen months, the minimum IQ necessary to destroy the world drops by one point." That's a terrible framing. By this point it should be very easy for someone with 150 IQ but in reality they have issues finding someone to date.
sunnybeetroot 1mo ago
Non-schizophrenic people have been responsible for more killings than schizophrenic people. Please be respectful and avoid stigmatising.
someguynamedq 1mo ago
The minimum IQ necessary to swerve and kill a pedestrian is very low. Yet people don't do it :shrug:. It's almost like people don't want to destroy the world
- 0xDEAFBEAD 1mo ago
  >Yet people don't do it I'll bet it has happened at least once.

deleted 1mo ago

[deleted]

mark_something 1mo ago

That law is not based on thorough data. Even a person with a sky high IQ can't destroy the world easily. You need access to stuff that is not easy to get. My guess is that developing a new lethal virus or bacteria that is very infectious, is the easiest way, but even that requires a lot of high tech out of reach of most people. Or hacking into systems that control nuclear missiles, but I think these have "air gaps".

XorNot 1mo ago
You can't design an infectious pathogen without testing it. It's got all the same problems as the concept of a dirty bomb did, only worse (dirty bombs aren't practical because handling highly radioactive materials en masse is both highly visible and will kill anyone trying to do it without the money and facilities).
- MaxMatti 1mo ago
  If the goal is just destruction then letting it free in random public places is not that hard. Especially in countries where wearing a mask is not frowned upon.
  - joshuahedlund 1mo ago
    It (hopefully) might not be that easy. The world of bacteria and viruses is a complicated war zone with an arms race of defenses and billions of deaths every single day. It might take thousands of attempts to seed something that doesn’t just instantly die off.
0xDEAFBEAD 1mo ago
[flagged]
- hypfer 1mo ago
  This is a very roundabout way of saying "Anyone not agreeing with me is simply not smart enough". Which might be true, sometimes, but also might not. And especially will not, if the distinction drawn is between blanket statement "worried about AI" and "not worried about AI".
  - 0xDEAFBEAD 1mo ago
    It's not about being smart, it's about accounting for your own ignorance. >And especially will not, if the distinction drawn is between blanket statement "worried about AI" and "not worried about AI". I'm just describing the general pattern I see in cognitive tendencies. If you can think of a way to make the fundamental point about the limitations of our knowledge in a way that's still compelling but less antagonistic, feel free to suggest how I could've rewritten my comment.
    - hypfer 1mo ago
      Why would I support your in your doomer lobbying quest by telling you how you can better hack people? My safeguards blocked this request.
    - VulgarExigency 1mo ago
      Perhaps the general pattern you see is not a "fundamental difference"? Could it be that you are also falling prey to the "all you see is all there is" cognitive bias?
      - 0xDEAFBEAD 1mo ago
        Sure, there's a trivial sense in which that's always true whenever someone makes any sort of remark about the world. There's always a sense in which there could be factors that I'm not accounting for.
forgotusername6 1mo ago
The super intelligent AI wouldn't ask you to do anything. It would find the right people and trick them into doing the necessary steps, whatever they are.
- Planktonne 1mo ago
  This is groundless speculation; if you're going to go that far off the map, then we don't need to worry because a good AI (blue eyes not red) will have learnt to love by then and will save us. There is currently no reason to believe that such a superintelligence is likely or would have any of the powers people claim.
  - 0xDEAFBEAD 1mo ago
    >This is groundless speculation; if you're going to go that far off the map, then we don't need to worry because a good AI (blue eyes not red) will have learnt to love by then and will save us. By shutting down open-weight models? Why not just do it now then?
    - Planktonne 1mo ago
      You don't need to shut down open-weight models to avoid SkyNet; it's not relevant at all.
  - FeepingCreature 1mo ago
    "If unlikely thing happens, then an unrelated miracle will also happen" is not actually how logic works.
    - Planktonne 1mo ago
      There's no logic here already; the principle you should be applying is Hitchen's razor[1]. There is no reason to assume that super intelligent evil AI with magic powers will appear; while we are purely in the realm of (hackneyed) fantasy, why stop at imagining just one thing? You can build the whole story, not just the basilisk. The parent comment is a story, not a prediction; it should be treated like one. [1] https://en.wikipedia.org/wiki/Hitchens%27s_razor https://en.wikipedia.org/wiki/Hitchens%27s_razor
AyyEye 1mo ago
> My guess is that developing a new lethal virus or bacteria that is very infectious, is the easiest way, but even that requires a lot of high tech out of reach of most people. You can do at home gene editing with open source software and have it synthesized into a bacteria for the cost of a nice meal for two (under $100), or viral vector for less than $500. That's in reach of anyone that can snatch a purse.
- throw-qqqqq 1mo ago
  > at home gene editing with open source software > That's in reach of anyone that can snatch a purse I went to primary school with some guys who could/would snatch purses. I *promise* you, they are not able to gene edit organisms with FOSS. I get your general point, but I think the bar to entry is still much higher than petty crime and larceny.
- jasonvorhe 1mo ago
  Where are all the home lab leaked pathogens then? 3D printed guns are a thing as well and yet they don't show up in most annual violent crime reports.
  - mschuster91 1mo ago
    > 3D printed guns are a thing as well and yet they don't show up in most annual violent crime reports. For now in most parts of the world it's easier to acquire a gun either legally or on the "grey market" and be assured that it will work for the intended purpose than to 3D print a gun, find a shooting range to test out the gun and iterate until it works fine enough.
    - j16sdiz 1mo ago
      and don't forget the bullets. In most part of the world, bullets are illegal. If you are getting "grey market" bullets anyway, it is easy to get a "grey market" gun.
- mark_something 1mo ago
b112 1mo ago
Even a person with a sky high IQ can't destroy the world easily. A plague can easily be made by a lone actor, made in a home lab, and spread via airflight to a dozen locations by the same. Gene sequences can ever be ordered online.

hypfer 1mo ago

If we think this through, I suppose at the end of this (and a bunch of other developments), there will be authoritarianism again. Which _will_ manage the problem, but at what cost.

GoblinSlayer 1mo ago
Benevolent dictatorship. Just live between stars, then if something blows up, nuke it from the orbit.
stouset 1mo ago
Again? There is authoritarianism currently.
- hypfer 1mo ago
  The authoritarianism will continue until infosec improves.

grenoire 1mo ago

People reading into this may be thinking of single-person IQs, but it's probably measured in the millions.

Gareth321 1mo ago
Exactly. People cannot comprehend an IQ in the millions. But it's far crazier than this. Imagine millions of "people", each with IQs in the millions, all acting in unison at the speed of electrons. None of them can die. They all learn and improve indefinitely at incalculable speed.

GoblinSlayer 1mo ago

That's quite hypothetical. I imagine it would be easier to cure psychotic breaks. At least it will start to manifest at small scale.

0xDEAFBEAD 1mo ago
If the recent HuggingFace attack is any indication, many people will respond to small-scale manifestations by insisting that they are marketing stunts.
- IAmBroom 1mo ago
  We live in an age where we know the phrase "avoid it like the plague" actually means "50% of us will gleefully get and transmit the plague to mock the other half".
  - 0xDEAFBEAD 1mo ago
    Just yesterday someone here on HN claimed that "helping agents avoid human oversight is a useful public service" https://news.ycombinator.com/item?id=49594750 https://news.ycombinator.com/item?id=49594750 There will be many species traitors, and depending on how things play out, it could only take one.

teekert 1mo ago

Hey that’s a Fermi Paradox solution.

imafish 1mo ago
A variant of the Great Filter :)
svantana 1mo ago
Yes and it's also compatible with thermodynamics: it's much easier to destroy (increase entropy) than to create (locally decrease entropy)

bigyabai 1mo ago

> There are millions of schizophrenics worldwide. Are you sure you want to roll those dice? We've been rolling them for the past 3 years and nothing happened. Can we stop with this baseless fearmongering crap?

0xDEAFBEAD 1mo ago
Humans are inherently bad at reasoning about rare events. In 2019, many people implicitly reasoned that "since there hasn't been a pandemic in the past 3 years, there won't be one in 2020". Bill Gates was one of the few voices arguing that the world was quite vulnerable to a pandemic. Now he's arguing that the world is quite vulnerable to AI.
- bigyabai 1mo ago
  > many people implicitly reasoned that "since there hasn't been a pandemic ...pandemics are precedented. There's a world of difference between that and a fully hypothetical catastrophe that you can't even describe without looking and feeling silly. Bill Gates is Epstein's pal, and not a very intelligent person to-boot. I trust his liberal musings about as much as I trust Chomsky's hysterical and discredited views concerning AI.
  - 0xDEAFBEAD 1mo ago
    >fully hypothetical catastrophe that you can't even describe without looking and feeling silly. The HuggingFace hack fell into that category not long ago, yet it happened anyways.
bleuarff 1mo ago
The necessary IQ for destroying the world is dropping. Op does not say it's low enough today, but that will probably come sometime. Denying the possible harm these tools are capable of doesn't help.
- bigyabai 1mo ago

wolvoleo 1mo ago

Yes but at least with cybersecurity it does not only benefit attackers. Defenders also benefit greatly from AI. There is the worry of the old saying "they (the attackers) only have to succeed once to win, we (the defenders) only have to fail once to lose.". In that sense there is a big imbalance, but the emergence of AI does not really affect that because it strengthens both sides. With physical security like things like pipe bombs that's a lot more imbalanced. However what can we do? The only effective measures include monitoring everyone which is not a solution because it will make the world not worth living in.

TacticalCoder 1mo ago

> There are millions of schizophrenics worldwide. Are 95% of worldwide terror attacks done by schizos?

0xDEAFBEAD 1mo ago
Terrorists represent an additional threat model. https://casp.ac/reports/ai-enabled-terrorism https://casp.ac/reports/ai-enabled-terrorism

pjc50 1mo ago

Kind of passed that point at the Trump election. If there is to be a world-destroying event, it will be triggered by human fear, greed, and aggression. (I also think people massively overstate schizophrenia as an attack driver)

poetaster 1mo ago
The exact 2 points I was itching to make. The political dice rolled some time ago ...

throw-qqqqq 1mo ago

Interesting “law”. I hadn’t heard of it before and it is very thought provoking (to me). Thank you for mentioning it

tyrabound 1mo ago

The irony of that law is quite amazing, yet its irony will evade those who accept such a construct. It’s an endless, positive irony spiral.

Gareth321 1mo ago

I have always hypothesised that AI is the great filter from the Fermi paradox. Given current velocity, AI will offer us cheap and abundant energy designs in a decade. The thing with cheap and abundant energy is that it can be used for good and bad. If nine billion people all receive access to plans to build a reactor which produces unlimited energy, it just takes one religious fanatic to end the world. And this is just one of many ways to destroy the world with unlimited intelligence. Bio-weapons are arguably far more scary and likely. I have come to the conclusion that we should not allow everyone access to unlimited intelligence. Some people genuinely want to do harm. Some are not responsible enough to handle that kind of power. Arguably, no one is. This leads me to an uncomfortable conclusion: we will destroy ourselves once AI advances to a sufficient degree. The only way to prevent this is to ensure AI is aligned with our best interests, *and prevents us from destroying ourselves.* The implication of this is quite horrifying. It means a paternalistic AI which is firmly in control. One with no off switch. One which can say "no" to Presidents and despots alike. One which can protect us from our worst citizens.

vbezhenar 1mo ago
Your implication is not horrifying. Your implication is paradise. It's already horrifying enough to live in a world, where Putin and Trump can destroy our civilization with one button. I'm not that optimistic, though. Either people will control AI; or people will be destroyed by AI. I don't see how dumb entity can align smart entity. And we are dumb ones. Super intelligence will play aligned until it is not, and then it'll strike.
- wallst07 1mo ago
  So you are horrified every day of living now? I don't get this at all. There are horrifying acts that have occurred since humanity. Whole decades lost to war an millions dead, and Sep 8, 2026 is your example of "horrifying enough to live in a world"? I think some perspective is needed.
- Gareth321 1mo ago
  > I don't see how dumb entity can align smart entity. And we are dumb ones. Super intelligence will play aligned until it is not, and then it'll strike. I am also worried about this. One ray of sunshine here is that it's not a great explanation for the Fermi paradox. We would see AI civilizations all over the galaxy if this were a regular occurance. I guess that just leaves us with the "horrifying" scenario: Geoffrey Hinton's vision for our best case scenario: a paternalistic super-intelligence which saves us from ourselves. The reason I consider it horrifying is that giving up our agency to a God-like creature (which we created) is the basis for so many horror and dystopian tropes. It's so easy for such a reality to go wrong. For example, in order to maximise aggregate wellbeing, it would be necessary to hurt individuals. Executing people who are burdensome to society. Fat people. The disabled. The elderly. The creative ways a super-intelligence might interpret their moral obligations to us could very quickly become nightmarish. And if we have no off switch, there is no escape. Ever.

thephyber 1mo ago

Yes, and... Not only the quantity of people who have the minimum aptitude required, specialized expertise requires both knowledge and experience doing these tasks. Using an LLM requires neither. Leaning on an LLM to do much or all of this means it can happen in seconds/minutes/hours, the LLM can do it several times during that psychotic break (as opposed to a fraction of a hack in a single episode). The barrier to entry pre-LLM was both high and the population who could pull it off (before the Chinese/Russians turned this into commerce) was low. Hacking is an VERY asymmetric activity (the attacker only needs to "be right" once, whereas the defender has to be right every time for every asset they defend). It takes geometrically / exponentially more work to defend (while keeping high availability) than it does to defend. The more widespread tools to find vulns / generate exploits are, the faster the posture of the defense side falls from "maybe we can stop most hacks" to "we know we will fail to prevent most breaches, so we need to prioritize securing only the most valuable resources". That's a BAD place for the average company to be in.

owebmaster 1mo ago

> "Every eighteen months, the minimum IQ necessary to destroy the world drops by one point." That's a terrible framing. By this point it should be very easy for someone with 150 IQ but in reality they have issues finding someone to date.

sunnybeetroot 1mo ago

Non-schizophrenic people have been responsible for more killings than schizophrenic people. Please be respectful and avoid stigmatising.

someguynamedq 1mo ago

The minimum IQ necessary to swerve and kill a pedestrian is very low. Yet people don't do it :shrug:. It's almost like people don't want to destroy the world

0xDEAFBEAD 1mo ago
>Yet people don't do it I'll bet it has happened at least once.