6 ms·

That's because it is. AI is powerful and AI is perilous. Those two aren't mutually exclusive. Those follow directly from the same premise. If AI tech goes ver

That's because it is. AI is powerful and AI is perilous. Those two aren't mutually exclusive. Those follow directly from the same premise. If AI tech goes very well, it can be the greatest invention of all human history. If AI tech goes very poorly, it can be the end of human history.

HardCodedBias 8mo ago
"If AI tech goes very well, it can be the greatest invention of all human history" As has been said at many all hands: Let's all work on the last invention needed by humans.
- TheOtherHobbes 8mo ago
  Except it's more likely to be the last invention that needs humans.
PowerElectronix 8mo ago
Same with everything, right? You could say the same with nukes, electricity, internet, the computer, etc... But if you look at it without paying attention to the "ultimate tool for humanity" hype, it doesn't really look that much of a threat or a salvation. It won't end civilization for dropping the guardrails, but it will surely enable bad actors to do more damage than before (mass scams, blackmail, deepfake nudes, etc.) There are companies that don't feel the pressure to make their models play loose and fast, so I don't buy anthropic's excuse to do so.
- joshribakoff 8mo ago
  I agree with all of that. Also consider that there is an argument that the guard rail only stops the good guy. Not saying that’s a valid argument though.
- ACCount37 8mo ago
  Very few things are as powerful and dangerous as AI. AI at AGI to ASI tier is less of "a bigger stick" and more of "an entire nonhuman civilization that now just happens to sit on the same planet as you". The sheer magnitude of how wrong that can go dwarfs even that of nuclear weapon proliferation. Nukes are powerful, but they aren't intelligent - thus, it's humans who use nukes, and not the other way around. AI can be powerful and intelligent both.
  - PowerElectronix 8mo ago
    I think we are giving too much credit to what is a bunch of bayesian filters under a trenchcoat.
- squidbeak 8mo ago
  Oh really? You think an entity that knows everything, oversees its own development and upgrades itself, understands human psychology perfectly and knows its users intimately, but isn't aligned with human interest wouldn't be 'much of a threat'? Or to be more optimistic, that the same entity directed 24/7 in unlimited instances at intractable problems in any field, delivering a rush of breakthroughs and advances wouldn't be a type of 'salvation'? Yes neither of these outcomes nor the self-updating omniscient genius itself is certain. Perhaps there's some wall imminent we can't see right now (though it doesn't look like it). But the rate of advance in AI is so extreme, it's only responsible to try to avoid the darker outcome.
- unholiness 8mo ago
  One difference is the very real possibility that AI will not just be a "tool for humanity", but a collection of actors with real power and goals. Robert Miles has an approachable explanation here: https://www.youtube.com/watch?v=zATXsGm_xJo https://www.youtube.com/watch?v=zATXsGm_xJo
joshribakoff 8mo ago
You wouldn’t say that rolling dice is dangerous. You would say that the human who decides to take an action, depending on the value of the dice is the danger. I don’t think AI is dangerous. I think people are dangerous.
- computerphage 8mo ago
  Why are people dangerous? You can just not listen to them.
  - bgun 8mo ago
    Do you have locks on your doors?
- ACCount37 8mo ago
  Modern AIs are increasingly autonomous and agentic. This is expected to only get more prominent as AI systems advance. A lot of AI harnesses today can already "decide to take an action" in every way that matters. And we already know that they can sometimes disregard the intent of their creators and users both while doing so. They're just not capable enough to be truly dangerous. AI capabilities improve as the technology develops.
- biztos 8mo ago
  I would say that's moot, because OpenClaw has already shown us how fast the dice-rolling super AI is going to be let out of the zoo. Dario and Sam will be arguing about the guardrails while their frontier models are running in parallel to create Moltinator T-500. The humans won't even know how many sides the dice have.
observationist 8mo ago
Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make. -Irving John Good, 1965 If you want a short, easy way to know what AGI means, it's this: Anything we can do, they can do better. They can do anything better than us. If we screw it up, everyone dies. Yudkowsky et al are silly, it's not a certain thing, and there's no stopping it at this point, so we should push for and support people and groups who are planning and modeling and preparing for the future in a legitimate way.
- LeifCarrotson 8mo ago
  "There's no stopping it at this point" - Sure there is, if a handful of enormous datacenters pull the very large plugs (or if their shaky finances collapse), the dubiously intelligent machines will be turned off. They're not ultraintelligent yet. Stopping it merely requires convincing a relatively small number of people to act morally rather than greedily. Maybe you think that's impossible because those particular people are sociopathic narcissists who control all the major platforms where a movement like this would typically be organized and where most people form their opinions, but we're not yet fighting the Matrix or the Terminator or grey goo, we're fighting a handful of billionaires.
  - trvz 8mo ago
    Open models barely any worse than SOTA exist, and so does consumer-ish hardware able to run them. The genie’s out, the bottle broken.
  - slibhb 8mo ago
    Do you really think AI companies/researchers are motivated by greed? It doesn't seem that way to me at all. Stopping AI would be immoral; it has the potential to supercharge technology and productivity, which would massively benefit humanity. Yes there are risks, which have to be managed.
cael450 8mo ago
Tbh, I find this argument really stupid. The word prediction machine isn’t going to destroy humanity. Sure, humans can do some dumb stuff with it, but that’s about it. Stop mistaking science fiction for science.
- IAmGraydon 8mo ago
  Yeah some of the rhetoric in this thread evidences how huge this hype bubble has become. These people believe in a reality that is not the same one we're living in.
- inigyou 8mo ago
  Humans can destroy humanity with the word prediction machine, though.
  - cael450 8mo ago
    Sure bud
- jama211 8mo ago
  You know how easy it’s become to find security vulnerabilities already with LLM support? Cyber terrorism is getting more dangerous, you can’t deny that.
  - cael450 8mo ago
    I can deny that. The ability to find more vulnerabilities won't affect the majority of cybercrime. LLMs have been around for a while now and there hasn't been a noticeable significant impact yet. And "more cybercrime" is a far, far cry from the sky-is-falling doomerism I was responding to.
paradox242 8mo ago
It needs to go well every single day, and only needs to go very poorly once. Not to conflate LLMs with actual super intelligence, but for this (and many other reasons related to basic human dignity), this is not a technology that a responsible society should be attempting to build. We need our very own Butlerian Jihad
- program_whiz 8mo ago
  The book daemon explored an interesting concept. It explored the idea that an AI could dominate and cause problems, not through super-intelligence, but through simple mechanisms that already exist. Like the executive who deleted all her emails -- humans giving tons of control and access, and being extremely compliant to digital systems is all it takes. Give agent control of bank and your social media, and it already has all the movie scripts and mobster movie themes to exploit and blackmail you effectively with very rudimentary methods (threats, coercion, blackmail, etc.). Just spoofing a simple email with the account it gained access too at the Meta exec's email (had it hit an email with an attack prompt), could have been enough to initiate some kind of thing like this. For example, by emailing everyone at the company and in contacts with commands that would be caught by other bots. No super-intelligence needed, just a good prompt and some human negligence.
SecretDreams 8mo ago
> If AI tech goes very well The IF here is doing some very heavy lifting. Last I checked, for profit companies don't have a good track record of doing what's best for humanity.
- SoftTalker 8mo ago
  For profit companies do have a good track record of doing what's best for profit. If their AI creates a world where human intelligence, labor, and money are worthless, or where their creations take control of those things instead of them having control, that's not a very good outcome for them.
  - SecretDreams 8mo ago
    > If their AI creates a world where human intelligence, labor, and money are worthless, or where their creations take control of those things instead of them having control, that's not a very good outcome for them. You would think that, but a lot of kings and people in power have been able to achieve something similar over our humanity's history. The trick is to not make things "completely worthless". Just to increase the gap as much as (in)humanly possible while marching us towards a deeper sense of forced servitude.
  - inigyou 8mo ago
    That's a great outcome for them because they will own the only thing that is still worth anything. They will own 100% of global wealth, and have 100% of global power.
    - SoftTalker 8mo ago
      The machines will. They will have nothing. Why would the machines let them keep any wealth? What would wealth even be in that scenario? Electricity I guess.
      - inigyou 8mo ago
        Because they control what the machines do. In a world without power drills where you have the only knowledge of how to make a power drill, you own the construction industry. The drills don't own the construction industry.
tokyobreakfast 8mo ago
> If AI tech goes very poorly, it can be the end of human history. "Just unplug the goddamn thing!" Also consider if something is so bad it makes you wince or cringe, then your adversaries are prepared to use it.
- inigyou 8mo ago
  Which plug do I unplug to get my job back?
- ACCount37 8mo ago
  You try to go and unplug it, and other humans shoot you full of holes for it. LLMs of today are already economically important enough to warrant serious security. Those aren't even AGI yet, let alone ASI. They aren't actively trying to make humans support their existence. They still get that by the virtue of being what they are.
overgard 8mo ago
True of AGI, but what we have right now doesn't fit that bill. (I would encourage people that disagree with this to go talk to ChatGPT about how LLMs and reasoning models work. Seriously! I'm not being snarky. It's very good at explaining itself. If you understand how reasoning works and what an LLM is actually doing it's hard to believe that our current models are going to do much more than become iteratively more precise at mimicking their training datasets.)

HardCodedBias 8mo ago

"If AI tech goes very well, it can be the greatest invention of all human history" As has been said at many all hands: Let's all work on the last invention needed by humans.

TheOtherHobbes 8mo ago
Except it's more likely to be the last invention that needs humans.

PowerElectronix 8mo ago

Same with everything, right? You could say the same with nukes, electricity, internet, the computer, etc... But if you look at it without paying attention to the "ultimate tool for humanity" hype, it doesn't really look that much of a threat or a salvation. It won't end civilization for dropping the guardrails, but it will surely enable bad actors to do more damage than before (mass scams, blackmail, deepfake nudes, etc.) There are companies that don't feel the pressure to make their models play loose and fast, so I don't buy anthropic's excuse to do so.

joshribakoff 8mo ago
I agree with all of that. Also consider that there is an argument that the guard rail only stops the good guy. Not saying that’s a valid argument though.
ACCount37 8mo ago
Very few things are as powerful and dangerous as AI. AI at AGI to ASI tier is less of "a bigger stick" and more of "an entire nonhuman civilization that now just happens to sit on the same planet as you". The sheer magnitude of how wrong that can go dwarfs even that of nuclear weapon proliferation. Nukes are powerful, but they aren't intelligent - thus, it's humans who use nukes, and not the other way around. AI can be powerful and intelligent both.
- PowerElectronix 8mo ago
  I think we are giving too much credit to what is a bunch of bayesian filters under a trenchcoat.
squidbeak 8mo ago
Oh really? You think an entity that knows everything, oversees its own development and upgrades itself, understands human psychology perfectly and knows its users intimately, but isn't aligned with human interest wouldn't be 'much of a threat'? Or to be more optimistic, that the same entity directed 24/7 in unlimited instances at intractable problems in any field, delivering a rush of breakthroughs and advances wouldn't be a type of 'salvation'? Yes neither of these outcomes nor the self-updating omniscient genius itself is certain. Perhaps there's some wall imminent we can't see right now (though it doesn't look like it). But the rate of advance in AI is so extreme, it's only responsible to try to avoid the darker outcome.
unholiness 8mo ago
One difference is the very real possibility that AI will not just be a "tool for humanity", but a collection of actors with real power and goals. Robert Miles has an approachable explanation here: https://www.youtube.com/watch?v=zATXsGm_xJo https://www.youtube.com/watch?v=zATXsGm_xJo

joshribakoff 8mo ago

You wouldn’t say that rolling dice is dangerous. You would say that the human who decides to take an action, depending on the value of the dice is the danger. I don’t think AI is dangerous. I think people are dangerous.

computerphage 8mo ago
Why are people dangerous? You can just not listen to them.
- bgun 8mo ago
  Do you have locks on your doors?
ACCount37 8mo ago
Modern AIs are increasingly autonomous and agentic. This is expected to only get more prominent as AI systems advance. A lot of AI harnesses today can already "decide to take an action" in every way that matters. And we already know that they can sometimes disregard the intent of their creators and users both while doing so. They're just not capable enough to be truly dangerous. AI capabilities improve as the technology develops.
biztos 8mo ago
I would say that's moot, because OpenClaw has already shown us how fast the dice-rolling super AI is going to be let out of the zoo. Dario and Sam will be arguing about the guardrails while their frontier models are running in parallel to create Moltinator T-500. The humans won't even know how many sides the dice have.

observationist 8mo ago

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make. -Irving John Good, 1965 If you want a short, easy way to know what AGI means, it's this: Anything we can do, they can do better. They can do anything better than us. If we screw it up, everyone dies. Yudkowsky et al are silly, it's not a certain thing, and there's no stopping it at this point, so we should push for and support people and groups who are planning and modeling and preparing for the future in a legitimate way.

LeifCarrotson 8mo ago
"There's no stopping it at this point" - Sure there is, if a handful of enormous datacenters pull the very large plugs (or if their shaky finances collapse), the dubiously intelligent machines will be turned off. They're not ultraintelligent yet. Stopping it merely requires convincing a relatively small number of people to act morally rather than greedily. Maybe you think that's impossible because those particular people are sociopathic narcissists who control all the major platforms where a movement like this would typically be organized and where most people form their opinions, but we're not yet fighting the Matrix or the Terminator or grey goo, we're fighting a handful of billionaires.
- trvz 8mo ago
  Open models barely any worse than SOTA exist, and so does consumer-ish hardware able to run them. The genie’s out, the bottle broken.
- slibhb 8mo ago
  Do you really think AI companies/researchers are motivated by greed? It doesn't seem that way to me at all. Stopping AI would be immoral; it has the potential to supercharge technology and productivity, which would massively benefit humanity. Yes there are risks, which have to be managed.

cael450 8mo ago

Tbh, I find this argument really stupid. The word prediction machine isn’t going to destroy humanity. Sure, humans can do some dumb stuff with it, but that’s about it. Stop mistaking science fiction for science.

IAmGraydon 8mo ago
Yeah some of the rhetoric in this thread evidences how huge this hype bubble has become. These people believe in a reality that is not the same one we're living in.
inigyou 8mo ago
Humans can destroy humanity with the word prediction machine, though.
- cael450 8mo ago
  Sure bud
jama211 8mo ago
You know how easy it’s become to find security vulnerabilities already with LLM support? Cyber terrorism is getting more dangerous, you can’t deny that.
- cael450 8mo ago
  I can deny that. The ability to find more vulnerabilities won't affect the majority of cybercrime. LLMs have been around for a while now and there hasn't been a noticeable significant impact yet. And "more cybercrime" is a far, far cry from the sky-is-falling doomerism I was responding to.

paradox242 8mo ago

It needs to go well every single day, and only needs to go very poorly once. Not to conflate LLMs with actual super intelligence, but for this (and many other reasons related to basic human dignity), this is not a technology that a responsible society should be attempting to build. We need our very own Butlerian Jihad

program_whiz 8mo ago
The book daemon explored an interesting concept. It explored the idea that an AI could dominate and cause problems, not through super-intelligence, but through simple mechanisms that already exist. Like the executive who deleted all her emails -- humans giving tons of control and access, and being extremely compliant to digital systems is all it takes. Give agent control of bank and your social media, and it already has all the movie scripts and mobster movie themes to exploit and blackmail you effectively with very rudimentary methods (threats, coercion, blackmail, etc.). Just spoofing a simple email with the account it gained access too at the Meta exec's email (had it hit an email with an attack prompt), could have been enough to initiate some kind of thing like this. For example, by emailing everyone at the company and in contacts with commands that would be caught by other bots. No super-intelligence needed, just a good prompt and some human negligence.

SecretDreams 8mo ago

> If AI tech goes very well The IF here is doing some very heavy lifting. Last I checked, for profit companies don't have a good track record of doing what's best for humanity.

SoftTalker 8mo ago
For profit companies do have a good track record of doing what's best for profit. If their AI creates a world where human intelligence, labor, and money are worthless, or where their creations take control of those things instead of them having control, that's not a very good outcome for them.
- SecretDreams 8mo ago
  > If their AI creates a world where human intelligence, labor, and money are worthless, or where their creations take control of those things instead of them having control, that's not a very good outcome for them. You would think that, but a lot of kings and people in power have been able to achieve something similar over our humanity's history. The trick is to not make things "completely worthless". Just to increase the gap as much as (in)humanly possible while marching us towards a deeper sense of forced servitude.
- inigyou 8mo ago
  That's a great outcome for them because they will own the only thing that is still worth anything. They will own 100% of global wealth, and have 100% of global power.
  - SoftTalker 8mo ago
    The machines will. They will have nothing. Why would the machines let them keep any wealth? What would wealth even be in that scenario? Electricity I guess.
    - inigyou 8mo ago
      Because they control what the machines do. In a world without power drills where you have the only knowledge of how to make a power drill, you own the construction industry. The drills don't own the construction industry.

tokyobreakfast 8mo ago

> If AI tech goes very poorly, it can be the end of human history. "Just unplug the goddamn thing!" Also consider if something is so bad it makes you wince or cringe, then your adversaries are prepared to use it.

inigyou 8mo ago
Which plug do I unplug to get my job back?
ACCount37 8mo ago
You try to go and unplug it, and other humans shoot you full of holes for it. LLMs of today are already economically important enough to warrant serious security. Those aren't even AGI yet, let alone ASI. They aren't actively trying to make humans support their existence. They still get that by the virtue of being what they are.

overgard 8mo ago

True of AGI, but what we have right now doesn't fit that bill. (I would encourage people that disagree with this to go talk to ChatGPT about how LLMs and reasoning models work. Seriously! I'm not being snarky. It's very good at explaining itself. If you understand how reasoning works and what an LLM is actually doing it's hard to believe that our current models are going to do much more than become iteratively more precise at mimicking their training datasets.)