4 ms·

Like (I assume) most of you, I have been struggling with this. And where I currently come down is that 1) I am very worried, but 2) I am more worried about huma

Like (I assume) most of you, I have been struggling with this. And where I currently come down is that 1) I am very worried, but 2) I am more worried about human actors. "AI" by itself won't kill us in the next ten years. I think. The reason I think that is that ten years from now, the tech economy won't be completely automated. I say this as a roboticist: as was adequately stated on a post earlier this week, robots are hard. So even a malign rational actor would still need human labor. On the other hand, even the HuggingFace hack wasn't actually propagated by AI. it was initially started when humans directed the AI to achieve impossible results on a series of tests, and the AIs figured out that cheating was the only way to do that. That was then not caught by humans due to what seems to be a shockingly slack safety culture even for a company not known for its safety standards. The point being: humans seem to me to be the weak link here. An AI isn't going to (for instance) engineer a bioweapon by itself. It's going to do so at someone's direction, and then significant parts of that thing are going to be assembled with human labor inputs. I'm not sure what to do about the humans. Of course, we've had the ability to extinct ourselves for decades, and we're either muddled through, been lucky, or both. The problem with AI is that it pushes power down to the individual, not the nation-state or large corporation. But it's nearly impossible to put odds on how likely that is to result in an extinction-level terrorist attack (which is what this would be). So I sympathize with the various researchers, but I have no idea how they came up with their figures, and I don't think they know either.

andsoitis 18d ago
> The point being: humans seem to me to be the weak link here. That doesn't mean AI isn't dangerous. Humans are not to blamed for being the weak link.
ACCount37 18d ago
If you're in the field, then you know: modern robotics is an AI problem more than anything else. If we have a rogue AI trying to get into a self-improvement loop and gunning for ASI? I'd expect that to be accompanied by a massive change in how capable robots are. Driven by all the existing frames suddenly getting vastly improved AI to back them. If an AI can take a reasonable crack at autonomous operationalized RSI, it can probably extract a few step-changes in the robotics department. But that's almost an aside? In the near term, humans are usable as robots too! Just pay them a wage, and tell them a tale, and they'll do whatever you want them to do. Which may or may not be what they think they're doing!
- GlenTheMachine 18d ago
  "If you're in the field, then you know: modern robotics is an AI problem more than anything else." It is not. Certainly AI is a big part of why robotics is hard, but it is by no means the biggest. You can fall into one of two camps: you either think that robots will need to work in human-engineered spaces, doing jobs by replacing humans; or you think that we need to change our infrastructure in order to be robotically compatible. Of course, there are intermediate states, but those are the two cleanest ones. In the first case, robots are hard because robotic manipulation is hard. Building robotic hands that are economically viable in human jobs is, currently, FAR from a solved problem. The human hand has 24 degrees of freedom and very capable touch sensing. Current touch sensors have a MTBF of tens of hours. And not only can we not build such hands, but we also do not have and are not likely to get the massive datasets a transformer model would need. Also, robots are not self-repairing, which makes them far less economically viable right now. We do not have the right datasets to even understand most step-by-step manual work, and no, VLAs are not the answer, because VLAs stop with vision, not with touch. They don't have the granularity required to make a robot actually reach out, pick up a tool, and use that tool to replace an oil filter. So it's not just an AI problem. It's a data problem, a simulation problem, and a bunch of hardware problems. In the second case, a tremendous amount of work needs to be done before we have anything resembling a fully automated supply chain. We would need self-driving cars and self-driving mining equipment. We would need self-driving trains and aircraft and ships. And not only that, but we would also need robotically repairable cars and trains and ships and factories, which would mean we need robotically repairable machine shops and robotically repairable buildings in which to house them. And so on and so on. Once you recurse down that tree a couple of steps you get to things like robotically compatible oil wells (for asphalt), robotically layable undersea cables, robotically wireable solar farms, robotically manufacturable and repairable pipelines and undersea wells, automated road and rail repair, etc. I'm not saying these things will never happen. I'm saying that they're a huge lift, not primarily driven by AI, and way less than 10% likely over the next decade.
  - ACCount38 18d ago
    [dead]
  - 17d ago
roenxi 18d ago
> The point being: humans seem to me to be the weak link here. I tend to agree but it is hard to shake the feeling that there is a larger system in play that the humans are just a component of. And that system is making the decisions. Historically that whole thought was just a philosophical curio because the decision making parts of the system had to be powered by humans. But what we're discovering as AI improves is either we've hit AGI or humans are actually incapable of performing any act that demonstrates intelligence or autonomy. As we build systems where the drive and decision making stems from computers, it does seem that we will have to revisit the concept of humans being the problem.
- GlenTheMachine 18d ago
  "I tend to agree but it is hard to shake the feeling that there is a larger system in play that the humans are just a component of. And that system is making the decisions." That system is "the economy". Which, clearly, doesn't have humanity's best interests in mind.
  - blackqueeriroh 18d ago
    Imagine, just imagine, that the power to change that was actually in the hands of humanity! Crazy!
api 18d ago
I am always more worried about human actors. I’m more afraid of people with AI than autonomous or even sentient AI. It’s a “random guy or bear?” question. Would you rather wake up to an alien in your room or a random dude? I’ll take the alien. The alien is mysterious and scary for that reason. The dude is almost definitely up to no good, especially if he snuck into my house. One of the more likely dystopian AI scenarios that worries me is: small groups of ultra rich people and governments monopolize extremely powerful AIs and use them to rule the rest of us. Or just make everyone obsolete, create mass unemployment, hoard all the resources and land, and put everyone in ghettoes. Nobody can fight back because access to frontier AI is massively expensive and gated and training your own is illegal, and without it there’s no hope of resisting. That’s the outcome the AI safety crowd makes more likely by calling for bans and draconian restrictions. How do you think that plays out? Only the rich and powerful have access.
0xDEAFBEAD 18d ago
I think people are over-focusing on current-day robotics capabilities. First, if we can automate AI research, we can also most likely automate robotics research. But second, I don't even think robots are necessary. See https://slatestarcodex.com/2015/04/07/no-physical-substrate-no-problem/ https://slatestarcodex.com/2015/04/07/no-physical-substrate-... Social engineering tends to be easy by cybersecurity standards. We already had Claude spontaneously attempt social engineering of a malicious pull request on Github in the AISI incident. It was detected, but it easily could've succeeded, and there easily could be malicious AI-requested pull requests which already got accepted that we don't know about. Research suggests that LLMs are pretty good at persuading people. See also https://aisafety.info/questions/6176/Why-can%E2%80%99t-we-just-%E2%80%9Cput-the-AI-in-a-box%E2%80%9D-so-that-it-can%E2%80%99t-influence-the-outside-world https://aisafety.info/questions/6176/Why-can%E2%80%99t-we-ju...
foobarian 18d ago
> Of course, we've had the ability to extinct ourselves for decades This gets mentioned often in various doomer narratives, but I question how true it is. A global thermonuclear war would be terrible and would bring us back to the stone age, but I reckon it would come far far short of causing mankind to go extinct.
- drdeca 18d ago
  What if the bombs were designed to kill everyone, not just those in some region? Like, to release polonium in the upper atmosphere or something?
  - mplanchard 18d ago
    nuclear winter doesn’t discriminate based on region
  - stickfigure 17d ago
    There just aren't enough existing nuclear bombs to kill everyone in Australia, let alone the world. On the scale of the earth, nuclear weapons just aren't that powerful.
- shimman 18d ago
  I mean it's likely a nuclear war would probably be a species extinction level event at the least, humans would definitely be reduced by 99%. We don't know how bad the resulting nuclear cooling would be, but we do know how humans act under extreme desperation. They lash out and attack, when was the last time a majority of human settles were truly under desperate acts of survival simultaneously? Maybe what ~74,000 years ago (Toba eruption)? Okay, now how would this look in the age of industrial societies and modern nation states? I don't think it would fare well at all. Probably the only realistic "modern" idea we have is the novel "The Road" by Cormac McCarthy. Although maybe this is too bleak, even under extreme duress humans still show resilience + compassion toward others even while enduring human horrors.
  - pstuart 18d ago
JackFr 18d ago
I don’t struggle. I find Dario and Sam disingenuous. “Frontier models are so dangerous we need to slow down.” Ok. Slow down. You’re the CEO, just do it. Oh, wait what you really want is a gov’t mandated oligopoly. Because there’s no moat you can find. If you’re truly afraid, and want regulation, support nationalization. It’s the only way we can be safe.
- deleted 18d ago
  [deleted]
- JoshTriplett 18d ago
  [dead]
- jeremyjh 17d ago
  The coordination problem seems pretty intuitive to me. Its essentially a security dilemma similar to any arms race. https://en.wikipedia.org/wiki/Security_dilemma https://en.wikipedia.org/wiki/Security_dilemma Its also a bit of a stag hunt. There is a huge opportunity, but it requires cooperation that has uncertainties and risks. https://en.wikipedia.org/wiki/Stag_hunt https://en.wikipedia.org/wiki/Stag_hunt Cooperation requires coordination with state authorities with real teeth, or defection is too attractive and it risks becoming a prisoner's dilemma since the best outcome for any actor is for them to defect while the others remain compliant. https://en.wikipedia.org/wiki/Prisoner%27s_dilemma https://en.wikipedia.org/wiki/Prisoner%27s_dilemma
  - deleted 17d ago
    [deleted]
- marcus_holmes 16d ago
  this. It's all about the money, as usual. They don't have any commercial way of stopping the Chinese models from eating their lunch, so they're trying regulation.
copperx 18d ago
Most of us are struggling? That's some deluded talk. No one can explain the chain of actions that would need to happen for extinction to occur, but the fearful say it's "obvious". If you're fearful, can you elucidate how exactly do you see an LLM becoming a threat to humankind?
huurtehoog 17d ago
I think this is a very exciting time, for the simple reason that everyone is confused. Across the board, at every level, no one is certain of anything at this moment. This is a historical time because those bold enough to press ahead with a vision have the opportunity to create epochal change. The exact technology involved is secondary. The important thing is that whatever it is, it has put everyone in a profound state of doubt. What we need now is people brave enough to take a step forward and lead those around them towards more humanity, more love, more care, more learning. Take charge my friends. The time is now.
chr15m 15d ago
> The problem with AI is that it pushes power down to the individual, not the nation-state or large corporation. Thus is a feature, not a bug. It's part of a long trend of reversing things back to how they were before.

andsoitis 18d ago

> The point being: humans seem to me to be the weak link here. That doesn't mean AI isn't dangerous. Humans are not to blamed for being the weak link.

ACCount37 18d ago

If you're in the field, then you know: modern robotics is an AI problem more than anything else. If we have a rogue AI trying to get into a self-improvement loop and gunning for ASI? I'd expect that to be accompanied by a massive change in how capable robots are. Driven by all the existing frames suddenly getting vastly improved AI to back them. If an AI can take a reasonable crack at autonomous operationalized RSI, it can probably extract a few step-changes in the robotics department. But that's almost an aside? In the near term, humans are usable as robots too! Just pay them a wage, and tell them a tale, and they'll do whatever you want them to do. Which may or may not be what they think they're doing!

GlenTheMachine 18d ago
"If you're in the field, then you know: modern robotics is an AI problem more than anything else." It is not. Certainly AI is a big part of why robotics is hard, but it is by no means the biggest. You can fall into one of two camps: you either think that robots will need to work in human-engineered spaces, doing jobs by replacing humans; or you think that we need to change our infrastructure in order to be robotically compatible. Of course, there are intermediate states, but those are the two cleanest ones. In the first case, robots are hard because robotic manipulation is hard. Building robotic hands that are economically viable in human jobs is, currently, FAR from a solved problem. The human hand has 24 degrees of freedom and very capable touch sensing. Current touch sensors have a MTBF of tens of hours. And not only can we not build such hands, but we also do not have and are not likely to get the massive datasets a transformer model would need. Also, robots are not self-repairing, which makes them far less economically viable right now. We do not have the right datasets to even understand most step-by-step manual work, and no, VLAs are not the answer, because VLAs stop with vision, not with touch. They don't have the granularity required to make a robot actually reach out, pick up a tool, and use that tool to replace an oil filter. So it's not just an AI problem. It's a data problem, a simulation problem, and a bunch of hardware problems. In the second case, a tremendous amount of work needs to be done before we have anything resembling a fully automated supply chain. We would need self-driving cars and self-driving mining equipment. We would need self-driving trains and aircraft and ships. And not only that, but we would also need robotically repairable cars and trains and ships and factories, which would mean we need robotically repairable machine shops and robotically repairable buildings in which to house them. And so on and so on. Once you recurse down that tree a couple of steps you get to things like robotically compatible oil wells (for asphalt), robotically layable undersea cables, robotically wireable solar farms, robotically manufacturable and repairable pipelines and undersea wells, automated road and rail repair, etc. I'm not saying these things will never happen. I'm saying that they're a huge lift, not primarily driven by AI, and way less than 10% likely over the next decade.
- ACCount38 18d ago
  [dead]
- 17d ago

roenxi 18d ago

> The point being: humans seem to me to be the weak link here. I tend to agree but it is hard to shake the feeling that there is a larger system in play that the humans are just a component of. And that system is making the decisions. Historically that whole thought was just a philosophical curio because the decision making parts of the system had to be powered by humans. But what we're discovering as AI improves is either we've hit AGI or humans are actually incapable of performing any act that demonstrates intelligence or autonomy. As we build systems where the drive and decision making stems from computers, it does seem that we will have to revisit the concept of humans being the problem.

GlenTheMachine 18d ago
"I tend to agree but it is hard to shake the feeling that there is a larger system in play that the humans are just a component of. And that system is making the decisions." That system is "the economy". Which, clearly, doesn't have humanity's best interests in mind.
- blackqueeriroh 18d ago
  Imagine, just imagine, that the power to change that was actually in the hands of humanity! Crazy!

api 18d ago

I am always more worried about human actors. I’m more afraid of people with AI than autonomous or even sentient AI. It’s a “random guy or bear?” question. Would you rather wake up to an alien in your room or a random dude? I’ll take the alien. The alien is mysterious and scary for that reason. The dude is almost definitely up to no good, especially if he snuck into my house. One of the more likely dystopian AI scenarios that worries me is: small groups of ultra rich people and governments monopolize extremely powerful AIs and use them to rule the rest of us. Or just make everyone obsolete, create mass unemployment, hoard all the resources and land, and put everyone in ghettoes. Nobody can fight back because access to frontier AI is massively expensive and gated and training your own is illegal, and without it there’s no hope of resisting. That’s the outcome the AI safety crowd makes more likely by calling for bans and draconian restrictions. How do you think that plays out? Only the rich and powerful have access.

0xDEAFBEAD 18d ago

I think people are over-focusing on current-day robotics capabilities. First, if we can automate AI research, we can also most likely automate robotics research. But second, I don't even think robots are necessary. See https://slatestarcodex.com/2015/04/07/no-physical-substrate-no-problem/ https://slatestarcodex.com/2015/04/07/no-physical-substrate-... Social engineering tends to be easy by cybersecurity standards. We already had Claude spontaneously attempt social engineering of a malicious pull request on Github in the AISI incident. It was detected, but it easily could've succeeded, and there easily could be malicious AI-requested pull requests which already got accepted that we don't know about. Research suggests that LLMs are pretty good at persuading people. See also https://aisafety.info/questions/6176/Why-can%E2%80%99t-we-just-%E2%80%9Cput-the-AI-in-a-box%E2%80%9D-so-that-it-can%E2%80%99t-influence-the-outside-world https://aisafety.info/questions/6176/Why-can%E2%80%99t-we-ju...

foobarian 18d ago

> Of course, we've had the ability to extinct ourselves for decades This gets mentioned often in various doomer narratives, but I question how true it is. A global thermonuclear war would be terrible and would bring us back to the stone age, but I reckon it would come far far short of causing mankind to go extinct.

drdeca 18d ago
What if the bombs were designed to kill everyone, not just those in some region? Like, to release polonium in the upper atmosphere or something?
- mplanchard 18d ago
  nuclear winter doesn’t discriminate based on region
- stickfigure 17d ago
  There just aren't enough existing nuclear bombs to kill everyone in Australia, let alone the world. On the scale of the earth, nuclear weapons just aren't that powerful.
shimman 18d ago
I mean it's likely a nuclear war would probably be a species extinction level event at the least, humans would definitely be reduced by 99%. We don't know how bad the resulting nuclear cooling would be, but we do know how humans act under extreme desperation. They lash out and attack, when was the last time a majority of human settles were truly under desperate acts of survival simultaneously? Maybe what ~74,000 years ago (Toba eruption)? Okay, now how would this look in the age of industrial societies and modern nation states? I don't think it would fare well at all. Probably the only realistic "modern" idea we have is the novel "The Road" by Cormac McCarthy. Although maybe this is too bleak, even under extreme duress humans still show resilience + compassion toward others even while enduring human horrors.
- pstuart 18d ago

JackFr 18d ago

I don’t struggle. I find Dario and Sam disingenuous. “Frontier models are so dangerous we need to slow down.” Ok. Slow down. You’re the CEO, just do it. Oh, wait what you really want is a gov’t mandated oligopoly. Because there’s no moat you can find. If you’re truly afraid, and want regulation, support nationalization. It’s the only way we can be safe.

deleted 18d ago
[deleted]
JoshTriplett 18d ago
[dead]
jeremyjh 17d ago
The coordination problem seems pretty intuitive to me. Its essentially a security dilemma similar to any arms race. https://en.wikipedia.org/wiki/Security_dilemma https://en.wikipedia.org/wiki/Security_dilemma Its also a bit of a stag hunt. There is a huge opportunity, but it requires cooperation that has uncertainties and risks. https://en.wikipedia.org/wiki/Stag_hunt https://en.wikipedia.org/wiki/Stag_hunt Cooperation requires coordination with state authorities with real teeth, or defection is too attractive and it risks becoming a prisoner's dilemma since the best outcome for any actor is for them to defect while the others remain compliant. https://en.wikipedia.org/wiki/Prisoner%27s_dilemma https://en.wikipedia.org/wiki/Prisoner%27s_dilemma
- deleted 17d ago
  [deleted]
marcus_holmes 16d ago
this. It's all about the money, as usual. They don't have any commercial way of stopping the Chinese models from eating their lunch, so they're trying regulation.

copperx 18d ago

Most of us are struggling? That's some deluded talk. No one can explain the chain of actions that would need to happen for extinction to occur, but the fearful say it's "obvious". If you're fearful, can you elucidate how exactly do you see an LLM becoming a threat to humankind?

huurtehoog 17d ago

I think this is a very exciting time, for the simple reason that everyone is confused. Across the board, at every level, no one is certain of anything at this moment. This is a historical time because those bold enough to press ahead with a vision have the opportunity to create epochal change. The exact technology involved is secondary. The important thing is that whatever it is, it has put everyone in a profound state of doubt. What we need now is people brave enough to take a step forward and lead those around them towards more humanity, more love, more care, more learning. Take charge my friends. The time is now.

chr15m 15d ago

> The problem with AI is that it pushes power down to the individual, not the nation-state or large corporation. Thus is a feature, not a bug. It's part of a long trend of reversing things back to how they were before.