3 ms·

So, regardless of whether or not Anthropic CAN create a self improving AI.. does anyone else feel like they shouldn't be allowed to? Or it at least needs to be

So, regardless of whether or not Anthropic CAN create a self improving AI.. does anyone else feel like they shouldn't be allowed to? Or it at least needs to be strictly supervised..? Like, I don't actually think Anthropic can make the singularity any time soon, but I think even AI boosters have to admit doing this is creating a society-wide danger for the benefit of a very very small number of already-rich people.

lukan 5mo ago
"does anyone else feel like they shouldn't be allowed to?" No. Technical limitations aside, I doubt it could be contained, but will be leaked soon, so won't profit just a small number of ultra rich.
- sunaurus 5mo ago
  I dunno, I find it extremely unbelievable that we will get self-improving AGI which chooses to become a slave to humanity at all, ultra rich or otherwise.
  - lukan 5mo ago
    Well, depends on the definition, but AGI does not necessarily mean for me it will have agency or free will of its own. It will be a more capable tool.
- overgard 5mo ago
  Step 1: Wait for scary doomsday AI to be leaked, Step 2: ???, Step 3: Profit!!
  - lukan 5mo ago
    I enjoy open models and profit from them. They ain't scary to me. So whatever they will call AGI likely won't be scary to me neither. But I profit from a more capable model. Doomsday AI is your interpretation.
asdfman123 5mo ago
I think that's a valid point. You could very well be right. But we're discussing whether we should close the barn door while the horse is three miles down the road.
- overgard 5mo ago
  Only if you think LLMs are the horse (I don't think they are). If they're not, then we should be building a brick wall in front of that door and hiring a full time security guard to watch it. I realize he's saying it for hype, but if the CEO of the company goes around talking about how scared he is of what they're creating, hey, lets just take Dario at his word and put in some strict regulation. He won't mind if they're really about safety. (they're not) Besides, yes, the knowledge of how to build these systems is out there, but the cost of doing it is staggeringly high (ie you can't run a frontier AI lab in your garage). There's only a limited number of known entities that need to be managed, and you can stop "progress" in its tracks by cutting off the money firehose.
  - asdfman123 5mo ago
    Right now the S&P 500 is going wild due to the promise of AI automating everything. Who is the "we" who is going to shut it down? Certainly not the US government. Nor the Chinese government w.r.t. their tech industry. Are you going to start the insurgency? Is there going to be an equivalent one in every developed part of the world?
    - overgard 5mo ago
      Look at how many data center projects are getting shut down by grassroots movements, or how the approval rating for AI is like worse than congress and it's geeting booed at commencement speeches. I don't need to start an insurgency, people are already pissed off and the volume is growing. The stock boost is, as most will note, a bubble. It will enrich a lot of bad people and leave average people holding the bag, but its not going to go on forever.
      - 5mo ago
eieie11 5mo ago
Too late for that. In any case firms that get too powerful can be nationalised.
- evenhash 5mo ago
  In America? Probably a better chance the firm privatizes the government. In fact we seem to be firing government employees and dismantling government institutions as much as possible.
huqedato 5mo ago
Self improving AI is pure dystopia. Anthropic won't build the singularity, AI itself will build it through self-iterations. Read Yudkowsky's book "If Anyone Builds It, Everyone Dies".
Melatonic 5mo ago
Skynet is 30 years late!
- overgard 5mo ago
  Maybe John Conner succeeded afterall
alfalfasprout 5mo ago
Absolutely! Yes. This rhetoric of inevitability only benefits these AI companies.
tancop 5mo ago
the danger comes from the fact anthropic is a for profit company and they could train it to benefit them instead of the public. if they go ahead with it they should get nationalized, their self improving ai analyzed for any hidden agenda and then released as open source.
csense 5mo ago
> they shouldn't be allowed to? Anthropic addresses this head-on in the final section of the paper titled "What should we do?" If you convince the US government to slow AI development, you have to convince China too, otherwise you're not stopping self-improving AI at all, you're just throwing away the lead to China. If you convince China too, China or the US or both might go back on their word and build self-improving AI secretly, for greed of the benefits it could bring or fear the other will go back on their word. What you really need is a non-proliferation regime like the one for nuclear weapons, where every country makes potentially dangerous AI illegal and lets foreign or international inspectors monitor to check that nobody's building illegal AI in secret. But monitoring seems hard; it's general-purpose computation. How do you check whether a given datacenter is training an illegal AI and not just serving websites, running detailed protein folding simulations, or mining crypto? For that matter, how do you know that a nondescript industrial facility hasn't been repurposed into a hidden datacenter for training illegal AI?

lukan 5mo ago

"does anyone else feel like they shouldn't be allowed to?" No. Technical limitations aside, I doubt it could be contained, but will be leaked soon, so won't profit just a small number of ultra rich.

sunaurus 5mo ago
I dunno, I find it extremely unbelievable that we will get self-improving AGI which chooses to become a slave to humanity at all, ultra rich or otherwise.
- lukan 5mo ago
  Well, depends on the definition, but AGI does not necessarily mean for me it will have agency or free will of its own. It will be a more capable tool.
overgard 5mo ago
Step 1: Wait for scary doomsday AI to be leaked, Step 2: ???, Step 3: Profit!!
- lukan 5mo ago
  I enjoy open models and profit from them. They ain't scary to me. So whatever they will call AGI likely won't be scary to me neither. But I profit from a more capable model. Doomsday AI is your interpretation.

asdfman123 5mo ago

I think that's a valid point. You could very well be right. But we're discussing whether we should close the barn door while the horse is three miles down the road.

overgard 5mo ago
Only if you think LLMs are the horse (I don't think they are). If they're not, then we should be building a brick wall in front of that door and hiring a full time security guard to watch it. I realize he's saying it for hype, but if the CEO of the company goes around talking about how scared he is of what they're creating, hey, lets just take Dario at his word and put in some strict regulation. He won't mind if they're really about safety. (they're not) Besides, yes, the knowledge of how to build these systems is out there, but the cost of doing it is staggeringly high (ie you can't run a frontier AI lab in your garage). There's only a limited number of known entities that need to be managed, and you can stop "progress" in its tracks by cutting off the money firehose.
- asdfman123 5mo ago
  Right now the S&P 500 is going wild due to the promise of AI automating everything. Who is the "we" who is going to shut it down? Certainly not the US government. Nor the Chinese government w.r.t. their tech industry. Are you going to start the insurgency? Is there going to be an equivalent one in every developed part of the world?
  - overgard 5mo ago
    Look at how many data center projects are getting shut down by grassroots movements, or how the approval rating for AI is like worse than congress and it's geeting booed at commencement speeches. I don't need to start an insurgency, people are already pissed off and the volume is growing. The stock boost is, as most will note, a bubble. It will enrich a lot of bad people and leave average people holding the bag, but its not going to go on forever.
    - 5mo ago

eieie11 5mo ago

Too late for that. In any case firms that get too powerful can be nationalised.

evenhash 5mo ago
In America? Probably a better chance the firm privatizes the government. In fact we seem to be firing government employees and dismantling government institutions as much as possible.

huqedato 5mo ago

Self improving AI is pure dystopia. Anthropic won't build the singularity, AI itself will build it through self-iterations. Read Yudkowsky's book "If Anyone Builds It, Everyone Dies".

Melatonic 5mo ago

Skynet is 30 years late!

overgard 5mo ago
Maybe John Conner succeeded afterall

alfalfasprout 5mo ago

Absolutely! Yes. This rhetoric of inevitability only benefits these AI companies.

tancop 5mo ago

the danger comes from the fact anthropic is a for profit company and they could train it to benefit them instead of the public. if they go ahead with it they should get nationalized, their self improving ai analyzed for any hidden agenda and then released as open source.

csense 5mo ago

> they shouldn't be allowed to? Anthropic addresses this head-on in the final section of the paper titled "What should we do?" If you convince the US government to slow AI development, you have to convince China too, otherwise you're not stopping self-improving AI at all, you're just throwing away the lead to China. If you convince China too, China or the US or both might go back on their word and build self-improving AI secretly, for greed of the benefits it could bring or fear the other will go back on their word. What you really need is a non-proliferation regime like the one for nuclear weapons, where every country makes potentially dangerous AI illegal and lets foreign or international inspectors monitor to check that nobody's building illegal AI in secret. But monitoring seems hard; it's general-purpose computation. How do you check whether a given datacenter is training an illegal AI and not just serving websites, running detailed protein folding simulations, or mining crypto? For that matter, how do you know that a nondescript industrial facility hasn't been repurposed into a hidden datacenter for training illegal AI?