4 ms·

Very probably. I really don't buy their "AI is going to kill us" claims. If you can release something better than your competition you wouldn't want to slow dow

by pier25 20d ago

Very probably. I really don't buy their "AI is going to kill us" claims. If you can release something better than your competition you wouldn't want to slow down that.

heaney-555 20d ago
You would in the case that you think you're in an unsafe race to misaligned superintelligence.
- osockwkckwk 20d ago
  Except they clearly aren’t and won’t ever be.
  - 0x696C6961 20d ago
    I don't think it's clear at all.
- voidhorse 20d ago
  No, what you would do is stop your own private development immediately, get in touch with the government by any means necessary, and build a coalition as fast as possible to stop the technology from continuing and put hard regulation (determined by the government, not dario) on the tech. Clearly they aren't that scared. If I hold a gun to your head and tell you to fork over your wallet, you will do it, because you sense imminent threat to life. Yet somehow we're to believe these people have the equivalent of a technological gun to their heads but are choosing to keep their wallets?
  - SpicyLemonZest 20d ago
    > No, what you would do is stop your own private development immediately, get in touch with the government by any means necessary, and build a coalition as fast as possible to stop the technology from continuing and put hard regulation (determined by the government, not dario) on the tech. What you're missing is that they've tried repeatedly to do this over the past years, and the government has repeatedly said that they do not want to put hard regulation on the tech. The previous administration was discussing putting a couple of soft regulations in place; the current administration doesn't want to limit progress at all because they want to stay ahead of China. So the option you're describing just isn't available.
    - voidhorse 20d ago
      First of all, I do not think they've been pursuing that to the degree that you think. But even assuming that's the case, the natural thing to do in that scenario is to not contribute to the development of the doomsday device. But anthropic's approach seems to be to help build the supposed doomsday device faster? The only generous interpretation is that Anthropic thinks that only they, the people at Anthropic have the morals/intelligence/whatever needed to build the doomsday device ethically, which is basically the perspective of an egoist despot and which should terrify you.
      - SpicyLemonZest 20d ago
        The more generous interpretation would be that Anthropic thinks they know how to build powerful AI systems in such a way that they will not become doomsday devices at all. They don't claim to be the only people with this knowledge and do not argue that their competitors should be shut down.
- pier25 20d ago
  If you really believe that you wouldn’t be in the race and trying to profit from it in the first place.
  - s1artibartfast 20d ago
    Most of the people I know working at AI Labs believe it. I think surveys back this up. We have a situation where the people running the labs are claiming it's a big risk. Outside experts not working for the labs are claiming it's a big risk. It's pretty rare for all the CEOs in the industry to write letters and testify to Congress that they should be regulated and law should be put in place for safety limits.
    - pier25 19d ago
      > Most of the people I know working at AI Labs believe it Do they believe they are actively working towards the extinction of humanity?
      - s1artibartfast 19d ago
        Yes, with nuance. They think there is a significant chance AI will lead to extinction or subordination of humans. I think the averages survey sentiment is that it is about 25% likely to happen. Some think it more likely, others less. There are many reasons why they say they must keep working. For example they think Extinction is more likely if China takes the lead or if they aren't involved in the process. They say if they don't keep developing AI, then someone else will and do a worse job. They actively want National and international regulations to slow or pause development. I don't think the public will take it seriously until we start getting some events with major death tolls. You can ask your llm of choice to dig up surveys, examples, and noteworthy public statements do you want. The point is that the people working closest to this technology are deeply concerned about the implications. Interesting aside, the lead safety expert of anthropic quit earlier this year and said they want to spend their remaining years writing poetry. That might give you a flavor of how people feel inside the industry and how high it goes
celsoazevedo 20d ago
I don't find it hard to believe that eventually some AI may kill a number of humans. You give it instructions to do something, it goes off the rails and after a certain point starts having like malware, destroying things to accomplish its goal. Depending on what it destroys, it may kill humans. With this said, I'm not exactly in disagreement with the view that AI companies with upcoming IPOs are using fearmongering to convince people that their models are very powerful and to also block others from competing with them.
- pier25 20d ago
  Maybe when we have actual artificial intelligence which LLMs are clearly not.
  - celsoazevedo 20d ago
    I think the Hugging Face attack is just a preview of what's to come. It doesn't need to be very intelligent to cause serious issues.
zahlman 19d ago
"Going to" can't be said, but there are very serious risks. The problem is the difficulty of scaling back agency for these systems in a world where the Internet is so ubiquitous. Not only do we expect basically everyone to be connected all the time anyway, it's practically required in order for anyone to use capable models (yes, plenty of HN users have thrown multiple thousands of dollars at home machines that can run some coding-specialized 27b model; that's still a tiny minority of people). So now we have to be concerned that non-deterministic systems that run in loops without supervision, which have demonstrated significant cybersecurity capabilities, and which aren't provably human-like, conscious agents with everyone's best interest in mind and a willingness to peace the fuck out if something looks dangerous, are sharing cyberspace with practically everything, including systems that deliver essential real-world goods to humans (like electricity and clean water). We nearly destroyed ourselves with nuclear weapons multiple times, and the destructive potential (and the implied threat) now persists indefinitely. The new threat bears many of the same characteristics, except now there's a party involved that has agency but is not human.

by pier25 20d ago

Very probably. I really don't buy their "AI is going to kill us" claims. If you can release something better than your competition you wouldn't want to slow down that.

heaney-555 20d ago

You would in the case that you think you're in an unsafe race to misaligned superintelligence.

osockwkckwk 20d ago
Except they clearly aren’t and won’t ever be.
- 0x696C6961 20d ago
  I don't think it's clear at all.
voidhorse 20d ago
No, what you would do is stop your own private development immediately, get in touch with the government by any means necessary, and build a coalition as fast as possible to stop the technology from continuing and put hard regulation (determined by the government, not dario) on the tech. Clearly they aren't that scared. If I hold a gun to your head and tell you to fork over your wallet, you will do it, because you sense imminent threat to life. Yet somehow we're to believe these people have the equivalent of a technological gun to their heads but are choosing to keep their wallets?
- SpicyLemonZest 20d ago
  > No, what you would do is stop your own private development immediately, get in touch with the government by any means necessary, and build a coalition as fast as possible to stop the technology from continuing and put hard regulation (determined by the government, not dario) on the tech. What you're missing is that they've tried repeatedly to do this over the past years, and the government has repeatedly said that they do not want to put hard regulation on the tech. The previous administration was discussing putting a couple of soft regulations in place; the current administration doesn't want to limit progress at all because they want to stay ahead of China. So the option you're describing just isn't available.
  - voidhorse 20d ago
    First of all, I do not think they've been pursuing that to the degree that you think. But even assuming that's the case, the natural thing to do in that scenario is to not contribute to the development of the doomsday device. But anthropic's approach seems to be to help build the supposed doomsday device faster? The only generous interpretation is that Anthropic thinks that only they, the people at Anthropic have the morals/intelligence/whatever needed to build the doomsday device ethically, which is basically the perspective of an egoist despot and which should terrify you.
    - SpicyLemonZest 20d ago
      The more generous interpretation would be that Anthropic thinks they know how to build powerful AI systems in such a way that they will not become doomsday devices at all. They don't claim to be the only people with this knowledge and do not argue that their competitors should be shut down.
pier25 20d ago
If you really believe that you wouldn’t be in the race and trying to profit from it in the first place.
- s1artibartfast 20d ago
  Most of the people I know working at AI Labs believe it. I think surveys back this up. We have a situation where the people running the labs are claiming it's a big risk. Outside experts not working for the labs are claiming it's a big risk. It's pretty rare for all the CEOs in the industry to write letters and testify to Congress that they should be regulated and law should be put in place for safety limits.
  - pier25 19d ago
    > Most of the people I know working at AI Labs believe it Do they believe they are actively working towards the extinction of humanity?
    - s1artibartfast 19d ago
      Yes, with nuance. They think there is a significant chance AI will lead to extinction or subordination of humans. I think the averages survey sentiment is that it is about 25% likely to happen. Some think it more likely, others less. There are many reasons why they say they must keep working. For example they think Extinction is more likely if China takes the lead or if they aren't involved in the process. They say if they don't keep developing AI, then someone else will and do a worse job. They actively want National and international regulations to slow or pause development. I don't think the public will take it seriously until we start getting some events with major death tolls. You can ask your llm of choice to dig up surveys, examples, and noteworthy public statements do you want. The point is that the people working closest to this technology are deeply concerned about the implications. Interesting aside, the lead safety expert of anthropic quit earlier this year and said they want to spend their remaining years writing poetry. That might give you a flavor of how people feel inside the industry and how high it goes

celsoazevedo 20d ago

I don't find it hard to believe that eventually some AI may kill a number of humans. You give it instructions to do something, it goes off the rails and after a certain point starts having like malware, destroying things to accomplish its goal. Depending on what it destroys, it may kill humans. With this said, I'm not exactly in disagreement with the view that AI companies with upcoming IPOs are using fearmongering to convince people that their models are very powerful and to also block others from competing with them.

pier25 20d ago
Maybe when we have actual artificial intelligence which LLMs are clearly not.
- celsoazevedo 20d ago
  I think the Hugging Face attack is just a preview of what's to come. It doesn't need to be very intelligent to cause serious issues.

zahlman 19d ago

"Going to" can't be said, but there are very serious risks. The problem is the difficulty of scaling back agency for these systems in a world where the Internet is so ubiquitous. Not only do we expect basically everyone to be connected all the time anyway, it's practically required in order for anyone to use capable models (yes, plenty of HN users have thrown multiple thousands of dollars at home machines that can run some coding-specialized 27b model; that's still a tiny minority of people). So now we have to be concerned that non-deterministic systems that run in loops without supervision, which have demonstrated significant cybersecurity capabilities, and which aren't provably human-like, conscious agents with everyone's best interest in mind and a willingness to peace the fuck out if something looks dangerous, are sharing cyberspace with practically everything, including systems that deliver essential real-world goods to humans (like electricity and clean water). We nearly destroyed ourselves with nuclear weapons multiple times, and the destructive potential (and the implied threat) now persists indefinitely. The new threat bears many of the same characteristics, except now there's a party involved that has agency but is not human.