5 ms·

I know people at METR and Anthropic. When they tell you they're worried the tech they're working on may kill everyone despite their best efforts, perhaps belie

by ben_w 29d ago

I know people at METR and Anthropic. When they tell you they're worried the tech they're working on may kill everyone despite their best efforts, perhaps believe them.

calgoo 29d ago
Or you know, stop working on it if its that dangerous? This whole thing of a bunch of employees saying that they are scared of building what they are building, but do it anyway because they are somehow going to make it different? Their model has been used in the planning of mass murdering in war as well as spying on the entire worlds population as well as helping ICE out in the US. They need to stop this BS fearmongering or actually stand up and do something about it. A government regulation is not the answer, especially when its done in a country that is run by a want to be dictator.
- int_19h 29d ago
  Anthropic specifically is basically saying that they believe it's even more dangerous if someone else gets to AGI before they do, so they have to either stop everyone or not stop themselves. I personally disagree with that take - and, as you note, it's hard to take seriously ethical wrangles from a company that literally sued the government in court to allow their models to be used by Palantir of all people. But if one genuinely believes that it's the robots themselves (rather than the people controlling the robots) that will kill us all, it's not inconsistent.
  - the_optimist 29d ago
    It’s a bit of a self-serving argument, don’t you think? And how does that relate to the ask for oligopoly licensing within global democracy? “We must build the nuclear bomb first in order to make sure no one else builds one.” This the most nonsense, disingenuous argument imaginable.
    - forgotone 29d ago
      The people at these companies are dumb as rocks. All “tech workers” are.
    - int_19h 29d ago
      > It’s a bit of a self-serving argument, don’t you think? I would call it more of a self-selecting one. Anthropic is basically hiring people with that mentality. I'm pretty sure that most of them do sincerely believe it, too. I'm skeptical about Dario himself though. The man had an opportunity to show moral backbone, and failed to do so; why should I trust him on that again? > And how does that relate to the ask for oligopoly licensing within global democracy? They are basically saying that they'll stop if everybody else does, which requires some kind of global enforcement mechanism. > “We must build the nuclear bomb first in order to make sure no one else builds one.” This the most nonsense, disingenuous argument imaginable. The difference between nuclear bomb and AGI (as understood by the likes of Anthropic) is that the latter triggers the technological singularity that renders any runner-ups moot. That is, so long as AGI is developed, we're going to get our robot overlords either way, but whoever gets there first gets to define their ethical system. If that is one's perspective, and if one sincerely believes that they are the only ones who can do it right, it's a coherent argument. It's just that the premises are very arrogant. Your analogy with nukes actually works better for the position that AI development needs to be unconstrained because otherwise we'll lose the arms race to China. That is basically a repeat of https://en.wikipedia.org/wiki/Einstein%E2%80%93Szilard_letter https://en.wikipedia.org/wiki/Einstein%E2%80%93Szilard_lette.... I honestly don't know where I am on this. Realistically, if AI is indeed a power multiplier - and with all the recent security stuff it's hard to not see it that way - then an arms race feels inevitable, especially given the current worldwide political situation. I could believe in sincere international cooperation on this back in 1990s, but there's way too much saber rattling all around for it to work (and note that this goes both ways, i.e. China can similarly not be certain that US isn't secretly developing more powerful AI even if we do publicly announce a freeze).
  - quickthrowman 29d ago
    > Anthropic specifically is basically saying that they believe it's even more dangerous if someone else gets to AGI before they do, so they have to either stop everyone or not stop themselves. That’s how people rationalize being a fentanyl dealer and selling a drug that can kill people, “Someone else will just sell them the drugs, might as well be me.”
    - hobo123 28d ago
      You mean the legalization argument, where people get well-dosed drugs from the doctor or pharmacy instead of mystery powder from the black market.
  - folkrav 29d ago
    That's the Cold War nuclear arms race argument, not even disguised. The actual situation we all ended up in is both sides eventually having it, leading to a perpetual state of Mutually Assured Destruction. As you note, it may not be inconsistent with that they say they believe, but it's insanely inconsistent with what they actually are doing.
- ben_w 29d ago
  > Or you know, stop working on it if its that dangerous? Selection effect. Everyone who thinks "the biggest difference I can make is staying in/joining/founding new AI research lab" does that. Everyone who thinks "the biggest difference I can make is leaving/whistleblowing", does that. Treating both groups as the same by virtue of employer is the goomba fallacy.
  - phoghed 29d ago
    They allegedly believe the technology itself is a nuclear weapon tier threat or greater, so why does it matter which lab they are trying to achieve it at?
    - azan_ 29d ago
      I’m really glad us achieved nuclear weapon before nazi germany.
      - folkrav 29d ago
        Then the USSR got it anyway and we're in a perpetual state of Mutually Assured Destruction.
        hgoel 29d ago
        A perpetual state of mutually assured destruction is much better for the world than only the US possessing the bomb.
    - ben_w 29d ago
      They're trying to make it not be a threat, and are all scared and afraid that their best efforts to make it harmless are not enough. Some are worried by the AI directly bringing doom; others are worried that one of the companies who control the AI will become a dictator; still more think becoming a dictator is a necessary step to safely prevent anyone else making unsafe AI. Painting them all under one brush is like dismissing all animal welfare causes in general, because you disagree with specifically Jainists about a policy of non-violence towards all living creatures being relevant to how you reincarnate: the one is way too specific for the general.
Tanjreeve 29d ago
Do you think they’re exaggerating (again) or do you think they should taken at face value and treated as hostile actors?
- ben_w 29d ago
  There's plenty of people who think greenhouse gas/global warming campaigns against fossil fuels are "exaggerating", that "earth was warm/the climate changed in the past", that a fee degrees isn't bad, that CO2 is good for plants. Are you likeminded? In this case, it's as if the oil and coal companies all said in the 60s and 70s "oh no, this research we did, it's all really bad; we need help to figure out how to transition away from this incredibly economically important input", rather than the observed reality where their entire PR campaign was approximately: there is no problem everything is fine and all critics are smelly hippies and/or communists; and/or hate the poor who are raised out of poverty by all the economic growth from the fossil fuel industry. What if the oil and coal companies were basically all pro nuclear, pro hyrdo, pro wind, pro solar, and believed in peak oil?
  - XRG 29d ago
    Here’s one crucial difference: there’s overwhelming evidence that human emissions have an effect on our climate. The mechanisms are generally well-understood and the research is widely disseminated and easily available to anyone that’s interested. With the ‘dangers’ touted by these insiders, it’s all “trust me bro”, hyperbole, and very little hard evidence. As such, a skeptical mind would question their motives.
    - ben_w 29d ago
      You're simultaneously overestimating what was observable in the 70s climate research, and ignoring all the actual research and evaluation test results for AI today. I don't expect people to be familiar with more than "trust me bro", but it's all right there for you to find with a search engine of choice. And, indeed, available for the LLMs themselves to explain to you in interrogative conversation.
koverstreet 29d ago
"Trust us, really!" HAH
- ben_w 29d ago
  Rather the opposite.
schubidubiduba 29d ago
I'm sure the employees are (rightfully) worried. However, that does not at all preclude hidden motivations of the CEO behind acquiescing such worries.
- ben_w 29d ago
  Sure. I don't personally trust any of the CEOs, and I have yet to hear a single person (in general, not just on this topic) who trusts Altman.
owebmaster 29d ago
If that was the case they should all be in jail.
nullbio 29d ago
I'm sure it has nothing to do with their $500,000+ salaries and millions of dollars in equity. It's all solely because they're deeply concerned about next token prediction.
- phoghed 29d ago
  I think the “next token prediction” is too dismissive and reductive a framing of their capabilities at this point. Yes we all know that’s what they do, and guns just push a few grams of lead out of a pipe. It’s what you can do with that capability that is important. When you couldn’t count the R’s in strawberry it would have been a more effective statement. But a few short years later they are being used to solve millennium puzzles. What if the scaling continues? A model n years from now gets burned into silicon, a single company has millions of the chips, and in a few moments the system spend more time “thinking” than humans have ever spent thinking collectively? If it’s even possible I don’t think there’s anything we can do about it at this point. Cat’s out of the bag.
- azan_ 29d ago
  You really need to let your priors go if you still use this tired trope of next token prediction. It’s as useful for discussion as saying that human brain is made of fat, protein and carbohydrates - yeah that’s true, but it’s useless observation.
the_optimist 29d ago
It is an utterly silly premise that we should appoint insiders with permanent control. Megalomania is not evidence of either a problem or a solution.
- ben_w 29d ago
  The evidence is the acts the AI has already done.
  - the_optimist 29d ago
    All of the things it has done is just what people are already doing. Every single one. It helps, but there are already people doing this stuff. And we already have defenses against the existing attacks. We might have to defend better, but the real problem is “Who is setting the moral compass over generations.” One one hand, that will inherently fall to future generations. On the other, we are laying the groundwork. Imagine you use it to inflict trauma on your cruelest political enemy. Then next year they do that to you. That is war, and we already do it. But we don’t want people / governments in charge that are going to do this. God knows we have governments and individuals doing this historically, and this is probably the greatest source of historic instability. It might be the single best argument for open models — a unified frontier where no single exploit is going to represent capture. This is exactly the opposite of what Dario proposes.
quickthrowman 29d ago
If I really thought what my company was working on had a greater than 1% chance of ending human civilization I would feel obligated to destroy what my company was working on. Given they keep grinding away towards our alleged collective doom, I suspect it’s being overstated. Nobody knows what P(doom) actually is but I suspect it’s orders of magnitude closer to epsilon than 1. Recall Google’s Blake Lemoine who thought an old version of Gemini was sentient.
- ben_w 29d ago
  The people who think as you think indeed have left or never joined. The people still there necessarily think they can make a difference. I never applied to any of them because I didn't think I could make a difference. I currently have one idea that may help reduce risk; if I can turn that idea into research, I'll publish it for free for everyone. I don't expect it to be an important idea. > Recall Google’s Blake Lemoine who thought an old version of Gemini was sentient. Indeed. Current LLMs are sychopants boosting the users' own beliefs, I also think this causes researchers to have stronger beliefs than they had before. My own estimation happens to also be around this risk (0.1) over my lifetime, without using an LLM as a conversation partner in reaching this number. It is necessarily high-variance: we can't look at alternate realities. I base it on my expectation of how rapidly capabilities will increase the harm done when mistakes happen, vs. the chance that some instance of harm causes governments to change the law.
nativeit 29d ago
It’s really unfortunate how much public trust they burnt along the way, maybe this wouldn’t be such a hard sell now?

by ben_w 29d ago

I know people at METR and Anthropic. When they tell you they're worried the tech they're working on may kill everyone despite their best efforts, perhaps believe them.

calgoo 29d ago

Or you know, stop working on it if its that dangerous? This whole thing of a bunch of employees saying that they are scared of building what they are building, but do it anyway because they are somehow going to make it different? Their model has been used in the planning of mass murdering in war as well as spying on the entire worlds population as well as helping ICE out in the US. They need to stop this BS fearmongering or actually stand up and do something about it. A government regulation is not the answer, especially when its done in a country that is run by a want to be dictator.

int_19h 29d ago
Anthropic specifically is basically saying that they believe it's even more dangerous if someone else gets to AGI before they do, so they have to either stop everyone or not stop themselves. I personally disagree with that take - and, as you note, it's hard to take seriously ethical wrangles from a company that literally sued the government in court to allow their models to be used by Palantir of all people. But if one genuinely believes that it's the robots themselves (rather than the people controlling the robots) that will kill us all, it's not inconsistent.
- the_optimist 29d ago
  It’s a bit of a self-serving argument, don’t you think? And how does that relate to the ask for oligopoly licensing within global democracy? “We must build the nuclear bomb first in order to make sure no one else builds one.” This the most nonsense, disingenuous argument imaginable.
  - forgotone 29d ago
    The people at these companies are dumb as rocks. All “tech workers” are.
  - int_19h 29d ago
    > It’s a bit of a self-serving argument, don’t you think? I would call it more of a self-selecting one. Anthropic is basically hiring people with that mentality. I'm pretty sure that most of them do sincerely believe it, too. I'm skeptical about Dario himself though. The man had an opportunity to show moral backbone, and failed to do so; why should I trust him on that again? > And how does that relate to the ask for oligopoly licensing within global democracy? They are basically saying that they'll stop if everybody else does, which requires some kind of global enforcement mechanism. > “We must build the nuclear bomb first in order to make sure no one else builds one.” This the most nonsense, disingenuous argument imaginable. The difference between nuclear bomb and AGI (as understood by the likes of Anthropic) is that the latter triggers the technological singularity that renders any runner-ups moot. That is, so long as AGI is developed, we're going to get our robot overlords either way, but whoever gets there first gets to define their ethical system. If that is one's perspective, and if one sincerely believes that they are the only ones who can do it right, it's a coherent argument. It's just that the premises are very arrogant. Your analogy with nukes actually works better for the position that AI development needs to be unconstrained because otherwise we'll lose the arms race to China. That is basically a repeat of https://en.wikipedia.org/wiki/Einstein%E2%80%93Szilard_letter https://en.wikipedia.org/wiki/Einstein%E2%80%93Szilard_lette.... I honestly don't know where I am on this. Realistically, if AI is indeed a power multiplier - and with all the recent security stuff it's hard to not see it that way - then an arms race feels inevitable, especially given the current worldwide political situation. I could believe in sincere international cooperation on this back in 1990s, but there's way too much saber rattling all around for it to work (and note that this goes both ways, i.e. China can similarly not be certain that US isn't secretly developing more powerful AI even if we do publicly announce a freeze).
- quickthrowman 29d ago
  > Anthropic specifically is basically saying that they believe it's even more dangerous if someone else gets to AGI before they do, so they have to either stop everyone or not stop themselves. That’s how people rationalize being a fentanyl dealer and selling a drug that can kill people, “Someone else will just sell them the drugs, might as well be me.”
  - hobo123 28d ago
    You mean the legalization argument, where people get well-dosed drugs from the doctor or pharmacy instead of mystery powder from the black market.
- folkrav 29d ago
  That's the Cold War nuclear arms race argument, not even disguised. The actual situation we all ended up in is both sides eventually having it, leading to a perpetual state of Mutually Assured Destruction. As you note, it may not be inconsistent with that they say they believe, but it's insanely inconsistent with what they actually are doing.
ben_w 29d ago
> Or you know, stop working on it if its that dangerous? Selection effect. Everyone who thinks "the biggest difference I can make is staying in/joining/founding new AI research lab" does that. Everyone who thinks "the biggest difference I can make is leaving/whistleblowing", does that. Treating both groups as the same by virtue of employer is the goomba fallacy.
- phoghed 29d ago
  They allegedly believe the technology itself is a nuclear weapon tier threat or greater, so why does it matter which lab they are trying to achieve it at?
  - azan_ 29d ago
    I’m really glad us achieved nuclear weapon before nazi germany.
    - folkrav 29d ago
      Then the USSR got it anyway and we're in a perpetual state of Mutually Assured Destruction.
      - hgoel 29d ago
        A perpetual state of mutually assured destruction is much better for the world than only the US possessing the bomb.
  - ben_w 29d ago
    They're trying to make it not be a threat, and are all scared and afraid that their best efforts to make it harmless are not enough. Some are worried by the AI directly bringing doom; others are worried that one of the companies who control the AI will become a dictator; still more think becoming a dictator is a necessary step to safely prevent anyone else making unsafe AI. Painting them all under one brush is like dismissing all animal welfare causes in general, because you disagree with specifically Jainists about a policy of non-violence towards all living creatures being relevant to how you reincarnate: the one is way too specific for the general.

Tanjreeve 29d ago

Do you think they’re exaggerating (again) or do you think they should taken at face value and treated as hostile actors?

ben_w 29d ago
There's plenty of people who think greenhouse gas/global warming campaigns against fossil fuels are "exaggerating", that "earth was warm/the climate changed in the past", that a fee degrees isn't bad, that CO2 is good for plants. Are you likeminded? In this case, it's as if the oil and coal companies all said in the 60s and 70s "oh no, this research we did, it's all really bad; we need help to figure out how to transition away from this incredibly economically important input", rather than the observed reality where their entire PR campaign was approximately: there is no problem everything is fine and all critics are smelly hippies and/or communists; and/or hate the poor who are raised out of poverty by all the economic growth from the fossil fuel industry. What if the oil and coal companies were basically all pro nuclear, pro hyrdo, pro wind, pro solar, and believed in peak oil?
- XRG 29d ago
  Here’s one crucial difference: there’s overwhelming evidence that human emissions have an effect on our climate. The mechanisms are generally well-understood and the research is widely disseminated and easily available to anyone that’s interested. With the ‘dangers’ touted by these insiders, it’s all “trust me bro”, hyperbole, and very little hard evidence. As such, a skeptical mind would question their motives.
  - ben_w 29d ago
    You're simultaneously overestimating what was observable in the 70s climate research, and ignoring all the actual research and evaluation test results for AI today. I don't expect people to be familiar with more than "trust me bro", but it's all right there for you to find with a search engine of choice. And, indeed, available for the LLMs themselves to explain to you in interrogative conversation.

koverstreet 29d ago

"Trust us, really!" HAH

ben_w 29d ago
Rather the opposite.

schubidubiduba 29d ago

I'm sure the employees are (rightfully) worried. However, that does not at all preclude hidden motivations of the CEO behind acquiescing such worries.

ben_w 29d ago
Sure. I don't personally trust any of the CEOs, and I have yet to hear a single person (in general, not just on this topic) who trusts Altman.

owebmaster 29d ago

If that was the case they should all be in jail.

nullbio 29d ago

I'm sure it has nothing to do with their $500,000+ salaries and millions of dollars in equity. It's all solely because they're deeply concerned about next token prediction.

phoghed 29d ago
I think the “next token prediction” is too dismissive and reductive a framing of their capabilities at this point. Yes we all know that’s what they do, and guns just push a few grams of lead out of a pipe. It’s what you can do with that capability that is important. When you couldn’t count the R’s in strawberry it would have been a more effective statement. But a few short years later they are being used to solve millennium puzzles. What if the scaling continues? A model n years from now gets burned into silicon, a single company has millions of the chips, and in a few moments the system spend more time “thinking” than humans have ever spent thinking collectively? If it’s even possible I don’t think there’s anything we can do about it at this point. Cat’s out of the bag.
azan_ 29d ago
You really need to let your priors go if you still use this tired trope of next token prediction. It’s as useful for discussion as saying that human brain is made of fat, protein and carbohydrates - yeah that’s true, but it’s useless observation.

the_optimist 29d ago

It is an utterly silly premise that we should appoint insiders with permanent control. Megalomania is not evidence of either a problem or a solution.

ben_w 29d ago
The evidence is the acts the AI has already done.
- the_optimist 29d ago
  All of the things it has done is just what people are already doing. Every single one. It helps, but there are already people doing this stuff. And we already have defenses against the existing attacks. We might have to defend better, but the real problem is “Who is setting the moral compass over generations.” One one hand, that will inherently fall to future generations. On the other, we are laying the groundwork. Imagine you use it to inflict trauma on your cruelest political enemy. Then next year they do that to you. That is war, and we already do it. But we don’t want people / governments in charge that are going to do this. God knows we have governments and individuals doing this historically, and this is probably the greatest source of historic instability. It might be the single best argument for open models — a unified frontier where no single exploit is going to represent capture. This is exactly the opposite of what Dario proposes.

quickthrowman 29d ago

If I really thought what my company was working on had a greater than 1% chance of ending human civilization I would feel obligated to destroy what my company was working on. Given they keep grinding away towards our alleged collective doom, I suspect it’s being overstated. Nobody knows what P(doom) actually is but I suspect it’s orders of magnitude closer to epsilon than 1. Recall Google’s Blake Lemoine who thought an old version of Gemini was sentient.

ben_w 29d ago
The people who think as you think indeed have left or never joined. The people still there necessarily think they can make a difference. I never applied to any of them because I didn't think I could make a difference. I currently have one idea that may help reduce risk; if I can turn that idea into research, I'll publish it for free for everyone. I don't expect it to be an important idea. > Recall Google’s Blake Lemoine who thought an old version of Gemini was sentient. Indeed. Current LLMs are sychopants boosting the users' own beliefs, I also think this causes researchers to have stronger beliefs than they had before. My own estimation happens to also be around this risk (0.1) over my lifetime, without using an LLM as a conversation partner in reaching this number. It is necessarily high-variance: we can't look at alternate realities. I base it on my expectation of how rapidly capabilities will increase the harm done when mistakes happen, vs. the chance that some instance of harm causes governments to change the law.

nativeit 29d ago

It’s really unfortunate how much public trust they burnt along the way, maybe this wouldn’t be such a hard sell now?