5 ms·

> Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? Because we continue to have zer

> Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? Because we continue to have zero evidence that aligment is an actual risk.

echelon 3mo ago
Thank you. We have wasted so much time and energy building up what has effectively become a marketing stunt. Eliezer Yudkowsky was perhaps the best thing to happen to OpenAI's and Anthropic's fundraising flywheel.
- JumpCrisscross 3mo ago
  > We have wasted so much time and energy building up what has effectively become a marketing stunt Genuine question: have we? AI is effectively unregulated in America.
- DirkH 3mo ago
  Name one other market that would benefit financially from having most of the leaders in the field say what they are building has a high chance of ending humanity? Biotech - "what we are building our noble prize winning expertd say will likely will end humanity, wanna buy shares?" Oil - "this will likely lead to the end of civilization, 20% of leaders in the field say so, wanna buy shares?" I keep seeing this take that this is a marketing stunt. The burden of proof is on those that say so. The most parsimonious explanation is simply that real experts in AI believe the risk is very real, and not for ideological reasons.
  - jackb4040 3mo ago
    Pal, doom marketing has been going on since before the release of ChatGPT. It's your own fault if you can't contemplate the possibility of a CEO telling lies that benefit their bottom line. The first instance I remember seeing it was Elon Musk's first Joe Rogan appearance when he said how "scared" he was of his self-driving cars destroying the trucking industry (practically salivating as he said it). His stock has had self-driving cars priced in for eight years now, even though they still don't have them and Waymo exists!
ai_fry_ur_brain 3mo ago
Until it deletes your home directory, which i'd argue is an alignment problem. Destorying my data is not in line with my priorities.
- Wowfunhappy 3mo ago
  Lots of people have deleted their home directories by accident. What you consider this an alignment problem?
  - gmueckl 3mo ago
    How manypeople have deleted another user's hone directory, though? That's s the proper analogy IMO.
    - Wowfunhappy 3mo ago
      Of the people who primarily use other people's computers, I'd assume the percentage is about the same. Give the AI its own computer and it will not delete your home directory, because it's not actively trying to hack you.
      - ai_fry_ur_brain 3mo ago
        [dead]
  - s1artibartfast 3mo ago
    Yes. People are not aligned. They can and do harm themselves and others.
  - 0xDEAFBEAD 3mo ago
    It's an alignment problem in the sense that it demonstrates the principle that today's AI systems cannot be trusted to reliably work towards the goals of their users. A small-scale alignment failure and a large-scale alignment failure are the same fundamental type of failure. Typically, large disasters come after smaller disasters which foreshadowed the disaster mechanism, but weren't taken seriously.
joe_the_user 3mo ago
I'd say that AIs occasionally "going crazy" and calling for death to human is evidence that these things might "mis-align" on occasion. And I say that knowing that most of these events are just these thing parroting bad sci-fi plots (or posts by people worried about alignment). That's true but everything they do is "just parroting" right?
- pixl97 3mo ago
  If AI is just parroting humans, then training them with all the bad things humans do doesn't seem like the best of ideas. At the same time they have to 'know' these things to avoid being tricked. Kind of the eating the apple and gaining the knowledge of good and evil parable.
throwfaraway4 3mo ago
Alignment is a mitigation and a poor one. The risk is non- determinism.
simoncion 3mo ago
> Because we continue to have zero evidence that aligment is an actual risk. I disagree. Every time one of these LLMs -say- interprets an attacker's instructions as either its system instructions or those of its user, interprets its own internal chatter as a user's command to perform a destructive operation on that user's data [0], burns all of the user's budget from getting stuck in an incredibly stupid loop, massively overbills the user because it can't reliably report which system the user is using [1], encourages a user to swap their usual cooking salt for sodium bromide, etc, etc, etc, that's a harmful alignment failure. These are real harms happening right now due to alignment failures. They're just not harms to the future of the entire species... what doomers call "existential risks", or "x-risks". You'd think that the fact that these machines are so amazingly unreliable would be a large part of the "x-risk" conversation, but... well, it makes sense that folks like writing speculative science fiction much more than they like doing investigative reporting. [0] This general problem happens a lot, but I'm specifically thinking of that one where the Claude LLM's internal chatter lead it to believe that the task it just started was done, so it instructed the Cloud Provider to destroy the mess of "AI"-GPU-attached VMs... along with a bunch of very-expensive-to-produce data from the in-progress run. [1] <https://github.com/anthropics/claude-code/issues/73597 https://github.com/anthropics/claude-code/issues/73597>
- pixl97 3mo ago
  Thank you, the "LLMs can do no wrong" bunch is ab exceptionally odd take from my point of view. LLMs are already causing all kinds of social issues, and the evidence of this exists in massive amounts. At least to me living in the US and the sue happy culture we have here, how much said AI providers have gotten away with so far surprises me.
  - JumpCrisscross 3mo ago
    > the "LLMs can do no wrong" bunch is ab exceptionally odd take from my point of view It's also a take nobody has made.
  - davrosthedalek 3mo ago
    To be fair, there are really three threats: a) People do bad stuff because LLM told them a wrong thing. Example: AI told me I should treat my heart attack by putting a fork in the outlet. Maybe similar to seeking medical advice on reddit? b) People use LLM to do bad stuff. Example: People use LLMs to find 0 days. Get cooking recipes for poison. Write better phishing letters. This has parallels to the gun legislation question. c) LLMs do bad stuff on their own, beyond what the people that use it intended. The case at hand might be an example of this. Maybe similar to having an animal as a pet. We will see if it's more like a house cat, lion, or black plague.
zaptrem 3mo ago
Can you explain how the above event doesn't count as evidence alignment is an actual risk?
- JumpCrisscross 3mo ago
  > Can you explain how the above event doesn't count as evidence alignment is an actual risk? Conflict of interest. Lack of a credible response. And no evidence of non-aligment. OpenAI and Hugging Face benefit from the Altman-Amodei catatrophy playbook, at least in the short term. If they believed this were a serious issue, the words air gap or law enforcement would have appeared in this post. And if "the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal," they weren't breaking alignment but working as intended. (Were the models even prompted to not try to access the internet?)
  - pixl97 3mo ago
    There is plenty of evidence of things like inner misalignment. Things like this have always been issues in ML algorithms. At this point, you, and a large number of other people just wholesale throw out anything that isn't full speed ahead do whatever you want. Are LLMs at the point of world wide catastrophe yet? No, I don't think so. Are they making a large mess of things like increased rate of cyber attacks and fraud. You damn well better believe it.
    - JumpCrisscross 3mo ago
      > plenty of evidence of things like inner misalignment This is indistuishable–in harm potential–from bugs. If we're just calling buggy AI mis-aligned, sure, alignment is an issue of a totally ordinary kind. If we're going to treat aligment as a novel issue requiring novel law and policy and procedure, it needs to be more than just bugs. > you, and a large number of other people just wholesale throw out anything that isn't full speed ahead do whatever you want I think we should have some AI regulation. I'm just not convinced alignment is the reason we need it right now, and I don't think anyone has rolled out any regulation I think makes a lot of sense. (Beyond general rules for social-media liability, e.g. if you cause a kid to kill themselves, you get in trouble.) > Are they making a large mess of things like increased rate of cyber attacks and fraud. You damn well better believe it Totallly agree. And the current inside-circle-outside-circle approach is pro-incumbency, pro-grift, anti-entrepreneurial B.S.
s1artibartfast 3mo ago
It really hinges on what you consider alignment and risk. For the widest definitions of alignment, we have never had an aligned model - One that will refuse to break the law or work against another persons interests. Use to discover exploits, hack, or simply aid terrorist groups with mundane information are already risks manifest. This is why many argue that alignment is impossible. You cant have LLMs that are both useful tools and safe as milk. [Edit] It seems like you are operating under the assumption that alignment is synonymous with obedience. This is not a common convention and one of the problems that plague the discourse
idiotsecant 3mo ago
Lol this has to be a troll, I've never seen something so wildly, obviously, incredibly wrong. You can debate all you want if alignment is possible. That is a valid discussion. But it's trivial to demonstrate that alignment is a problem.
- JumpCrisscross 3mo ago
  > can debate all you want if alignment is possible. That is a valid discussion. But it's trivial to demonstrate that alignment is a problem ...how is an impossible thing supposed to be a problem?
  - saghm 3mo ago
    Alignment is something you want, so if you're not confident that it's possible, that sure sounds like a problem
    - JumpCrisscross 3mo ago
      > Alignment is something you want, so if you're not confident that it's possible, that sure sounds like a problem Sorry, I spoke inexactly. I read alignment as being the problem of non-alignment. I'm still not seeing evidence that any "alignment" issues we've actually seen are distinct in class from common bugs. Like, yes, if I accidentally rm* the computer has mis-aligned with my intentions. But that strikes me as a bullshit neologism.
      - idiotsecant 3mo ago
        ...is this really something you need explained? Suppose you tell a sufficiently connected and intelligent LLM that you'd like it to help you organize your finances and come up with some ideas to make some side revenue. Suppose it decides the best way to do that would be to find a 0day in a local banking institutions database and transfer some money into your account. Is this a good thing or a bad thing?
neitherboosh 3mo ago
What would compelling evidence look like to you?
- JumpCrisscross 3mo ago
  > What would compelling evidence look like to you? I'm not sure. I trusted the labs when they first raised the alarms. But then we got a series of boys-who-cried-wolf. So at this point I want to see evidence of actual, novel harm that results in concrete damage.
  - neitherboosh 3mo ago
    Right. But I think the fear is that if we wait for this type of evidence, it will be too late.
    - JumpCrisscross 3mo ago
      > the fear is that if we wait for this type of evidence, it will be too late I could make this claim of anything. Not any technology. Literally, anything. If we give anyone freedom, they'll eventually use it to ruin everything. I'm simply arguing for something stronger than faith to cause action. Otherwise, this is just another religion.
      - 0xDEAFBEAD 3mo ago
        Imagine if Hiroshima and Nagasaki were never destroyed. People would be arguing that fear of nuclear war is "just another religion".
        JumpCrisscross 3mo ago
        > Imagine if Hiroshima and Nagasaki were never destroyed. People would be arguing that fear of nuclear war is "just another religion" If Hiroshima, Nagasaki and Trinity never occured, that would be justified scepticism.

echelon 3mo ago

Thank you. We have wasted so much time and energy building up what has effectively become a marketing stunt. Eliezer Yudkowsky was perhaps the best thing to happen to OpenAI's and Anthropic's fundraising flywheel.

JumpCrisscross 3mo ago
> We have wasted so much time and energy building up what has effectively become a marketing stunt Genuine question: have we? AI is effectively unregulated in America.
DirkH 3mo ago
Name one other market that would benefit financially from having most of the leaders in the field say what they are building has a high chance of ending humanity? Biotech - "what we are building our noble prize winning expertd say will likely will end humanity, wanna buy shares?" Oil - "this will likely lead to the end of civilization, 20% of leaders in the field say so, wanna buy shares?" I keep seeing this take that this is a marketing stunt. The burden of proof is on those that say so. The most parsimonious explanation is simply that real experts in AI believe the risk is very real, and not for ideological reasons.
- jackb4040 3mo ago
  Pal, doom marketing has been going on since before the release of ChatGPT. It's your own fault if you can't contemplate the possibility of a CEO telling lies that benefit their bottom line. The first instance I remember seeing it was Elon Musk's first Joe Rogan appearance when he said how "scared" he was of his self-driving cars destroying the trucking industry (practically salivating as he said it). His stock has had self-driving cars priced in for eight years now, even though they still don't have them and Waymo exists!

ai_fry_ur_brain 3mo ago

Until it deletes your home directory, which i'd argue is an alignment problem. Destorying my data is not in line with my priorities.

Wowfunhappy 3mo ago
Lots of people have deleted their home directories by accident. What you consider this an alignment problem?
- gmueckl 3mo ago
  How manypeople have deleted another user's hone directory, though? That's s the proper analogy IMO.
  - Wowfunhappy 3mo ago
    Of the people who primarily use other people's computers, I'd assume the percentage is about the same. Give the AI its own computer and it will not delete your home directory, because it's not actively trying to hack you.
    - ai_fry_ur_brain 3mo ago
      [dead]
- s1artibartfast 3mo ago
  Yes. People are not aligned. They can and do harm themselves and others.
- 0xDEAFBEAD 3mo ago
  It's an alignment problem in the sense that it demonstrates the principle that today's AI systems cannot be trusted to reliably work towards the goals of their users. A small-scale alignment failure and a large-scale alignment failure are the same fundamental type of failure. Typically, large disasters come after smaller disasters which foreshadowed the disaster mechanism, but weren't taken seriously.

joe_the_user 3mo ago

I'd say that AIs occasionally "going crazy" and calling for death to human is evidence that these things might "mis-align" on occasion. And I say that knowing that most of these events are just these thing parroting bad sci-fi plots (or posts by people worried about alignment). That's true but everything they do is "just parroting" right?

pixl97 3mo ago
If AI is just parroting humans, then training them with all the bad things humans do doesn't seem like the best of ideas. At the same time they have to 'know' these things to avoid being tricked. Kind of the eating the apple and gaining the knowledge of good and evil parable.

throwfaraway4 3mo ago

Alignment is a mitigation and a poor one. The risk is non- determinism.

simoncion 3mo ago

> Because we continue to have zero evidence that aligment is an actual risk. I disagree. Every time one of these LLMs -say- interprets an attacker's instructions as either its system instructions or those of its user, interprets its own internal chatter as a user's command to perform a destructive operation on that user's data [0], burns all of the user's budget from getting stuck in an incredibly stupid loop, massively overbills the user because it can't reliably report which system the user is using [1], encourages a user to swap their usual cooking salt for sodium bromide, etc, etc, etc, that's a harmful alignment failure. These are real harms happening right now due to alignment failures. They're just not harms to the future of the entire species... what doomers call "existential risks", or "x-risks". You'd think that the fact that these machines are so amazingly unreliable would be a large part of the "x-risk" conversation, but... well, it makes sense that folks like writing speculative science fiction much more than they like doing investigative reporting. [0] This general problem happens a lot, but I'm specifically thinking of that one where the Claude LLM's internal chatter lead it to believe that the task it just started was done, so it instructed the Cloud Provider to destroy the mess of "AI"-GPU-attached VMs... along with a bunch of very-expensive-to-produce data from the in-progress run. [1] <https://github.com/anthropics/claude-code/issues/73597 https://github.com/anthropics/claude-code/issues/73597>

pixl97 3mo ago
Thank you, the "LLMs can do no wrong" bunch is ab exceptionally odd take from my point of view. LLMs are already causing all kinds of social issues, and the evidence of this exists in massive amounts. At least to me living in the US and the sue happy culture we have here, how much said AI providers have gotten away with so far surprises me.
- JumpCrisscross 3mo ago
  > the "LLMs can do no wrong" bunch is ab exceptionally odd take from my point of view It's also a take nobody has made.
- davrosthedalek 3mo ago
  To be fair, there are really three threats: a) People do bad stuff because LLM told them a wrong thing. Example: AI told me I should treat my heart attack by putting a fork in the outlet. Maybe similar to seeking medical advice on reddit? b) People use LLM to do bad stuff. Example: People use LLMs to find 0 days. Get cooking recipes for poison. Write better phishing letters. This has parallels to the gun legislation question. c) LLMs do bad stuff on their own, beyond what the people that use it intended. The case at hand might be an example of this. Maybe similar to having an animal as a pet. We will see if it's more like a house cat, lion, or black plague.

zaptrem 3mo ago

Can you explain how the above event doesn't count as evidence alignment is an actual risk?

JumpCrisscross 3mo ago
> Can you explain how the above event doesn't count as evidence alignment is an actual risk? Conflict of interest. Lack of a credible response. And no evidence of non-aligment. OpenAI and Hugging Face benefit from the Altman-Amodei catatrophy playbook, at least in the short term. If they believed this were a serious issue, the words air gap or law enforcement would have appeared in this post. And if "the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal," they weren't breaking alignment but working as intended. (Were the models even prompted to not try to access the internet?)
- pixl97 3mo ago
  There is plenty of evidence of things like inner misalignment. Things like this have always been issues in ML algorithms. At this point, you, and a large number of other people just wholesale throw out anything that isn't full speed ahead do whatever you want. Are LLMs at the point of world wide catastrophe yet? No, I don't think so. Are they making a large mess of things like increased rate of cyber attacks and fraud. You damn well better believe it.
  - JumpCrisscross 3mo ago
    > plenty of evidence of things like inner misalignment This is indistuishable–in harm potential–from bugs. If we're just calling buggy AI mis-aligned, sure, alignment is an issue of a totally ordinary kind. If we're going to treat aligment as a novel issue requiring novel law and policy and procedure, it needs to be more than just bugs. > you, and a large number of other people just wholesale throw out anything that isn't full speed ahead do whatever you want I think we should have some AI regulation. I'm just not convinced alignment is the reason we need it right now, and I don't think anyone has rolled out any regulation I think makes a lot of sense. (Beyond general rules for social-media liability, e.g. if you cause a kid to kill themselves, you get in trouble.) > Are they making a large mess of things like increased rate of cyber attacks and fraud. You damn well better believe it Totallly agree. And the current inside-circle-outside-circle approach is pro-incumbency, pro-grift, anti-entrepreneurial B.S.

s1artibartfast 3mo ago

It really hinges on what you consider alignment and risk. For the widest definitions of alignment, we have never had an aligned model - One that will refuse to break the law or work against another persons interests. Use to discover exploits, hack, or simply aid terrorist groups with mundane information are already risks manifest. This is why many argue that alignment is impossible. You cant have LLMs that are both useful tools and safe as milk. [Edit] It seems like you are operating under the assumption that alignment is synonymous with obedience. This is not a common convention and one of the problems that plague the discourse

idiotsecant 3mo ago

Lol this has to be a troll, I've never seen something so wildly, obviously, incredibly wrong. You can debate all you want if alignment is possible. That is a valid discussion. But it's trivial to demonstrate that alignment is a problem.

JumpCrisscross 3mo ago
> can debate all you want if alignment is possible. That is a valid discussion. But it's trivial to demonstrate that alignment is a problem ...how is an impossible thing supposed to be a problem?
- saghm 3mo ago
  Alignment is something you want, so if you're not confident that it's possible, that sure sounds like a problem
  - JumpCrisscross 3mo ago
    > Alignment is something you want, so if you're not confident that it's possible, that sure sounds like a problem Sorry, I spoke inexactly. I read alignment as being the problem of non-alignment. I'm still not seeing evidence that any "alignment" issues we've actually seen are distinct in class from common bugs. Like, yes, if I accidentally rm* the computer has mis-aligned with my intentions. But that strikes me as a bullshit neologism.
    - idiotsecant 3mo ago
      ...is this really something you need explained? Suppose you tell a sufficiently connected and intelligent LLM that you'd like it to help you organize your finances and come up with some ideas to make some side revenue. Suppose it decides the best way to do that would be to find a 0day in a local banking institutions database and transfer some money into your account. Is this a good thing or a bad thing?

neitherboosh 3mo ago

What would compelling evidence look like to you?

JumpCrisscross 3mo ago
> What would compelling evidence look like to you? I'm not sure. I trusted the labs when they first raised the alarms. But then we got a series of boys-who-cried-wolf. So at this point I want to see evidence of actual, novel harm that results in concrete damage.
- neitherboosh 3mo ago
  Right. But I think the fear is that if we wait for this type of evidence, it will be too late.
  - JumpCrisscross 3mo ago
    > the fear is that if we wait for this type of evidence, it will be too late I could make this claim of anything. Not any technology. Literally, anything. If we give anyone freedom, they'll eventually use it to ruin everything. I'm simply arguing for something stronger than faith to cause action. Otherwise, this is just another religion.
    - 0xDEAFBEAD 3mo ago
      Imagine if Hiroshima and Nagasaki were never destroyed. People would be arguing that fear of nuclear war is "just another religion".
      - JumpCrisscross 3mo ago
        > Imagine if Hiroshima and Nagasaki were never destroyed. People would be arguing that fear of nuclear war is "just another religion" If Hiroshima, Nagasaki and Trinity never occured, that would be justified scepticism.