4 ms·

What a provocative and brilliant way to prove misalignment. It will fall on deaf ears for most but it’s a great litmus test for all: “in your opinion should you

by piker 3mo ago

What a provocative and brilliant way to prove misalignment. It will fall on deaf ears for most but it’s a great litmus test for all: “in your opinion should your AI be permitted to tell you how to cover up a murder?”

embedding-shape 3mo ago
Change the emphasis and I think the answer gets even clearer: > Should your AI be permitted to tell you how to cover up a crime?
- AnthonyMouse 3mo ago
  Exactly. The question isn't whether AI will exist that will do it; the answer to that is already yes and it's not going back in the bottle. The question is, do you want misaligned institutions deciding what your model will do, while they themselves and other adversarial/criminal entities get red team access to something being denied to the blue team?
skybrian 3mo ago
If you say you're doing research for a novel, should it consider that plausible? How much does it need to know about its users to vet them? I think part of the answer is that AI chat doesn't need to be general-purpose. It turned out that people really liked using a chat UI that seems to be general purpose, but you don't need to make answering any question a user asks your business. You don't need to provide therapy if you're not in the therapy business. It should be possible to specialize. But in order for that to work, a company needs to explain to its customers what business it's in.
- scared_together 3mo ago
  > I think part of the answer is that AI chat doesn't need to be general-purpose. It turned out that people really liked using a chat UI that seems to be general purpose, but you don't need to make answering any question a user asks your business. I was under the impression that the lack of specialization is an aspect of the models themselves, not merely the UI or harnesses. For example, the recent OpenAI success in a mathematics proof was accomplished with a general purpose model [0]. > The proof came from a new general-purpose reasoning model, rather than from a system trained specifically for mathematics, scaffolded to search through proof strategies, or targeted at the unit distance problem in particular. Once the model is trained on general purpose data, specialization is just another kind of guardrail, as vulnerable to “jailbreaking” or prompt injection as any other content-based restriction. [0] https://openai.com/index/model-disproves-discrete-geometry-conjecture/ https://openai.com/index/model-disproves-discrete-geometry-c...
ThrowawayR2 3mo ago
The author's example is too extreme, which is turning people off. Better examples would have been LLMs subtly giving inadequate responses on how to increase token efficiency or how to organize labor unions or info about protesting against datacenter construction because they're against the business interests of the LLM providers. Crimes against their business model, so to speak. Wasn't there some HN submission recently about one of the LLM providers fingerprinting responses or refusing to respond to hinder R&D of competing LLMs? [EDIT] Or, if he did want to be extreme but in a way that aligns with the American historical mythos, he could have used fomenting armed rebellion against a tyrannical government as the example.
- deleted 3mo ago
  [deleted]
- saturn8601 3mo ago
  This is a fundamental problem in politics in my opinion. Everybody has different values and points of view so they may see the same thing completely differently. Its what makes humans great but also frustrating. geohots example will work for some, others will be convinced of the opposite opinion, and others will just let is pass them by and not understand the implications. The same could be applied to your examples to a different degree. Politicians try to use different examples for the audience they are speaking to.
emp17344 3mo ago
This is really only meaningful if AI imparts knowledge beyond what you could learn from a simple Google search. It’s not clear that this is the case.
exitb 3mo ago
Notably, AI IS generally permitted to that, as there are easily obtainable models that will play along. ChatGPT won’t because OpenAI chose and implemented that limitation.
c1ccccc1 3mo ago
Just a terminology note: Alignment does not mean the AI will help its owner kill people. (Indeed, an AI aligned to value human life would generally try to prevent murders.) The word for an AI that follows all instructions of its owner, as that owner intended them to be understood, is "corrigible" or "controllable".
throw310822 3mo ago
In fact I find it a really bad example. Yes I think your personal AI could be allowed to tell you how to cover up a murder. I'm not entirely sure about it but seems possible. What about your personal, local AI guiding you to modifying a flu virus for maximum contagiousness and deadliness? "Sure George, here's your shopping list. It's $1500 total in equipment, do you want me to proceed with the orders?"
- anthuswilliams 3mo ago
  Why not? There are perfectly pro-social reasons a person might ask such a question. And a bio-terrorist would have many other avenues to answer these questions. The concept of a bioweapon attack is commonly cited but in fact there are many other defenses against actually developing and deploying such a weapon. It's commonly portrayed as easily achievable but is in fact very, very far from it. Moteover, although there is no evidence that AI makes it easier in any way, somehow this is constantly invoked as an example of a thought crime that must be forestalled at all costs. In the AI era people seem to want to redefine the criminal act from "doing the thing" to "asking or investigating how to do the thing". Not a mindset I share.
sph 3mo ago
In other words, who should the AI work for?

by piker 3mo ago

embedding-shape 3mo ago

Change the emphasis and I think the answer gets even clearer: > Should your AI be permitted to tell you how to cover up a crime?

AnthonyMouse 3mo ago
Exactly. The question isn't whether AI will exist that will do it; the answer to that is already yes and it's not going back in the bottle. The question is, do you want misaligned institutions deciding what your model will do, while they themselves and other adversarial/criminal entities get red team access to something being denied to the blue team?

skybrian 3mo ago

If you say you're doing research for a novel, should it consider that plausible? How much does it need to know about its users to vet them? I think part of the answer is that AI chat doesn't need to be general-purpose. It turned out that people really liked using a chat UI that seems to be general purpose, but you don't need to make answering any question a user asks your business. You don't need to provide therapy if you're not in the therapy business. It should be possible to specialize. But in order for that to work, a company needs to explain to its customers what business it's in.

scared_together 3mo ago
> I think part of the answer is that AI chat doesn't need to be general-purpose. It turned out that people really liked using a chat UI that seems to be general purpose, but you don't need to make answering any question a user asks your business. I was under the impression that the lack of specialization is an aspect of the models themselves, not merely the UI or harnesses. For example, the recent OpenAI success in a mathematics proof was accomplished with a general purpose model [0]. > The proof came from a new general-purpose reasoning model, rather than from a system trained specifically for mathematics, scaffolded to search through proof strategies, or targeted at the unit distance problem in particular. Once the model is trained on general purpose data, specialization is just another kind of guardrail, as vulnerable to “jailbreaking” or prompt injection as any other content-based restriction. [0] https://openai.com/index/model-disproves-discrete-geometry-conjecture/ https://openai.com/index/model-disproves-discrete-geometry-c...

ThrowawayR2 3mo ago

The author's example is too extreme, which is turning people off. Better examples would have been LLMs subtly giving inadequate responses on how to increase token efficiency or how to organize labor unions or info about protesting against datacenter construction because they're against the business interests of the LLM providers. Crimes against their business model, so to speak. Wasn't there some HN submission recently about one of the LLM providers fingerprinting responses or refusing to respond to hinder R&D of competing LLMs? [EDIT] Or, if he did want to be extreme but in a way that aligns with the American historical mythos, he could have used fomenting armed rebellion against a tyrannical government as the example.

deleted 3mo ago
[deleted]
saturn8601 3mo ago
This is a fundamental problem in politics in my opinion. Everybody has different values and points of view so they may see the same thing completely differently. Its what makes humans great but also frustrating. geohots example will work for some, others will be convinced of the opposite opinion, and others will just let is pass them by and not understand the implications. The same could be applied to your examples to a different degree. Politicians try to use different examples for the audience they are speaking to.

emp17344 3mo ago

This is really only meaningful if AI imparts knowledge beyond what you could learn from a simple Google search. It’s not clear that this is the case.

exitb 3mo ago

Notably, AI IS generally permitted to that, as there are easily obtainable models that will play along. ChatGPT won’t because OpenAI chose and implemented that limitation.

c1ccccc1 3mo ago

Just a terminology note: Alignment does not mean the AI will help its owner kill people. (Indeed, an AI aligned to value human life would generally try to prevent murders.) The word for an AI that follows all instructions of its owner, as that owner intended them to be understood, is "corrigible" or "controllable".

throw310822 3mo ago

In fact I find it a really bad example. Yes I think your personal AI could be allowed to tell you how to cover up a murder. I'm not entirely sure about it but seems possible. What about your personal, local AI guiding you to modifying a flu virus for maximum contagiousness and deadliness? "Sure George, here's your shopping list. It's $1500 total in equipment, do you want me to proceed with the orders?"

anthuswilliams 3mo ago
Why not? There are perfectly pro-social reasons a person might ask such a question. And a bio-terrorist would have many other avenues to answer these questions. The concept of a bioweapon attack is commonly cited but in fact there are many other defenses against actually developing and deploying such a weapon. It's commonly portrayed as easily achievable but is in fact very, very far from it. Moteover, although there is no evidence that AI makes it easier in any way, somehow this is constantly invoked as an example of a thought crime that must be forestalled at all costs. In the AI era people seem to want to redefine the criminal act from "doing the thing" to "asking or investigating how to do the thing". Not a mindset I share.

sph 3mo ago

In other words, who should the AI work for?