3 ms·

Serious question, what is there to ai safety? So far I think it's just pretty helpful when coding certain things, but most of what it gives is low quality. What

by ppppo 2y ago

Serious question, what is there to ai safety? So far I think it's just pretty helpful when coding certain things, but most of what it gives is low quality. What are the real risks involved? Maybe it's just me but so far I don't see risks, maybe once there's GPT 42o.

tempodox 2y ago
> what is there to ai safety? Bullshit jobs where you get payed for pretending. It is supposed to help “Open”“AI” build a moat based on “safety”.
- ppppo 2y ago
  I do see people already creating this new niche, as we have airport security and unnecessary bureaucracy (Germany as an example) we will probably have something like "AI supervisors" who basically chat with an llm for a few minutes a day for the state or local municipality and make a career out of that.
deleted 2y ago
[deleted]
moritzwarhier 2y ago
When GPT-4 was released, they published examples of prompts and agent-like scenarios that are considered malicious. They are even categorized. The appendix also contains full responses before they implemented changes and further training in reaction to the red-teaming results, at least for most of the prompts. (starts on p. 44) https://cdn.openai.com/papers/gpt-4-system-card.pdf https://cdn.openai.com/papers/gpt-4-system-card.pdf I think the categories of prompts are still relevant, and some of the used of LLMs described are already widely deployed (e.g. the one about propaganda on social media) People tend to laugh about the increasingly strict guardrails of commercial chatbots, and it has proved to be at least in large parts an cat and mouse game. But I wouldn't underestimate the viciousness of an unfiltered LLM instructed to generate propaganda, hate speech or convincing instructions for humans to perform harmful acts.
elzbardico 2y ago
Basically people that are going to do reinforcement learning to ensure the model generates output according to a set of pre-defined values. So, this people defines what those values and principles are, and then they make sure the model generate stuff aligned with those. We would usually call this moderation, but for branding, marketing and financing purposes it is far more interesting to call it safety because it reinforces on the lay mind an idea that those models are far more powerful than they really are and that we are on the cusp of General AI. It is like lowering the suspension, painting the wheels black, and putting rear flaps and badges that say "Sport" and "Turbo" in a unmodified Toyota Corola.
yoz-y 2y ago
Ai telling you to eat rocks might be funny. Ai telling you to mix bleach and vinegar is not. With reach any type of an authoritative answer will find enough people that take it at face value. Computer mistakes have been costly in the past (excel and Greece?), AI encourages lack of scrutiny despite all the lip service given to mark the answers as “potentially” wrong. It’s not too hard to imagine a future where AI will be used to make decisions that no one will try to understand or verify.
- ppppo 2y ago
  Thanks, I see. I guess the real issue is the reach. For now at least I don't see ai having that sort of reach that would make me uneasy, like interacting with kids in video games. I can see how it should have guard rails there on what to say. However, since llms are approximations of the same neural nets as humans, I wonder if it will ever be 100% safe. Even with humans who we assume to be for sure good have bad intent sometimes (eg doctors, teachers abusing kids).
- genewitch 2y ago
  > imagine a future where AI will be used to make decisions i'd have to check my journal but i am fairly certain this has been the case for at least a decade and a half - AI making decisions that negatively affect people without justification or recourse. I know the (apocryphal) "go-to" is amazon binning resumes but there are other ones; but ALPR (mixed with human error,) other traffic management systems, the weird "racist" "AI" in boston or baltimore (or whatever), lending/credit. nevermind the chaos that is online advertising.
tivert 2y ago
> Serious question, what is there to ai safety? ... What are the real risks involved? One set of risks are hypothetical, and basically are about avoiding things like the paperclip maximizer example. Another set of risks are more practical and involve the existing technology, like how to you keep it from being abused for ill. Realistically, that mostly amounts to trying to avoid PR problems, because I can't imagine those guardrails would stay on when the company doesn't want them there.
almazzz 2y ago
[dead]

by ppppo 2y ago

tempodox 2y ago

> what is there to ai safety? Bullshit jobs where you get payed for pretending. It is supposed to help “Open”“AI” build a moat based on “safety”.

ppppo 2y ago
I do see people already creating this new niche, as we have airport security and unnecessary bureaucracy (Germany as an example) we will probably have something like "AI supervisors" who basically chat with an llm for a few minutes a day for the state or local municipality and make a career out of that.

deleted 2y ago

[deleted]

moritzwarhier 2y ago

When GPT-4 was released, they published examples of prompts and agent-like scenarios that are considered malicious. They are even categorized. The appendix also contains full responses before they implemented changes and further training in reaction to the red-teaming results, at least for most of the prompts. (starts on p. 44) https://cdn.openai.com/papers/gpt-4-system-card.pdf https://cdn.openai.com/papers/gpt-4-system-card.pdf I think the categories of prompts are still relevant, and some of the used of LLMs described are already widely deployed (e.g. the one about propaganda on social media) People tend to laugh about the increasingly strict guardrails of commercial chatbots, and it has proved to be at least in large parts an cat and mouse game. But I wouldn't underestimate the viciousness of an unfiltered LLM instructed to generate propaganda, hate speech or convincing instructions for humans to perform harmful acts.

elzbardico 2y ago

Basically people that are going to do reinforcement learning to ensure the model generates output according to a set of pre-defined values. So, this people defines what those values and principles are, and then they make sure the model generate stuff aligned with those. We would usually call this moderation, but for branding, marketing and financing purposes it is far more interesting to call it safety because it reinforces on the lay mind an idea that those models are far more powerful than they really are and that we are on the cusp of General AI. It is like lowering the suspension, painting the wheels black, and putting rear flaps and badges that say "Sport" and "Turbo" in a unmodified Toyota Corola.

yoz-y 2y ago

Ai telling you to eat rocks might be funny. Ai telling you to mix bleach and vinegar is not. With reach any type of an authoritative answer will find enough people that take it at face value. Computer mistakes have been costly in the past (excel and Greece?), AI encourages lack of scrutiny despite all the lip service given to mark the answers as “potentially” wrong. It’s not too hard to imagine a future where AI will be used to make decisions that no one will try to understand or verify.

ppppo 2y ago
Thanks, I see. I guess the real issue is the reach. For now at least I don't see ai having that sort of reach that would make me uneasy, like interacting with kids in video games. I can see how it should have guard rails there on what to say. However, since llms are approximations of the same neural nets as humans, I wonder if it will ever be 100% safe. Even with humans who we assume to be for sure good have bad intent sometimes (eg doctors, teachers abusing kids).
genewitch 2y ago
> imagine a future where AI will be used to make decisions i'd have to check my journal but i am fairly certain this has been the case for at least a decade and a half - AI making decisions that negatively affect people without justification or recourse. I know the (apocryphal) "go-to" is amazon binning resumes but there are other ones; but ALPR (mixed with human error,) other traffic management systems, the weird "racist" "AI" in boston or baltimore (or whatever), lending/credit. nevermind the chaos that is online advertising.

tivert 2y ago

> Serious question, what is there to ai safety? ... What are the real risks involved? One set of risks are hypothetical, and basically are about avoiding things like the paperclip maximizer example. Another set of risks are more practical and involve the existing technology, like how to you keep it from being abused for ill. Realistically, that mostly amounts to trying to avoid PR problems, because I can't imagine those guardrails would stay on when the company doesn't want them there.

almazzz 2y ago

[dead]