4 ms·

I'm not sure that's accurate. Most companies use "AI safety" as a shorthand for "brand safety", i.e., making sure that your LLM can't be trolled into praising H

I'm not sure that's accurate. Most companies use "AI safety" as a shorthand for "brand safety", i.e., making sure that your LLM can't be trolled into praising Hitler or denying climate change. Basically, do what needs to be done to avoid bad press. There is definitely that second and more abstract take - AI doomsday safety - that Altman liked to talk about to get favorable regulation passed. But it doesn't seem to be much of a real field. "Hard" AI doomers think that superhuman AI can't be reliably constrained and we should just stop. "Soft" doomers think we need to solve "alignment". But we don't really know what's going on inside an LLM, so we test for alignment by asking it what it thinks about Hitler. In other words, back to brand safety.

upwardbound 3y ago
My company Preamble markets ourselves as an AI Safety company so I’ll opine on what I think is a reasonable middle-ground definition. We interpret near-term AI safety to mean solving the problems that already exist today but will also exist in AGI. As a key example, we were the first team to discover Prompt Injection, which we responsibly disclosed to OpenAI leadership in early May of 2022, four months before it was independently rediscovered by Riley Goodside. (See the citation at the bottom of this page https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/1_1_vulns/LLM01_PromptInjection.md https://github.com/OWASP/www-project-top-10-for-large-langua... ) Prompt Injection is a perfect example of what we consider a near-term “AI safety” issue: it’s a real issue that affects current systems, and it will also affect any instruction-based AGI systems if a solution is not found by the time AGI is reached. There is currently no known general solution to Prompt Injection, and it’s #1 on OWASP’s recent list of the top vulns in LLMs. (See link above. This vuln is designated LLM01.)
kmeisthax 3y ago
You're not wrong. AI safety originally meant the doomsday variant and has sort of morphed into the brand safety variant. My understanding was that safety research was treating both as the same problem, or at least that they believed brand safety was a worthwhile subset of the doomsday problem to attack.

upwardbound 3y ago

My company Preamble markets ourselves as an AI Safety company so I’ll opine on what I think is a reasonable middle-ground definition. We interpret near-term AI safety to mean solving the problems that already exist today but will also exist in AGI. As a key example, we were the first team to discover Prompt Injection, which we responsibly disclosed to OpenAI leadership in early May of 2022, four months before it was independently rediscovered by Riley Goodside. (See the citation at the bottom of this page https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/1_1_vulns/LLM01_PromptInjection.md https://github.com/OWASP/www-project-top-10-for-large-langua... ) Prompt Injection is a perfect example of what we consider a near-term “AI safety” issue: it’s a real issue that affects current systems, and it will also affect any instruction-based AGI systems if a solution is not found by the time AGI is reached. There is currently no known general solution to Prompt Injection, and it’s #1 on OWASP’s recent list of the top vulns in LLMs. (See link above. This vuln is designated LLM01.)

kmeisthax 3y ago

You're not wrong. AI safety originally meant the doomsday variant and has sort of morphed into the brand safety variant. My understanding was that safety research was treating both as the same problem, or at least that they believed brand safety was a worthwhile subset of the doomsday problem to attack.