3 ms·

I think what's already obvious is that most people in AI research care only for solving some hard problems in computer science. Both AI safety and ethics[0] be

I think what's already obvious is that most people in AI research care only for solving some hard problems in computer science. Both AI safety and ethics[0] be damned. So the amount of control OpenAI had on the research arm is less than you'd think. [0] For the record, AI safety is the branch of research worried about controlling computer God, while AI ethics is the branch of research worried about whether or not it's OK to pay Pakistani kids pennies to label training data.

lukev 3y ago
Genuinely, great distinction. Would you also include in “AI ethics” a considered rollout of automation to maximize its benefit to everyone? Because there is real benefit. The kind of work AI can automate is shit work and it would be to the theoretical benefit of society for it to be automated… unless all the benefits accrued to the owners of the tech instead of the people whose (stupid) jobs are being replaced. Is that “AI ethics” or still a third (anti capitalist?) thing?
- giantrobot 3y ago
  > unless all the benefits accrued to the owners of the tech instead of the people whose (stupid) jobs are being replaced. The workers doing the jobs will never benefit. They'll be shown the door and the owners will reap all the profits.
pardoned_turkey 3y ago
I'm not sure that's accurate. Most companies use "AI safety" as a shorthand for "brand safety", i.e., making sure that your LLM can't be trolled into praising Hitler or denying climate change. Basically, do what needs to be done to avoid bad press. There is definitely that second and more abstract take - AI doomsday safety - that Altman liked to talk about to get favorable regulation passed. But it doesn't seem to be much of a real field. "Hard" AI doomers think that superhuman AI can't be reliably constrained and we should just stop. "Soft" doomers think we need to solve "alignment". But we don't really know what's going on inside an LLM, so we test for alignment by asking it what it thinks about Hitler. In other words, back to brand safety.
- upwardbound 3y ago
  My company Preamble markets ourselves as an AI Safety company so I’ll opine on what I think is a reasonable middle-ground definition. We interpret near-term AI safety to mean solving the problems that already exist today but will also exist in AGI. As a key example, we were the first team to discover Prompt Injection, which we responsibly disclosed to OpenAI leadership in early May of 2022, four months before it was independently rediscovered by Riley Goodside. (See the citation at the bottom of this page https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/1_1_vulns/LLM01_PromptInjection.md https://github.com/OWASP/www-project-top-10-for-large-langua... ) Prompt Injection is a perfect example of what we consider a near-term “AI safety” issue: it’s a real issue that affects current systems, and it will also affect any instruction-based AGI systems if a solution is not found by the time AGI is reached. There is currently no known general solution to Prompt Injection, and it’s #1 on OWASP’s recent list of the top vulns in LLMs. (See link above. This vuln is designated LLM01.)
- kmeisthax 3y ago
  You're not wrong. AI safety originally meant the doomsday variant and has sort of morphed into the brand safety variant. My understanding was that safety research was treating both as the same problem, or at least that they believed brand safety was a worthwhile subset of the doomsday problem to attack.
morkalork 3y ago
I thought AI ethics was not creating racist bail predictors or health insurance bots that automatically deny claims with a 90% error rate.
- bertil 3y ago
  It's a broad field. Also, the two are tied: those underpaid, overworked Filipinas? Surprisingly reactionaries.

lukev 3y ago

Genuinely, great distinction. Would you also include in “AI ethics” a considered rollout of automation to maximize its benefit to everyone? Because there is real benefit. The kind of work AI can automate is shit work and it would be to the theoretical benefit of society for it to be automated… unless all the benefits accrued to the owners of the tech instead of the people whose (stupid) jobs are being replaced. Is that “AI ethics” or still a third (anti capitalist?) thing?

giantrobot 3y ago
> unless all the benefits accrued to the owners of the tech instead of the people whose (stupid) jobs are being replaced. The workers doing the jobs will never benefit. They'll be shown the door and the owners will reap all the profits.

pardoned_turkey 3y ago

I'm not sure that's accurate. Most companies use "AI safety" as a shorthand for "brand safety", i.e., making sure that your LLM can't be trolled into praising Hitler or denying climate change. Basically, do what needs to be done to avoid bad press. There is definitely that second and more abstract take - AI doomsday safety - that Altman liked to talk about to get favorable regulation passed. But it doesn't seem to be much of a real field. "Hard" AI doomers think that superhuman AI can't be reliably constrained and we should just stop. "Soft" doomers think we need to solve "alignment". But we don't really know what's going on inside an LLM, so we test for alignment by asking it what it thinks about Hitler. In other words, back to brand safety.

upwardbound 3y ago
My company Preamble markets ourselves as an AI Safety company so I’ll opine on what I think is a reasonable middle-ground definition. We interpret near-term AI safety to mean solving the problems that already exist today but will also exist in AGI. As a key example, we were the first team to discover Prompt Injection, which we responsibly disclosed to OpenAI leadership in early May of 2022, four months before it was independently rediscovered by Riley Goodside. (See the citation at the bottom of this page https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/1_1_vulns/LLM01_PromptInjection.md https://github.com/OWASP/www-project-top-10-for-large-langua... ) Prompt Injection is a perfect example of what we consider a near-term “AI safety” issue: it’s a real issue that affects current systems, and it will also affect any instruction-based AGI systems if a solution is not found by the time AGI is reached. There is currently no known general solution to Prompt Injection, and it’s #1 on OWASP’s recent list of the top vulns in LLMs. (See link above. This vuln is designated LLM01.)
kmeisthax 3y ago
You're not wrong. AI safety originally meant the doomsday variant and has sort of morphed into the brand safety variant. My understanding was that safety research was treating both as the same problem, or at least that they believed brand safety was a worthwhile subset of the doomsday problem to attack.

morkalork 3y ago

I thought AI ethics was not creating racist bail predictors or health insurance bots that automatically deny claims with a 90% error rate.

bertil 3y ago
It's a broad field. Also, the two are tied: those underpaid, overworked Filipinas? Surprisingly reactionaries.