4 ms·

I’ve never heard this term “AI alignment” before and they also don’t define it…

by onos 4y ago

carapace 4y ago
You know, chaotic neutral or lawful good, etc...
layer8 4y ago
https://en.wikipedia.org/wiki/AI_alignment https://en.wikipedia.org/wiki/AI_alignment
deleted 4y ago
[deleted]
codeflo 4y ago
(Added: People really shouldn't flag you for asking a question.) The point is "aligning the AI's goals with human interests", i.e. make the AI non-evil. You're somewhat lucky to never have gone down that particular rabbit hole. Some strange, but also seemingly very intelligent people on the internet argue that it's the most significant extinction risk humanity faces. My impression is that actual AI/ML researchers believe this to be mostly bullshit. (I'm personally not qualified to decide, just explaining the positions.) The author of this curriculum seems to define it as: > Within the coming decades, artificial general intelligence (AGI) may surpass human capabilities at a wide range of important tasks. We outline a case for expecting that, without substantial effort to prevent it, AGIs could learn to pursue goals which are undesirable (i.e. misaligned) from a human perspective. We argue that if AGIs are trained in ways similar to today’s most capable models, they could learn to act deceptively to receive higher reward, learn internally-represented goals which generalize beyond their training distributions, and pursue those goals using power-seeking strategies. We outline how the deployment of misaligned AGIs might irreversibly undermine human control over the world, and briefly review research directions aimed at preventing this outcome. This is from here, which I haven't read, but found three links deep on the posted site: https://arxiv.org/abs/2209.00626 https://arxiv.org/abs/2209.00626
- random314 4y ago
  To be clear the author of the AGI course is a recent physics grad with no experience in AI
  - mitthrowaway2 4y ago
    You've mentioned this ad-hominem several times. What's your background?
    - random314 4y ago
      ML and data science :) Have a decade of experience including FAANG and a published peer reviewed paper.
  - deleted 4y ago
    [deleted]
- Max_Limelihood 4y ago
  Actual AI researchers have wildly different opinions on this; IME, going off all the AI researchers I’ve talked to, they tend to split about 50/50.
random314 4y ago
It's a "well defined" term in the crackpot side of AI, represented by lesswrong et al, who are self appointed AI experts who have never published any actual peer reviewed research in AI.
- Max_Limelihood 4y ago
  Self-appointed by the boards of directors at major companies and universities? https://slatestarcodex.com/2015/05/22/ai-researchers-on-ai-risk/ https://slatestarcodex.com/2015/05/22/ai-researchers-on-ai-r... And no peer-reviewed research, apart from the dozens of widely-cited academic papers on this? All of these are just from one lab, not even including groups like CHAI: https://www.anthropic.com/#papers https://www.anthropic.com/#papers
  - random314 4y ago
    > And no peer-reviewed research, apart from the dozens of widely-cited academic papers on this? All of these are just from one lab, not even including groups like CHAI: https://www.anthropic.com/#papers https://www.anthropic.com/#papers Which of these papers track AGI Safety and are also published in peer reviewed journals like SIGKDD, NeurIPS, AAAI etc?
- mchaver 4y ago
  OpenAI mentions AI Alignment and they are currently at the forefront of pushing AI on the world so it is not just that community, though I wouldn't be surprised if many employees at OpenAI participate there. https://openai.com/blog/our-approach-to-alignment-research https://openai.com/blog/our-approach-to-alignment-research
  - random314 4y ago
    Open AI is one of the actual research organizations promoting AI Safety work. However, this has more to do with funding and the money put up front by the original founders - Elon Musk, Sam Altman etc, who themselves are not AI researchers, yet AGI enthusiasts. Open AI published a number of papers, essentially and implicitly supporting the position of the AI inexperienced founders. They try to highlight AI safety aspects quite frequently. But I would not take it too seriously, just like you wouldn't take a Facebook employees endorsement of Zucks vision. If you are familiar with ML, you would know that L2 norms are a key optimization metric in ML. L2 norms are basically Pythagorean hypotenuse computations in high dimensions. I will take AI safety research seriously when they start discussing the risk factors associated with Pythagoras theorem and how we as a society must deal with it before we become a slave to L2 norm driven super intelligence. If you want to really talk about human risk from AI, Tesla's self driving killing machines are already here along with ML guided drone bombers. It's no different from Therac 25. If we are not discussing these and creating a course about super intelligence, it's just mental masturbation.

I’ve never heard this term “AI alignment” before and they also don’t define it…

by onos 4y ago

I’ve never heard this term “AI alignment” before and they also don’t define it…

carapace 4y ago

You know, chaotic neutral or lawful good, etc...

layer8 4y ago

https://en.wikipedia.org/wiki/AI_alignment https://en.wikipedia.org/wiki/AI_alignment

deleted 4y ago

[deleted]

codeflo 4y ago

(Added: People really shouldn't flag you for asking a question.) The point is "aligning the AI's goals with human interests", i.e. make the AI non-evil. You're somewhat lucky to never have gone down that particular rabbit hole. Some strange, but also seemingly very intelligent people on the internet argue that it's the most significant extinction risk humanity faces. My impression is that actual AI/ML researchers believe this to be mostly bullshit. (I'm personally not qualified to decide, just explaining the positions.) The author of this curriculum seems to define it as: > Within the coming decades, artificial general intelligence (AGI) may surpass human capabilities at a wide range of important tasks. We outline a case for expecting that, without substantial effort to prevent it, AGIs could learn to pursue goals which are undesirable (i.e. misaligned) from a human perspective. We argue that if AGIs are trained in ways similar to today’s most capable models, they could learn to act deceptively to receive higher reward, learn internally-represented goals which generalize beyond their training distributions, and pursue those goals using power-seeking strategies. We outline how the deployment of misaligned AGIs might irreversibly undermine human control over the world, and briefly review research directions aimed at preventing this outcome. This is from here, which I haven't read, but found three links deep on the posted site: https://arxiv.org/abs/2209.00626 https://arxiv.org/abs/2209.00626

random314 4y ago
To be clear the author of the AGI course is a recent physics grad with no experience in AI
- mitthrowaway2 4y ago
  You've mentioned this ad-hominem several times. What's your background?
  - random314 4y ago
    ML and data science :) Have a decade of experience including FAANG and a published peer reviewed paper.
- deleted 4y ago
  [deleted]
Max_Limelihood 4y ago
Actual AI researchers have wildly different opinions on this; IME, going off all the AI researchers I’ve talked to, they tend to split about 50/50.

random314 4y ago

It's a "well defined" term in the crackpot side of AI, represented by lesswrong et al, who are self appointed AI experts who have never published any actual peer reviewed research in AI.

Max_Limelihood 4y ago
Self-appointed by the boards of directors at major companies and universities? https://slatestarcodex.com/2015/05/22/ai-researchers-on-ai-risk/ https://slatestarcodex.com/2015/05/22/ai-researchers-on-ai-r... And no peer-reviewed research, apart from the dozens of widely-cited academic papers on this? All of these are just from one lab, not even including groups like CHAI: https://www.anthropic.com/#papers https://www.anthropic.com/#papers
- random314 4y ago
  > And no peer-reviewed research, apart from the dozens of widely-cited academic papers on this? All of these are just from one lab, not even including groups like CHAI: https://www.anthropic.com/#papers https://www.anthropic.com/#papers Which of these papers track AGI Safety and are also published in peer reviewed journals like SIGKDD, NeurIPS, AAAI etc?
mchaver 4y ago
OpenAI mentions AI Alignment and they are currently at the forefront of pushing AI on the world so it is not just that community, though I wouldn't be surprised if many employees at OpenAI participate there. https://openai.com/blog/our-approach-to-alignment-research https://openai.com/blog/our-approach-to-alignment-research
- random314 4y ago
  Open AI is one of the actual research organizations promoting AI Safety work. However, this has more to do with funding and the money put up front by the original founders - Elon Musk, Sam Altman etc, who themselves are not AI researchers, yet AGI enthusiasts. Open AI published a number of papers, essentially and implicitly supporting the position of the AI inexperienced founders. They try to highlight AI safety aspects quite frequently. But I would not take it too seriously, just like you wouldn't take a Facebook employees endorsement of Zucks vision. If you are familiar with ML, you would know that L2 norms are a key optimization metric in ML. L2 norms are basically Pythagorean hypotenuse computations in high dimensions. I will take AI safety research seriously when they start discussing the risk factors associated with Pythagoras theorem and how we as a society must deal with it before we become a slave to L2 norm driven super intelligence. If you want to really talk about human risk from AI, Tesla's self driving killing machines are already here along with ML guided drone bombers. It's no different from Therac 25. If we are not discussing these and creating a course about super intelligence, it's just mental masturbation.