(Added: People really shouldn't flag you for asking a question.) The point is "aligning the AI's goals with human interests", i.e. make the AI non-evil. You'r

(Added: People really shouldn't flag you for asking a question.) The point is "aligning the AI's goals with human interests", i.e. make the AI non-evil. You're somewhat lucky to never have gone down that particular rabbit hole. Some strange, but also seemingly very intelligent people on the internet argue that it's the most significant extinction risk humanity faces. My impression is that actual AI/ML researchers believe this to be mostly bullshit. (I'm personally not qualified to decide, just explaining the positions.) The author of this curriculum seems to define it as: > Within the coming decades, artificial general intelligence (AGI) may surpass human capabilities at a wide range of important tasks. We outline a case for expecting that, without substantial effort to prevent it, AGIs could learn to pursue goals which are undesirable (i.e. misaligned) from a human perspective. We argue that if AGIs are trained in ways similar to today’s most capable models, they could learn to act deceptively to receive higher reward, learn internally-represented goals which generalize beyond their training distributions, and pursue those goals using power-seeking strategies. We outline how the deployment of misaligned AGIs might irreversibly undermine human control over the world, and briefly review research directions aimed at preventing this outcome. This is from here, which I haven't read, but found three links deep on the posted site: https://arxiv.org/abs/2209.00626 https://arxiv.org/abs/2209.00626