I'm uncertain where I stand on this but I'm going to give a shot at a less dismissive answer than most of the replies so far. There's a very abstract argument

I'm uncertain where I stand on this but I'm going to give a shot at a less dismissive answer than most of the replies so far. There's a very abstract argument that goes like this: * "Orthogonality thesis": an arbitrarily smart agent could in theory value ~anything, e.g. maximize Facebook's valuation (say as measured by some specific dataset published by S&P). * "Instrumental convergence thesis": most things that you might conceivably value are easier to optimize if you have a lot of power and resources (e.g. if you just run S&P, and hey maybe also control the worlds militaries so you can ensure nobody challenges that). * Belief in the power of intelligence: it's theoretically possible to be much more capable at ~anything than the most capable human just by thinking better and faster, including building even better AIs. Eventually some human will build an agent that does that, and plausibly this will happen in the current century. * Alignment problem: just because we make something doesn't automatically mean it wants exactly what we want, and in fact getting it to share our values is hard. Children disagree with parents, human values aren't those of natural selection, and ML language models repeat white supremacist ideas about Black people from their Internet training data. Extrapolate out these points - which FWIW I find individually plausible - and you get doom: sooner or later humanity loses control of its destiny to whatever superintelligence we stumble on first, which in the worst case re-uses our atoms for something else (digits in some high-density storage representation that meets the loss function's technical criterion for Facebook's S&P valuation) about even in the best case seems fairly dystopian (whatever human Zuckerberg would actually want). Now this is all pretty scifi and that's about all there was to it a few decades ago, but more recently we've started hitting ~parity milestones (chess, Go, protein folding, translation, arguably even fiction and visual art) that many experts once thought would require fully general human-level intelligence. It's clear to me that we haven't built that yet, but (as I said in another comment upthread) it also seems to me that our models are advancing in capability much more rapidly than we are gaining the ability to understand their internals or make guarantees of any kind about their worst-case or out-of-training-distribution behavior. Seems entirely sane to be worried by that trajectory and to start thinking seriously about how it might connect to the scary abstract speculations. (edited for typos)