If AGI is possible, then the described concerns about risk hold up. Either you think AGI is impossible for some reason or you think it is possible, but the goa

If AGI is possible, then the described concerns about risk hold up. Either you think AGI is impossible for some reason or you think it is possible, but the goal alignment issues don’t matter. You haven’t given any explanation for either. Maybe you think something else? I wouldn’t know because you don’t say anything. Obviously there isn’t “hard scientific backing” since AGI doesn’t exist yet. There are still unknowns about what would be required. That doesn’t mean it’s impossible to think about what its existence could mean. It was possible to think about the consequences of flight before airplanes existed. I think AGI is possible because we’re surrounded by general intelligence and there’s nothing magical about brains or biology. It’s just the timeframe that’s unclear. We already see issues with poorly optimized reward functions on a smaller scale, both in ML models (classifying based on unintended things) and in humans (drug addiction).