Ok, hear me out on the (albeit unlikely) doomsday flow chart… It’s not about the replaced workers, that’s the kind of change humanity can handle no matter how

Ok, hear me out on the (albeit unlikely) doomsday flow chart… It’s not about the replaced workers, that’s the kind of change humanity can handle no matter how messy it gets, worst case it’s some kind of class uprising worker rebellion and we have ourselves some war or social unrest and a variable number of people die… not great but definitely not an existential risk. It’s equal to or less than climate change on the “how bad will this be to humanity” mathematics of civilisation. Now the danger scenario… so we invent an AGI, and unlike the just going to replace some basic minimum wage type jobs … this one’s actually “clever” enough it can be put to work on building more sophisticated things, like perhaps an even smarter AGI, built faster because it’s worked on by a group of first generation AGI units that never sleep, and so we get to the second generation much faster, and then perhaps a third… etc… classic singularity scenario, with all the attendant risks of the AI viewing us ass irrelevant before we realise it yada yada yada… But that’s not the only risk, there’s the “photocopy burn” risk that we run by having technology we don’t fully understand build technology we barely comprehend… even if the first gen AGI is smart, if it’s remotely human like using inference to get ingenuity, it can probably still make the odd mistake or two, so we get our careful well thought out safety instructions conveyed to the second generation systems… just a little wrong… and the little wrong turns into more wrong with the third because the second gen systems doesn’t quite think the way we expect it to and doesn’t have the safeguards working the way we think… and then eventually we wind up with a sufficiently smart AGI that is fully either sociopathic or psychopathic and has zero compunction doing things we find abhorrent in order to achieve the goals it’s set, and it may be in control of systems (or able to take control) that are sufficiently important that this can also turn into an existential risk… It’s entirely plausible, but I don’t think the doom sayers are being realistic about the timelines… we’re barely able to get the AI/ML to reliably stick to a script and not hallucinate fake shit… the best results at human behaviour require sophisticated supporting logic and memory systems to augment the black box ML stuff… and these support systems give us levers we can use to control the rate at which we allow systems to progress… I don’t think we’re ever going to get a black box set of weights for an ML model, no matter how sophisticated the code, that somehow is self aware… there’s probably going to need to be a lot of that support software, and I’m pretty sure we’re going to develop things to filter stuff out as part of them like “never write “kill all humans” into the short term memory” seems smart.