Well, one should consider that the deep learning system being developed today are developed through a process that's said to be "closer to alchemy than chemistr

Well, one should consider that the deep learning system being developed today are developed through a process that's said to be "closer to alchemy than chemistry". The best deep learning developers aren't thinking about "what is intelligence, really?" but rather figuring ways to optimize systems on benchmarks. And at the same time, current system definitely have made advances in seeming intelligence. Moreover, the Eliza Effect [1] means it's easy for human being interpret a machine that interacts appropriately as intelligent even given very simple programming. If the engineers were engineering these systems, one would expect some immunity to this but as mentioned, they are doing something subtly different. You can see this in action with the Google engineer who tried to get a lawyer for an unreleased LLM and Geoffrey Hinton's turn to worrying after an AI explained a joke to him might arguably be included. And oppositely, because these are developed like alchemy, maybe further qualities will just pop up the way transformers changed language models and maybe this will yield all relevant human capacities. I'm doubtful but not as doubtful as I was five years ago. [1] https://en.wikipedia.org/wiki/ELIZA_effect https://en.wikipedia.org/wiki/ELIZA_effect