Humans learn. LLM context windows are vastly larger than our short-term memory, but vastly smaller than our long-term recollection. LLMs can recall vastly more

Humans learn. LLM context windows are vastly larger than our short-term memory, but vastly smaller than our long-term recollection. LLMs can recall vastly more information than our long-term memory, but only from their static training data. Also, even though LLMs can generate text much faster than humans, we may be internally thinking much faster. Each adult human brain has over 100 billion neurons and 100 trillion synapses, and each has been working every moment, for decades. This is what separates human reasoning from LLM reasoning, and it can’t be solved by scaling the latter to anything feasible. I wish AI companies would take a decent chunk of their billions, and split it into 1000+ million-dollar projects that each try a different idea to overcome these issues, and others (like emotion and alignment). Many of these projects would certainly fail, but some may produce breakthroughs. Meanwhile, spending the entire billion on scaling compute has failed and will continue to fail, because everyone else does that, so the resulting model has no practical advantages and makes less money than it cost to train before it becomes obsoleted by other people’s breakthroughs.