To the layman, sure, this stuff seems high tech. However, for anyone that has been exposed to neural nets in the past decade, the solution of "throwing more co

To the layman, sure, this stuff seems high tech. However, for anyone that has been exposed to neural nets in the past decade, the solution of "throwing more compute" at it has been obvious. The big driver of this has been the actual compute availability. Even in early 2000s, it was simply unfeasible to just make computers iterate on a bunch of things until they got it right. When deep learning took hold and showed promise, hardware start to catch up, and now a rack of GPUs can pretty much out-compute buildings of servers back from 2000s. Same thing is going to happen in the future. If economy works out in such a way that someone with median salary can buy a box that allows them to train a model like Llama in a week (with of course cheap electricity and updated power grid), we are going to get much cooler things. Because in that case, it wouldn't be a matter of figuring out what the best architecture is for LLM - it would be a matter of generalizing the compute graph, which means that the actual architecture would be a learnable parameter. I mean if you look at the human brain, and take the neural activations per second (which is essentially a floating point operation of add/multiply) times the number of neurons, you get some petaflop number. We have racks of hardware that can match that number right now. We just don't know the correct structure of all the compute graph, and to figure it out is a matter of iteration, and computers are better at iteration than humans are.