> But other than scaling that from 1000 neurons to billions, I'm curious what has changed about the concepts of pathing or tolerance to make these models better

> But other than scaling that from 1000 neurons to billions, I'm curious what has changed about the concepts of pathing or tolerance to make these models better? Maybe my concept of the principle behind modern LLMs is too archaic or rooted in a cartoon understanding of our own wetware that I tried to reproduce. Realastically... the problem is a data problem. The math is mostly there. Transformers et al would have been figured out in no time had we had the sort of data tools we have today. I'm talking about the ease with which you can take gigabytes of data, throw it in S3, and analyze it in minutes. That, combined with cheap and accessible compute (cloud) and the maturation of CUDA meant it was all a matter of time before this took place.