I believe you are correct, I mentioned "There is also a limit to the amount of times you can "just make it bigger" before it actually performs more poorly, as I

I believe you are correct, I mentioned "There is also a limit to the amount of times you can "just make it bigger" before it actually performs more poorly, as I understand it." But there may also be a certain sweet spot we can slowly increase as we do more research. We lose accuracy when we quantize from a larger model to a smaller one (such as 16b quantized to 4b) and it's usually done for speed, but a larger model quantized down tends to be more accurate than a model trained to be that particular size. So it would be interesting to see how it would work if researchers build up to much larger parameter sizes, but quantized them down so they were easier to run but still kept some of the benefits of the increased parameter count. I also imagine parameter count is currently more limited by how much we can process at once, so we could be looking at the NES and trying to imagine PS5 games based on what we know about the NES :)