I work with LLMs, won't say where, but smaller models stop performing better on benchmarks after a certain point, e.g. they seem to hit their learning capacity

I work with LLMs, won't say where, but smaller models stop performing better on benchmarks after a certain point, e.g. they seem to hit their learning capacity (at least with current techniques). Small models struggle to keep context the way larger models do, their outputs are impressive, but lack a certain amount of logical consistency and flow. Whether this is a fundamental issue with the model size or some combination of training technique and model size is yet to be known. But for now, we know what works and are exploring that until we squeeze all the 'easy' perf we can.