>I'm not convinced that simply increasing the number of parameters improves the models How so? Llama 33B is clearly worse than llama-64B and that's 'only' a 2x

>I'm not convinced that simply increasing the number of parameters improves the models How so? Llama 33B is clearly worse than llama-64B and that's 'only' a 2x increase. Are you aware of any example where a larger model fails to outperform a smaller one when all else is equal (tokens, architecture, data quality, etc.)? Obviously for a fixed amount of training compute more parameters can be bad, but there's a trade off where more parameters means you train on fewer tokens.