Just on your very last point, I think you've nailed why a 1-bit quant of a bigger LM can't beat 3-bit quants of an LM a third the size, if what you mean is that

Just on your very last point, I think you've nailed why a 1-bit quant of a bigger LM can't beat 3-bit quants of an LM a third the size, if what you mean is that more extreme compression of a LM is more likely to introduce harmful artefacts, so you need a better quantisation method to produce 1-bit than you do at 3-bit to end up with a model with the same information content retained. What I don't think that tells us anything about is directly trained 1-bit LMs versus 3-bit LMs, because in that case there's no compression step to introduce quantisation artefacts. There might be an analogous training data size argument but it's not clear to me that there needs to be: a 3X parameter 1-bit LLM and a 1X parameter 3-bit LLM ought to be equivalent in terms of their information capacity.