Quantization is a category error, the thing you care about is not in weight space, so there’s unbounded error introduced by doing it. The thing you actually wan

Quantization is a category error, the thing you care about is not in weight space, so there’s unbounded error introduced by doing it. The thing you actually want to preserve is the knowledge manifold, but that is in a different vector space. Until we have some better understanding of how to interact with that space directly, rather than inferring it through distillation of reasoning traces, I would not anticipate truly low bit models to be useful.