No one outside the factory knows how the sausage is made. Watercooler chat is that it's a 1.75 terraweight 16-choose-4 mixture model quantized at 2.5-3bpw with

No one outside the factory knows how the sausage is made. Watercooler chat is that it's a 1.75 terraweight 16-choose-4 mixture model quantized at 2.5-3bpw with a mundane rotary encoding and the kind of KV cache that only an Azure or GCP disagg rack can deliver. There's little doubt that they're hanging on by a fingernail with an MMLU lead over the French that is nonetheless real. So something "bleeding-edge" is going on. (Now whether or not MMLU is a useful metric for the usefulness of a model that won't do what you tell it to is debatable, I think it's not). But is that bleeding edge innovation like Galois and the resulting group theory? Or bleeding edge like "holy shit they could keep track of enough people to build the Pyramids at Giza with a low-precision protractor and infinite motivation"? I tend to think that there's a lot of "straight ahead and damn the torpedoes" brute-force arbitrary compute budget subsidized below cost by a hyperscaler in the mix. But who knows. The effectiveness of distillation is suggestive but not conclusive that we can "Shannon" the process recursively up to some unknown bound without anything much fancier than Mixtral from an arch standpoint.