5 ms·
>It is almost guaranteed that a 60-90B model can outperform current SOTA in coding tasks within 2-3 years. I don't disagree, but how much of this ends up being
by supern0va 5mo ago
>It is almost guaranteed that a 60-90B model can outperform current SOTA in coding tasks within 2-3 years.
I don't disagree, but how much of this ends up being distillation? I can't help but imagine that 4.8 was probably trained in part by leveraging Mythos.
If the very large models turn out to be very expensive to run relative to the benefits, it's possible that they could end up still being trained, but ultimately used as a tool to create smaller models that are nearly as effective.
I'm curious if someone here with a stronger background in the space has a similar intuition or not.
- onlyrealcuzzo 5mo ago> I don't disagree, but how much of this ends up being distillation? You don't need distillation. They already have the training sets. It's MLA + MoE + Medusa (a better version of Speculative Decoding) + 1.58b (possibly - maybe nothing) + GRAM (which will almost certainly not turn out to be a nothing burger, but no one has quickly turned this around yet to prove it).
- Philpax 5mo agoIt wouldn't be data distillation: instead, it would be teacher-student distillation. The teacher model has stronger representations that the student can mimic, which would give it more capability over training on the data itself.
- minimaltom 5mo agoFrontier labs have their own variants of MLA and certainly their own balance/scaling-laws for things like MoE vs FC vs Attn. MoE scales really well for inference with horizontal scaling + batching, which these guys luv. On the architectures side, I'm a lot more interesting in attention residuals than anything else, one of those things that seems obvious in hindsight and Kimi have proven it at scale.
- onlyrealcuzzo 5mo ago> Frontier labs have their own variants of MLA Yes, variants typically 2-3x less good... Same with speculative decoding... They all do something, but there are known techniques that are substantially better - that just were't known when they started development of the previous models.
- amluto 5mo agoHow useful is speculative decoding in a batched setting where you get paid for throughput (aggregated across users) and you mostly don’t get paid for latency or single-session throughput?
- onlyrealcuzzo 5mo agoIt's useful at the local level, where there will be SOTA models developed...
- zozbot234 5mo agoLocal models are moving towards batched inference too, if only for non-interactive use. An early experimental patchset for DS4 (running DeepSeek V4 Flash) seems to show 2x aggregate tok/s decode when processing 8 streams concurrently, and more than 3x when processing as many as 32 streams concurrently. Note that prefill (which is not helped significantly by this change) then becomes a larger fraction of total wall-clock time, so the overall gain is lower (i.e. prefill is akin to a 'serial' task wrt. Amdahl's law). MTP will still be highly valuable for interactive use of course.
- semiquaver 5mo agoThe frontier labs distill their own base models all day long. It’s not just something done by nefarious Chinese copycats. The knowledge embodied by the internal base models that we never see is much more powerful and useful than the much sparser raw training data
- supern0va 5mo agoI think you replied to the wrong parent.
- coldtea 5mo ago>It’s not just something done by nefarious Chinese copycats And even that would be rich as a accusation from SOTAs that depend on explicitly disregarding millions of training data intellectual property..
- manmal 5mo agoBut how? The training data is the unadulterated content those models are based on? I genuinely don’t understand, no snark.
- wtallis 5mo agoRaw training data is raw. A really big model trained on it has already done a first-pass of finding patterns and squeezing out redundancy. Re-ingesting the full training set to train a smaller model is probably more expensive, for marginal quality improvement over distilling from the large model.
- adgjlsfhk1 5mo agoDistilling from a larger model is not only probably cheaper than from data, it's also likely higher quality. There's pretty strong support for the proposition that NNs learn a smoothed and regularized version of the data. The NNs are likely higher quality than most of the data they are training from.
- 4mo ago
- spwa4 5mo ago> I don't disagree, but how much of this ends up being distillation? A lot, so you can bet tens of millions are flowing to congress to have distillation declared illegal before this happens. And then it'll happen anyway.
- lambda 5mo agoDistillation isn't only between different labs. A lab can train a large model, and then distill a smaller model from it that retains the majority of the useful capbility. I don't know well enough if there's any benefit of that over just training the smaller model directly, but I'll bet there are some times where that is useful. I could easily see it being easier to do the initial pre-training on a larger model but be able to distill everything useful down into a smaller model, essentially filtering out a lot of noise in the process.
- spwa4 5mo agoThere used to be training methods like that but I think they've been phased out in favor of letting small models evolve by rewriting their own training material. Surprisingly that's actually cheaper.
- bandrami 5mo agoI think the idea is you sink the pretraining costs once and then you can distill multiple specialized models from that
- rao-v 5mo agoIt’s really worth distinguishing between old-fashioned student teacher distillation (ie at the level of layers, weights and distributions) and large scale synthetic dataset creation. The latter is much better (since you can clean up, review, update responses and filter your datasets). I suspect nobody is doing real student teacher distillation, it’s just easier to do a bunch of training on the same giant corpus then post train on the synthetic corpus with its reasoning traces etc. (which might have been generated by a bigger better LLM)
- thisisaman408 5mo ago[dead]
- ACCount37 5mo agoA reason to do student-teacher distillation is that soft target logits in general are a richer medium than text that tokenizes to hard targets. More steering signal per teacher token. And running ultra large 10T tier models in autoregressive generation mode can get expensive. So there are reasons not to reduce to text only synthetics.
- rao-v 5mo agoI agree, and if my suspicion is right, it’s rarer because it’s much easier to deploy the large LLM and filter for it’s best output than to waste time running it on arbitary output just to train the student. Though you could argue that perhaps labs just save the per token distribution and use that during fine tuning … which starts looking more like student teacher fine tuning if not classic distillation from random weights
- ACCount37 5mo agoFull distributions are a fucking pain to save - at this point just save the hidden states. But there are lossy compression tricks there.
- 5mo ago
- ACCount37 5mo agoScale is always desirable, and there are always gains from scale. It's a matter of whether you can afford training and inference at increased scale. There is a real trend of smaller models becoming more "capability-dense" - i.e. the best 8Bs of today beat the best 32Bs of 2 years ago. This is in part a product of distillation being used to train the smaller models. But people consistently underestimate how "capability hungry" the world is. There are diminishing returns on model capabilities in narrow "summarize the search results" sorts of applications - but as capabilities improve, LLMs enter, get their footing in and begin to dominate new niches. At times, expensive, highly desirable niches. I do not expect anyone at the frontier to pop up and say "no reason to train a new model" within the following decade. There will always be a demand for an LLM that's 5-10% more capable and more reliable at some highly advanced task, and generational upgrades will keep delivering those 5-10%. From increased scale and improved training both.
- IgorPartola 5mo agoI think this is exactly right. Basically when I am coding, having an agent that roughly matches my intelligence is a feature, not a bug. Having one that is 10x as smart would actively slow me down because I would have to spend the time understanding what it is doing or hand over all architecture to it and just vibe code everything, hoping that it doesn’t do the PhD version of fizzbuzz instead of the maintainable one. But for some classes of problems I think a model that is 10-100x smarter than the smartest expert is a huge boon. These would be problems that are very hard to solve but easy to verify that the solution is correct. Protein folding, sudoku, etc. Because of this I see the really smart models going to biomedical and pharma first and maybe a few high profit verticals rather than being widely deployed. I am sure Pfizer would be happy to pay for a 100x smarter than the smartest researcher model. But I am not certain that this kind of market fit would justify trillion dollar valuations in the long run. And in the meantime normal “human companion” models will go from Sonnet to some open weight model running on a Dell tower in your closet to maybe even on your phone in the next few years.
- willsmith72 5mo agoaren't you conflating being 10x as smart with code that is 10x more complicated? the relationship should be the opposite, the smartest people can write the most readable solutions