3 ms·
I'd argue we jumped that shark since the shift in focus to post training. Labs focus on getting good at specific formats and tasks. The generalization argument
by ianbutler 9mo ago
I'd argue we jumped that shark since the shift in focus to post training. Labs focus on getting good at specific formats and tasks. The generalization argument was ceded (not in the long term but in the short term) to the need to produce immediate value.
Now if a format dominates it will be post trained for and then it is in fact better.
- Der_Einzige 9mo agoAnthropic and Gemini still release new pre-training checkpoints regularly. It's just OpenAI who got stupid on that. RIP GPT-4.5
- ianbutler 9mo agoAll models released from those providers go through stages of post training too, none of the models you interact with go from pre-training to release. An example of the post training pipeline is tool calling, that is to my understanding a part of post training and not pre training in general. I can't speak to what the exact split is or what is a part of post training versus pre training at various labs but I am exceedingly confident all labs post train for effectiveness in specific domains.
- Der_Einzige 9mo agoI did not claim that post training doesn't happen on these models, and you are being extremely patronizing (I publish quite a bit of research on LLMs at top conferences). I claimed that OpenAI overindexed on getting away with aggressive post-training on old pre-training checkpoints. Gemini / Anthropic correctly realized that new pre-training checkpoints need to happen to get the best gains in their latest model releases (which get post-trained too).
- ianbutler 9mo agoIf you read that as patronizing that says more about you than me personally, I have no idea who you are so your own insecurity at what is a rather unloaded explanation perplexes me.