The "biggest model that fits" instinct is just wrong now. Compact models routinely beat massive predecessors from 12 months ago. Scaling laws only reliably pred

The "biggest model that fits" instinct is just wrong now. Compact models routinely beat massive predecessors from 12 months ago. Scaling laws only reliably predict pre-training loss anyway, not how the model actually performs on your task. Dug into the research behind this: https://philippdubach.com/posts/the-most-expensive-assumption-in-ai/ https://philippdubach.com/posts/the-most-expensive-assumptio...