I don’t think it’s all that surprising. VRAM is much smaller than all the data the bot was trained on, so some stuff gets dropped or “lossily compressed”. In th

I don’t think it’s all that surprising. VRAM is much smaller than all the data the bot was trained on, so some stuff gets dropped or “lossily compressed”. In the case of foundation models, this is likely a bit of everything. Fine tunes take that and change what gets dropped or compressed. Rather than somewhat evenly forgetting things, it forgets very little of the knowledge they’re targeting (like coding) and in exchange drops much more of “everything else”. I don’t believe it works this way, but in a sense, it’s like GPT4o is a 400B parameter model but only devotes 20B parameters to coding because the rest are taken up by Wikipedia and knowing French and what not. A 70B coding fine-tune might be able to devote more than 20B parameters to coding, in exchange for only speaking English, having very little encyclopedic knowledge, etc. It’s kind of like CPUs vs ASICs. CPUs (GPT4o) will perform better on average at random tasks, while ASICs (finetunes) do dramatically better on the task they were built for, even if they have a lower transistor count.