4 ms·
Well, it makes sense. The smaller the niche, the lesser weight in the overall training loss. At the end of the day, LLMs are (literally) classifiers that assign
by jurgenaut23 2y ago
Well, it makes sense. The smaller the niche, the lesser weight in the overall training loss. At the end of the day, LLMs are (literally) classifiers that assign probabilities to tokens given some previous tokens.
- svantana 2y agoYes, but o1, o3 and sonnet are not necessarily pure language models - they are opaque services. For all we know they could do syntax-aware processing or run compilers on code behind the scenes.
- skissane 2y agoThe fact they make mistakes like this implies they probably don’t, since surely steps like that would catch many of these
- bubblyworld 2y agoSOTA language models are trained with reinforcement learning now too, so the model distributions can be far removed from the underlying data distribution. Not that I think your overall point is wrong (it's obviously a very strong bias), but things are getting more complex with time.