3 ms·
Llama-1, 1T tokens, dumb as a box of rocks Llama-2, 2T tokens, smarter than a box of rocks Mistral-7B, 8T tokens, way smarter than llama-2 Llama-3, 15T token
by moffkalast 2y ago
Llama-1, 1T tokens, dumb as a box of rocks
Llama-2, 2T tokens, smarter than a box of rocks
Mistral-7B, 8T tokens, way smarter than llama-2
Llama-3, 15T tokens, smarter than anything a few times its size
Gemma-2, 13T synthetic tokens, slightly better than llama-3
(for the same approximate parameter size)
I think it roughly tracks that moar data = moar betterer.
- cerved 2y agobut the OP was talking about the size of the context window, not the size of the training corpus
- moffkalast 2y agoHmm right, I read that wrong. Still, interesting data I think.
- SideburnsOfDoom 2y ago> not smart > slightly smarter > way smarter > Last one, "slightly smarter" So, the the usual s-curve, that has an exponential phase, then topping out?
- moffkalast 2y agoPretty much, yep. There was definitely a more significant jump there in the middle where 7B models went from being a complete waste of time to actually useful. Then going from being able to craft a sensible response to 80% of questions to 90% is a much smaller apparent increase but takes a lot more compute to achieve as per the pareto principle.
- hdhshdhshdjd 2y agoI see giant models like Intel chips over the last decade: big, powerful, expensive, energy hogs. Small models are like arm: you get much of the performance you actually need for common consumer tasks, very cheap to run, and energy efficient. We need both but I personally spend most of my ML time on small models training and I’m very happy with the results.