4 ms·
> not smart > slightly smarter > way smarter > Last one, "slightly smarter" So, the the usual s-curve, that has an exponential phase, then topping out?
by SideburnsOfDoom 2y ago
> not smart
> slightly smarter
> way smarter
> Last one, "slightly smarter"
So, the the usual s-curve, that has an exponential phase, then topping out?
- moffkalast 2y agoPretty much, yep. There was definitely a more significant jump there in the middle where 7B models went from being a complete waste of time to actually useful. Then going from being able to craft a sensible response to 80% of questions to 90% is a much smaller apparent increase but takes a lot more compute to achieve as per the pareto principle.
- hdhshdhshdjd 2y agoI see giant models like Intel chips over the last decade: big, powerful, expensive, energy hogs. Small models are like arm: you get much of the performance you actually need for common consumer tasks, very cheap to run, and energy efficient. We need both but I personally spend most of my ML time on small models training and I’m very happy with the results.