"Better" is such a subjective word in AI. I don't think it will be better than GPT4 in most areas. It's very difficult to imagine 3b or less being good in the g

"Better" is such a subjective word in AI. I don't think it will be better than GPT4 in most areas. It's very difficult to imagine 3b or less being good in the general domain without some kind of current-day-math-subverting breakthroughs. However, something like Siri would need a very specialized model and a lot of value would come from its integration with apps. So in that sense, it could be a lot better at these specific tasks through training data curation than even very large models. 3B models can run quite well on consumer hardware, probably they'd work reasonably locally on iPhones. 250M would fly on an iPhone. I wonder if that's what they're aiming for. The models could also be so small because that makes sense in R&D, at this stage. But ultimately, most of the value to the user would probably come from how these models integrate with the app ecosystem. What's the point of making them generally smart if that will just cost exponentially more compute? From what we've seen with Llama2, for example, the compute costs go up exponentially for any sort of cleverness in the broad general domain of knowledge. In other words, the power law of the long tail/strongly diminishing returns apply for utility per kWh spent. Hope Apple is thinking this, I'd rather not tack on unnecessarily large compute costs to my iCloud subscription.