4 ms·
It would be cool to understand when the system will use one or the other (the ~3 billion on-device model or the bigger one on Apple servers).
by GaggiX 2y ago
It would be cool to understand when the system will use one or the other (the ~3 billion on-device model or the bigger one on Apple servers).
- swatcoder 2y agoConceivably, they don't have precise answers for that yet, and won't until after they see what real-world usage looks like. They built out a system that's ready to scale to deliver features that may not work on available hardware, but they're also incentivized to minimize actual reliance on that cloud stuff as it incurs per-use costs that local runs don't.
- GaggiX 2y agoYeah this is probably right. If it works well enough during real-world usage it will be using the on-device model, if not then there is the bigger one on the servers. There is also GPT-4o, so they have 3 different models to use depending on the task.
- aixpert 2y agoif you have ever used a 3 billion or 7 billion parameter model you know that they are really bad at text generation, so this will be done in the cloud