5 ms·
The latency will make or break this. I run all the models on-device as much as possible.
by xena 3y ago
The latency will make or break this. I run all the models on-device as much as possible.
- Terr_ 3y agoPlus there's the whole privacy/autonomy aspect. Notwithstanding the last few decades of trend, many people still want certain data/behavior to stay firmly within a device they (supposedly) own and control, as opposed to owning a dumb-terminal.
- johnfn 3y agoI don't think that's true. Latency is critical when the limiting factor is that you know what you want to do but in order to do it, you need to issue many requests all at once. With GPT4 I can issue requests at a high enough level that the limiting factor is actually me thinking of what the next request will be.