4 ms·
This is a local device LLM, outdated of a whole year, which is massive for LLMs, running on servers. Privacy side, there is little reason to run this on serve
by Kuinox 2y ago
This is a local device LLM, outdated of a whole year, which is massive for LLMs, running on servers.
Privacy side, there is little reason to run this on servers, a Pixel 6 could run 7B models at 5 token/s a year ago.
It's bad by incompetence, 7B models of a year ago were terribly bad.
It's not privacy-first enough, as it's possible to run the AI directly in the browser, but for some reason they didnt do it.
- mentalgear 2y agoIt's unlikely that they are doing only a plain old LLM query. It's probably a more advanced setup (RAG, Page/LLM Summary Result Querying, maybe even some sort of Summarization validation checking). Also consider accessibility for low-end devices and development countries.
- Kuinox 2y agoI don't see any indication that they use any RAG. It still doesn't explain why they use an old model. More recent models can run on any phone and have better quality than the model they are currently using.
- zozbot234 2y ago> it's possible to run the AI directly in the browser, but for some reason they didnt do it. Perhaps this is a proof of concept and they will have optional Firefox integration at a later time. Firefox uses local AI for webpage translation already.
- aucisson_masque 2y agoNot everyone has pixel 6 device, same for computer. There are still lots and lots of old device in third world countries. Running a year old model, if it's power efficient and economically constrained makes complete sense to me.