2 ms·
I think this could work really well. LLMs on mobile phones can be surprisingly capable. I've been having fun trying out this one on my iPhone: https://llm.mlc.
by simonw 3y ago
I think this could work really well.
LLMs on mobile phones can be surprisingly capable. I've been having fun trying out this one on my iPhone: https://llm.mlc.ai/#ios https://llm.mlc.ai/#ios
Getting that to work with a local copy of Wikipedia - where it can essentially run a "search_wikipedia(...)" function any time it needs extra information - would definitely be feasible, using this pattern: https://til.simonwillison.net/llms/python-react-pattern#user-content-what-does-england-share-borders-with https://til.simonwillison.net/llms/python-react-pattern#user...
I think it would need to use regular text search though - you could try calculating embedding vectors for the whole of Wikipedia but even with a small embedding model I doubt you could fit those on the device, they'd take up too much space.
- WhitneyLand 3y agoThe hybrid approach looks powerful, however I wonder how it would compare to taking the best ~7B size model and fine tuning it directly on Wikipedia? Maybe impractical, but seems like it could potentially be more seamlessly integrated. Keeping in mind: - Many models already include WP in their training, at least parts of it. - The parts that are missing or a subset of what’s missing could be supplemented via finetuning. - Data is commonly compressed about 100x during training (it’s lossy) so the total size might still run on a phone. You’d then have a fully capable LLM, but enhanced to be comprehensively encyclopedic.