4 ms·

Makes me curious on how useful a tiny AI model based just on wikipedia would be on mobile phones. Especially if it could also cite sources.

llmnonsense 3y ago
A (tiny) AI model can not cite sources, it can only hallucinate citations. A (tiny) AI model can not create a representation of knowledge or truths based on sources it knows. It can not provide any truths except by random chance. You would have an hallucinating make-up-some-fake-facts machine on your phone.
- simonw 3y ago
  "A (tiny) AI model can not cite sources, it can only hallucinate citations." I don't think that's definitely true, if you build the system around it well. It would look something like this: 1. User asks a question. LLM extracts key concepts from that question to use as search terms. 2. LLM triggers a search of Wikipedia, getting back snippets of pages, each with their page identifier. 3. LLM is fed those snippets along with the user's question and instructions to cite pages that it uses content from. 4. LLM generates a response which includes formatted citations. This response may be complete garbage, but... 5. Your code can at least confirm that the citations correspond to the pages that you fed into the LLM. I've seen this approach work well with larger models. The open question is if it could work with smaller ones. The 7B models (Mistral and its variants in particular) are getting VERY effective. I'm confident they could mostly-work for the above sequence... and you can just about run a 7B model on a phone. The bigger question for me is if you could get this to work with a 3B model, since those are much more mobile-device friendly than 7B.
  - johnsillings 3y ago
    agree. sounds doable and awesome.
  - deleted 3y ago
    [deleted]
- KMnO4 3y ago
  Just for curiosity, why did you create a new account just for this comment?
  - llmnonsense 3y ago
    I try not to engage in discussions on social media but some days I read something and duty calls.
- Erratic6576 3y ago
  I suspect that chatGPT was shy about giving me exact quotes of Latin works for fears of disclosing its training sources and risk copyright infringement lawsuits.
- KeplerBoy 3y ago
  Meh, you could always embed the relevant section of articles into your prompt and have the model rephrase and cite that.
toomuchtodo 3y ago
Looking forward to Wikipedia building these models of their corpus and hosting them alongside the content. They could even offer as a mutable torrent cryptographically signed [1] (or rsync if you want to go old school). [1] https://news.ycombinator.com/item?id=29513547 https://news.ycombinator.com/item?id=29513547
simonw 3y ago
I think this could work really well. LLMs on mobile phones can be surprisingly capable. I've been having fun trying out this one on my iPhone: https://llm.mlc.ai/#ios https://llm.mlc.ai/#ios Getting that to work with a local copy of Wikipedia - where it can essentially run a "search_wikipedia(...)" function any time it needs extra information - would definitely be feasible, using this pattern: https://til.simonwillison.net/llms/python-react-pattern#user-content-what-does-england-share-borders-with https://til.simonwillison.net/llms/python-react-pattern#user... I think it would need to use regular text search though - you could try calculating embedding vectors for the whole of Wikipedia but even with a small embedding model I doubt you could fit those on the device, they'd take up too much space.
- WhitneyLand 3y ago
  The hybrid approach looks powerful, however I wonder how it would compare to taking the best ~7B size model and fine tuning it directly on Wikipedia? Maybe impractical, but seems like it could potentially be more seamlessly integrated. Keeping in mind: - Many models already include WP in their training, at least parts of it. - The parts that are missing or a subset of what’s missing could be supplemented via finetuning. - Data is commonly compressed about 100x during training (it’s lossy) so the total size might still run on a phone. You’d then have a fully capable LLM, but enhanced to be comprehensively encyclopedic.

llmnonsense 3y ago

A (tiny) AI model can not cite sources, it can only hallucinate citations. A (tiny) AI model can not create a representation of knowledge or truths based on sources it knows. It can not provide any truths except by random chance. You would have an hallucinating make-up-some-fake-facts machine on your phone.

simonw 3y ago
"A (tiny) AI model can not cite sources, it can only hallucinate citations." I don't think that's definitely true, if you build the system around it well. It would look something like this: 1. User asks a question. LLM extracts key concepts from that question to use as search terms. 2. LLM triggers a search of Wikipedia, getting back snippets of pages, each with their page identifier. 3. LLM is fed those snippets along with the user's question and instructions to cite pages that it uses content from. 4. LLM generates a response which includes formatted citations. This response may be complete garbage, but... 5. Your code can at least confirm that the citations correspond to the pages that you fed into the LLM. I've seen this approach work well with larger models. The open question is if it could work with smaller ones. The 7B models (Mistral and its variants in particular) are getting VERY effective. I'm confident they could mostly-work for the above sequence... and you can just about run a 7B model on a phone. The bigger question for me is if you could get this to work with a 3B model, since those are much more mobile-device friendly than 7B.
- johnsillings 3y ago
  agree. sounds doable and awesome.
- deleted 3y ago
  [deleted]
KMnO4 3y ago
Just for curiosity, why did you create a new account just for this comment?
- llmnonsense 3y ago
  I try not to engage in discussions on social media but some days I read something and duty calls.
Erratic6576 3y ago
I suspect that chatGPT was shy about giving me exact quotes of Latin works for fears of disclosing its training sources and risk copyright infringement lawsuits.
KeplerBoy 3y ago
Meh, you could always embed the relevant section of articles into your prompt and have the model rephrase and cite that.

toomuchtodo 3y ago

Looking forward to Wikipedia building these models of their corpus and hosting them alongside the content. They could even offer as a mutable torrent cryptographically signed [1] (or rsync if you want to go old school). [1] https://news.ycombinator.com/item?id=29513547 https://news.ycombinator.com/item?id=29513547

simonw 3y ago

I think this could work really well. LLMs on mobile phones can be surprisingly capable. I've been having fun trying out this one on my iPhone: https://llm.mlc.ai/#ios https://llm.mlc.ai/#ios Getting that to work with a local copy of Wikipedia - where it can essentially run a "search_wikipedia(...)" function any time it needs extra information - would definitely be feasible, using this pattern: https://til.simonwillison.net/llms/python-react-pattern#user-content-what-does-england-share-borders-with https://til.simonwillison.net/llms/python-react-pattern#user... I think it would need to use regular text search though - you could try calculating embedding vectors for the whole of Wikipedia but even with a small embedding model I doubt you could fit those on the device, they'd take up too much space.

WhitneyLand 3y ago
The hybrid approach looks powerful, however I wonder how it would compare to taking the best ~7B size model and fine tuning it directly on Wikipedia? Maybe impractical, but seems like it could potentially be more seamlessly integrated. Keeping in mind: - Many models already include WP in their training, at least parts of it. - The parts that are missing or a subset of what’s missing could be supplemented via finetuning. - Data is commonly compressed about 100x during training (it’s lossy) so the total size might still run on a phone. You’d then have a fully capable LLM, but enhanced to be comprehensively encyclopedic.