45 ms·
How big does an LLM need to be to support natural language queries with RAG?
by nuc1e0n 11mo ago
How big does an LLM need to be to support natural language queries with RAG?
- mattnewton 11mo agoMy hot (maybe just warm these days) take is, the problem with voice assistants on phones is they have to be able to have reasonable responses to a long tail or users will learn not to use them, since the use cases aren’t discoverable and the primarily value is talking to it like a person. So voice assistants backed by very large LLMs over the network are going to win even if we solve the (substantial) battery usage issue.
- nuc1e0n 11mo agoWhy even bother with the text generation then? You could just make a phone call to an LLM with a TTS frontend. Like with directory enquiries back in the day. Which can be set up as easily as a BBS if you have a home server rack like Jeff Geerling makes youtube videos about.
- jazzyjackson 11mo ago1 800 CHAT GPT exists, I use it often enough to put it on speed dial on my flip phone