3 ms·
Would it not be better to ask models to search the topic on the Internet and then answer? I do not understand why we expect small LLMs to answer from own knowle
by brainless 2mo ago
Would it not be better to ask models to search the topic on the Internet and then answer? I do not understand why we expect small LLMs to answer from own knowledge.
- walrus01 2mo agoI don't, really, but 20B is also not that small... It's an intentionally weird question to see how confidently incorrect something will be. It certainly writes a plausible sounding explanation that could fool someone for whom English is their 2nd or 3rd language, or is not familiar with specific North American slang. It's also something I've seen has great results with esoteric individual pieces of knowledge that works fine in a Q6 or Q8 quantized LLM but breaks down in a bad way at worse quantization.
- sznio 2mo agoParameter count is not everything. 20b parameters * 1.5 bits per parameter is just 30 billion bits, about 3.75gb a full 20b fp16 is about 40GB. I find it weird how a smaller model still produces decent text, except it bullshits all the way.
- HelloUsername 2mo ago> Ask models to search on the Internet and answer? To me, the benefit of running small models is that they fit on your device exactly that you don't need any internet connection. It's all local and offline, so you can still consult for information in any scenario.
- CTDOCodebases 2mo agoI thought the benefit of small models is that they are a natural language UI to whatever they are connected to. It seems strange to me to expect a small model to answer everything correctly when every device that they are being used on support networking and knowledge is constantly evolving.
- nl 2mo agoThis is a very bad way of thinking of it. Small LLMs have clues about real knowledge but only surface level answers will be accurate.
- dofm 2mo agoThe real target of small models is surely the opposite: knowing with clarity how to conversationally or otherwise reason about your requirements in natural language, sound and imagery, consult APIs that interact with apps, and then have a strong, generic capability to reason about what it finds. If you want it to know more things locally, you could hopefully distribute wikipedia or other appropriate large sources on a flash storage, properly pre-indexed, with a local tool API, and let it work its magic on that. Not least because that is as I understand it, considerably easier to incrementally update. This is, without doubt, a very difficult target, not least because it is rather at odds with the way LLMs work on a crude level.
- unglaublich 2mo agoTrue but they could query a local 50GB Wikipedia dump for factual info’s via embedding search. All still local.
- em500 2mo agoI wonder if someone already made a somewhat turnkey setup like this? Small models should ideally not waste a lot of parameters on trying to memorize too much knowledge and then hallucinate / interpolate a lot of the gaps.
- spider-mario 2mo agoMaybe we don’t necessarily expect them to answer from their own knowledge, but to either do that or say “I don’t know”.
- nozzlegear 2mo agoIt would indeed be better to have them search the internet for an answer, but the issue (as I see it) is overconfident models don't know when they should search the internet because they think they already know the answer.