4 ms·
The logic seems mad to me because SLMs can simply not hold as much information as an LLM. Maybe if you combine an SLM with a database (as a tool) then it could
by amelius 2mo ago
The logic seems mad to me because SLMs can simply not hold as much information as an LLM.
Maybe if you combine an SLM with a database (as a tool) then it could work, but someone should first prove that.
- fph 2mo agoBut do you really need a model that has the complete Duran Duran discography memorized and preloaded in RAM at all time?
- amelius 2mo agoThat's a different question. Probably not. But: 1. Training a large model with lots of information, then stripping the "useless" information from that model to obtain a small model => nobody has shown this. 2. Training a small model, letting it use a database tool so it scores the same as a large model without database => nobody has shown this.
- js8 2mo agoI partly agree. Theory (of computation) shows it must be possible, however nobody has produced the small model (well, depends who you ask, what is small, article disputes that) and the database yet. To go very small (thousands of rules) so that the reasoner can be understood by humans and proven sound - might be computationally quite difficult.
- slfnflctd 2mo agoThis really elegantly sums up two of the most major areas of research in the field right now. I expect people are going to hammer at both problems for a while. It will be interesting to see how much useful stuff is built as a result.
- embedding-shape 2mo agoI mean maybe yes? The hypothesis from the early GPT days was (and in a small way still remains): "If we just chuck more data into the training, does it get better at X, even if the data was seemingly unrelated to X?", and the workings of LLMs seem to kind be pointing in that direction, although with some ceiling. But seemingly models good at programming for example, would get worse at programming if you removed everything not-programming. Train a model solely on syntax, and it'll be worse than a general purpose LLM on syntax, in general at least.
- js8 2mo ago"But seemingly models good at programming for example, would get worse at programming if you removed everything not-programming. Train a model solely on syntax, and it'll be worse than a general purpose LLM on syntax, in general at least." That might be because emergence of capabilities to reason about programs requires abstractions (such as fuzzy and modal logic) that are rarely present in software sources. That doesn't mean the reasoning model itself has to be large; neither does it have to emerge from the ML training on large language corpus, we might construct it by different means.
- medwards666 2mo agoBut what if I _really like_ Duran Duran???
- sandytoast 2mo agoThen you’re hungry like the wolf
- rotis 2mo agoSure. I don't give a damn about them. But Phil Collins man. I love reviewing his body of work before I get worked up. Obvously we cannot cut him, because I need him. So how do you decide what to leave out?
- js8 2mo agoThe cost you pay is in additional reasoning the SLMs have to do. As I write elsewhere, LLM "remembers" that "Socrates is mortal", or other commonly useful deduction. SLM might need to derive it first by reasoning from the DB, which slows it down. (Or worse, it might miss the correct reasoning because it's just too much side quests to follow.) But the advantage is flexibility.