3 ms·
They're not blenders. This is clear from the fact that you can distill the logic ability from a 700b parameter model into a 14b model and maintain almost all o
by literalAardvark 10mo ago
They're not blenders.
This is clear from the fact that you can distill the logic ability from a 700b parameter model into a 14b model and maintain almost all of it.
You just lose knowledge, which can be provided externally, and which is the actual "pirated" part.
The logic is _learned_
- bayindirh 10mo agoAre there any recent publications about it so I can refresh myself on the matter?
- D-Machine 10mo agoYou won't find any trustworthy papers on the topic because GP is simply wrong here. That models can be distilled has no bearing whatsoever on whether a model has learned actual knowledge or understanding ("logic"). Models have always learned sparse/approximately-sparse and/or redundant weights, but they are still all doing manifold-fitting. The resulting embeddings from such fitting reflect semantics and semantic patterns. For LLMs trained on the internet, the semantic patterns learned are linguistic, which are not just strictly logical, but also reflect emotional, connotational, conventional, and frequent patterns, all of which can be illogical or just wrong. While linguistic semantic patterns are correlated with logical patterns in some cases, this is simply not true in general.
- encyclopedism 10mo agoIt hasn't learned any LOGIC. It has 'learned' patterns from the input.
- theshrike79 10mo agoWhat is logic other than applying patterns?
- encyclopedism 10mo agoThe definition is broad for now this will do: Logic is the study of correct reasoning.
- vidarh 10mo agoHow is that different from applying patterns?