3 ms·
There is no concept of "knowledge" in LLM as it is on Wikipedia. The question-tokens define the answer-tokens. That's it. The art relies in clustering the rele
by SeriousM 4mo ago
There is no concept of "knowledge" in LLM as it is on Wikipedia.
The question-tokens define the answer-tokens. That's it. The art relies in clustering the relevant weights together.
- tempaccount420 4mo ago> There is no concept of "knowledge" in LLM as it is on Wikipedia. There can be, you don't know if the closed source models aren't using something like DeepSeek's Engram.
- HarHarVeryFunny 4mo agoThe name "Engram" (n-gram) says it all - this is just another type of statistical word association, not a factual knowledge store. While DeepSeek describe this as "knowledge lookup", what Engram is really trying to do is separate dynamic reasoning from static pattern recall, with the static patterns just being word-level n-gram statistics, not declarative facts/knowledge. Just because 2-3 words often appear together in a sequence doesn't mean they represent a fact or truth (or falsehood) - it is just an n-gram statistical regularity. If Engram helps reduce LLM GPU memory and FLOP requirements then that is great, but it's not a solution for Hallucination.
- baq 4mo agoIf it were that simple we’d all be talking with sql and yet this isn’t happening. Circuits which emerge in the layers during training are much more complicated than a simple Bayesian relation.
- solid_fuel 4mo agoCorrect, LLMs are not ontologically capable of “knowing”. That is why I put “know” in quotes.