5 ms·

> Hallucination is not a bug that scaling can fix. It is the structural consequence of operating without concepts. NNs are as close to continuous as we can get

> Hallucination is not a bug that scaling can fix. It is the structural consequence of operating without concepts. NNs are as close to continuous as we can get with discrete computing. They’re flexible and adaptable and can contain many “concepts.” But their chief strength is also their chief weakness: these “concepts” are implicit. I wonder if we can get a hybrid architecture that has the flexibility of NNs while retaining discrete concepts like a knowledge base does.

measurablefunc 7mo ago
There is nothing continuous on the computer, it's all bit strings & boolean arithmetic. The semantics imposed on the bit strings does not exist anywhere in the arithmetic operations, i.e. there is no arithmetic operation corresponding to something as simple as the color red.
- kelseyfrog 7mo ago
  It sounds like you're saying that if a computer had infinite precision then hallucinations would not occur?
  - measurablefunc 7mo ago
    The way neural networks work is that the base neural network is embedded in a sampling loop, i.e. a query is fed into the network & the driver samples output tokens to append to the query so that it can be re-fed back into the network (q → nn → [a, b, c, ...] → q + sample([a, b, c, ...])). There is no way to avoid hallucinations b/c hallucinations are how the entire network works at the implementation level. The precision makes no difference b/c the arithmetic operations are semantically void & only become meaningful after they are interpreted by someone who knows to associated 1 /w red, 2 w/ blue, 3 w/ clouds, & so on & so forth. The mapping between the numbers & concepts does not exist in the arithmetic.
    - kelseyfrog 7mo ago
      Oh, I thought that the embedding space of the residual stream was precisely that.
      - measurablefunc 7mo ago
        The arithmetic is meaningless, it doesn't matter what you call it b/c on the computer it's all bit strings & boolean arithmetic. You can call some sequence of operations residual & others embeddings but that is all imposed top-down. There is nothing in the arithmetic that indicates it is somehow special & corresponds to embeddings or residuals.
        kelseyfrog 7mo ago
        Ah ok, so if we had such a mapping then models wouldn't hallucinate?
        measurablefunc 7mo ago
        Maybe it's better if you define the terms b/c what I mean by hallucination is that the arithmetic operations + sampling mean that it's all hallucinations. The output is a trajectory of a probabilistic computation over some set of symbols (0s & 1s). Those symbols are meaningless, the only reason they have meaning is b/c everyone has agreed that the number 97 is the ascii code for "a" & every conformant text processor w/ a conformant video adapter will convert 97 (0b1100001) into the display pattern for the letter "a".
        kelseyfrog 7mo ago
        So kind of like if you flip a coin, the sampling means the heads or tails you get isn't real?
        measurablefunc 7mo ago
        It's when you define heads or tails however you want & then tell me you have objective semantics for each side of the coin when all you've really done is established a convention about which side is which. The coin is real, what you call each side is a convention & what semantics you attach to a sequence of flips is also a convention that has nothing to do with the reality of the coin.
- naasking 7mo ago
  > The semantics imposed on the bit strings does not exist anywhere in the arithmetic operations, Correct, the semantics is actually in the network of relations between the nodes. That has been one of the major lessons of LLMs, and it validates the systems response to Searle's Chinese Room.
AdieuToLogic 7mo ago
> NNs are as close to continuous as we can get with discrete computing. This is incorrect. For example, fuzzy logic[0] can model analog ("continuous") truth beyond discrete digital representations, such as 1/0, true/false, etc. 0 - https://en.wikipedia.org/wiki/Fuzzy_logic https://en.wikipedia.org/wiki/Fuzzy_logic

measurablefunc 7mo ago

There is nothing continuous on the computer, it's all bit strings & boolean arithmetic. The semantics imposed on the bit strings does not exist anywhere in the arithmetic operations, i.e. there is no arithmetic operation corresponding to something as simple as the color red.

kelseyfrog 7mo ago
It sounds like you're saying that if a computer had infinite precision then hallucinations would not occur?
- measurablefunc 7mo ago
  The way neural networks work is that the base neural network is embedded in a sampling loop, i.e. a query is fed into the network & the driver samples output tokens to append to the query so that it can be re-fed back into the network (q → nn → [a, b, c, ...] → q + sample([a, b, c, ...])). There is no way to avoid hallucinations b/c hallucinations are how the entire network works at the implementation level. The precision makes no difference b/c the arithmetic operations are semantically void & only become meaningful after they are interpreted by someone who knows to associated 1 /w red, 2 w/ blue, 3 w/ clouds, & so on & so forth. The mapping between the numbers & concepts does not exist in the arithmetic.
  - kelseyfrog 7mo ago
    Oh, I thought that the embedding space of the residual stream was precisely that.
    - measurablefunc 7mo ago
      The arithmetic is meaningless, it doesn't matter what you call it b/c on the computer it's all bit strings & boolean arithmetic. You can call some sequence of operations residual & others embeddings but that is all imposed top-down. There is nothing in the arithmetic that indicates it is somehow special & corresponds to embeddings or residuals.
      - kelseyfrog 7mo ago
        Ah ok, so if we had such a mapping then models wouldn't hallucinate?
        measurablefunc 7mo ago
        Maybe it's better if you define the terms b/c what I mean by hallucination is that the arithmetic operations + sampling mean that it's all hallucinations. The output is a trajectory of a probabilistic computation over some set of symbols (0s & 1s). Those symbols are meaningless, the only reason they have meaning is b/c everyone has agreed that the number 97 is the ascii code for "a" & every conformant text processor w/ a conformant video adapter will convert 97 (0b1100001) into the display pattern for the letter "a".
        kelseyfrog 7mo ago
        So kind of like if you flip a coin, the sampling means the heads or tails you get isn't real?
        measurablefunc 7mo ago
        It's when you define heads or tails however you want & then tell me you have objective semantics for each side of the coin when all you've really done is established a convention about which side is which. The coin is real, what you call each side is a convention & what semantics you attach to a sequence of flips is also a convention that has nothing to do with the reality of the coin.
naasking 7mo ago
> The semantics imposed on the bit strings does not exist anywhere in the arithmetic operations, Correct, the semantics is actually in the network of relations between the nodes. That has been one of the major lessons of LLMs, and it validates the systems response to Searle's Chinese Room.

AdieuToLogic 7mo ago

> NNs are as close to continuous as we can get with discrete computing. This is incorrect. For example, fuzzy logic[0] can model analog ("continuous") truth beyond discrete digital representations, such as 1/0, true/false, etc. 0 - https://en.wikipedia.org/wiki/Fuzzy_logic https://en.wikipedia.org/wiki/Fuzzy_logic