3 ms·

I couldn’t help but ask Claude 2 for some better words to name this phenomenon. The response was fascinating: “Here are some alternative words or concepts that

I couldn’t help but ask Claude 2 for some better words to name this phenomenon. The response was fascinating: “Here are some alternative words or concepts that could potentially provide a more nuanced or accurate description of the phenomenon compared to solely using "confabulation": - Imputation: Generating explanatory responses by imputing or attributing plausible details, characteristics, or causal relationships even when factually incorrect. - Speculation: The responses reflect a type of informed speculation when direct factual knowledge is lacking, as opposed to confabulation which often implies a pathological or impaired process. - Heuristic reasoning: The behavior emerges from the model's trained heuristic techniques for reasoning and responding rather than any psychological process like confabulation per se. - Plausible surmise: The responses take the form of plausible but unfounded surmises put forth due to the system design and limitations rather than an intent to confabulate. - Bridging inference: The model draws bridging or gap-filling inferences to connect related concepts or complete a response, even if factually ungrounded. - Suppositional fabrication: The language hints more strongly that the responses are fabricated suppositions not intended to deliberately deceive but still warranting cautious interpretation. - Educated approximation: The responses could be viewed as a kind of educated approximation or speculation given training-imposed response pressures rather than confabulation due to impairment. Designating it as solely "confabulation" may over-attribute psychological realism without addressing countervailing technical factors.” Personally, I thought “Suppositional fabrication” came closest. Maybe just fabrication would work.

intended 3y ago
Approximation. Imputation, Reasoning, Fabrication will end up sending people down the anthropomorphization rabbit hole. Approximation is "good enough".
- ImHereToVote 3y ago
  I think some anthropomorphizing is apt. It is trained on anthropic data from the anthropocene after all.
- wizofaus 3y ago
  But when we approximate we use appropriate language to make it clear that's what we're doing. ChatGPT will confidently give very precise answers based around completely fabricated "facts".
  - Jensson 3y ago
    It will also sometimes use approximate language when stating facts, so you can't use how it says things even as a hint.
  - intended 3y ago
    Your example highlights the issue! Chat GPT isnt even giving "precise answers". Its just presenting the next best token. It would do the same thing if the tokens were converted to another set of symbols instead of ASCII/UTF. The fact that the tokens happen to look like english, and that combinations of english have meaning.. not really a concern to the token generator. Its never inaccurate - its always mathematically faithful. Its just created the next token to suffice its reward function. And No reward function can magically tell you what reality is* We can discuss this on HN though. In reality, there are enough managers and CEOs who are planning to do silly things with Gen AI. To survive those conversations, this idea of "thinking system in a server" needs to go. ---- If some code can do that, we could just give up on science entirely and let the algorithm solve everything for us.
trashtester 3y ago
The way I think of the phenomenon, neural networks interpolate. Facts form only points in this space, while the network weights represents the patterns of the points by "curves". They are so good at it, though, that it almost seem like they remember each point. Their ability to generalize, though, comes from representing the learning as these "curves". Sometimes they produce generalizations that are poor or false. Even when they are false, though, they may be good. In fact, Stephen Wolfram is claiming that when GPT-4 "halucinates" a function name for the Wolfram Language, the team often agrees with GPT-4 that precisely that call SHOULD be part of the language, so they go on and implement it AFTER it came up as a halucination. One might (and I would) argue that this behavior is precisely what most human creativity (in technical domains) is about.
masswerk 3y ago
I'm still not happy with any of this: the model is traversing a semantic field according to inherent probabilities, which are in turn a function of its specific aggregation. Hence, as an alleged "speaker", it is more like coerced to these "hallucinations", which are more "in the faults" of the semantic field, rather than being a production of any agency. We may say, this phenomenon is "oozing" from the semantic aggregation, as the model is under stress.

intended 3y ago

Approximation. Imputation, Reasoning, Fabrication will end up sending people down the anthropomorphization rabbit hole. Approximation is "good enough".

ImHereToVote 3y ago
I think some anthropomorphizing is apt. It is trained on anthropic data from the anthropocene after all.
wizofaus 3y ago
But when we approximate we use appropriate language to make it clear that's what we're doing. ChatGPT will confidently give very precise answers based around completely fabricated "facts".
- Jensson 3y ago
  It will also sometimes use approximate language when stating facts, so you can't use how it says things even as a hint.
- intended 3y ago
  Your example highlights the issue! Chat GPT isnt even giving "precise answers". Its just presenting the next best token. It would do the same thing if the tokens were converted to another set of symbols instead of ASCII/UTF. The fact that the tokens happen to look like english, and that combinations of english have meaning.. not really a concern to the token generator. Its never inaccurate - its always mathematically faithful. Its just created the next token to suffice its reward function. And No reward function can magically tell you what reality is* We can discuss this on HN though. In reality, there are enough managers and CEOs who are planning to do silly things with Gen AI. To survive those conversations, this idea of "thinking system in a server" needs to go. ---- If some code can do that, we could just give up on science entirely and let the algorithm solve everything for us.

trashtester 3y ago

The way I think of the phenomenon, neural networks interpolate. Facts form only points in this space, while the network weights represents the patterns of the points by "curves". They are so good at it, though, that it almost seem like they remember each point. Their ability to generalize, though, comes from representing the learning as these "curves". Sometimes they produce generalizations that are poor or false. Even when they are false, though, they may be good. In fact, Stephen Wolfram is claiming that when GPT-4 "halucinates" a function name for the Wolfram Language, the team often agrees with GPT-4 that precisely that call SHOULD be part of the language, so they go on and implement it AFTER it came up as a halucination. One might (and I would) argue that this behavior is precisely what most human creativity (in technical domains) is about.

masswerk 3y ago

I'm still not happy with any of this: the model is traversing a semantic field according to inherent probabilities, which are in turn a function of its specific aggregation. Hence, as an alleged "speaker", it is more like coerced to these "hallucinations", which are more "in the faults" of the semantic field, rather than being a production of any agency. We may say, this phenomenon is "oozing" from the semantic aggregation, as the model is under stress.