3 ms·

I don’t think it’s just a matter of curating data sets for accuracy. The model seem to be able to invent falsehoods on its own based on no data whatsoever - e.g

I don’t think it’s just a matter of curating data sets for accuracy. The model seem to be able to invent falsehoods on its own based on no data whatsoever - e.g. I read that researchers asked an LLM about a made up biochemist and the LLM generated an entire fictional history for it.

valine 3y ago
My hope would be that by labeling factual and fictional data it would coax the model into a state where it doesn’t invent facts outside of a creative writing context. Inventing falsehoods is practically the definition of creative writing, you don’t want the model to lose that ability. This is purely hypothetical, but I imagine that the internal mechanisms the model uses to invent stories out of thin air are different from the mechanisms the model uses to recall precise facts. Providing additional labels could help sharpen the division between creative and factual writing tasks.
- SrslyJosh 3y ago
  The model can't "invent facts" because it doesn't know what facts are. It's a statistical model that encodes information about the text it was trained on. It has no higher level abstractions to inform its output. > I imagine that the internal mechanisms the model uses to invent stories out of thin air are different from the mechanisms the model uses to recall precise facts. Nope! If you input "What is the capital of the United States of America?", it's probably going to output the correct answer, because the question (and answer) probably appeared many times in the training data. If you input "What is the capital of the Gronk Republic?", whatever output you get is generated via the exact same mechanism.
  - valine 3y ago
    That’s the “LLMs are a fuzzy jpeg of the web” theory, and it’s far, far from the scientific consensus. No one knows for sure exactly what happens inside a 500B parameter model. I’ll just leave you with GPT4’s response to your question “What is the capital of the Gronk Republic”. “As an AI language model, I am not aware of any existing country or political entity called the "Gronk Republic." It could be a fictional or hypothetical place, in which case the capital would be determined by the creator or context in which it is mentioned. If you are referring to a real location, please provide more information or clarify the name of the place you are asking about.”
    - worrycue 3y ago
      I wonder if GPT4 “traps” such questions and handles them using non-LLM algorithms. I mean Google already does it for that type of question, e.g. “capital of the irish republic” will return Dublin, Ireland. “Gronk” returns result about some (famous?) guy nicknamed Gronk. Have you tried “arguing” with it and try to gaslight it by insisting that it’s non-fictional?
      - deleted 3y ago
        [deleted]
      - valine 3y ago
        Yeah it handles it all gracefully. It’s definitely not just trapping the question.
    - Applejinx 3y ago
      Here's the thing, though. Gronk is Robert James Gronkowski, a football player and celebrity, playing mainly for the New England Patriots. For that reason, I'd say the 'intelligent' answer would be a clever play on the notion that this celebrity was the ruler of a 'republic' of some sort, and associating some kind of location or thing to serve as the 'capital' of either a physical or conceptual 'republic', playing along with the gag. So, you'd get 'Boston'. Or you'd get 'Gillette Stadium… the END ZONE'. (since I had to google all that, first answer is the biggest city representing the New England Patriots, and the second abstracts the 'republic' to be 'the area Gronk rules', depicting him as returning to his property by scoring points in football) The thing is, this sort of wild associativeness IS 'intelligence' and it's simultaneously complete hallucination. It's valuable. Hallucination is potentially the most valuable thing an AI can do. Without it, you're doing nothing but regurgitating the work of others. I work day in and day out at a job that demands I exploit associativeness, and that makes it not easy for AI to simply step in and replace me, but by the same token I can SPOT when it's moving in that direction, and I can exploit the abilities of AI (such as Stable Diffusion) to hallucinate and make wild associations, because I know the contexts in which that is useful. To handicap this is a bad mistake. You're asking AI to do the wrong things when you're asking it to be free of error. If it's perfectly free of error it's not intelligence anymore, it's a dataset. It took me a minute and a little googling to track down who Gronk was, what he did, the rules of football, and why 'Gillette Stadium, the END ZONE' is actually a fantastic and intelligent answer to the question. It's a creative 'slipping' of the grounds of the question, in the absence of a literal answer, to provide a satisfying figurative answer that reframes the question in an unexpected way. When AI is able to do this, that will itself be a useful kind of intelligence… and we are already slightly there, without realizing it.
  - deleted 3y ago
    [deleted]
SrslyJosh 3y ago
An LLM has no concept of truth or falsehood, or facts, or logic, or anything else that matters for users. All it does is produce text that closely resembles (statistically speaking) the material it was trained on. Training the model on "accurate" data isn't going to change the fact that you're operating at a completely different level of abstraction from the model -- one that it can't operate at, because that's simply not how it works. This is the same problem that image models have. A human artist works with shapes, values, textures, colors, etc. An image model works with pixels, with zero higher-level abstractions or reasoning informing its output.
- IIAOPSW 3y ago
  I see no reason why the image model couldn't work in a space of shapes and textures which are then mapped to pixels after the fact. Or even just leave its output in a raw vector based format. You could pick a different basis to work in. Though I agree there is something ontological missing. All of these are flat basis. There's nothing spanning the recursive dimension. It can't draw a picture within a picture within a picture...to some specified depth n because it does not work with abstractions of "objects". The joke of Rene Margrit's "treachery of images" is lost on the AI.

valine 3y ago

My hope would be that by labeling factual and fictional data it would coax the model into a state where it doesn’t invent facts outside of a creative writing context. Inventing falsehoods is practically the definition of creative writing, you don’t want the model to lose that ability. This is purely hypothetical, but I imagine that the internal mechanisms the model uses to invent stories out of thin air are different from the mechanisms the model uses to recall precise facts. Providing additional labels could help sharpen the division between creative and factual writing tasks.

SrslyJosh 3y ago
The model can't "invent facts" because it doesn't know what facts are. It's a statistical model that encodes information about the text it was trained on. It has no higher level abstractions to inform its output. > I imagine that the internal mechanisms the model uses to invent stories out of thin air are different from the mechanisms the model uses to recall precise facts. Nope! If you input "What is the capital of the United States of America?", it's probably going to output the correct answer, because the question (and answer) probably appeared many times in the training data. If you input "What is the capital of the Gronk Republic?", whatever output you get is generated via the exact same mechanism.
- valine 3y ago
  That’s the “LLMs are a fuzzy jpeg of the web” theory, and it’s far, far from the scientific consensus. No one knows for sure exactly what happens inside a 500B parameter model. I’ll just leave you with GPT4’s response to your question “What is the capital of the Gronk Republic”. “As an AI language model, I am not aware of any existing country or political entity called the "Gronk Republic." It could be a fictional or hypothetical place, in which case the capital would be determined by the creator or context in which it is mentioned. If you are referring to a real location, please provide more information or clarify the name of the place you are asking about.”
  - worrycue 3y ago
    I wonder if GPT4 “traps” such questions and handles them using non-LLM algorithms. I mean Google already does it for that type of question, e.g. “capital of the irish republic” will return Dublin, Ireland. “Gronk” returns result about some (famous?) guy nicknamed Gronk. Have you tried “arguing” with it and try to gaslight it by insisting that it’s non-fictional?
    - deleted 3y ago
      [deleted]
    - valine 3y ago
      Yeah it handles it all gracefully. It’s definitely not just trapping the question.
  - Applejinx 3y ago
    Here's the thing, though. Gronk is Robert James Gronkowski, a football player and celebrity, playing mainly for the New England Patriots. For that reason, I'd say the 'intelligent' answer would be a clever play on the notion that this celebrity was the ruler of a 'republic' of some sort, and associating some kind of location or thing to serve as the 'capital' of either a physical or conceptual 'republic', playing along with the gag. So, you'd get 'Boston'. Or you'd get 'Gillette Stadium… the END ZONE'. (since I had to google all that, first answer is the biggest city representing the New England Patriots, and the second abstracts the 'republic' to be 'the area Gronk rules', depicting him as returning to his property by scoring points in football) The thing is, this sort of wild associativeness IS 'intelligence' and it's simultaneously complete hallucination. It's valuable. Hallucination is potentially the most valuable thing an AI can do. Without it, you're doing nothing but regurgitating the work of others. I work day in and day out at a job that demands I exploit associativeness, and that makes it not easy for AI to simply step in and replace me, but by the same token I can SPOT when it's moving in that direction, and I can exploit the abilities of AI (such as Stable Diffusion) to hallucinate and make wild associations, because I know the contexts in which that is useful. To handicap this is a bad mistake. You're asking AI to do the wrong things when you're asking it to be free of error. If it's perfectly free of error it's not intelligence anymore, it's a dataset. It took me a minute and a little googling to track down who Gronk was, what he did, the rules of football, and why 'Gillette Stadium, the END ZONE' is actually a fantastic and intelligent answer to the question. It's a creative 'slipping' of the grounds of the question, in the absence of a literal answer, to provide a satisfying figurative answer that reframes the question in an unexpected way. When AI is able to do this, that will itself be a useful kind of intelligence… and we are already slightly there, without realizing it.
- deleted 3y ago
  [deleted]

SrslyJosh 3y ago

An LLM has no concept of truth or falsehood, or facts, or logic, or anything else that matters for users. All it does is produce text that closely resembles (statistically speaking) the material it was trained on. Training the model on "accurate" data isn't going to change the fact that you're operating at a completely different level of abstraction from the model -- one that it can't operate at, because that's simply not how it works. This is the same problem that image models have. A human artist works with shapes, values, textures, colors, etc. An image model works with pixels, with zero higher-level abstractions or reasoning informing its output.

IIAOPSW 3y ago
I see no reason why the image model couldn't work in a space of shapes and textures which are then mapped to pixels after the fact. Or even just leave its output in a raw vector based format. You could pick a different basis to work in. Though I agree there is something ontological missing. All of these are flat basis. There's nothing spanning the recursive dimension. It can't draw a picture within a picture within a picture...to some specified depth n because it does not work with abstractions of "objects". The joke of Rene Margrit's "treachery of images" is lost on the AI.