8 ms·

Trying to encode truthfulness or biasless-ness into the weights of a model is a fools errand from the start. Consider an extremely optimistic (near term) case

by lukev 3y ago

Trying to encode truthfulness or biasless-ness into the weights of a model is a fools errand from the start. Consider an extremely optimistic (near term) case for AI; that we can build a model with intelligence, knowledge and recall equivalent to a Ph.D. educated individual with a top 1% percentile IQ. Even if you have access to such an individual, it would be absurd to rely on their knowledge and intuition alone as your primary search engine or decision maker. As a society, we as humans have decided a long time ago that we need supplemental information and collaborative, multi-perspective decision making. Therefore, the correct role of AI is not as an oracle that gives you the answers from its weights, but as an information processor that helps you sort through answers from a known corpus of information and ground facts. We will never get language models to exhibit a sufficient level of "truthfulness" from training alone. The way we encode knowledge retrieval patterns and multi-agent systems matters way, way more.

kelseyfrog 3y ago
This is why "critical thinking" is often promoted in education. It's the idea that it's better to teach students "how to discern the truth" than teach them "the truth". Is it possible to teach students critical thinking? Is it easier or harder to teach AI critical thinking? Curious to hear your thoughts.
- lukev 3y ago
  I don't know if it can be taught critical thinking. LLM token-generation is only vaguely analogous to "thinking" at all. Recursive self prompting with both internal and external calibration might be analogous to "critical thinking" though.
  - ddmma 3y ago
    Worth trying to show some examples of critical thinking to LLM and then see what happens on new set of data points
    - lukev 3y ago
      This definitely works. There's lots of papers on this. Heck, you can verify this with ChatGPT yourself... just ask it a couple questions, then (in a separate thread) ask the same questions but tell it to "think step by step" as an instruction. It performs visibly better. The point is it fundamentally, at the end of the day, it's generating one token at a time with only its prompt and previously generated tokens for input. If you can inject other information/facts into the context, or even context from another model instance that have a different "train of thought", you can in theory get better results than the model itself generating a sequence. Look up Chain of Thought / Tree of Thought / Graph of Thought for an intro, as well as basically anything on retrieval augmented generation (RAG) or LLM tool usage.
      - ddmma 3y ago
        Thanks for pointing out the evaluation methods and actually being doing some metrics on the outcome from LLMs myself. I bet more on now adjusted knowledge on top of generated content based on facts/ nonsense scoring that can be harvested for feeding reasoning graphs with kind of auto balancing. Less resources and high impact will make the difference on long term.
- WalterBright 3y ago
  > Is it possible to teach students critical thinking? I doubt it. Universities claim to teach it, but I don't see it in the graduates. Consider designing airplanes. (I.e. engineering.) If the result doesn't fly, no amount of rhetoric will convince anyone otherwise. This is not true of the humanities, where educated people can and do convince themselves of endless bad ideas. I'll take the reality checks of engineering over "critical thinking".
  - maxbond 3y ago
    Once I was riding a train, and the conductor announced that only some cars would open their doors at the upcoming station, because the train was longer than the platform. "I wonder why?" A stranger mused. "I know," her companion sneered, "it's because some engineer designed it." She wasn't wrong per say, an engineer had designed the platform - a hundred years ago, for a different train. The train we were riding had squeeked through the legislature on a shoestring budget, reusing the existing tracks and with only a few new platforms slated to be built. And many of those were axed along the way due to budget overruns and such. It was a political miracle the train existed at all. But this passenger wasn't really looking for context. She had stereotypes and narratives that explained the situation to her satisfaction. Are you quite sure that you are engaging your faculties of critical thinking here, rather than relying on narratives and stereotypes? Does it really seem likely to you that the people doing the work you're most familiar with are the only people capable of thinking critically? Or that the feedback loops that you've had in your career are the only ones available?
    - WalterBright 3y ago
      Let me put it this way. Marxism doesn't fly, and nobody has yet found an example of it flying. But academics continue to push Marxism. That's a massive failure of critical thinking by the people who teach it. As for engineers, they make mistakes all the time. And the mistakes are very hard to talk one's way out of. Our society is built by engineers making things that work. Your car doesn't move by talking it into moving.
      - 3y ago
OfSanguineFire 3y ago
Collaborative decision-making often draws criticism as creating paralysis and undermining social harmony. At least some people calling for one-man rule is a perennial trait of human societies, from the ancient desire for a king or caliph to modern supporters of authoritarianism. This phenomenon makes me wonder if some of the population would happily surrender decision-making to an AI model that they feel is adequately smarter than them.
- lukev 3y ago
  The size of the set intersection between people fearing/hoping for the AI singularity (SF "rationalists", mostly) and neo-reactionary monarchists (yes, in 2023) should give you all the answer you need.
ChatGTP 3y ago
I've thought a lot about this myself, it's one thing to have an "AI" system tell you something, it's a very different thing to trust it without doing the (often arduous) task of verification.
- 15457345234 3y ago
  I've thought a lot about this and can't figure out why people aren't realizing that an 'AI agent' which lies constantly about facts which are _trivial_ to validate has _negative_ value. I asked it 'which cherry keyboard switches are clicky, the red ones or the blue ones' and it lied and said that the red ones are clicky. That's information that's contained on about ten thousand blogs because of the mechanical keyboard fad but it still got the answer wrong. That's negative value, right there. An immediately and obviously unreliable misinformation generator. Haven't used it since. Seems to me a bit like the Flipper Zero - it's a good tool for a certain type of person.
  - ChatGTP 3y ago
    It’s a barrier for me, I often think I should “outsource” something to ChatGPT and then I realise I’ve been lied to before. Apparently ChatGPT 4 is better but I’m not paying for it after my crappy experiences with 3.
    - 15457345234 3y ago
      > Apparently ChatGPT 4 is better but I’m not paying for it The thing about this is, even if 4 was infinitely better than 3 (it never lied, never hallucinated) leaving 3 open to the world is such an obvious unmitigated hazard that I'd never ethically be able to do business with them. 3 is a mischief machine, it should not be available to the public. People will trivially use it to generate horseshit websites that will clutter google with shit. We're already seeing this happen. It's like buying caltrops from a company that advertises them by leaving 'demonstration samples' in roads.

by lukev 3y ago

kelseyfrog 3y ago

This is why "critical thinking" is often promoted in education. It's the idea that it's better to teach students "how to discern the truth" than teach them "the truth". Is it possible to teach students critical thinking? Is it easier or harder to teach AI critical thinking? Curious to hear your thoughts.

lukev 3y ago
I don't know if it can be taught critical thinking. LLM token-generation is only vaguely analogous to "thinking" at all. Recursive self prompting with both internal and external calibration might be analogous to "critical thinking" though.
- ddmma 3y ago
  Worth trying to show some examples of critical thinking to LLM and then see what happens on new set of data points
  - lukev 3y ago
    This definitely works. There's lots of papers on this. Heck, you can verify this with ChatGPT yourself... just ask it a couple questions, then (in a separate thread) ask the same questions but tell it to "think step by step" as an instruction. It performs visibly better. The point is it fundamentally, at the end of the day, it's generating one token at a time with only its prompt and previously generated tokens for input. If you can inject other information/facts into the context, or even context from another model instance that have a different "train of thought", you can in theory get better results than the model itself generating a sequence. Look up Chain of Thought / Tree of Thought / Graph of Thought for an intro, as well as basically anything on retrieval augmented generation (RAG) or LLM tool usage.
    - ddmma 3y ago
      Thanks for pointing out the evaluation methods and actually being doing some metrics on the outcome from LLMs myself. I bet more on now adjusted knowledge on top of generated content based on facts/ nonsense scoring that can be harvested for feeding reasoning graphs with kind of auto balancing. Less resources and high impact will make the difference on long term.
WalterBright 3y ago
> Is it possible to teach students critical thinking? I doubt it. Universities claim to teach it, but I don't see it in the graduates. Consider designing airplanes. (I.e. engineering.) If the result doesn't fly, no amount of rhetoric will convince anyone otherwise. This is not true of the humanities, where educated people can and do convince themselves of endless bad ideas. I'll take the reality checks of engineering over "critical thinking".
- maxbond 3y ago
  Once I was riding a train, and the conductor announced that only some cars would open their doors at the upcoming station, because the train was longer than the platform. "I wonder why?" A stranger mused. "I know," her companion sneered, "it's because some engineer designed it." She wasn't wrong per say, an engineer had designed the platform - a hundred years ago, for a different train. The train we were riding had squeeked through the legislature on a shoestring budget, reusing the existing tracks and with only a few new platforms slated to be built. And many of those were axed along the way due to budget overruns and such. It was a political miracle the train existed at all. But this passenger wasn't really looking for context. She had stereotypes and narratives that explained the situation to her satisfaction. Are you quite sure that you are engaging your faculties of critical thinking here, rather than relying on narratives and stereotypes? Does it really seem likely to you that the people doing the work you're most familiar with are the only people capable of thinking critically? Or that the feedback loops that you've had in your career are the only ones available?
  - WalterBright 3y ago
    Let me put it this way. Marxism doesn't fly, and nobody has yet found an example of it flying. But academics continue to push Marxism. That's a massive failure of critical thinking by the people who teach it. As for engineers, they make mistakes all the time. And the mistakes are very hard to talk one's way out of. Our society is built by engineers making things that work. Your car doesn't move by talking it into moving.
    - 3y ago

OfSanguineFire 3y ago

Collaborative decision-making often draws criticism as creating paralysis and undermining social harmony. At least some people calling for one-man rule is a perennial trait of human societies, from the ancient desire for a king or caliph to modern supporters of authoritarianism. This phenomenon makes me wonder if some of the population would happily surrender decision-making to an AI model that they feel is adequately smarter than them.

lukev 3y ago
The size of the set intersection between people fearing/hoping for the AI singularity (SF "rationalists", mostly) and neo-reactionary monarchists (yes, in 2023) should give you all the answer you need.

ChatGTP 3y ago

I've thought a lot about this myself, it's one thing to have an "AI" system tell you something, it's a very different thing to trust it without doing the (often arduous) task of verification.

15457345234 3y ago
I've thought a lot about this and can't figure out why people aren't realizing that an 'AI agent' which lies constantly about facts which are _trivial_ to validate has _negative_ value. I asked it 'which cherry keyboard switches are clicky, the red ones or the blue ones' and it lied and said that the red ones are clicky. That's information that's contained on about ten thousand blogs because of the mechanical keyboard fad but it still got the answer wrong. That's negative value, right there. An immediately and obviously unreliable misinformation generator. Haven't used it since. Seems to me a bit like the Flipper Zero - it's a good tool for a certain type of person.
- ChatGTP 3y ago
  It’s a barrier for me, I often think I should “outsource” something to ChatGPT and then I realise I’ve been lied to before. Apparently ChatGPT 4 is better but I’m not paying for it after my crappy experiences with 3.
  - 15457345234 3y ago
    > Apparently ChatGPT 4 is better but I’m not paying for it The thing about this is, even if 4 was infinitely better than 3 (it never lied, never hallucinated) leaving 3 open to the world is such an obvious unmitigated hazard that I'd never ethically be able to do business with them. 3 is a mischief machine, it should not be available to the public. People will trivially use it to generate horseshit websites that will clutter google with shit. We're already seeing this happen. It's like buying caltrops from a company that advertises them by leaving 'demonstration samples' in roads.