5 ms·

Reminder that these models have no concept of truth, no abstract reasoning abilities, and there's no guarantee that the plausible-looking text they spit out isn

Reminder that these models have no concept of truth, no abstract reasoning abilities, and there's no guarantee that the plausible-looking text they spit out isn't complete bullshit. Their output is impressive considering that they're incapable of reasoning, but it desperately needs to be sanity-checked by someone with relevant knowledge before using it for anything serious.

EGreg 4y ago
I think it's "top-down reasoning" It does in fact create some sort of model out of those billions of dimensions, and it's able to re-use it across many different fields, too It's not "bottom-up reasoning" so it misses a bunch of little details. Basically AI is a search through a vast space, and most of the approaches have focused on top-down search. Reasoning bottom-up is what AlphaBeta search did (rybka and all those other chess programs, that calculated all possible combinations 10 moves ahead). Those missed some of the "top-down positional" stuff, whereas AlphaGo missed some weird-ass edge cases https://www.engadget.com/alphago-pushed-human-go-players-to-become-more-creative-231703950.html https://www.engadget.com/alphago-pushed-human-go-players-to-... But it is a sort of intelligence. Just not one you can trust when details matter. You can use it for pretty pictures for now, and essays. Generative AI is not deep but extremely broad. It knows many subjects. And can discuss more than any specific person. But guess who produced all that content for it and made it publicly available. Humans! (And in cases where it wasn't publicly available, I guess they can sue the company)
KyeRussell 4y ago
Ironically this “convincing bullshit” comment is turning into something everyone parrots in every LLM discussion thread. I think that someone that’s gone as far to built this with GPT-n is aware of its limitations.
- jacquesm 4y ago
  The question isn't really whether someone that has built it is aware of the limitation, but whether the users of that thing that someone built is aware of them. In one study (see reference) users were able to identify the AI about 2/3rd of the times (and identical for the human experts). So 1/3rd of the time the AI is not detected and may well be spouting nonsense. https://www.medrxiv.org/content/10.1101/2023.01.23.23284735v2 https://www.medrxiv.org/content/10.1101/2023.01.23.23284735v...
famouswaffles 4y ago
Real time post shifting and mental gymnastics is truly a sight to behold. No the output isn't perfect and should be double checked (as with any professional domain - you think what an Engineer with decades of experience signs off hasn't been checked multiple times by multiple parties ?, you think a professional programmer writes code and..just deploys it ?) but LLMs reason just fine. If you have anything to say about that other than the nonsensical - "it's not true understanding just because", i'd love to hear it.
- Madmallard 4y ago
  This line of reasoning sort of goes both ways. Advocates and claimants of the technology arguing that humans reason no differently and that LLMs just given the right data will be better than everyone at everything related is just as hokey.
  - famouswaffles 4y ago
    I didn't say anything about reasoning no differently. Suppose you have 2 equations. You don't know what these equations are. However, you know that for any input, the output is the same. Any mathematician worth his salt will tell you that given said information, those 2 equations are equal or equivalent. It doesn't actually matter what those equations are. Equation 1 could be say a+b and Equation 2 could be (a-5) + (b-5) + 10 Or more realistically, both equations could look vastly different. The point I'm driving home here is that true distinction reveals itself in results. The fallacy of the philosophical zombie is that there is this supposed important distinction between "true understanding" and "fake/mimicry/whatever understanding" and yet you can't actually test for it. You can't show this supposed huge difference. A distinction that can't be tested for is not a distinction.
    - underanalyzer 4y ago
      This analogy does not make sense to me. We do not have equivalence between all of the infinite inputs and outputs here, we have equivalence in a finite number of cases and known cases where the two functions (human output and llm output) diverge drastically. Any mathematician worth their salt would tell you these functions are definitely not equal. Now you could make the argument that these functions are close enough most of the time so it won't matter but unless you want to get really rigorous that's more of a stats / engineering perceptive not mathematics. Any more importantly that's very up for debate, especially in a high pressure situation like medicine. Of course these models are wild and I'm quite impressed with them. I still can be worried about the damage someone who doesn't think things through could cause by assuming GPT-4 has human or super human level intelligence in a niche, high impact field.
pyuser583 4y ago
I experimented with asking the models to rewrite text using odd and specific rules - “all sentences must start with the letter b and end with the letter d.” It was pretty good, but sometimes it wouldn’t even try. It would spit out a sentence that could have been easily rewritten to start with b and end with d - but it just didn’t. These sorts of hallucinations are a big problem is high stakes practice.
- duskwuff 4y ago
  LLMs like GPT4 operate on tokens, not characters, so they're operating with a severe handicap when playing those sorts of games -- they see a word like "robot" as a single token, not as a collection of letters, so they don't know that it starts with the letter R unless that fact appeared in its training material. Interestingly, they do better at rhyming games, because those are based on associations between tokens which are easier to infer from usage in poetry, or by reading rhyming dictionaries.
  - pyuser583 4y ago
    Why was it able to still do a good job? The hallucinations were about 10%, but the rest often improved the original sentence, despite the silly rules? Also, wouldn’t learning from a dictionary make it pretty easy to know what word a letter starts with? Everything in chapter “B” starts with b.
mgfist 4y ago
> Reminder that these models have no concept of truth, no abstract reasoning abilities This has positive potential too - bias prevention. Give it an ultrasound and it'll do its work regardless of the patient. None of this is guaranteed (there could be training pitfalls, i.e. maybe patient data is fed to AI, including race/sex, and if training data has bias toward non-white males then so will AI), but it's a potentially positive aspect to explore!
- refurb 4y ago
  But bias has a negative impact too. It’s not unusual for clinicians to interpret differently based on patient history and other risk factors. The same CT scan is going to be interpreted very differently for a patient with no history of cancer versus one who went into remission 1 year ago.
  - idopmstuff 4y ago
    That's not bias, it's use of relevant context. If you're going to have an LLM doing diagnoses, you're definitely going to feed it that sort of context.
    - refurb 4y ago
      It's a matter of degree though. Do you just tell the AI "this person had cancer?", or "this person went into remission 1 year ago?" or "this person was treated with 4 lines of chemo, partial remission for 1 line, non-response for 2nd line, strong response for cycles 1-3 for 3rd line, but then discontinuation due to tolerability, then 4th line complete remission after 6 cycles plus two courses of radiotherapy"? Theoretically you could do this, but capturing all this and correctly interpreting it is far harder for AI than for a human who has been directing treatment the entire time.
      - notch898a 4y ago
        Meanwhile for late childhood to to mid adulthood, 99% of my visits have been a doctor listening to me for 2 minutes tops while applying what amounts to a flow chart of how to diagnose and treat something like strep. Yes a doctor is best, but remember in many places we're competing with "no access do doctor so just pray it doesn't go to Rheumatic fever."
fnordpiglet 4y ago
Except we have very powerful reasoning systems at the moment. Attaching a fully trained LLM on a speciality outputting in a structured form into a semantic reasoning system and feeding the results back through the LLM in a feedback cycle should control for hallucinations. It’s pretty clear though to all but the most cynical these LLM are producing amazing results regardless of whether “reasoning” is happening. The degree to which they produce insightful, cogent, and complex synthesis of concepts in a rational seeming form is startling. I’d challenge anyone to demonstrate humans don’t “reason” in a similar way. Are we really anything more than expectation engines powered by gradient descent expectation juice? How come our reasoning fails so often and we have to be taught logic through reinforcement learning? Why do we fall for fallacies so easily when we “know” about them and can “reason” so well? I will wager $5 that as we build ensemble models that can leverage the reasoning AI tech of the last 80 years in a feedback cycle with LLMs and other generative AI models we will be staring at intelligence that far surpasses our own, and perhaps be so alien we can’t recognize it because it actually is backed by reasoning - unlike ours.
muzani 4y ago
I wonder how many people who think they have no reasoning abilities have actually seen GPT-4. There's clearly some form of reasoning with GPT-4. There are situations where a human can reason better. But these are usually identified as bugs and will probably just be fixed later. Yes, they're basically autocomplete machines but a human is a bundle of neurons optimized for survival anyway. But humans can become irrational to things that threaten survival. Classic medical example is the drama that emerged when one doctor suggested that washing hands would reduce mortality rates. Humans and LLM-based AI evolve rationality based around different factors and form different "bugs" and I love that they're capable of fact checking each other now.
- malborodog 4y ago
  Just because it produces sequences of tokens that we typically experience as evidence of reasoning does not mean that it reasoned to produce them. We know -- generally -- the process by which those tokens were produced. It is not deterministic, but its properties are understood. It's a language model! It is not a mind.
  - stavros 4y ago
    I would be very interested to hear how exactly a mind produces its reasoning.
- jml2 4y ago
  https://ekjsgrjelrbno.substack.com/p/language-models https://ekjsgrjelrbno.substack.com/p/language-models
alfor 4y ago
At the speed things go, give it a few months.
ngngngng 4y ago
> there's no guarantee that the plausible-looking text they spit out isn't complete bullshit. Sounds just like Doctors.
nopinsight 4y ago
GPT-4 can do some reasoning, just not at a smart/trained human level. You cannot explain its results in USMLE, Uniform Bar Exam, and several other tests otherwise. This is a major improvement from earlier models. This assumes there is not too much data contamination, but other people’s experiences with it suggest that the capability is real even if it might be a bit below what the test performance suggests. The latter is also explainable by a broader distribution of real world problems relative to tests. See my other recent comments for more examples and elaboration, eg https://news.ycombinator.com/item?id=35301499 https://news.ycombinator.com/item?id=35301499

EGreg 4y ago

I think it's "top-down reasoning" It does in fact create some sort of model out of those billions of dimensions, and it's able to re-use it across many different fields, too It's not "bottom-up reasoning" so it misses a bunch of little details. Basically AI is a search through a vast space, and most of the approaches have focused on top-down search. Reasoning bottom-up is what AlphaBeta search did (rybka and all those other chess programs, that calculated all possible combinations 10 moves ahead). Those missed some of the "top-down positional" stuff, whereas AlphaGo missed some weird-ass edge cases https://www.engadget.com/alphago-pushed-human-go-players-to-become-more-creative-231703950.html https://www.engadget.com/alphago-pushed-human-go-players-to-... But it is a sort of intelligence. Just not one you can trust when details matter. You can use it for pretty pictures for now, and essays. Generative AI is not deep but extremely broad. It knows many subjects. And can discuss more than any specific person. But guess who produced all that content for it and made it publicly available. Humans! (And in cases where it wasn't publicly available, I guess they can sue the company)

KyeRussell 4y ago

Ironically this “convincing bullshit” comment is turning into something everyone parrots in every LLM discussion thread. I think that someone that’s gone as far to built this with GPT-n is aware of its limitations.

jacquesm 4y ago
The question isn't really whether someone that has built it is aware of the limitation, but whether the users of that thing that someone built is aware of them. In one study (see reference) users were able to identify the AI about 2/3rd of the times (and identical for the human experts). So 1/3rd of the time the AI is not detected and may well be spouting nonsense. https://www.medrxiv.org/content/10.1101/2023.01.23.23284735v2 https://www.medrxiv.org/content/10.1101/2023.01.23.23284735v...

famouswaffles 4y ago

Real time post shifting and mental gymnastics is truly a sight to behold. No the output isn't perfect and should be double checked (as with any professional domain - you think what an Engineer with decades of experience signs off hasn't been checked multiple times by multiple parties ?, you think a professional programmer writes code and..just deploys it ?) but LLMs reason just fine. If you have anything to say about that other than the nonsensical - "it's not true understanding just because", i'd love to hear it.

Madmallard 4y ago
This line of reasoning sort of goes both ways. Advocates and claimants of the technology arguing that humans reason no differently and that LLMs just given the right data will be better than everyone at everything related is just as hokey.
- famouswaffles 4y ago
  I didn't say anything about reasoning no differently. Suppose you have 2 equations. You don't know what these equations are. However, you know that for any input, the output is the same. Any mathematician worth his salt will tell you that given said information, those 2 equations are equal or equivalent. It doesn't actually matter what those equations are. Equation 1 could be say a+b and Equation 2 could be (a-5) + (b-5) + 10 Or more realistically, both equations could look vastly different. The point I'm driving home here is that true distinction reveals itself in results. The fallacy of the philosophical zombie is that there is this supposed important distinction between "true understanding" and "fake/mimicry/whatever understanding" and yet you can't actually test for it. You can't show this supposed huge difference. A distinction that can't be tested for is not a distinction.
  - underanalyzer 4y ago
    This analogy does not make sense to me. We do not have equivalence between all of the infinite inputs and outputs here, we have equivalence in a finite number of cases and known cases where the two functions (human output and llm output) diverge drastically. Any mathematician worth their salt would tell you these functions are definitely not equal. Now you could make the argument that these functions are close enough most of the time so it won't matter but unless you want to get really rigorous that's more of a stats / engineering perceptive not mathematics. Any more importantly that's very up for debate, especially in a high pressure situation like medicine. Of course these models are wild and I'm quite impressed with them. I still can be worried about the damage someone who doesn't think things through could cause by assuming GPT-4 has human or super human level intelligence in a niche, high impact field.

pyuser583 4y ago

I experimented with asking the models to rewrite text using odd and specific rules - “all sentences must start with the letter b and end with the letter d.” It was pretty good, but sometimes it wouldn’t even try. It would spit out a sentence that could have been easily rewritten to start with b and end with d - but it just didn’t. These sorts of hallucinations are a big problem is high stakes practice.

duskwuff 4y ago
LLMs like GPT4 operate on tokens, not characters, so they're operating with a severe handicap when playing those sorts of games -- they see a word like "robot" as a single token, not as a collection of letters, so they don't know that it starts with the letter R unless that fact appeared in its training material. Interestingly, they do better at rhyming games, because those are based on associations between tokens which are easier to infer from usage in poetry, or by reading rhyming dictionaries.
- pyuser583 4y ago
  Why was it able to still do a good job? The hallucinations were about 10%, but the rest often improved the original sentence, despite the silly rules? Also, wouldn’t learning from a dictionary make it pretty easy to know what word a letter starts with? Everything in chapter “B” starts with b.

mgfist 4y ago

> Reminder that these models have no concept of truth, no abstract reasoning abilities This has positive potential too - bias prevention. Give it an ultrasound and it'll do its work regardless of the patient. None of this is guaranteed (there could be training pitfalls, i.e. maybe patient data is fed to AI, including race/sex, and if training data has bias toward non-white males then so will AI), but it's a potentially positive aspect to explore!

refurb 4y ago
But bias has a negative impact too. It’s not unusual for clinicians to interpret differently based on patient history and other risk factors. The same CT scan is going to be interpreted very differently for a patient with no history of cancer versus one who went into remission 1 year ago.
- idopmstuff 4y ago
  That's not bias, it's use of relevant context. If you're going to have an LLM doing diagnoses, you're definitely going to feed it that sort of context.
  - refurb 4y ago
    It's a matter of degree though. Do you just tell the AI "this person had cancer?", or "this person went into remission 1 year ago?" or "this person was treated with 4 lines of chemo, partial remission for 1 line, non-response for 2nd line, strong response for cycles 1-3 for 3rd line, but then discontinuation due to tolerability, then 4th line complete remission after 6 cycles plus two courses of radiotherapy"? Theoretically you could do this, but capturing all this and correctly interpreting it is far harder for AI than for a human who has been directing treatment the entire time.
    - notch898a 4y ago
      Meanwhile for late childhood to to mid adulthood, 99% of my visits have been a doctor listening to me for 2 minutes tops while applying what amounts to a flow chart of how to diagnose and treat something like strep. Yes a doctor is best, but remember in many places we're competing with "no access do doctor so just pray it doesn't go to Rheumatic fever."

fnordpiglet 4y ago

Except we have very powerful reasoning systems at the moment. Attaching a fully trained LLM on a speciality outputting in a structured form into a semantic reasoning system and feeding the results back through the LLM in a feedback cycle should control for hallucinations. It’s pretty clear though to all but the most cynical these LLM are producing amazing results regardless of whether “reasoning” is happening. The degree to which they produce insightful, cogent, and complex synthesis of concepts in a rational seeming form is startling. I’d challenge anyone to demonstrate humans don’t “reason” in a similar way. Are we really anything more than expectation engines powered by gradient descent expectation juice? How come our reasoning fails so often and we have to be taught logic through reinforcement learning? Why do we fall for fallacies so easily when we “know” about them and can “reason” so well? I will wager $5 that as we build ensemble models that can leverage the reasoning AI tech of the last 80 years in a feedback cycle with LLMs and other generative AI models we will be staring at intelligence that far surpasses our own, and perhaps be so alien we can’t recognize it because it actually is backed by reasoning - unlike ours.

muzani 4y ago

I wonder how many people who think they have no reasoning abilities have actually seen GPT-4. There's clearly some form of reasoning with GPT-4. There are situations where a human can reason better. But these are usually identified as bugs and will probably just be fixed later. Yes, they're basically autocomplete machines but a human is a bundle of neurons optimized for survival anyway. But humans can become irrational to things that threaten survival. Classic medical example is the drama that emerged when one doctor suggested that washing hands would reduce mortality rates. Humans and LLM-based AI evolve rationality based around different factors and form different "bugs" and I love that they're capable of fact checking each other now.

malborodog 4y ago
Just because it produces sequences of tokens that we typically experience as evidence of reasoning does not mean that it reasoned to produce them. We know -- generally -- the process by which those tokens were produced. It is not deterministic, but its properties are understood. It's a language model! It is not a mind.
- stavros 4y ago
  I would be very interested to hear how exactly a mind produces its reasoning.
jml2 4y ago
https://ekjsgrjelrbno.substack.com/p/language-models https://ekjsgrjelrbno.substack.com/p/language-models

alfor 4y ago

At the speed things go, give it a few months.

ngngngng 4y ago

> there's no guarantee that the plausible-looking text they spit out isn't complete bullshit. Sounds just like Doctors.

nopinsight 4y ago

GPT-4 can do some reasoning, just not at a smart/trained human level. You cannot explain its results in USMLE, Uniform Bar Exam, and several other tests otherwise. This is a major improvement from earlier models. This assumes there is not too much data contamination, but other people’s experiences with it suggest that the capability is real even if it might be a bit below what the test performance suggests. The latter is also explainable by a broader distribution of real world problems relative to tests. See my other recent comments for more examples and elaboration, eg https://news.ycombinator.com/item?id=35301499 https://news.ycombinator.com/item?id=35301499