3 ms·

LLMs can definitely be helpful, but you need your radar set to 11 because they will convincingly feed you smart sounding nonsense when you are most at risk.

ChrisMarshallNY 3mo ago
Oh yeah, but the same goes for almost any information source (especially these days).
- abalashov 3mo ago
  Yeah, but this is like saying that one needn't focus so much on LLMs making mistakes because humans also mistakes. They do, but the shape of the way LLMs will confidently mislead you is quite different to the way misinformed humans, or even the malevolent and mendacious humans, will mislead you.
  - brookst 3mo ago
    There are plenty of authoritative reference books full of errors. Teachers in every subject can be wrong, convincingly.
    - abalashov 3mo ago
      Yes, but human errors are based on genuine and elaborate misconceptions (or propagandistic intent), not squishy, facile "you're right to push back on that, I straight up made that up" type stuff.
      - ChrisMarshallNY 3mo ago
        I just apply the "Wikipedia model" to things like LLMs (and StackOverflow, which can be just as bad -but without the admitting error part). I look for citations and footnotes. In many cases, accuracy isn't something that I worry about, as bad info becomes apparent, almost immediately. In cases where it matters, though, I may try a couple of verifications. One problem with humans, is that we can be quite insecure, and will fight to the death to defend a provably wrong position, because we can't bear to be seen as in error.
        jcgl 3mo ago
        > bad info becomes apparent, almost immediately Can you elaborate on this? I suspect that you’re thinking mostly of cases in which you already have a fair bit of domain expertise. But in the general case, this seems to be very untrue. Which is why it’s so pernicious that LLMs can generate such quantities of syntactically-plausible-but-factually-untrue text.
        ChrisMarshallNY 3mo ago
        Valid point, but I don’t really just start learning whole new vocations, cold. I’m an engineer/software developer, with over 40 years’ experience, and my learning is generally some branch off that (like learning asynchronous programming, or a new UI framework). Can’t really put it into precise terminology, but I get a “gut feeling,” that something is/is not plausible, and it happens pretty quickly; usually when I start some implementation. I tend to learn by do[0] (note the “sample playground,” in each essay), so the “reality filter” gets applied fairly rapidly. In any case, I have been dealing with misinformation (sometimes, deliberate), for a long time, and I’m still working fairly effectively, so I guess it works. Can’t argue with results. [0] https://littlegreenviper.com/series/swiftwater/ https://littlegreenviper.com/series/swiftwater/
        jcgl 3mo ago
        Sure, I take your point that the smell-test works reasonably well for domains related or adjacent to one's own. But (not speaking about your use specifically here) many people (most, I'd wager) use LLMs for many things beyond their own expertise. And it's there that they're most likely to be ensnared without even knowing it. I definitely agree with your notion that learning-by-doing is a helpful salve for LLM falsehoods. It's no panacea (working != correct (an incorrect solution can appear correct over a given interval)), but it's a good way of working in general that helps keep LLMs in check. And it's very natural to code or other things that can be immediately applied. But learning-by-doing of course doesn't work with topics that aren't immediately applied. Which includes lots of topics that people use LLMs for (Wikipedia too, for that matter). The set of unfamiliar-or-unapplied is practically a lot larger than the set of familiar-or-applied.
      - dpkirchner 3mo ago
        I wish humans that make up "facts" would admit to it as readily as LLMs do. Even though I know it's just a training trick to ground the response.
    - varjag 3mo ago
      It's not anywhere close. Like, by four orders of magnitude.

ChrisMarshallNY 3mo ago

Oh yeah, but the same goes for almost any information source (especially these days).

abalashov 3mo ago
Yeah, but this is like saying that one needn't focus so much on LLMs making mistakes because humans also mistakes. They do, but the shape of the way LLMs will confidently mislead you is quite different to the way misinformed humans, or even the malevolent and mendacious humans, will mislead you.
- brookst 3mo ago
  There are plenty of authoritative reference books full of errors. Teachers in every subject can be wrong, convincingly.
  - abalashov 3mo ago
    Yes, but human errors are based on genuine and elaborate misconceptions (or propagandistic intent), not squishy, facile "you're right to push back on that, I straight up made that up" type stuff.
    - ChrisMarshallNY 3mo ago
      I just apply the "Wikipedia model" to things like LLMs (and StackOverflow, which can be just as bad -but without the admitting error part). I look for citations and footnotes. In many cases, accuracy isn't something that I worry about, as bad info becomes apparent, almost immediately. In cases where it matters, though, I may try a couple of verifications. One problem with humans, is that we can be quite insecure, and will fight to the death to defend a provably wrong position, because we can't bear to be seen as in error.
      - jcgl 3mo ago
        > bad info becomes apparent, almost immediately Can you elaborate on this? I suspect that you’re thinking mostly of cases in which you already have a fair bit of domain expertise. But in the general case, this seems to be very untrue. Which is why it’s so pernicious that LLMs can generate such quantities of syntactically-plausible-but-factually-untrue text.
        ChrisMarshallNY 3mo ago
        Valid point, but I don’t really just start learning whole new vocations, cold. I’m an engineer/software developer, with over 40 years’ experience, and my learning is generally some branch off that (like learning asynchronous programming, or a new UI framework). Can’t really put it into precise terminology, but I get a “gut feeling,” that something is/is not plausible, and it happens pretty quickly; usually when I start some implementation. I tend to learn by do[0] (note the “sample playground,” in each essay), so the “reality filter” gets applied fairly rapidly. In any case, I have been dealing with misinformation (sometimes, deliberate), for a long time, and I’m still working fairly effectively, so I guess it works. Can’t argue with results. [0] https://littlegreenviper.com/series/swiftwater/ https://littlegreenviper.com/series/swiftwater/
        jcgl 3mo ago
        Sure, I take your point that the smell-test works reasonably well for domains related or adjacent to one's own. But (not speaking about your use specifically here) many people (most, I'd wager) use LLMs for many things beyond their own expertise. And it's there that they're most likely to be ensnared without even knowing it. I definitely agree with your notion that learning-by-doing is a helpful salve for LLM falsehoods. It's no panacea (working != correct (an incorrect solution can appear correct over a given interval)), but it's a good way of working in general that helps keep LLMs in check. And it's very natural to code or other things that can be immediately applied. But learning-by-doing of course doesn't work with topics that aren't immediately applied. Which includes lots of topics that people use LLMs for (Wikipedia too, for that matter). The set of unfamiliar-or-unapplied is practically a lot larger than the set of familiar-or-applied.
    - dpkirchner 3mo ago
      I wish humans that make up "facts" would admit to it as readily as LLMs do. Even though I know it's just a training trick to ground the response.
  - varjag 3mo ago
    It's not anywhere close. Like, by four orders of magnitude.