3 ms·

> People have made the point many times before, that “hallucination” is mainly what generative AI does and not the exception, but most of the time it’s a useful

> People have made the point many times before, that “hallucination” is mainly what generative AI does and not the exception, but most of the time it’s a useful hallucination. Oh yeah that's exactly what I want from a machine intelligence, a "best friend who knows everything about me," is that they just make shit up that they think I'd like to hear. I'd really love a personal assistant that gets me and my date a reservation at a restaurant that doesn't exist. That'll really spice up the evening. The mental gymnastics involved in the AI community are truly pushing the boundaries of parody at this point. If your machines mainly generate bullshit, they cannot be serious products. If on the other hand they're intelligent, why do they make up so much shit? You just can't have this both ways and expect to be taken seriously.

simonw 2y ago
One of the main reason LLMs are unintuitive and difficult to use is that you have to learn how to get useful results out of fundamentally unreliable technology. Once you figure out how to do that they're absurdly useful. Maybe a good analogy here is working with animals? Guide dogs, sniffer dogs, falconry... all cases where you can get great results but you have to learn how to work with a very unpredictable partner.
- ToucanLoucan 2y ago
  > One of the main reason LLMs are unintuitive and difficult to use is that you have to learn how to get useful results out of fundamentally unreliable technology. Name literally any other technology that works this way. > Guide dogs, sniffer dogs, falconry... Guide dogs are an imperfect solution to an actual problem: the inability for people to see. And dogs respond to training far more reliably than LLMs respond to prompts. Sniffer dogs are at least in part bullshit and have been shown in many studies to respond to the subtle cues of their handlers far more reliably than anything they actually smell. And the best of part of them is they also (completely outside their own control mind you) ruin lives when falsely detecting drugs on cars that look a way the officer handling them thinks means they have drugs inside. And falconry is a hobby.
  - simonw 2y ago
    "Name literally any other technology that works this way" Since you don't like my animal examples, how about power tools? Chainsaws, table saws, lathes... all examples of tools where you have to learn how to use them before they'll be useful to you. (My inability to come up with an analogy you find convincing shouldn't invalidate my claim that "LLMs are unreliable technology that is still useful if you learn how to work with it" - maybe this is the first time that's ever been true for an unreliable technology, though I find that doubtful.)
    - marcosdumay 2y ago
      The correct name for unreliable power tools is "trash".
      - blibble 2y ago
        which happens to be the correct name for A"I" too
  - netruk44 2y ago
    > Name literally any other technology that works this way. The internet for one. Not the internet itself (although it certainly can be unreliable), but rather the information on it. Which I think is more relevant to the argument anyway, as LLM’s do in fact reliably function exactly the way they were built to. Information on the internet is inherently unreliable. It’s only when you consider externalities (like reputation of source) that its information can then be made “reliable”. Information that comes out of LLM’s is inherently unreliable. It’s only through externalities (such as online research) that its information can be made reliable. Unless you can invent a truth machine that somehow can tell truth from fiction, I don’t see either of these things becoming reliable, stand-alone sources of information.
  - bluesnowmonkey 2y ago
    > Name literally any other technology that works this way. How about people? They make mistakes all the time, disobey instructions, don’t show up to work, occasionally attempt to embezzle or sabotage their employers. Yet we manage to build huge successful companies out of them.
- mdp2021 2y ago
  > Once you figure out how to do that they're absurdly useful I have read some posts of yours advancing that but I never met those with the details: do you mean more "prompt engineering", or "application selection", or "system integration"...?
  - simonw 2y ago
    Typing code faster. Building quick illustrative prototypes. Researching options for libraries (that are old and stable enough to be in the training data). Porting code from one language to another (surprisingly) [1]. Using as a thesaurus. Answering questions about code (like piping in a whole codebase and asking about it) [2]. Writing an initial set of unit tests. Finding the most interesting new ideas in a paper or online discussion thread without reading the whole thing. Building one-off tools for converting data. Writing complex SQL queries. Finding potential causes of difficult bugs. [3] [1] I built https://tools.simonwillison.net/hacker-news-thread-export https://tools.simonwillison.net/hacker-news-thread-export this morning from my phone using that trick: https://claude.ai/share/7d0de887-5ff8-4b8c-90b1-b5d4d4ca9b84 https://claude.ai/share/7d0de887-5ff8-4b8c-90b1-b5d4d4ca9b84 [2] Examples of that here: https://simonwillison.net/2025/Mar/11/using-llms-for-code/#bonus-answering-questions-about-codebases https://simonwillison.net/2025/Mar/11/using-llms-for-code/#b... [3] https://simonwillison.net/2024/Sep/25/o1-preview-llm/ https://simonwillison.net/2024/Sep/25/o1-preview-llm/ is an early example of using a "reasoning" model for that Or if you meant "what do you have to figure out to use them effectively despite their flaws?", that's a huge topic. It's mostly about building a deep intuition for what they can and cannot help with, then figuring out how to prompt them (including managing their context of inputs) to get good results. The most I've written about that is probably this piece: https://simonwillison.net/2025/Mar/11/using-llms-for-code/ https://simonwillison.net/2025/Mar/11/using-llms-for-code/
    - mdp2021 2y ago
      All of that is very interesting. Side note: don't you agree that "answering about documentation with 100% reliability" would be a more than desirable further feature? (Think of those options in the shell commands which can be so confusing they made it to xkcd material.) But that would mean achieving production-level RAG; and that in turn would be a revolution in LLMs, which would revise your list above...

simonw 2y ago

One of the main reason LLMs are unintuitive and difficult to use is that you have to learn how to get useful results out of fundamentally unreliable technology. Once you figure out how to do that they're absurdly useful. Maybe a good analogy here is working with animals? Guide dogs, sniffer dogs, falconry... all cases where you can get great results but you have to learn how to work with a very unpredictable partner.

ToucanLoucan 2y ago
> One of the main reason LLMs are unintuitive and difficult to use is that you have to learn how to get useful results out of fundamentally unreliable technology. Name literally any other technology that works this way. > Guide dogs, sniffer dogs, falconry... Guide dogs are an imperfect solution to an actual problem: the inability for people to see. And dogs respond to training far more reliably than LLMs respond to prompts. Sniffer dogs are at least in part bullshit and have been shown in many studies to respond to the subtle cues of their handlers far more reliably than anything they actually smell. And the best of part of them is they also (completely outside their own control mind you) ruin lives when falsely detecting drugs on cars that look a way the officer handling them thinks means they have drugs inside. And falconry is a hobby.
- simonw 2y ago
  "Name literally any other technology that works this way" Since you don't like my animal examples, how about power tools? Chainsaws, table saws, lathes... all examples of tools where you have to learn how to use them before they'll be useful to you. (My inability to come up with an analogy you find convincing shouldn't invalidate my claim that "LLMs are unreliable technology that is still useful if you learn how to work with it" - maybe this is the first time that's ever been true for an unreliable technology, though I find that doubtful.)
  - marcosdumay 2y ago
    The correct name for unreliable power tools is "trash".
    - blibble 2y ago
      which happens to be the correct name for A"I" too
- netruk44 2y ago
  > Name literally any other technology that works this way. The internet for one. Not the internet itself (although it certainly can be unreliable), but rather the information on it. Which I think is more relevant to the argument anyway, as LLM’s do in fact reliably function exactly the way they were built to. Information on the internet is inherently unreliable. It’s only when you consider externalities (like reputation of source) that its information can then be made “reliable”. Information that comes out of LLM’s is inherently unreliable. It’s only through externalities (such as online research) that its information can be made reliable. Unless you can invent a truth machine that somehow can tell truth from fiction, I don’t see either of these things becoming reliable, stand-alone sources of information.
- bluesnowmonkey 2y ago
  > Name literally any other technology that works this way. How about people? They make mistakes all the time, disobey instructions, don’t show up to work, occasionally attempt to embezzle or sabotage their employers. Yet we manage to build huge successful companies out of them.
mdp2021 2y ago
> Once you figure out how to do that they're absurdly useful I have read some posts of yours advancing that but I never met those with the details: do you mean more "prompt engineering", or "application selection", or "system integration"...?
- simonw 2y ago
  Typing code faster. Building quick illustrative prototypes. Researching options for libraries (that are old and stable enough to be in the training data). Porting code from one language to another (surprisingly) [1]. Using as a thesaurus. Answering questions about code (like piping in a whole codebase and asking about it) [2]. Writing an initial set of unit tests. Finding the most interesting new ideas in a paper or online discussion thread without reading the whole thing. Building one-off tools for converting data. Writing complex SQL queries. Finding potential causes of difficult bugs. [3] [1] I built https://tools.simonwillison.net/hacker-news-thread-export https://tools.simonwillison.net/hacker-news-thread-export this morning from my phone using that trick: https://claude.ai/share/7d0de887-5ff8-4b8c-90b1-b5d4d4ca9b84 https://claude.ai/share/7d0de887-5ff8-4b8c-90b1-b5d4d4ca9b84 [2] Examples of that here: https://simonwillison.net/2025/Mar/11/using-llms-for-code/#bonus-answering-questions-about-codebases https://simonwillison.net/2025/Mar/11/using-llms-for-code/#b... [3] https://simonwillison.net/2024/Sep/25/o1-preview-llm/ https://simonwillison.net/2024/Sep/25/o1-preview-llm/ is an early example of using a "reasoning" model for that Or if you meant "what do you have to figure out to use them effectively despite their flaws?", that's a huge topic. It's mostly about building a deep intuition for what they can and cannot help with, then figuring out how to prompt them (including managing their context of inputs) to get good results. The most I've written about that is probably this piece: https://simonwillison.net/2025/Mar/11/using-llms-for-code/ https://simonwillison.net/2025/Mar/11/using-llms-for-code/
  - mdp2021 2y ago
    All of that is very interesting. Side note: don't you agree that "answering about documentation with 100% reliability" would be a more than desirable further feature? (Think of those options in the shell commands which can be so confusing they made it to xkcd material.) But that would mean achieving production-level RAG; and that in turn would be a revolution in LLMs, which would revise your list above...