3 ms·

> This was my first exposure to any kind of outdoor plumbing, I'd never touched PVC before. I find it telling that the highest praise for LLMs comes from peopl

> This was my first exposure to any kind of outdoor plumbing, I'd never touched PVC before. I find it telling that the highest praise for LLMs comes from people using it for something where they admittedly have very little domain knowledge. Domain experts usually mention major caveats. I've been testing them on subjects where I already understand the problem well, and I've yet to see any outputs that would make me trust them on things I don't already know. https://en.wikipedia.org/wiki/Gell-Mann_Amnesia https://en.wikipedia.org/wiki/Gell-Mann_Amnesia https://en.wiktionary.org/wiki/confidence_trick https://en.wiktionary.org/wiki/confidence_trick

joenot443 2mo ago
That's a good point - I've heard people make the same one whenever LLMs are brought up. I hope someday you're able to get more value from them. Anyway, my pool's looking great and I gained some new skills. I probably could have gotten there with books and YouTube alone, but having another tool at my disposal made me a bit more confident.
- ryandrake 2mo ago
  I remember this was brought up about a month or so on HN, in the context of describing how one person's experience using LLMs can be so vastly different than another person's: "LLMs seem good at things you are not good at." So, if you've never touched PVC before, LLM sounds plausibly competent--it may actually be or it may not be, but you'll walk away from it thinking you learned something. If you are a professional plumber and ask an LLM the same thing, the output will more look flawed and possibly dangerous. Same for software writing: If you're not a good software developer, you probably think an LLM is great and writes much better code faster than a human developer can. But if you are a good software developer, LLM output is slop and requires huge rework to be passable.
  - avgDev 2mo ago
    This sounds accurate. I am not a mechanic by trade, but I know nearly everything there is to know about working on an ICE car. I did paint cars professionally for a bit. LLMs are absolutely full of garbage advice, when I try to use it for troubleshooting. However, people who don't know anything about cars are telling me it helped them fix issues. I am assuming their issues were maybe surface level and something I would just know without even looking at any manuals, because when I use it for complex problems it just doesn't work for me.
    - chasd00 2mo ago
      if it helped them fix issues then it helped them fix issues. Good for them. Maybe their problems didn't rise to your bar but at least they were able to get it solved. That's very useful and empowering to people without the direct knowledge and experience themselves.
  - ff317 2mo ago
    LLMs seem good at things you are not good at... primarily because you lack the skill to actually judge the goodness of their work, and they present their work confidently with an air of authority.
  - chasd00 2mo ago
    > So, if you've never touched PVC before, LLM sounds plausibly competent well to be fair, it's sounds about as competent as your average homedepot employee. He's wasn't doing something super complicated, cutting and gluing PVC for an above ground pool is very common and doesn't require a plumber. I used some youtube videos to fix my dishwasher, i didn't need a professional service agent from the manufacturer I just needed some pointers. as for software writing, for standard everyday enterprise app work which is typically just CRUD and moving data around it works fine. That kind of software does not need to be a highly tuned work of art to meet the requirements.
- mrguyorama 2mo ago
  That's the entire problem. The main thing LLMs gave you was confidence, and the thing to understand is that the confidence an LLM gives you is often utterly false and baseless. Why were you not confident with literal how to videos and documentation, but became confident when a chatbot generated probable text?
iamacyborg 2mo ago
LLMs are infecting us all with Nobelitis.
NewsaHackO 2mo ago
Well, I guess another plausible explanation is that usually "domain experts" are people that get paid for their expertise, and have a vested interest in saying that LLMs cannot replicate what they want people to pay them for.
yunwal 2mo ago
The greatest living mathematician uses it for Math https://siliconreckoner.substack.com/p/terence-tao-on-machine-assisted-proofs https://siliconreckoner.substack.com/p/terence-tao-on-machin... To me it seems like people who are confident in their expertise generally find it useful, even if it's imperfect
ff317 2mo ago
When I apply LLMs to domains where I'm already an expert, I find it particularly lacking when I want it to do deep, difficult, novel work that requires precision. On the surface, the output looks pretty amazing at first. But when I turn a critical eye to every detail, I end up finding a lot of flawed "thinking", and the lengthy process of fully understanding what it generated and cleaning it up to my standards makes me question the entire value proposition. However, I find it does a great job being a low-level automaton sort of assistant. For instance, in the domain of software engineering: I would not trust it to implement a major architectural change, or a groundbreaking, complex new feature. I would trust it more (but not completely) on something like a refactoring that may touch thousands of lines in a fairly mechanistic way, but that was a little too-complicated for simpler tools likes regexes. While that's kind of a nifty use, I think it's fair to say that non-LLM software purpose-built for such tasks can probably do the same thing more effectively for less real cost (meaning the currently-subsidized real cost of all the training and inference power burn, etc)

joenot443 2mo ago

That's a good point - I've heard people make the same one whenever LLMs are brought up. I hope someday you're able to get more value from them. Anyway, my pool's looking great and I gained some new skills. I probably could have gotten there with books and YouTube alone, but having another tool at my disposal made me a bit more confident.

ryandrake 2mo ago
I remember this was brought up about a month or so on HN, in the context of describing how one person's experience using LLMs can be so vastly different than another person's: "LLMs seem good at things you are not good at." So, if you've never touched PVC before, LLM sounds plausibly competent--it may actually be or it may not be, but you'll walk away from it thinking you learned something. If you are a professional plumber and ask an LLM the same thing, the output will more look flawed and possibly dangerous. Same for software writing: If you're not a good software developer, you probably think an LLM is great and writes much better code faster than a human developer can. But if you are a good software developer, LLM output is slop and requires huge rework to be passable.
- avgDev 2mo ago
  This sounds accurate. I am not a mechanic by trade, but I know nearly everything there is to know about working on an ICE car. I did paint cars professionally for a bit. LLMs are absolutely full of garbage advice, when I try to use it for troubleshooting. However, people who don't know anything about cars are telling me it helped them fix issues. I am assuming their issues were maybe surface level and something I would just know without even looking at any manuals, because when I use it for complex problems it just doesn't work for me.
  - chasd00 2mo ago
    if it helped them fix issues then it helped them fix issues. Good for them. Maybe their problems didn't rise to your bar but at least they were able to get it solved. That's very useful and empowering to people without the direct knowledge and experience themselves.
- ff317 2mo ago
  LLMs seem good at things you are not good at... primarily because you lack the skill to actually judge the goodness of their work, and they present their work confidently with an air of authority.
- chasd00 2mo ago
  > So, if you've never touched PVC before, LLM sounds plausibly competent well to be fair, it's sounds about as competent as your average homedepot employee. He's wasn't doing something super complicated, cutting and gluing PVC for an above ground pool is very common and doesn't require a plumber. I used some youtube videos to fix my dishwasher, i didn't need a professional service agent from the manufacturer I just needed some pointers. as for software writing, for standard everyday enterprise app work which is typically just CRUD and moving data around it works fine. That kind of software does not need to be a highly tuned work of art to meet the requirements.
mrguyorama 2mo ago
That's the entire problem. The main thing LLMs gave you was confidence, and the thing to understand is that the confidence an LLM gives you is often utterly false and baseless. Why were you not confident with literal how to videos and documentation, but became confident when a chatbot generated probable text?

iamacyborg 2mo ago

LLMs are infecting us all with Nobelitis.

NewsaHackO 2mo ago

Well, I guess another plausible explanation is that usually "domain experts" are people that get paid for their expertise, and have a vested interest in saying that LLMs cannot replicate what they want people to pay them for.

yunwal 2mo ago

The greatest living mathematician uses it for Math https://siliconreckoner.substack.com/p/terence-tao-on-machine-assisted-proofs https://siliconreckoner.substack.com/p/terence-tao-on-machin... To me it seems like people who are confident in their expertise generally find it useful, even if it's imperfect

ff317 2mo ago

When I apply LLMs to domains where I'm already an expert, I find it particularly lacking when I want it to do deep, difficult, novel work that requires precision. On the surface, the output looks pretty amazing at first. But when I turn a critical eye to every detail, I end up finding a lot of flawed "thinking", and the lengthy process of fully understanding what it generated and cleaning it up to my standards makes me question the entire value proposition. However, I find it does a great job being a low-level automaton sort of assistant. For instance, in the domain of software engineering: I would not trust it to implement a major architectural change, or a groundbreaking, complex new feature. I would trust it more (but not completely) on something like a refactoring that may touch thousands of lines in a fairly mechanistic way, but that was a little too-complicated for simpler tools likes regexes. While that's kind of a nifty use, I think it's fair to say that non-LLM software purpose-built for such tasks can probably do the same thing more effectively for less real cost (meaning the currently-subsidized real cost of all the training and inference power burn, etc)