9 ms·

The biggest frustration with LLMs for me is people telling me I'm not promoting it in a good way. Just think about any product where they are selling a half bak

The biggest frustration with LLMs for me is people telling me I'm not promoting it in a good way. Just think about any product where they are selling a half baked product, and repeatedly telling the user you are not using it properly.

simonw 9mo ago
But that's not how most products work. If you buy a table saw and can't figure out how to cut a straight line in a piece of wood with it - or keep cutting your fingers off - but didn't take any time at all to learn how to use it, that's on you. Likewise a car, you have to take lessons and a test before you can use those! Why should LLMs be any different?
- what 9mo ago
  It’s more like the iPhone “you’re holding it wrong”.
  - deleted 9mo ago
    [deleted]
- tjr 9mo ago
  It seems generally agreed that LLMs (currently) do better or worse with different programming languages at least, and maybe with other project logistical differences. The fact that an LLM works great for one user on one project does not mean it will work equally great for another user on a different project. It might! It might work better. It might work worse. And both users might be using the tool equally well, with equal skill, insofar as their part goes.
- consp 9mo ago
  A table saw does not advertise to be a panacea which will make everyone obsolete.
  - simonw 9mo ago
    You should ignore anyone who says that LLMs are a panacea that will make everyone obsolete.
    - achierius 9mo ago
      Even if they're your boss? Remember that most people here are not independently wealthy, they're stuck answering to someone who may not have so level a take on these things as you do.
      - mckn1ght 9mo ago
        The problem there is the boss, not the technology. If it isn’t an insane take on AI, it’d be on something else, and eventually will be. People quit bad managers, not bad jobs. If you have a bad manager, work on quitting them.
        theappsecguy 9mo ago
        I think the problem is the techno fascist oligarchs that are peddling the snake oil that LLMs will wipe out all white collar jobs tomorrow. Managers usually answer to C suite, and the C suite is salivating at the idea of laying off 80% of staff
      - enraged_camel 9mo ago
        >> Even if they're your boss? Especially if they are your boss.
      - AstroBen 9mo ago
        Your boss can't magic things into reality. If the LLM can't do your job they can't replace you with it They can try. Which they'll then fail, and you'll be rehired and have to clean up the mess, then continue on
        achierius 9mo ago
- troupo 9mo ago
  Table saws and cars are deterministic. Once uou learn how to use them, the experience is repeatable. The various magic incantations that LLMs require cannot be learned or repeated. Whatever the "just one more prompt bro" du jour you're thinking of may or may not work at any given time for any given project in any given language.
  - simonw 9mo ago
    I'm finding the prompting techniques I've learned over the last six months continue to work just fine.
    - troupo 9mo ago
      Have you run the "same prompting technique" on the same problem in the same code base and got the same result all the time? I also have prompting techniques that work better than other magical incantations. They do also fail often. Or stop working in a new context. Or...
  - jonas21 9mo ago
    Operating a car (i.e. driving) is certainly not deterministic. Even if you take the same route over and over, you never know exactly what other drivers or pedestrians are going to do, or whether there will be unexpected road conditions, construction, inclement weather, etc. But through experience, you build up intuition and rules of thumb that allow you to drive safely, even in the face of uncertainty. It's the same programming with LLMs. Through experience, you build up intuition and rules of thumb that allow you to get good results, even if you don't get exactly the same result every time.
    - troupo 9mo ago
      > Operating a car (i.e. driving) is certainly not deterministic. Yes. Operating a car or a table saw is deterministic. If you turn your steering wheel left, the car will turn left every time with very few exceptions that can also be explained deterministically (e.g. hardware fault or ice on road). Operating LLMs is completly non-deterministic.
- lelanthran 9mo ago
  > But that's not how most products work. That's exactly how most products work :-/ > If you buy a table saw and can't figure out how to cut a straight line in a piece of wood with it - or keep cutting your fingers off - but didn't take any time at all to learn how to use it, that's on you. Of course - that's deterministic, so if you make a mistake and it comes out wrong, you can fix the mistake you made. > Why should LLMs be any different? Because they are not deterministic; you can't use experience with LLMs in any meaningful way. They may give you a different result when you run the same spec through the LLM a second time.
  - embedding-shape 9mo ago
    > They may give you a different result when you run the same spec through the LLM a second time. Yes kind of, but only different results (maybe) for the things you didn't specify. If you ask for A, B and C, and the LLM automatically made the choice to implement C in "the wrong way" (according to you), you can retry but specify exactly how you want C to be implemented, and it should follow that. Once you've nailed your "spec" enough so there isn't any ambiguity, the LLM won't have to make any choices for you, and then you'll get exactly what you expected. Learning this process, and learning how much and what exactly you have to instruct it to do, is you building up your experience learning how to work with an LLM, and that's meaningful, and something you get better with as you practice it.
    - troupo 9mo ago
      > Yes kind of, but only different results (maybe) for the things you didn't specify. No. They will produce a different result for everything, including the things you specify. It's so easy to verify that I'm surprised you're even making this claim. > Once you've nailed your "spec" enough so there isn't any ambiguity, the LLM won't have to make any choices for you, and then you'll get exactly what you expected 1. There's always ambiguity, or else you'll end up an eternity writing specs 2. LLMs will always produce different results even if the spec is 100% unambiguous for a huge variety of reasons, the main one being: their output is non-deterministic. Except in the most trivial of cases. And even then the simple fact of "your context window is 80% full" can lead to things like "I've rewritten half of your code even though the spec only said that the button color should be green"
- PunchyHamster 9mo ago
  Now imagine the table saw is really, REALLY shit at being table saw and saw no straight angle anywhere during its construction. And they come with new one every 6 months that is very slightly less crooked but controls are all moved over so you have to tweak your workflow Would you still blame the user ?
- torginus 9mo ago
  I'm glad you brought up the power tool analogy - I've bought a $40 soldering iron once, which looked just like the Weller that cost like 5x as much. There was nothing wrong with it on the surface, it was well built and heated up just fine. But every time i tried to solder with it, the results sucked. I couldn't articulate why, and assumed I was doing something wrong (I probably was). Then at my friends house, I got to try the real thing, and it worked like a dream. Again I can't pin down why, but everything just worked. This is how I felt with LLMs (and image generation) - sometimes it just doesn't feel right, and I can't put my finger on what should I fix, but I come away often with the feeling that I needed to do way more tweaking than necessary and the results were just still mediocre.
- switchbak 9mo ago
  It's not anyone's job to "promote it in a good way", we have no responsibility either for or against such tech. The analogy would be more like: "yeah, the motor blew up and burned your garage, but please don't be negative - we need you to promote this saw in a good way". Sure, it's important to "hold it right", but we're not in some cult here where we need to all sell this tech well beyond its current or future potential.
  - senordevnyc 9mo ago
    I think that was a typo and should have been "prompting", not "promoting".
- notnullorvoid 9mo ago
  No one knows what the actual "right way" to hold (prompt) an LLM is. A certain style or pattern to prompting may work in one scenario for one LLM, but change the scenario or model and it often loses any advantage and can give worse output than a different style/pattern. In contrast table saws and cars have pretty clear rules of operation.
tomjen3 9mo ago
If my mum buys a copy of Visual Studio, is it their fault if she cannot code?
- vignesh37 9mo ago
  its more like I buy Visual studio, it will crash at random time, and I get a response like you don't know how to use the ide.
  - simonw 9mo ago
    It's not like that though. It's like you buy Visual Studio and don't believe anyone who tells you that it's complex software with a lot of hidden features and settings that you need to explore in order to use it to its full potential.
    - vignesh37 9mo ago
      I feel it's not worth the effort to spend time and learn the hidden features. whenever I use it to plug something new into a existing codebase it either gives something good at first shot or repeat the non working solution again and again. after such session I only get a feeling instead of spending the last 15 minutes on prompting this, I should have learnt these stuff and this learning would be useful for me forever. I use LLMs as a better form of search engines and that's a useful product.
      - 9dev 9mo ago
        > I feel it's not worth the effort to spend time and learn the hidden features. And that's the only issue here. Many programmers feel offended by an AI threatening their livelihood, and are too arrogant to invest some time in a tool they do deem below themselves—then proceed to complain how useless the tool is on the internet. I'd really suggest taking antirez' advice at heart, and invest time in actually learning how to work with AI properly. Just because Claude Code has a text prompt like ChatGPT doesn't mean you know how to work with it yet. It is going to pay off.
      - 9mo ago
Auracle 9mo ago
Have you seen the way some people google/prompt? It can be a murder scene. Not coding related but my wife is certainly better than most and yet I’ve had to reprompt certain questions she’s asked ChatGPT because she gave it inadequate context. People are awful at that. Us coders are probably better off than most but just as with human communication if you’re not explaining things correctly you’re going to get garbage back.
- trinix912 9mo ago
  Until we get LLMs with deterministic output for a given prompt, there's no guarantee that you and me typing the same prompt will yield a working solution of similar quality. I agree that it helps to add context, but then again assuming people aren't already doing it doesn't help in any way. You can add all the context there is and still get a total smudge out of it. You can select regenerate a few times and it's no better. There's nothing indisputably proving which part of your prompt the LLM will fixate on more and which one it will silently forget (this one's even more apparent with longer prompts).
- lee_ars 9mo ago
  People are "awful at that" because when two people communicate, we're using a lot more than words. Each person participating in a conversation is doing a lot of active bridge-building. We're supplying and looking for extra nonverbal context; we're leaning on basic assumptions about the other speaker, their mood, their tone, their meanings; we're looking at not just syntax but the pragmatics of the convo (https://en.wikipedia.org/wiki/Pragmatics https://en.wikipedia.org/wiki/Pragmatics). The communication of meaning is a multi-dimensional thing that everyone in the conversation is continually contributing to and pushing on. In a way, LLMs are heavily exploitative of human linguistic abilities and expectations. We're wired so hard to actively engage and seek meaning in conversational exchanges that we tend to "helpfully" supply that meaning even when it's absent. We are "vulnerable" to LLMs because they supply all the "I'm talking to a person" linguistic cues, but without any form of underlying mind. Folks like your wife aren't necessarily "bad" at LLM prompting—they're simply responding to the signals they get. The LLM "seems smart." It seems like it "knows" things, so many folks engage with them naturally, as they would with another person, without painstakingly feeding in context and precisely defining all the edges. If anything, it speaks to just how good LLMs are at being LLMs.

simonw 9mo ago

But that's not how most products work. If you buy a table saw and can't figure out how to cut a straight line in a piece of wood with it - or keep cutting your fingers off - but didn't take any time at all to learn how to use it, that's on you. Likewise a car, you have to take lessons and a test before you can use those! Why should LLMs be any different?

what 9mo ago
It’s more like the iPhone “you’re holding it wrong”.
- deleted 9mo ago
  [deleted]
tjr 9mo ago
It seems generally agreed that LLMs (currently) do better or worse with different programming languages at least, and maybe with other project logistical differences. The fact that an LLM works great for one user on one project does not mean it will work equally great for another user on a different project. It might! It might work better. It might work worse. And both users might be using the tool equally well, with equal skill, insofar as their part goes.
consp 9mo ago
A table saw does not advertise to be a panacea which will make everyone obsolete.
- simonw 9mo ago
  You should ignore anyone who says that LLMs are a panacea that will make everyone obsolete.
  - achierius 9mo ago
    Even if they're your boss? Remember that most people here are not independently wealthy, they're stuck answering to someone who may not have so level a take on these things as you do.
    - mckn1ght 9mo ago
      The problem there is the boss, not the technology. If it isn’t an insane take on AI, it’d be on something else, and eventually will be. People quit bad managers, not bad jobs. If you have a bad manager, work on quitting them.
      - theappsecguy 9mo ago
        I think the problem is the techno fascist oligarchs that are peddling the snake oil that LLMs will wipe out all white collar jobs tomorrow. Managers usually answer to C suite, and the C suite is salivating at the idea of laying off 80% of staff
    - enraged_camel 9mo ago
      >> Even if they're your boss? Especially if they are your boss.
    - AstroBen 9mo ago
      Your boss can't magic things into reality. If the LLM can't do your job they can't replace you with it They can try. Which they'll then fail, and you'll be rehired and have to clean up the mess, then continue on
      - achierius 9mo ago
troupo 9mo ago
Table saws and cars are deterministic. Once uou learn how to use them, the experience is repeatable. The various magic incantations that LLMs require cannot be learned or repeated. Whatever the "just one more prompt bro" du jour you're thinking of may or may not work at any given time for any given project in any given language.
- simonw 9mo ago
  I'm finding the prompting techniques I've learned over the last six months continue to work just fine.
  - troupo 9mo ago
    Have you run the "same prompting technique" on the same problem in the same code base and got the same result all the time? I also have prompting techniques that work better than other magical incantations. They do also fail often. Or stop working in a new context. Or...
- jonas21 9mo ago
  Operating a car (i.e. driving) is certainly not deterministic. Even if you take the same route over and over, you never know exactly what other drivers or pedestrians are going to do, or whether there will be unexpected road conditions, construction, inclement weather, etc. But through experience, you build up intuition and rules of thumb that allow you to drive safely, even in the face of uncertainty. It's the same programming with LLMs. Through experience, you build up intuition and rules of thumb that allow you to get good results, even if you don't get exactly the same result every time.
  - troupo 9mo ago
    > Operating a car (i.e. driving) is certainly not deterministic. Yes. Operating a car or a table saw is deterministic. If you turn your steering wheel left, the car will turn left every time with very few exceptions that can also be explained deterministically (e.g. hardware fault or ice on road). Operating LLMs is completly non-deterministic.
lelanthran 9mo ago
> But that's not how most products work. That's exactly how most products work :-/ > If you buy a table saw and can't figure out how to cut a straight line in a piece of wood with it - or keep cutting your fingers off - but didn't take any time at all to learn how to use it, that's on you. Of course - that's deterministic, so if you make a mistake and it comes out wrong, you can fix the mistake you made. > Why should LLMs be any different? Because they are not deterministic; you can't use experience with LLMs in any meaningful way. They may give you a different result when you run the same spec through the LLM a second time.
- embedding-shape 9mo ago
  > They may give you a different result when you run the same spec through the LLM a second time. Yes kind of, but only different results (maybe) for the things you didn't specify. If you ask for A, B and C, and the LLM automatically made the choice to implement C in "the wrong way" (according to you), you can retry but specify exactly how you want C to be implemented, and it should follow that. Once you've nailed your "spec" enough so there isn't any ambiguity, the LLM won't have to make any choices for you, and then you'll get exactly what you expected. Learning this process, and learning how much and what exactly you have to instruct it to do, is you building up your experience learning how to work with an LLM, and that's meaningful, and something you get better with as you practice it.
  - troupo 9mo ago
    > Yes kind of, but only different results (maybe) for the things you didn't specify. No. They will produce a different result for everything, including the things you specify. It's so easy to verify that I'm surprised you're even making this claim. > Once you've nailed your "spec" enough so there isn't any ambiguity, the LLM won't have to make any choices for you, and then you'll get exactly what you expected 1. There's always ambiguity, or else you'll end up an eternity writing specs 2. LLMs will always produce different results even if the spec is 100% unambiguous for a huge variety of reasons, the main one being: their output is non-deterministic. Except in the most trivial of cases. And even then the simple fact of "your context window is 80% full" can lead to things like "I've rewritten half of your code even though the spec only said that the button color should be green"
PunchyHamster 9mo ago
Now imagine the table saw is really, REALLY shit at being table saw and saw no straight angle anywhere during its construction. And they come with new one every 6 months that is very slightly less crooked but controls are all moved over so you have to tweak your workflow Would you still blame the user ?
torginus 9mo ago
I'm glad you brought up the power tool analogy - I've bought a $40 soldering iron once, which looked just like the Weller that cost like 5x as much. There was nothing wrong with it on the surface, it was well built and heated up just fine. But every time i tried to solder with it, the results sucked. I couldn't articulate why, and assumed I was doing something wrong (I probably was). Then at my friends house, I got to try the real thing, and it worked like a dream. Again I can't pin down why, but everything just worked. This is how I felt with LLMs (and image generation) - sometimes it just doesn't feel right, and I can't put my finger on what should I fix, but I come away often with the feeling that I needed to do way more tweaking than necessary and the results were just still mediocre.
switchbak 9mo ago
It's not anyone's job to "promote it in a good way", we have no responsibility either for or against such tech. The analogy would be more like: "yeah, the motor blew up and burned your garage, but please don't be negative - we need you to promote this saw in a good way". Sure, it's important to "hold it right", but we're not in some cult here where we need to all sell this tech well beyond its current or future potential.
- senordevnyc 9mo ago
  I think that was a typo and should have been "prompting", not "promoting".
notnullorvoid 9mo ago
No one knows what the actual "right way" to hold (prompt) an LLM is. A certain style or pattern to prompting may work in one scenario for one LLM, but change the scenario or model and it often loses any advantage and can give worse output than a different style/pattern. In contrast table saws and cars have pretty clear rules of operation.

tomjen3 9mo ago

If my mum buys a copy of Visual Studio, is it their fault if she cannot code?

vignesh37 9mo ago
its more like I buy Visual studio, it will crash at random time, and I get a response like you don't know how to use the ide.
- simonw 9mo ago
  It's not like that though. It's like you buy Visual Studio and don't believe anyone who tells you that it's complex software with a lot of hidden features and settings that you need to explore in order to use it to its full potential.
  - vignesh37 9mo ago
    I feel it's not worth the effort to spend time and learn the hidden features. whenever I use it to plug something new into a existing codebase it either gives something good at first shot or repeat the non working solution again and again. after such session I only get a feeling instead of spending the last 15 minutes on prompting this, I should have learnt these stuff and this learning would be useful for me forever. I use LLMs as a better form of search engines and that's a useful product.
    - 9dev 9mo ago
      > I feel it's not worth the effort to spend time and learn the hidden features. And that's the only issue here. Many programmers feel offended by an AI threatening their livelihood, and are too arrogant to invest some time in a tool they do deem below themselves—then proceed to complain how useless the tool is on the internet. I'd really suggest taking antirez' advice at heart, and invest time in actually learning how to work with AI properly. Just because Claude Code has a text prompt like ChatGPT doesn't mean you know how to work with it yet. It is going to pay off.
    - 9mo ago

Auracle 9mo ago

Have you seen the way some people google/prompt? It can be a murder scene. Not coding related but my wife is certainly better than most and yet I’ve had to reprompt certain questions she’s asked ChatGPT because she gave it inadequate context. People are awful at that. Us coders are probably better off than most but just as with human communication if you’re not explaining things correctly you’re going to get garbage back.

trinix912 9mo ago
Until we get LLMs with deterministic output for a given prompt, there's no guarantee that you and me typing the same prompt will yield a working solution of similar quality. I agree that it helps to add context, but then again assuming people aren't already doing it doesn't help in any way. You can add all the context there is and still get a total smudge out of it. You can select regenerate a few times and it's no better. There's nothing indisputably proving which part of your prompt the LLM will fixate on more and which one it will silently forget (this one's even more apparent with longer prompts).
lee_ars 9mo ago
People are "awful at that" because when two people communicate, we're using a lot more than words. Each person participating in a conversation is doing a lot of active bridge-building. We're supplying and looking for extra nonverbal context; we're leaning on basic assumptions about the other speaker, their mood, their tone, their meanings; we're looking at not just syntax but the pragmatics of the convo (https://en.wikipedia.org/wiki/Pragmatics https://en.wikipedia.org/wiki/Pragmatics). The communication of meaning is a multi-dimensional thing that everyone in the conversation is continually contributing to and pushing on. In a way, LLMs are heavily exploitative of human linguistic abilities and expectations. We're wired so hard to actively engage and seek meaning in conversational exchanges that we tend to "helpfully" supply that meaning even when it's absent. We are "vulnerable" to LLMs because they supply all the "I'm talking to a person" linguistic cues, but without any form of underlying mind. Folks like your wife aren't necessarily "bad" at LLM prompting—they're simply responding to the signals they get. The LLM "seems smart." It seems like it "knows" things, so many folks engage with them naturally, as they would with another person, without painstakingly feeding in context and precisely defining all the edges. If anything, it speaks to just how good LLMs are at being LLMs.