4 ms·

No, it is a tooling problem. The tooling is telling laymen that they built wonderful things that definitely work and perfectly fix and add features. The tooli

No, it is a tooling problem. The tooling is telling laymen that they built wonderful things that definitely work and perfectly fix and add features. The tooling gasses them up and is simply wrong in these cases. If your tool regularly lies, gaslights and produces wrong results, that's a tooling issue.

fatata123 5mo ago
[dead]
deleted 5mo ago
[deleted]
deleted 5mo ago
[deleted]
Den_VR 5mo ago
Does the hammer lie to you that everything is a nail? Can a voltmeter _lie_ to you? EE are expected to know when their measurements are wrong. And Professional Engineers are legally accountable for consequences of such mistakes.
- possibleworlds 5mo ago
  If a hammer had a chat interface that said everything was a nail then the answer would be yes, the hammer lies to you about everything being a nail.
  - walletdrainer 5mo ago
    If someone believes a hammer when it tells them such things, they should probably have some sort of a caretaker assigned to help them through life.
    - mvid 5mo ago
      If hammer companies were suddenly the most valuable international companies, and spent millions on ad campaigns and lobbying about trusting the hammer interface, then you can assume a large amount of people might trust the hammer interface
      - walletdrainer 5mo ago
        Where are the ad campaigns telling me to trust LLMs? I don’t use an adblocker, do read traditional dead tree newspapers and do get exposed to satellite tv channels. I don’t think I’ve ever seen anyone anywhere telling me how reliable LLMs are. Pretty sure this tech sells itself to consumers, enterprise sales are what they’ve always been.
        card_zero 5mo ago
        So now you're pivoting away from the caretaker proposal? I thought it had potential but I don't know how you'd fund it.
        walletdrainer 5mo ago
        > I thought it had potential but I don't know how you'd fund it. The same way we fund other social services here in Europe. If an individual is incapable of caring for themselves, the state is expected to care for them.
        alnwlsn 5mo ago
        Literally saw a video ad the other day which went like "I've always been cautious using Google's AI because it sometimes gets things wrong, but this time, it got it right!"
      - leidenfrost 5mo ago
        Still, it's a tool. Even if your tool learns to talk and to make decisions, it's still a tool, not a person. You're the person and the one responsible for the decisions you make based on your tools. Going back from the analogy, the problem is that we conflated software <engineers> with "coders". A lot of people thought their job was to create code, we gave them a tool to generate a lot of code fast, and they truly think that "more code" = "more good"
  - digitalPhonix 5mo ago
    That wasn’t the question though? A hammer doesn’t have a chat interface, that’s the point.
    - kasabali 5mo ago
      Duh
- fao_ 5mo ago
  If software engineering wants to progress past being an "art" and be considered an engineering discipline, then it should adopt methods and practices from engineering. First and foremost, one of the universal methodologies is analysis of root cause in faults, and redundancies to avoid that. e.g. the FAA has two pilots for planes, and each system is built in redundantly so if an engineer misses a bolt or rivet, the plane won't crash. intersections are designed such that there is a forcing function[0] on the behaviour of the motorists to prevent fault. Or, to take your tool analogy, nail guns are designed to be pressed against something with a decent amount of pressure before you can fire them. All of these systems are designed around the core idea of "a human acting irrationally or improperly is not at fault" and, furthermore, that a human can have a bad day and still avoid a mistake. They all steer someone around a possible fault. Hell, the reason why we divide the road into lanes is itself a forcing function to avoid traffic collisions! So, where is the forcing function in large language models? What part of a large language model prevents gross misuse by laymen? I can think of examples here and there, maybe. OpenAI had to add guard rails to stop people from poisoning themselves with botulism and boron, etc. But the problem here is that the LLM is probabilistic, so there's really no guarantee that those guard rails will hold. I seem to remember there being a paper from a few months back, posted here, that show AI guardrails cannot be proven to work consistently. In that context, LLMs cannot be considered "safe" or "reliable" enough for use. Eddie Burback has a very, very good video showing an absolute worst case result of this[1], that was posted here last year. Even then, off the top of my head Angela Collier has a really, really good video demonstrating that there's an absolute plethora of people who have succumbed, in large ways or small, to the bullshit AI can spew[2]. I feel like if most developers were actually serious about being an engineering discipline, like we claim, then we wouldn't have all jumped on the LLM bandwagon until they'd been properly tested and had a certain level of reliability. Instead there are a sizable chunk of people saying they've stopped coding by hand entirely, and aren't even reviewing the code! i.e. They've thrown out a forcing function that existed to prevent errorenous PRs being committed! And for some bizzare reason, after about 2 decades of people talking about type safety and how we need formal verification to reduce error, everyone seems to be throwing "reduction of error" out the window! [0]: https://en.wikipedia.org/wiki/Behavior-shaping_constraint https://en.wikipedia.org/wiki/Behavior-shaping_constraint (if you're curious about the term) [1]: https://www.youtube.com/watch?v=VRjgNgJms3Q https://www.youtube.com/watch?v=VRjgNgJms3Q [2]: https://www.youtube.com/watch?v=7pqF90rstZQ https://www.youtube.com/watch?v=7pqF90rstZQ
- perching_aix 5mo ago
  > Can a voltmeter _lie_ to you? Hell fucking yes it can?
  - digitalPhonix 5mo ago
    When used according to it’s datasheet/user manual, how?
    - perching_aix 5mo ago
      When their precision mismatches their accuracy (or your expectations as driven by their design), just like with any other metrology tool. Now you might say: "but the datasheet will give you the tolerances, and the manual will tell you to mind it!" And yes, that's true. Just like how LLM providers also do: they tell you that outputs may be arbitrarily wrong, and that you should always check for mistakes. Is this bullshit? Yes. So are metrology tools that have a mismatching precision and accuracy, need calibration, and have designs that fail to make you mind either of these, sending you to reading duty instead. Which just so happens to be a whole lot of them. It is also absolutely not bullshit of course, because it is a fundamental limitation, just like those properties are for metrology devices. LLMs produce arbitrary natural language. Short of becoming able to perfectly read and predict the users' mind, they'll never be able to make any hard assurances, ever. Defective devices also exist, and so do incorrect / incomplete documentation.
      - digitalPhonix 5mo ago
        That's the difference - they have well defined and bounded errors. LLMs do not.
        perching_aix 5mo ago
        Which is a notably different argument than whether they can lie to you. Why are we skipping over the miscalibrated, defective, or ill-documented devices bit though? Those also all have arbitrary error.
    - 5mo ago
- heavyset_go 5mo ago
  If I had a hammer robot that I told to go hammer some nails in a birdhouse and it goes "Sure, I'm on it!" then it nails a cat to the wall and says "Here's you new complete birdhouse, it's perfect in everyway and will make everyone jealous", then yes, that is a tooling issue.
  - digitalPhonix 5mo ago
    The question wasn’t about a hammer robot, it was about a hammer
    - Tostino 5mo ago
      That's not a good analogy then. What benefit is provided by a hammer that just tells the operator (who has eyes and can see) that there is a nail under it (and I assume to swing)?
- Gud 5mo ago
  Yes, a voltmeter can lie to you. Full disclosure: I do high voltage testing for a living.
  - pardon_me 5mo ago
    It can misread, but meters cannot actively generate an incorrect output based on user expectations.
    - Gud 5mo ago
      …yet!
      - wookmaster 5mo ago
        Enshittification knows no bounds
onion2k 5mo ago
If your tool regularly lies, gaslights and produces wrong results, that's a tooling issue. It's a human issue if you don't recognise that the code it's generated is wrong. That will never change no matter how good the tooling gets.
- kiba 5mo ago
  The tooling is the issue because humans designed the tooling wrong. It's a chatbot interface fined tuned to sycophancy. That's not a coincidence.
- Hamuko 5mo ago
  Isn't part of the problem that these tools are advertised as allowing non-coders to code? How are you gonna recognise that the code is wrong when you don't know how to code and the product is telling you that you don't even need to?
- drw85 5mo ago
  Would it be a human issue, if you type something into a calculator and the calculated result is wrong? Would anyone use a calculator confidently, if the result was randomly generated?
  - allarm 5mo ago
    Why do you think this analogy is even remotely correct? It’s well-known that LLMs produce non-deterministic results. It’s also well-known that they hallucinate. To make it even clearer, all the top LLM players make sure to remind everyone of that behavior. If calculators had similar effects and warnings, it would have been a valid analogy. Instead, you're comparing apples and oranges.
    - drw85 5mo ago
      I think it's a valid analogy in some contexts. Like when talking to a person that is not aware of non-determinism and hallucinations. Which happens on this website very frequently. Many people here tell you to use AI like you use a calculator. With minimal or no oversight, with full access to production systems, etc. To let a non deterministic tool communicate on your behalf, or give it access to critical systems is evidence enough that a good number of people are not aware of these facts.
bingo-bongo 5mo ago
> ..laymen.. That’s the behavioral problem. When AI is assisting a professional, the outcome is vastly different.
- amiga386 5mo ago
  If investors invest heavily in lemon juice, then go around hyping it and selling it with the promise it makes you invisible to cameras (which it doesn't), it doesn't matter how stupid and gullible the rubes who fall for that are, the investors bear the responsibility for giving them that idea, when people start attempting to rob banks with lemon juice on their faces. (cf https://en.wikipedia.org/wiki/1995_Greater_Pittsburgh_bank_robberies https://en.wikipedia.org/wiki/1995_Greater_Pittsburgh_bank_r...) Hype is bad. Unwarranted hype is worse. Enabling people who can't do a thing to do what they think the thing is, but isn't, because they don't know any better, is inflicting a pox upon the world.
  - nh23423fefe 5mo ago
    What you say makes no sense and no one will act in the ways you proscribe.
AnthonBerg 5mo ago
By definition of responsibility it is a behavioral problem.
teo_zero 5mo ago
Technical analysis tells you that a stock is in its upwards trend. You invest all your money on it without thinking twice. The price goes down and you lose thousands of dollars. Is it a tool problem? LLMs spit out a sequence of tokens that is the most probable continuation of the input. LLMs don't lie any more than technical analysis does when it predicts the most likely trend of stock prices. It's up to you how to use this information.
- deleted 5mo ago
  [deleted]
potsandpans 5mo ago
Everyone, "please please please don't personify llms it's so harmful." Also everyone, "the inanimate tool is lying and tricking people into inundating open source developers with poorly thought out slop."
- heavyset_go 5mo ago
  I am personally everyone
zamadatix 5mo ago
There will usually be more than one issue, Sockbot's solution just happens to deal with more than one at a time.

fatata123 5mo ago

[dead]

deleted 5mo ago

[deleted]

deleted 5mo ago

[deleted]

Den_VR 5mo ago

Does the hammer lie to you that everything is a nail? Can a voltmeter _lie_ to you? EE are expected to know when their measurements are wrong. And Professional Engineers are legally accountable for consequences of such mistakes.

possibleworlds 5mo ago
If a hammer had a chat interface that said everything was a nail then the answer would be yes, the hammer lies to you about everything being a nail.
- walletdrainer 5mo ago
  If someone believes a hammer when it tells them such things, they should probably have some sort of a caretaker assigned to help them through life.
  - mvid 5mo ago
    If hammer companies were suddenly the most valuable international companies, and spent millions on ad campaigns and lobbying about trusting the hammer interface, then you can assume a large amount of people might trust the hammer interface
    - walletdrainer 5mo ago
      Where are the ad campaigns telling me to trust LLMs? I don’t use an adblocker, do read traditional dead tree newspapers and do get exposed to satellite tv channels. I don’t think I’ve ever seen anyone anywhere telling me how reliable LLMs are. Pretty sure this tech sells itself to consumers, enterprise sales are what they’ve always been.
      - card_zero 5mo ago
        So now you're pivoting away from the caretaker proposal? I thought it had potential but I don't know how you'd fund it.
        walletdrainer 5mo ago
        > I thought it had potential but I don't know how you'd fund it. The same way we fund other social services here in Europe. If an individual is incapable of caring for themselves, the state is expected to care for them.
      - alnwlsn 5mo ago
        Literally saw a video ad the other day which went like "I've always been cautious using Google's AI because it sometimes gets things wrong, but this time, it got it right!"
    - leidenfrost 5mo ago
      Still, it's a tool. Even if your tool learns to talk and to make decisions, it's still a tool, not a person. You're the person and the one responsible for the decisions you make based on your tools. Going back from the analogy, the problem is that we conflated software <engineers> with "coders". A lot of people thought their job was to create code, we gave them a tool to generate a lot of code fast, and they truly think that "more code" = "more good"
- digitalPhonix 5mo ago
  That wasn’t the question though? A hammer doesn’t have a chat interface, that’s the point.
  - kasabali 5mo ago
    Duh
fao_ 5mo ago
If software engineering wants to progress past being an "art" and be considered an engineering discipline, then it should adopt methods and practices from engineering. First and foremost, one of the universal methodologies is analysis of root cause in faults, and redundancies to avoid that. e.g. the FAA has two pilots for planes, and each system is built in redundantly so if an engineer misses a bolt or rivet, the plane won't crash. intersections are designed such that there is a forcing function[0] on the behaviour of the motorists to prevent fault. Or, to take your tool analogy, nail guns are designed to be pressed against something with a decent amount of pressure before you can fire them. All of these systems are designed around the core idea of "a human acting irrationally or improperly is not at fault" and, furthermore, that a human can have a bad day and still avoid a mistake. They all steer someone around a possible fault. Hell, the reason why we divide the road into lanes is itself a forcing function to avoid traffic collisions! So, where is the forcing function in large language models? What part of a large language model prevents gross misuse by laymen? I can think of examples here and there, maybe. OpenAI had to add guard rails to stop people from poisoning themselves with botulism and boron, etc. But the problem here is that the LLM is probabilistic, so there's really no guarantee that those guard rails will hold. I seem to remember there being a paper from a few months back, posted here, that show AI guardrails cannot be proven to work consistently. In that context, LLMs cannot be considered "safe" or "reliable" enough for use. Eddie Burback has a very, very good video showing an absolute worst case result of this[1], that was posted here last year. Even then, off the top of my head Angela Collier has a really, really good video demonstrating that there's an absolute plethora of people who have succumbed, in large ways or small, to the bullshit AI can spew[2]. I feel like if most developers were actually serious about being an engineering discipline, like we claim, then we wouldn't have all jumped on the LLM bandwagon until they'd been properly tested and had a certain level of reliability. Instead there are a sizable chunk of people saying they've stopped coding by hand entirely, and aren't even reviewing the code! i.e. They've thrown out a forcing function that existed to prevent errorenous PRs being committed! And for some bizzare reason, after about 2 decades of people talking about type safety and how we need formal verification to reduce error, everyone seems to be throwing "reduction of error" out the window! [0]: https://en.wikipedia.org/wiki/Behavior-shaping_constraint https://en.wikipedia.org/wiki/Behavior-shaping_constraint (if you're curious about the term) [1]: https://www.youtube.com/watch?v=VRjgNgJms3Q https://www.youtube.com/watch?v=VRjgNgJms3Q [2]: https://www.youtube.com/watch?v=7pqF90rstZQ https://www.youtube.com/watch?v=7pqF90rstZQ
perching_aix 5mo ago
> Can a voltmeter _lie_ to you? Hell fucking yes it can?
- digitalPhonix 5mo ago
  When used according to it’s datasheet/user manual, how?
  - perching_aix 5mo ago
    When their precision mismatches their accuracy (or your expectations as driven by their design), just like with any other metrology tool. Now you might say: "but the datasheet will give you the tolerances, and the manual will tell you to mind it!" And yes, that's true. Just like how LLM providers also do: they tell you that outputs may be arbitrarily wrong, and that you should always check for mistakes. Is this bullshit? Yes. So are metrology tools that have a mismatching precision and accuracy, need calibration, and have designs that fail to make you mind either of these, sending you to reading duty instead. Which just so happens to be a whole lot of them. It is also absolutely not bullshit of course, because it is a fundamental limitation, just like those properties are for metrology devices. LLMs produce arbitrary natural language. Short of becoming able to perfectly read and predict the users' mind, they'll never be able to make any hard assurances, ever. Defective devices also exist, and so do incorrect / incomplete documentation.
    - digitalPhonix 5mo ago
      That's the difference - they have well defined and bounded errors. LLMs do not.
      - perching_aix 5mo ago
        Which is a notably different argument than whether they can lie to you. Why are we skipping over the miscalibrated, defective, or ill-documented devices bit though? Those also all have arbitrary error.
  - 5mo ago
heavyset_go 5mo ago
If I had a hammer robot that I told to go hammer some nails in a birdhouse and it goes "Sure, I'm on it!" then it nails a cat to the wall and says "Here's you new complete birdhouse, it's perfect in everyway and will make everyone jealous", then yes, that is a tooling issue.
- digitalPhonix 5mo ago
  The question wasn’t about a hammer robot, it was about a hammer
  - Tostino 5mo ago
    That's not a good analogy then. What benefit is provided by a hammer that just tells the operator (who has eyes and can see) that there is a nail under it (and I assume to swing)?
Gud 5mo ago
Yes, a voltmeter can lie to you. Full disclosure: I do high voltage testing for a living.
- pardon_me 5mo ago
  It can misread, but meters cannot actively generate an incorrect output based on user expectations.
  - Gud 5mo ago
    …yet!
    - wookmaster 5mo ago
      Enshittification knows no bounds

onion2k 5mo ago

If your tool regularly lies, gaslights and produces wrong results, that's a tooling issue. It's a human issue if you don't recognise that the code it's generated is wrong. That will never change no matter how good the tooling gets.

kiba 5mo ago
The tooling is the issue because humans designed the tooling wrong. It's a chatbot interface fined tuned to sycophancy. That's not a coincidence.
Hamuko 5mo ago
Isn't part of the problem that these tools are advertised as allowing non-coders to code? How are you gonna recognise that the code is wrong when you don't know how to code and the product is telling you that you don't even need to?
drw85 5mo ago
Would it be a human issue, if you type something into a calculator and the calculated result is wrong? Would anyone use a calculator confidently, if the result was randomly generated?
- allarm 5mo ago
  Why do you think this analogy is even remotely correct? It’s well-known that LLMs produce non-deterministic results. It’s also well-known that they hallucinate. To make it even clearer, all the top LLM players make sure to remind everyone of that behavior. If calculators had similar effects and warnings, it would have been a valid analogy. Instead, you're comparing apples and oranges.
  - drw85 5mo ago
    I think it's a valid analogy in some contexts. Like when talking to a person that is not aware of non-determinism and hallucinations. Which happens on this website very frequently. Many people here tell you to use AI like you use a calculator. With minimal or no oversight, with full access to production systems, etc. To let a non deterministic tool communicate on your behalf, or give it access to critical systems is evidence enough that a good number of people are not aware of these facts.

bingo-bongo 5mo ago

> ..laymen.. That’s the behavioral problem. When AI is assisting a professional, the outcome is vastly different.

amiga386 5mo ago
If investors invest heavily in lemon juice, then go around hyping it and selling it with the promise it makes you invisible to cameras (which it doesn't), it doesn't matter how stupid and gullible the rubes who fall for that are, the investors bear the responsibility for giving them that idea, when people start attempting to rob banks with lemon juice on their faces. (cf https://en.wikipedia.org/wiki/1995_Greater_Pittsburgh_bank_robberies https://en.wikipedia.org/wiki/1995_Greater_Pittsburgh_bank_r...) Hype is bad. Unwarranted hype is worse. Enabling people who can't do a thing to do what they think the thing is, but isn't, because they don't know any better, is inflicting a pox upon the world.
- nh23423fefe 5mo ago
  What you say makes no sense and no one will act in the ways you proscribe.

AnthonBerg 5mo ago

By definition of responsibility it is a behavioral problem.

teo_zero 5mo ago

Technical analysis tells you that a stock is in its upwards trend. You invest all your money on it without thinking twice. The price goes down and you lose thousands of dollars. Is it a tool problem? LLMs spit out a sequence of tokens that is the most probable continuation of the input. LLMs don't lie any more than technical analysis does when it predicts the most likely trend of stock prices. It's up to you how to use this information.

deleted 5mo ago
[deleted]

potsandpans 5mo ago

Everyone, "please please please don't personify llms it's so harmful." Also everyone, "the inanimate tool is lying and tricking people into inundating open source developers with poorly thought out slop."

heavyset_go 5mo ago
I am personally everyone

zamadatix 5mo ago

There will usually be more than one issue, Sockbot's solution just happens to deal with more than one at a time.