3 ms·

This is kind of the reverse of https://en.wikipedia.org/wiki/Poka-yoke https://en.wikipedia.org/wiki/Poka-yoke . A lot of tools have affordances built in to mak

by pjc50 6mo ago

This is kind of the reverse of https://en.wikipedia.org/wiki/Poka-yoke https://en.wikipedia.org/wiki/Poka-yoke . A lot of tools have affordances built in to make "right" things easy and "wrong" or unsafe things harder. LLMs .. well, the text interface is uniquely flat. Everything is seemingly as easy as everything else. I worry about the use of humans as sacrificial accountability sinks. The "self-driving car" model already has this: a car which drives itself most of the time, but where a human user is required to be constantly alert so that the AI can transfer responsibility a few hundred miliseconds before the crash.

Avicebron 6mo ago
I think that might be the better definition between "engineering" and "vibing". Engineering follows and elevates Poka-yoke patterns, vibing ignores them.
BadBadJellyBean 6mo ago
I agree that LLMs could be more open about their dangers and that people are bad at judging risks sometimes. Still I think a band saw has very little warning on it and by it's design there is very little anyone can do about me cutting off my finger if I am not careful. LLM companies can do very little about the unpredictability of LLMs. So we have to choose how for we will let it go. In the end the LLM only produces texts. We are in control what tools we give it. The more tools the more useful and also the more dangerous. And maybe it's all worth it. Maybe the LLM deletes the database only sometimes but between that we make a lot of money. I don't think my employer would enjoy that so I will be more conservative.
- skydhash 6mo ago
  It’s possible to make AI safe, but that also throws most of the gains out of the windows, especially if the artifact is a diff which can take time to review. In IT, you often have to give access to possible malicious users, you just have to scope what they can do. But the push is agentic everything, where AI needs to be everywhere, not in its own sandbox.
  - BadBadJellyBean 6mo ago
    We don't have to blindly follow every trend. If agentic is not safe then it's on me if I use it and something breaks.
- clickety_clack 6mo ago
  A band saw is always a screaming band of bladed death. An LLM is sometimes a buddy, sometimes a mentor, and only sometimes a guy that drops your database.
  - cindyllm 6mo ago
    [dead]
  - Worf 6mo ago
    It's not easy to always remember it's a soulless tool. Sometimes I'm even about to say "thanks" before closing the chat window, until I realize I wouldn't say thanks to my saw or to a random CLI command. But AI, the saw and the random CLI command can all be helpful or destructive. Until the AI shows some signs of consciousness, I'll never treat it as a buddy or a mentor. I'll treat it like an advanced combination of grep, sort and other commands that manipulate text. It's hard to remember that when it works so amazingly well sometimes. I've been chatting with AI for a few years and every day I'm still amazed and how this is all possible. We've never had this in our lives until a few years ago and now it's changed the way we do a lot of things. But just like we have to remember the magical machine elves we hallucinate are not really there, we have to constantly remind ourselves that it's an unpredictable soulless tool with many rough edges. If it helps to treat it like a human, treat it like an idiot savant with autism, schizophrenia, ADHD, psychopathy and a personality disorder who sometimes forgets to take their pills and can start breaking things should a fly lands on their shoulder. You'd listen to them and value their input, but you wouldn't let them in your data center unsupervised as they have no ethics and no honor.
  - mattgreenrocks 6mo ago
    Maybe we can just not give it access to production databases ever? Not picking on you, but AI maximalism has infected tech to the point where we talk about how to stop AI from deleting prod instead of seeing that giving AI access to prod is a foolish idea to begin with.
    - clickety_clack 6mo ago
      I mean that it’s easy to be careful around a bandsaw because it’s clearly dangerous. The danger with LLMs is that they don’t seem overtly dangerous so you just go right ahead and throw your whole arm in there.
- muwtyhg 6mo ago
  > Still I think a band saw has very little warning on it and by it's design there is very little anyone can do about me cutting off my finger Most saws have a blade guard of some sort to prevent the blade from being over-exposed. They are also COVERED in warning signs and symbols, as well as having other safety features like emergency stop buttons/pedals. There has definitely been a maximal amount of effort taken to warn and keep people safe from saws. LLMs, conversely, have been shoved into everything with very little forethought or testing to make sure they are safe and perform the task correctly.
lelanthran 6mo ago
> This is kind of the reverse of https://en.wikipedia.org/wiki/Poka-yoke https://en.wikipedia.org/wiki/Poka-yoke . A lot of tools have affordances built in to make "right" things easy and "wrong" or unsafe things harder. I point to the first USB port as the harbinger of things to come - try it one way, fail, turn it around, fail again, then turn it around one more time. Just like AI, except there are unlimited axis upon which to turn it :-/
chrisweekly 6mo ago
^ which approach makes no logical sense; an inattentive or even partly-attentive driver simply cannot resume control and react accordingly within even 2 seconds.
polytely 6mo ago
This is so well put, and it not only happens on the user level but also on the organisational level. Where you can completely abdicate both responsibility and explanation by moving the complicated questions into the black box of an AI model.
eqvinox 6mo ago
> A lot of tools have affordances built in to make "right" things easy and "wrong" or unsafe things harder. This is true for almost anything handed to laypeople, but not for a lot of professional tools. Even a plain battery powered drill has very few protections against misuse. A soldering iron has none. Neither do sewing needles; sewing machines barely do, in the sense that you can't stick your fingers in a gap too narrow. A chemist's chemicals certainly have no protections, only warning labels. Etc. Also cf. the hierarchy of controls: https://www.cdc.gov/niosh/hierarchy-of-controls/about/index.html https://www.cdc.gov/niosh/hierarchy-of-controls/about/index.... people don't seem to want to eliminate AI → replacing it doesn't improve things → isolating it - yup, people are trying to put it in containers and not give it access to delete the production database → changing how people work with it: that's where we are now → PPE: no such thing for AI, sadly → production database is deleted.
- BadBadJellyBean 6mo ago
  Exactly this. I was talking about professionals. People who should know better. If we as professionals give away our agency and our accountability we make ourselves obsolete. If I just tell the LLM what to do and hope it doesn't go south then the Manager could probably do that as well. And if a non professional did it they should ask themselves why we have professionals. Maybe there was a reason and maybe they do have value.
- queenkjuul 6mo ago
  An LLM is a large and complex machine, not a screwdriver. Large and complex [physical] machines are built with safeguards to prevent misuse, injury, etc by regulation.
  - nuancebydefault 6mo ago
    LLM's are in principle text in / text out machines. If the user extends its capability to have agency over a production database or a machine, there's nothing that can safeguard the safety. Imagine I ask an LLM to instruct left/right/speed up/slow down while driving. I can simply bypass any safeguard by stating i suddenly became blind while driving a car. While in fact i'm blindfolded and doing an experiment on a highway.

by pjc50 6mo ago

Avicebron 6mo ago

I think that might be the better definition between "engineering" and "vibing". Engineering follows and elevates Poka-yoke patterns, vibing ignores them.

BadBadJellyBean 6mo ago

I agree that LLMs could be more open about their dangers and that people are bad at judging risks sometimes. Still I think a band saw has very little warning on it and by it's design there is very little anyone can do about me cutting off my finger if I am not careful. LLM companies can do very little about the unpredictability of LLMs. So we have to choose how for we will let it go. In the end the LLM only produces texts. We are in control what tools we give it. The more tools the more useful and also the more dangerous. And maybe it's all worth it. Maybe the LLM deletes the database only sometimes but between that we make a lot of money. I don't think my employer would enjoy that so I will be more conservative.

skydhash 6mo ago
It’s possible to make AI safe, but that also throws most of the gains out of the windows, especially if the artifact is a diff which can take time to review. In IT, you often have to give access to possible malicious users, you just have to scope what they can do. But the push is agentic everything, where AI needs to be everywhere, not in its own sandbox.
- BadBadJellyBean 6mo ago
  We don't have to blindly follow every trend. If agentic is not safe then it's on me if I use it and something breaks.
clickety_clack 6mo ago
A band saw is always a screaming band of bladed death. An LLM is sometimes a buddy, sometimes a mentor, and only sometimes a guy that drops your database.
- cindyllm 6mo ago
  [dead]
- Worf 6mo ago
  It's not easy to always remember it's a soulless tool. Sometimes I'm even about to say "thanks" before closing the chat window, until I realize I wouldn't say thanks to my saw or to a random CLI command. But AI, the saw and the random CLI command can all be helpful or destructive. Until the AI shows some signs of consciousness, I'll never treat it as a buddy or a mentor. I'll treat it like an advanced combination of grep, sort and other commands that manipulate text. It's hard to remember that when it works so amazingly well sometimes. I've been chatting with AI for a few years and every day I'm still amazed and how this is all possible. We've never had this in our lives until a few years ago and now it's changed the way we do a lot of things. But just like we have to remember the magical machine elves we hallucinate are not really there, we have to constantly remind ourselves that it's an unpredictable soulless tool with many rough edges. If it helps to treat it like a human, treat it like an idiot savant with autism, schizophrenia, ADHD, psychopathy and a personality disorder who sometimes forgets to take their pills and can start breaking things should a fly lands on their shoulder. You'd listen to them and value their input, but you wouldn't let them in your data center unsupervised as they have no ethics and no honor.
- mattgreenrocks 6mo ago
  Maybe we can just not give it access to production databases ever? Not picking on you, but AI maximalism has infected tech to the point where we talk about how to stop AI from deleting prod instead of seeing that giving AI access to prod is a foolish idea to begin with.
  - clickety_clack 6mo ago
    I mean that it’s easy to be careful around a bandsaw because it’s clearly dangerous. The danger with LLMs is that they don’t seem overtly dangerous so you just go right ahead and throw your whole arm in there.
muwtyhg 6mo ago
> Still I think a band saw has very little warning on it and by it's design there is very little anyone can do about me cutting off my finger Most saws have a blade guard of some sort to prevent the blade from being over-exposed. They are also COVERED in warning signs and symbols, as well as having other safety features like emergency stop buttons/pedals. There has definitely been a maximal amount of effort taken to warn and keep people safe from saws. LLMs, conversely, have been shoved into everything with very little forethought or testing to make sure they are safe and perform the task correctly.

lelanthran 6mo ago

> This is kind of the reverse of https://en.wikipedia.org/wiki/Poka-yoke https://en.wikipedia.org/wiki/Poka-yoke . A lot of tools have affordances built in to make "right" things easy and "wrong" or unsafe things harder. I point to the first USB port as the harbinger of things to come - try it one way, fail, turn it around, fail again, then turn it around one more time. Just like AI, except there are unlimited axis upon which to turn it :-/

chrisweekly 6mo ago

^ which approach makes no logical sense; an inattentive or even partly-attentive driver simply cannot resume control and react accordingly within even 2 seconds.

polytely 6mo ago

This is so well put, and it not only happens on the user level but also on the organisational level. Where you can completely abdicate both responsibility and explanation by moving the complicated questions into the black box of an AI model.

eqvinox 6mo ago

> A lot of tools have affordances built in to make "right" things easy and "wrong" or unsafe things harder. This is true for almost anything handed to laypeople, but not for a lot of professional tools. Even a plain battery powered drill has very few protections against misuse. A soldering iron has none. Neither do sewing needles; sewing machines barely do, in the sense that you can't stick your fingers in a gap too narrow. A chemist's chemicals certainly have no protections, only warning labels. Etc. Also cf. the hierarchy of controls: https://www.cdc.gov/niosh/hierarchy-of-controls/about/index.html https://www.cdc.gov/niosh/hierarchy-of-controls/about/index.... people don't seem to want to eliminate AI → replacing it doesn't improve things → isolating it - yup, people are trying to put it in containers and not give it access to delete the production database → changing how people work with it: that's where we are now → PPE: no such thing for AI, sadly → production database is deleted.

BadBadJellyBean 6mo ago
Exactly this. I was talking about professionals. People who should know better. If we as professionals give away our agency and our accountability we make ourselves obsolete. If I just tell the LLM what to do and hope it doesn't go south then the Manager could probably do that as well. And if a non professional did it they should ask themselves why we have professionals. Maybe there was a reason and maybe they do have value.
queenkjuul 6mo ago
An LLM is a large and complex machine, not a screwdriver. Large and complex [physical] machines are built with safeguards to prevent misuse, injury, etc by regulation.
- nuancebydefault 6mo ago
  LLM's are in principle text in / text out machines. If the user extends its capability to have agency over a production database or a machine, there's nothing that can safeguard the safety. Imagine I ask an LLM to instruct left/right/speed up/slow down while driving. I can simply bypass any safeguard by stating i suddenly became blind while driving a car. While in fact i'm blindfolded and doing an experiment on a highway.