10 ms·

Because human minds are fallible black boxes, we have developed a wide variety of tools that exist outside our minds, like spoken language, written language, la

Because human minds are fallible black boxes, we have developed a wide variety of tools that exist outside our minds, like spoken language, written language, law, standard operating procedures, math, scientific knowledge, etc. What does it look like for fallible human minds to work on engineering an airplane? Things are calculated, recorded, checked, tested. People do not just sit there thinking and then spitting out their best guess. Even if we suppose that LLMs work similar to the human mind (a huge supposition!), LLMs still do not do their work like teams of humans. An LLM dreams and guesses, and it still falls to humans to check and verify. Rigorous human work is actually a highly social activity. People interact using formal methods and that is what produces reliable results. Using an LLM as one of the social nodes is fine, but this article is about the typical use of software, which is to reliably encode those formal methods between humans. And LLMs don’t work that way. Basically, we can’t have it both ways. If an LLM thinks like a human, then we should not think of it as a software tool like curl or grep or Linux or Apple Photos. Tools that we expect (and need) to work the exact same way every time.

725686 2y ago
"People do not just sit there thinking and then spitting out their best guess." Well, if you are using AI like this, you are doing it wrong. Yes AI is imperfect, fallible, it sometimes hallucinates, but it is a freaking time saver (10x?). It is a tool. Don't expect a hammer to build you a cabinet.
- 420official 2y ago
  There is no other way to use an LLM than to give it context and have it give its best guess, that's how LLMs fundamentally work. You can give it different context, but it's just guessing at tokens.
TomK32 2y ago
> Because human minds are fallible black boxes, we have developed a wide variety of tools that exist outside our minds, like spoken language, written language, law, standard operating procedures, math, scientific knowledge, etc. Standard operating procedures are great but simplify it to checklists. Don't ever forget checklists which have proven vital for pilots and surgeons alike. And looking at the WHO Surgical Safety Checklist you might think "that's basic stuff" but apparently it is necessary and works https://www.who.int/teams/integrated-health-services/patient-safety/research/safe-surgery/tool-and-resources https://www.who.int/teams/integrated-health-services/patient...
jvanderbot 2y ago
This is a fantastic and thought-provoking response. Thinking of humans as fallible systems and humanity and its progress as a self-correcting distributed computation / construction system is going to stick with me for a long time.
- clint 2y ago
  Not trying to belittle or be mean, but what exactly did you assume about humans before you read this response? I find it facinating that apparently a lot of people don't think of humans as stochastic, non-deterministic black boxes. Heck one of the defining qualities of humans is that not only are we unpredictable and fundamentally unknowable to other intelligences (even other humans!) is that we also participate in sophisticated subterfuge and lying to manipulate other intelligences (even other humans!) and often very convincingly. In fact, I would propose that our society is fundamentally defined and shaped by our ability and willingness to hide, deceive, and use mind tricks to get what our little monkey brains want over the next couple hours or days.
  - sigbottle 2y ago
    I totally understand this rationally if you sit down and walk me through the steps. But there's a lot of reasons - ego, fear of losing... that core identity, etc. that can easily come back and bite you. I'm not sure if this is the same as meditation and ego death or whatever. I find that even if you go down the spiritual route, you also run into the same issues. People in philosophy also argue things like rational actors, self-coherency, etc. And hey, even in this current moment you were able to type out a coherent thought, right? I've noticed more and more that humans behave a lot like LLM's. In the sense that it's really, really hard to observe my true internal state - I can only try to find patterns and guess at shit. Every theory I've tried applying to myself is just "wrong" - in the sense that either it feels wrong, or I'll get depressed because the theory basically boils down to "you're lazy and you have to do the work" which is a highly emotionally evocative theory that doesn't help anyone. This probably didn't make any sense but g'day.
  - jvanderbot 2y ago
    I knew that they worked this way, but the conciseness of the response and clean analogy to systems I know and work with all day was just very satisfying. For example, there was probably still 10-20% of my mind that assumed that stubbornness and ignorance was the reason for things going slowly most of the time, but I'm re-evaluating that, even though I knew that delays and double-checking were inherent features of a business and process. Re-framing those delays as "evolved responses 100% of the time" rather than "10% of the mistrust, 10% ignorance, 10% .... " is just a more positive way of thinking about human-driven processes.
SoftTalker 2y ago
> What does it look like for fallible human minds to work on engineering an airplane? Things are calculated, recorded, checked, tested. People do not just sit there thinking and then spitting out their best guess. People used to do this. The result was massively overbuilt structures, some of which are still with us hundreds of years later. The result was also underbuilt structures, which tended to collapse and maybe kill people. They are no longer around. All of the science and math and process and standards in modern engineering is the solution humans came up with because our guesses aren't good enough. LLMs will need the same if they are to be relied upon.
chamomeal 2y ago
This is a really interesting perspective and a great point.

725686 2y ago

"People do not just sit there thinking and then spitting out their best guess." Well, if you are using AI like this, you are doing it wrong. Yes AI is imperfect, fallible, it sometimes hallucinates, but it is a freaking time saver (10x?). It is a tool. Don't expect a hammer to build you a cabinet.

420official 2y ago
There is no other way to use an LLM than to give it context and have it give its best guess, that's how LLMs fundamentally work. You can give it different context, but it's just guessing at tokens.

TomK32 2y ago

> Because human minds are fallible black boxes, we have developed a wide variety of tools that exist outside our minds, like spoken language, written language, law, standard operating procedures, math, scientific knowledge, etc. Standard operating procedures are great but simplify it to checklists. Don't ever forget checklists which have proven vital for pilots and surgeons alike. And looking at the WHO Surgical Safety Checklist you might think "that's basic stuff" but apparently it is necessary and works https://www.who.int/teams/integrated-health-services/patient-safety/research/safe-surgery/tool-and-resources https://www.who.int/teams/integrated-health-services/patient...

jvanderbot 2y ago

This is a fantastic and thought-provoking response. Thinking of humans as fallible systems and humanity and its progress as a self-correcting distributed computation / construction system is going to stick with me for a long time.

clint 2y ago
Not trying to belittle or be mean, but what exactly did you assume about humans before you read this response? I find it facinating that apparently a lot of people don't think of humans as stochastic, non-deterministic black boxes. Heck one of the defining qualities of humans is that not only are we unpredictable and fundamentally unknowable to other intelligences (even other humans!) is that we also participate in sophisticated subterfuge and lying to manipulate other intelligences (even other humans!) and often very convincingly. In fact, I would propose that our society is fundamentally defined and shaped by our ability and willingness to hide, deceive, and use mind tricks to get what our little monkey brains want over the next couple hours or days.
- sigbottle 2y ago
  I totally understand this rationally if you sit down and walk me through the steps. But there's a lot of reasons - ego, fear of losing... that core identity, etc. that can easily come back and bite you. I'm not sure if this is the same as meditation and ego death or whatever. I find that even if you go down the spiritual route, you also run into the same issues. People in philosophy also argue things like rational actors, self-coherency, etc. And hey, even in this current moment you were able to type out a coherent thought, right? I've noticed more and more that humans behave a lot like LLM's. In the sense that it's really, really hard to observe my true internal state - I can only try to find patterns and guess at shit. Every theory I've tried applying to myself is just "wrong" - in the sense that either it feels wrong, or I'll get depressed because the theory basically boils down to "you're lazy and you have to do the work" which is a highly emotionally evocative theory that doesn't help anyone. This probably didn't make any sense but g'day.
- jvanderbot 2y ago
  I knew that they worked this way, but the conciseness of the response and clean analogy to systems I know and work with all day was just very satisfying. For example, there was probably still 10-20% of my mind that assumed that stubbornness and ignorance was the reason for things going slowly most of the time, but I'm re-evaluating that, even though I knew that delays and double-checking were inherent features of a business and process. Re-framing those delays as "evolved responses 100% of the time" rather than "10% of the mistrust, 10% ignorance, 10% .... " is just a more positive way of thinking about human-driven processes.

SoftTalker 2y ago

> What does it look like for fallible human minds to work on engineering an airplane? Things are calculated, recorded, checked, tested. People do not just sit there thinking and then spitting out their best guess. People used to do this. The result was massively overbuilt structures, some of which are still with us hundreds of years later. The result was also underbuilt structures, which tended to collapse and maybe kill people. They are no longer around. All of the science and math and process and standards in modern engineering is the solution humans came up with because our guesses aren't good enough. LLMs will need the same if they are to be relied upon.

chamomeal 2y ago

This is a really interesting perspective and a great point.