4 ms·

poorly understood systems like humans?

chrismatic 4y ago
I guess they make up for it with accountability.
mindslight 4y ago
Yes... But humans generally have to explain ourselves. At the societal level, it's generally not good enough for another human to make an arbitrary decision. Rather they have to justify it with the reasoning leading to it. The better the reasoning, the more other humans believe them. Whereas with regards to tech, society is currently at the level of "computer says foo" (so it must be true). Requiring explanatory output to justify decisions is a likely path forward for making the use of LLM's accountable. But given how little we've been able to make traditional human-guided tech companies accountable for things like sharing scoring formulas and abusing personal information, I'm not hopeful.
- PeterisP 4y ago
  It is well known with experimental evidence that when people justify their decisions with the reasoning leading to it, it does not necessarily have anything to do with the actual reasons for the decision; even ignoring lying, we do not have privileged insight into all the factors our brain used to make a decision, and when asked for a justification for any decision which wasn't 100% the result of explicit rational planning (which is almost every decision, as even high-planning decisions are demonstrably influenced by non-rational factors, intuition, etc), we use our social skills to effectively invent a post-factum reasoning that both sounds plausible and is socially acceptable - just as language models can do. We often deceive ourselves, we have biases that discourage us from admitting and acknowledging certain reasons for why we did or do something; in some cases extensive therapy can help us recognize why some justification or reason we told others and ourselves was a false one, etc. In essence, dealing with plausible but potentially misleading justifications is something that we have had to do forever, and will still have to do for future artificial agents as well.
  - mindslight 4y ago
    I fully agree [0]. And I can see how my original comment was worded such that you'd think I wasn't taking that into account ("the reasoning"). When using ChatGPT, I can't help but thinking "this sounds like a high school essay". But really, it's not. Rather, it's someone who has college+ level of reading and studying, but has never had to have their ideas tested or scrutinized. A user of ChatGPT kind of mitigates this by asking for clarification, which is a form of learning in the specific session. But in the real world, ChatGPT-as-student would be adjusted with that feedback and then incorporate it into its overall model for serving the next user. [0] For example, the SSC post about "predictive processing" resonated strongly with me
    - jacquesm 4y ago
      If they only add a feedback loop it will already improve hundredfold in a very short time with respect to the output because then suddenly people rather than endless unstructured input will guide the answers towards the expected outcome. That is definitely still a very far cry from actual intelligence but it may well become a lot more useful that way.
  - mannykannot 4y ago
    I feel you are making a valid point here, but I have to demur where you say “…just as language models can do.” Current LLMs do not appear to have any concept of self, let alone a theory of mind - the notion that other people think of themselves the same way as one thinks about oneself (in fact, they don’t appear to have have any concept of language as being about an external world that exists independently of what is said about it.) An LLM can sometimes be prompted to give a response that looks like an explanation for how it came up with a previous response, but it is important to realize that the actual process by which it came to make the original response bears no relationship to the “explanation” it gave. Because everyone's experience with language has been with human-generated language, which is often not completely rational but, when it is not, often deviates in ways we intuitively understand, it is difficult to see the responses of current LLMs as being fundamentally different, even when we know they are.
hoseja 4y ago
Humans are: - fundamentally limited - rough edges sanded off in the ball tumble mill of evolution

I guess they make up for it with accountability.

Yes... But humans generally have to explain ourselves. At the societal level, it's generally not good enough for another human to make an arbitrary decision. Rather they have to justify it with the reasoning leading to it. The better the reasoning, the more other humans believe them. Whereas with regards to tech, society is currently at the level of "computer says foo" (so it must be true). Requiring explanatory output to justify decisions is a likely path forward for making the use of LLM's accountable. But given how little we've been able to make traditional human-guided tech companies accountable for things like sharing scoring formulas and abusing personal information, I'm not hopeful.

PeterisP 4y ago
It is well known with experimental evidence that when people justify their decisions with the reasoning leading to it, it does not necessarily have anything to do with the actual reasons for the decision; even ignoring lying, we do not have privileged insight into all the factors our brain used to make a decision, and when asked for a justification for any decision which wasn't 100% the result of explicit rational planning (which is almost every decision, as even high-planning decisions are demonstrably influenced by non-rational factors, intuition, etc), we use our social skills to effectively invent a post-factum reasoning that both sounds plausible and is socially acceptable - just as language models can do. We often deceive ourselves, we have biases that discourage us from admitting and acknowledging certain reasons for why we did or do something; in some cases extensive therapy can help us recognize why some justification or reason we told others and ourselves was a false one, etc. In essence, dealing with plausible but potentially misleading justifications is something that we have had to do forever, and will still have to do for future artificial agents as well.
- mindslight 4y ago
  I fully agree [0]. And I can see how my original comment was worded such that you'd think I wasn't taking that into account ("the reasoning"). When using ChatGPT, I can't help but thinking "this sounds like a high school essay". But really, it's not. Rather, it's someone who has college+ level of reading and studying, but has never had to have their ideas tested or scrutinized. A user of ChatGPT kind of mitigates this by asking for clarification, which is a form of learning in the specific session. But in the real world, ChatGPT-as-student would be adjusted with that feedback and then incorporate it into its overall model for serving the next user. [0] For example, the SSC post about "predictive processing" resonated strongly with me
  - jacquesm 4y ago
    If they only add a feedback loop it will already improve hundredfold in a very short time with respect to the output because then suddenly people rather than endless unstructured input will guide the answers towards the expected outcome. That is definitely still a very far cry from actual intelligence but it may well become a lot more useful that way.
- mannykannot 4y ago
  I feel you are making a valid point here, but I have to demur where you say “…just as language models can do.” Current LLMs do not appear to have any concept of self, let alone a theory of mind - the notion that other people think of themselves the same way as one thinks about oneself (in fact, they don’t appear to have have any concept of language as being about an external world that exists independently of what is said about it.) An LLM can sometimes be prompted to give a response that looks like an explanation for how it came up with a previous response, but it is important to realize that the actual process by which it came to make the original response bears no relationship to the “explanation” it gave. Because everyone's experience with language has been with human-generated language, which is often not completely rational but, when it is not, often deviates in ways we intuitively understand, it is difficult to see the responses of current LLMs as being fundamentally different, even when we know they are.

hoseja 4y ago

Humans are: - fundamentally limited - rough edges sanded off in the ball tumble mill of evolution