3 ms·

Talking about this in terms of exploration/exploitation may be a bit misleading, because from a pure exploration-exploitation perspective, biases wouldn't be a

Talking about this in terms of exploration/exploitation may be a bit misleading, because from a pure exploration-exploitation perspective, biases wouldn't be a problem if the groups were secretly all identical. If they are, you are "right" to spend zero effort on exploration, your initial inaccurate model that the X are better doctors than Y, will produce no worse results than the completely accurate model.

jb1991 27d ago
Isn’t exploration vs exploitation about the decision-making process, not about the actual reality in the world around you? It doesn’t matter if they are secretly identical or not. The exploration/exploitation trade-off is in the person making those decisions.
- vintermann 27d ago
  I don't understand what you suggest that implies?
  - ben_w 27d ago
    I think they're saying that while it doesn't matter, the agent and human "do not actutally know" that it does not matter. Philosophy sometimes says that knowledge is a "justified true belief"*; in this experiment, agents and humans have incorrectly justified a false belief that some applicants are better for certain roles. * other times, it says this isn't good enough
    - beepbooptheory 27d ago
      Seems quite odd to cite all of philosophy as saying something, as if it were a single person with contradictory beliefs.. And then its like you are both saying the justification is incorrect and the belief is false, so its not really like the bare nuance of the concept is adding to the point. Why feel the need to appeal to an (imaginary) authority at all in this case? "Oh well if philosophy said it, I better be taking this seriously!"
      - ben_w 27d ago
        I think you misunderstood my point, just as the other commentor misunderstood one level up. Perhaps a different approach to explain the problem here: "It ain't what they don't know, it's what they know for sure that just ain't so".
        beepbooptheory 27d ago
        Hm ok, but how are you mapping this, like, epistemological concept to what you are responding to re exploration/exploitation? Has exploration happened or not if it amounts to false beliefs? The whole point tradeoff doesn't seem to make sense if the person in fact can't actually successfully explore! Or even if there the possibility of that. But it is also very likely I am misunderstanding!
        ben_w 27d ago
        A flat distribution is still a distribution, and correct exploration would have revealed that the distribution is flat. The agent appears to have gained the false belief that it has learned something and done some exploring, when in fact it has not. c.f. Sally-Anne test: Sally thinks she knows where her toy is, we know that she doesn't, and indeed couldn't. The LLM (and humans in similar conditions) think they know what the distribution is, we know that they don't.
        beepbooptheory 27d ago
        Really not trying to be reductive here, but it feels like all you are trying to articulate here is that the LLM was wrong in this instance about something. Is that right? Is there something more we need to understand?
        ben_w 27d ago
        > Is there something more we need to understand? Only if you're interested in the specific failure modes that LLMs have. That's all this story is.
    - antonvs 26d ago
      Re your asterisk, the justified true belief (JTB) criteria are considered necessary, but no longer considered sufficient as a definition of knowledge. Because of that, JTB is often treated as a useful first approximation. It's easy to see why the JTB criteria are necessary: Belief: if you don't believe it then you can't count it as knowledge. Truth: a belief is not (valid) knowledge if it's false. Justification: accidentally getting the right answer isn't normally knowledge. But the original claim for JTB was that it was a sufficient definition of knowledge. Later critiques like Gettier's showed that this is not generally true, i.e. there are edge cases for which it fails. In many scenarios, those edge cases don't matter much. So you end up with JTB being an imperfect but useful definition.
      - ben_w 22d ago
        My take from the failure of JTB is that "knowledge" is an illusion. Best we can do is have a belief, any belief will unavoidably be justified, but we have no access to any oracle which can tell us if any particular belief was ever "true". When we say we "know" something, we simply have the belief that our certainly is 1, but this is half language and half self-delusion as a certainly of 1 should require infinite Bayesian evidence.
  - deleted 27d ago
    [deleted]
Turskarama 27d ago
Of course it does, if you start filtering people out at random then you have pointlessly introduced the possibility of randomly filtering out the best candidate.
- vintermann 27d ago
  You have no other information to go by in this scenario, so whatever you do you're equally likely to randomly exclude the best candidate.
freetime2 27d ago
I think this is implied by your comment that the issue is framed improperly, but just to point it out explicitly: the reason that this is a problem is because it would lead to segregation, inequality, and injustice in a society where the biased selection mechanism is used. Even if it happens to be “optimal” in this case at assigning employees to positions based purely on the information available and their likelihood to succeed, biases can present other issues.
- HappMacDonald 27d ago
  While that may be true in reality no such externality consequences appear to be baked into the game theory of the experiment. It's like having people or neural networks choose door 1 from door 2 without clear advantage to either of them and without making it clear that one door somehow represents "donating blood" while the other represents "kicking puppies".
- dinfinity 27d ago
  > Even if it happens to be “optimal” in this case at assigning employees to positions based purely on the information available and their likelihood to succeed, biases can present other issues. Yeah, even if the Aima people were 50% better at being a doctor than the Weku (or whatever) we still would not want Aima to be preferred over Weku just for being Aima. This is the core flaw of this study, imho. The whole equal treatment thing isn't supposed to be "everybody should be equally likely to be picked for a job", but rather "everybody's chances to be picked for a job should only rely on direct characteristics that influence their competence for the job". This study effectively forces the decision maker to use group membership as a proxy for competence due to the lack of information on direct characteristics. It is hard to see real world situations where there is no performance penalty for structurally choosing participants less fit for the job by using only group membership as a proxy.

jb1991 27d ago

Isn’t exploration vs exploitation about the decision-making process, not about the actual reality in the world around you? It doesn’t matter if they are secretly identical or not. The exploration/exploitation trade-off is in the person making those decisions.

vintermann 27d ago
I don't understand what you suggest that implies?
- ben_w 27d ago
  I think they're saying that while it doesn't matter, the agent and human "do not actutally know" that it does not matter. Philosophy sometimes says that knowledge is a "justified true belief"*; in this experiment, agents and humans have incorrectly justified a false belief that some applicants are better for certain roles. * other times, it says this isn't good enough
  - beepbooptheory 27d ago
    Seems quite odd to cite all of philosophy as saying something, as if it were a single person with contradictory beliefs.. And then its like you are both saying the justification is incorrect and the belief is false, so its not really like the bare nuance of the concept is adding to the point. Why feel the need to appeal to an (imaginary) authority at all in this case? "Oh well if philosophy said it, I better be taking this seriously!"
    - ben_w 27d ago
      I think you misunderstood my point, just as the other commentor misunderstood one level up. Perhaps a different approach to explain the problem here: "It ain't what they don't know, it's what they know for sure that just ain't so".
      - beepbooptheory 27d ago
        Hm ok, but how are you mapping this, like, epistemological concept to what you are responding to re exploration/exploitation? Has exploration happened or not if it amounts to false beliefs? The whole point tradeoff doesn't seem to make sense if the person in fact can't actually successfully explore! Or even if there the possibility of that. But it is also very likely I am misunderstanding!
        ben_w 27d ago
        A flat distribution is still a distribution, and correct exploration would have revealed that the distribution is flat. The agent appears to have gained the false belief that it has learned something and done some exploring, when in fact it has not. c.f. Sally-Anne test: Sally thinks she knows where her toy is, we know that she doesn't, and indeed couldn't. The LLM (and humans in similar conditions) think they know what the distribution is, we know that they don't.
        beepbooptheory 27d ago
        Really not trying to be reductive here, but it feels like all you are trying to articulate here is that the LLM was wrong in this instance about something. Is that right? Is there something more we need to understand?
        ben_w 27d ago
        > Is there something more we need to understand? Only if you're interested in the specific failure modes that LLMs have. That's all this story is.
  - antonvs 26d ago
    Re your asterisk, the justified true belief (JTB) criteria are considered necessary, but no longer considered sufficient as a definition of knowledge. Because of that, JTB is often treated as a useful first approximation. It's easy to see why the JTB criteria are necessary: Belief: if you don't believe it then you can't count it as knowledge. Truth: a belief is not (valid) knowledge if it's false. Justification: accidentally getting the right answer isn't normally knowledge. But the original claim for JTB was that it was a sufficient definition of knowledge. Later critiques like Gettier's showed that this is not generally true, i.e. there are edge cases for which it fails. In many scenarios, those edge cases don't matter much. So you end up with JTB being an imperfect but useful definition.
    - ben_w 22d ago
      My take from the failure of JTB is that "knowledge" is an illusion. Best we can do is have a belief, any belief will unavoidably be justified, but we have no access to any oracle which can tell us if any particular belief was ever "true". When we say we "know" something, we simply have the belief that our certainly is 1, but this is half language and half self-delusion as a certainly of 1 should require infinite Bayesian evidence.
- deleted 27d ago
  [deleted]

Turskarama 27d ago

Of course it does, if you start filtering people out at random then you have pointlessly introduced the possibility of randomly filtering out the best candidate.

vintermann 27d ago
You have no other information to go by in this scenario, so whatever you do you're equally likely to randomly exclude the best candidate.

freetime2 27d ago

I think this is implied by your comment that the issue is framed improperly, but just to point it out explicitly: the reason that this is a problem is because it would lead to segregation, inequality, and injustice in a society where the biased selection mechanism is used. Even if it happens to be “optimal” in this case at assigning employees to positions based purely on the information available and their likelihood to succeed, biases can present other issues.

HappMacDonald 27d ago
While that may be true in reality no such externality consequences appear to be baked into the game theory of the experiment. It's like having people or neural networks choose door 1 from door 2 without clear advantage to either of them and without making it clear that one door somehow represents "donating blood" while the other represents "kicking puppies".
dinfinity 27d ago
> Even if it happens to be “optimal” in this case at assigning employees to positions based purely on the information available and their likelihood to succeed, biases can present other issues. Yeah, even if the Aima people were 50% better at being a doctor than the Weku (or whatever) we still would not want Aima to be preferred over Weku just for being Aima. This is the core flaw of this study, imho. The whole equal treatment thing isn't supposed to be "everybody should be equally likely to be picked for a job", but rather "everybody's chances to be picked for a job should only rely on direct characteristics that influence their competence for the job". This study effectively forces the decision maker to use group membership as a proxy for competence due to the lack of information on direct characteristics. It is hard to see real world situations where there is no performance penalty for structurally choosing participants less fit for the job by using only group membership as a proxy.