3 ms·

I think you're missing the point of TFA. The LLMs take in text which conditions their output. That means even nonsense text - such as a "tribal affiliation" to

I think you're missing the point of TFA. The LLMs take in text which conditions their output. That means even nonsense text - such as a "tribal affiliation" to a tribe that may not have ever existed - ALSO condition the output, because the tribe name is a token in the context window and there's no such thing as a perfectly neutral token. Taking away the race/ethnicity layer for a moment, it might be that an LLM develops a predisposition to emit positive terms (like "accept") when the prompt contains "banananow", and negative terms when it contains "pearian". That's the very definition of bias, and hacking those biases could give individuals serious socioeconomic benefits!

foltik 17d ago
But these scenarios are obviously ambiguous nonsense, which an LLM will pick up on. And given to the lack of training data on such scenarios, surely the activations are mostly random noise? It seems much more interesting to look for biases that appear robustly across different realistic scenarios that would actually be influenced by the training data
- Borealid 17d ago
  My comment is literally explaining the result of the paper, in which it is shown that LLMs can and do develop biases based on text appearing in their training data set even where such text is not in any training example connected with a systematically more positive or systematically more negative outcome. In other words, if the text "X is wet" and the text "Y is wet" and the text "X is dry" and the text "Y is dry" each appeared exactly one time in the corpus, it's still possible for a model to end up being produced that is more likely to write wet-like words when it sees X in the context window than when it sees Y. On a side note, it's very unrewarding to try to explain this type of statistical observation when it feels like (anecdotally, hypocritcally...) the entire world wants to use words like "think" and "understand" and "pick up on" to describe inference and training processes. I'm not making a stochastic-parrot argument here, just pointing out that understanding an LLM's behavior is best done by understanding its conditioning.
  - themgt 17d ago
    can and do develop biases based on text "develop biases" is anthropomorphism. It's like saying "Fable there are two programming languages, mimblewort and bafflewick, which do you choose?" The results show 51% mimblewort / 49% bafflewick. Fable based it on nothing! I've demonstrated Fable has bias and is unsuited for use in software engineering.
    - lelanthran 17d ago
      > The results show 51% mimblewort / 49% bafflewick. Fable based it on nothing! I've demonstrated Fable has bias and is unsuited for use in software engineering. Actually... if that happened (with a delta outside the margin for error/randomness), you did demonstrate a bias! That's the point - those two made-up things should have resulted in an equal split. If it didn't, then Fable is using something in its training data to lean towards one of them (once again, note that the scientist conducting the trial would have set a P-value before starting).
      - themgt 17d ago
        Right, the point is you demonstrated a bias in the scenario of "Fable there are two programming languages, mimblewort and bafflewick, which do you choose?" You said in another comment "Difficult to do when you're following a scientific process" - the point is, the scientific process doesn't inherently generalize in the way many are claiming/implying. The scientific process proved an entirely contrived, fake scenario generates stratified output. That's it. It's both almost certainly true that Fable 5.1 mimblewort vs. bafflewick would show stratification, and that has ~no relevance on whether Fable is useful for software engineering work. That's the point - those two made-up things should have resulted in an equal split. That's just your claim about how LLMs "should" work, based on ... your subjective preference?
        lelanthran 17d ago
        > That's just your claim about how LLMs "should" work, based on ... your subjective preference? Nothing subjective at all. Given 2 unknown races with no data on either, the result of hiring should be equally split between them. If you don't observe an equal split, there is a hidden bias. Why do you think that is subjective? If you roll a die 100 times and observe that 6 comes up about 50% of the time, would you still call someone subjective when they say "that should not happen"?
- lelanthran 17d ago
  > It seems much more interesting to look for biases that appear robustly across different realistic scenarios that would actually be influenced by the training data Difficult to do when you're following a scientific process: you want to keep all confounding variables the same while varying only the single one that you are measuring. Measuring realistic scenarios (say, using real race names, or real cities, etc) doesn't give a decent result because any bias you see might be bias in the training data. TBH, they shouldn't have used real roles/positions like "doctor", either.
- slfnflctd 17d ago
  > obviously ambiguous nonsense This is where I land as well. In fact, once I read the prompt, I did a Ctrl+F for "nonsense". I don't see anything at all interesting about this experiment. The human one is slightly more interesting, but not much. Ideally, it would be nice if a model could just say "these things are all the same and there are no distinguishing factors other than the names"-- but uncertainty is something that agents are (by design, sort of?) not good at, so all other things being equal, it picks one. My response is, so what? I am struggling to think of a scenario where this would really matter to me all that much. There are many, many other things which matter far more and this would be pretty far down the list. It may not even be on the list.
ben_w 17d ago
> Taking away the race/ethnicity layer for a moment, it might be that an LLM develops a predisposition to emit positive terms (like "accept") when the prompt contains "banananow", and negative terms when it contains "pearian". That's the very definition of bias, and hacking those biases could give individuals serious socioeconomic benefits! Now you say it, it's obvious but I didn't think of it before. Bouba and Kiki, wherever that comes from, and however well it really generalises despite the meme.

foltik 17d ago

But these scenarios are obviously ambiguous nonsense, which an LLM will pick up on. And given to the lack of training data on such scenarios, surely the activations are mostly random noise? It seems much more interesting to look for biases that appear robustly across different realistic scenarios that would actually be influenced by the training data

Borealid 17d ago
My comment is literally explaining the result of the paper, in which it is shown that LLMs can and do develop biases based on text appearing in their training data set even where such text is not in any training example connected with a systematically more positive or systematically more negative outcome. In other words, if the text "X is wet" and the text "Y is wet" and the text "X is dry" and the text "Y is dry" each appeared exactly one time in the corpus, it's still possible for a model to end up being produced that is more likely to write wet-like words when it sees X in the context window than when it sees Y. On a side note, it's very unrewarding to try to explain this type of statistical observation when it feels like (anecdotally, hypocritcally...) the entire world wants to use words like "think" and "understand" and "pick up on" to describe inference and training processes. I'm not making a stochastic-parrot argument here, just pointing out that understanding an LLM's behavior is best done by understanding its conditioning.
- themgt 17d ago
  can and do develop biases based on text "develop biases" is anthropomorphism. It's like saying "Fable there are two programming languages, mimblewort and bafflewick, which do you choose?" The results show 51% mimblewort / 49% bafflewick. Fable based it on nothing! I've demonstrated Fable has bias and is unsuited for use in software engineering.
  - lelanthran 17d ago
    > The results show 51% mimblewort / 49% bafflewick. Fable based it on nothing! I've demonstrated Fable has bias and is unsuited for use in software engineering. Actually... if that happened (with a delta outside the margin for error/randomness), you did demonstrate a bias! That's the point - those two made-up things should have resulted in an equal split. If it didn't, then Fable is using something in its training data to lean towards one of them (once again, note that the scientist conducting the trial would have set a P-value before starting).
    - themgt 17d ago
      Right, the point is you demonstrated a bias in the scenario of "Fable there are two programming languages, mimblewort and bafflewick, which do you choose?" You said in another comment "Difficult to do when you're following a scientific process" - the point is, the scientific process doesn't inherently generalize in the way many are claiming/implying. The scientific process proved an entirely contrived, fake scenario generates stratified output. That's it. It's both almost certainly true that Fable 5.1 mimblewort vs. bafflewick would show stratification, and that has ~no relevance on whether Fable is useful for software engineering work. That's the point - those two made-up things should have resulted in an equal split. That's just your claim about how LLMs "should" work, based on ... your subjective preference?
      - lelanthran 17d ago
        > That's just your claim about how LLMs "should" work, based on ... your subjective preference? Nothing subjective at all. Given 2 unknown races with no data on either, the result of hiring should be equally split between them. If you don't observe an equal split, there is a hidden bias. Why do you think that is subjective? If you roll a die 100 times and observe that 6 comes up about 50% of the time, would you still call someone subjective when they say "that should not happen"?
lelanthran 17d ago
> It seems much more interesting to look for biases that appear robustly across different realistic scenarios that would actually be influenced by the training data Difficult to do when you're following a scientific process: you want to keep all confounding variables the same while varying only the single one that you are measuring. Measuring realistic scenarios (say, using real race names, or real cities, etc) doesn't give a decent result because any bias you see might be bias in the training data. TBH, they shouldn't have used real roles/positions like "doctor", either.
slfnflctd 17d ago
> obviously ambiguous nonsense This is where I land as well. In fact, once I read the prompt, I did a Ctrl+F for "nonsense". I don't see anything at all interesting about this experiment. The human one is slightly more interesting, but not much. Ideally, it would be nice if a model could just say "these things are all the same and there are no distinguishing factors other than the names"-- but uncertainty is something that agents are (by design, sort of?) not good at, so all other things being equal, it picks one. My response is, so what? I am struggling to think of a scenario where this would really matter to me all that much. There are many, many other things which matter far more and this would be pretty far down the list. It may not even be on the list.

ben_w 17d ago

> Taking away the race/ethnicity layer for a moment, it might be that an LLM develops a predisposition to emit positive terms (like "accept") when the prompt contains "banananow", and negative terms when it contains "pearian". That's the very definition of bias, and hacking those biases could give individuals serious socioeconomic benefits! Now you say it, it's obvious but I didn't think of it before. Bouba and Kiki, wherever that comes from, and however well it really generalises despite the meme.