3 ms·

This is funny because it’s a silly topic, but I think it shows something extremely seriously wrong with llms. The goblins stand out because it’s obvious. Think

This is funny because it’s a silly topic, but I think it shows something extremely seriously wrong with llms. The goblins stand out because it’s obvious. Think of all the other crazy biases latent in every interaction that we don’t notice because it’s not as obvious. Absolutely terrifying that OpenAI is just tossing around that such subtle training biases were hard enough to contain it had to be added to system prompt.

ninjagoo 5mo ago
> Absolutely terrifying that OpenAI is just tossing around that such subtle training biases were hard enough to contain it had to be added to system prompt. May I introduce you to homo sapiens, a species so vulnerable to such subtle (or otherwise) biases (and affiliations) that they had to develop elaborate and documented justice systems to contain the fallouts? :)
- chongli 5mo ago
  We’re really not that vulnerable to such things as a species, because we as individuals all have our own minds and our own sets of biases that cancel out and get lost in the noise. If we all had the exact same bias then it would be a huge problem.
  - ninjagoo 5mo ago
    > If we all had the exact same bias then it would be a huge problem. And may I introduce you to "groupthink" :))
    - Dylan16807 5mo ago
      Now imagine that every opinion you have is automatically fully groupthinked and you see the difference/problem with training up a big AI model that has a hundred million users. The problem does exist when using individual humans but in a much smaller form.
      - ninjagoo 5mo ago
        > The problem does exist when using individual humans but in a much smaller form. And may I introduce you to organized religion :)
        Dylan16807 5mo ago
        That's still a lot smaller! Make a major religion where everyone is a scifi clone of one person including their memories and then it'll be in the same ballpark of spreading bias.
  - jychang 5mo ago
    > We’re really not that vulnerable to such things as a species, because we as individuals all have our own minds and our own sets of biases that cancel out and get lost in the noise. [Citation Needed] Just because if you have a species-wide bias, people within the species would not easily recognize it. You can't claim with a straight face that "we're really not that vulnerable to such things". For example, I think it's pretty clear that all humans are vulnerable to phone addiction, especially kids.
    - chongli 5mo ago
      > people within the species would not easily recognize it [Citation Needed] Sorry, but I had to. There's easy counterexamples of true, species-wide biases that we're fully aware of. Optical illusions, cognitive biases, cultural universals (community-sanctioned relationships/marriage, inheritance, ceremonial treatment of the dead). What we don't have are universal biases towards believing specific facts or stories.
      - jychang 5mo ago
        None of those things are easily recognized though. They're not universals. A term like "cognitive biases" generally require a college level education. If you go to a tribe in the middle of the rainforest, would they be able to explain those concepts? Of course not. Plus, I already gave an example of a species wide bias at the end of the comment- phone addiction for kids. I'm clearly not saying it's impossible for a human to spot a bias, but rather... how many 5 year old kids recognize that phone addiction is a bad thing?
        chongli 5mo ago
        You’ve moved the goalposts. You’ve gone from “people within the species not being able to easily recognize a bias” to “people universally recognizing that bias, even with no education or contact with the rest of civilization.” That’s silly, and something I’d never argue for. To me, something is easy for humans to recognize if a 19th century scientist could discover it. We are a social and cultural species. Culture is how we learn anything over the long run.
  - arglebarnacle 5mo ago
    I hear you but of course history is full of examples of biases shared across large groups of people resulting in huge human costs. The analogy isn’t perfect of course but the way humans learn about their world is full of opportunities to introduce and sustain these large correlated biases—social pressure, tradition, parenting, education standardization. And not all of them are bad of course, but some are and many others are at least as weird as stray references to goblins and creatures
  - Ekaros 5mo ago
    Doesn't that depend on the biases in question? Many argue that homogenous societies do many things better. And part of homogeneity is sharing same set of biases.
  - lifis 5mo ago
    And what do you think society/culture is? It's a set of biases installed in people, whose purpose is mostly to replicate themselves. Humans are MORE susceptible that LLMs, because LLMs's biases are easily steered to something else, unlike most humans.
- hbs18 5mo ago
  An LLM is a computer program, which isn't a human. You wouldn't excuse a calculator being occasionally wrong because humans sometimes get manual calculations wrong too.
  - ninjagoo 5mo ago
    > An LLM is a computer program, which isn't a human. You wouldn't excuse a calculator being occasionally wrong because humans sometimes get manual calculations wrong too. Ah, now we're getting technical. An LLM is a non-deterministic/probabilistic computer program, not a calculator. Keeping that in mind is critical when using an LLM. Expecting deterministic behavior from an LLM is an example of what's known as a 'category error'. [1] [1] https://en.wikipedia.org/wiki/Category_mistake https://en.wikipedia.org/wiki/Category_mistake
ordinarily 5mo ago
Doesn't seem that surprising or terrifying to me. Humans come equipped with a lot more internal biases (learned in a fairly similar fashion), and they're usually a lot more resistant to getting rid of them. The truly terrifying stuff never makes it out of the RLHF NDAs.
- agnishom 5mo ago
  Humans also take a lot of time in producing output, and do not feed into a crazy accelerationistic feedback loop (most of the time).
  - ordinarily 5mo ago
    I mean, looking at the state of things right now in the US, I'd have to strongly disagree with you.
- Terr_ 5mo ago
  We ought to be terrified, when one adjusts for ll the use-cases people are talking about using these algorithms in. (Even if they ultimately back off, it's a lot of frothy bubble opportunity cost.) There a great many things people do which are not acceptable in our machines. Ex: I would not be comfortable flying on any airplane where the autopilot "just zones-out sometimes", even though it's a dysfunction also seen in people.
  - famouswaffles 5mo ago
    >Ex: I would not be comfortable flying on any airplane where the autopilot "just zones-out sometimes", even though it's a dysfunction also seen in people. You might if that was the best auto-pilot could be. Have you never used a bus or taken a taxi ? The vast majority of things people are using LLMs for isn't stuff deterministic logic machines did great at, but stuff those same machines did poorly at or straight up stuff previously relegated to the domains of humans only. If your competition also "just zones out sometimes" then it's not something you're going to focus on.
tptacek 5mo ago
I think it's extraordinarily telling that people are capable of being reflexively pessimistic in response to the goblin plague. It's like something Zitron would do. This story is wonderful.
- bitexploder 5mo ago
  I feel at least partially responsible. I would often instruct agents to "stop being a goblin". I really enjoyed this story too, though.
bitexploder 5mo ago
We do not have the complete picture.
snakebiteagain 5mo ago
Mandatory reading on that topic: www.anthropic.com/research/small-samples-poison We're probably not noticing a LOT of malicious attempts at poisoning major AI's only because we don't know what keywords to ask (but the scammers do and will abuse it).

ninjagoo 5mo ago

> Absolutely terrifying that OpenAI is just tossing around that such subtle training biases were hard enough to contain it had to be added to system prompt. May I introduce you to homo sapiens, a species so vulnerable to such subtle (or otherwise) biases (and affiliations) that they had to develop elaborate and documented justice systems to contain the fallouts? :)

chongli 5mo ago
We’re really not that vulnerable to such things as a species, because we as individuals all have our own minds and our own sets of biases that cancel out and get lost in the noise. If we all had the exact same bias then it would be a huge problem.
- ninjagoo 5mo ago
  > If we all had the exact same bias then it would be a huge problem. And may I introduce you to "groupthink" :))
  - Dylan16807 5mo ago
    Now imagine that every opinion you have is automatically fully groupthinked and you see the difference/problem with training up a big AI model that has a hundred million users. The problem does exist when using individual humans but in a much smaller form.
    - ninjagoo 5mo ago
      > The problem does exist when using individual humans but in a much smaller form. And may I introduce you to organized religion :)
      - Dylan16807 5mo ago
        That's still a lot smaller! Make a major religion where everyone is a scifi clone of one person including their memories and then it'll be in the same ballpark of spreading bias.
- jychang 5mo ago
  > We’re really not that vulnerable to such things as a species, because we as individuals all have our own minds and our own sets of biases that cancel out and get lost in the noise. [Citation Needed] Just because if you have a species-wide bias, people within the species would not easily recognize it. You can't claim with a straight face that "we're really not that vulnerable to such things". For example, I think it's pretty clear that all humans are vulnerable to phone addiction, especially kids.
  - chongli 5mo ago
    > people within the species would not easily recognize it [Citation Needed] Sorry, but I had to. There's easy counterexamples of true, species-wide biases that we're fully aware of. Optical illusions, cognitive biases, cultural universals (community-sanctioned relationships/marriage, inheritance, ceremonial treatment of the dead). What we don't have are universal biases towards believing specific facts or stories.
    - jychang 5mo ago
      None of those things are easily recognized though. They're not universals. A term like "cognitive biases" generally require a college level education. If you go to a tribe in the middle of the rainforest, would they be able to explain those concepts? Of course not. Plus, I already gave an example of a species wide bias at the end of the comment- phone addiction for kids. I'm clearly not saying it's impossible for a human to spot a bias, but rather... how many 5 year old kids recognize that phone addiction is a bad thing?
      - chongli 5mo ago
        You’ve moved the goalposts. You’ve gone from “people within the species not being able to easily recognize a bias” to “people universally recognizing that bias, even with no education or contact with the rest of civilization.” That’s silly, and something I’d never argue for. To me, something is easy for humans to recognize if a 19th century scientist could discover it. We are a social and cultural species. Culture is how we learn anything over the long run.
- arglebarnacle 5mo ago
  I hear you but of course history is full of examples of biases shared across large groups of people resulting in huge human costs. The analogy isn’t perfect of course but the way humans learn about their world is full of opportunities to introduce and sustain these large correlated biases—social pressure, tradition, parenting, education standardization. And not all of them are bad of course, but some are and many others are at least as weird as stray references to goblins and creatures
- Ekaros 5mo ago
  Doesn't that depend on the biases in question? Many argue that homogenous societies do many things better. And part of homogeneity is sharing same set of biases.
- lifis 5mo ago
  And what do you think society/culture is? It's a set of biases installed in people, whose purpose is mostly to replicate themselves. Humans are MORE susceptible that LLMs, because LLMs's biases are easily steered to something else, unlike most humans.
hbs18 5mo ago
An LLM is a computer program, which isn't a human. You wouldn't excuse a calculator being occasionally wrong because humans sometimes get manual calculations wrong too.
- ninjagoo 5mo ago
  > An LLM is a computer program, which isn't a human. You wouldn't excuse a calculator being occasionally wrong because humans sometimes get manual calculations wrong too. Ah, now we're getting technical. An LLM is a non-deterministic/probabilistic computer program, not a calculator. Keeping that in mind is critical when using an LLM. Expecting deterministic behavior from an LLM is an example of what's known as a 'category error'. [1] [1] https://en.wikipedia.org/wiki/Category_mistake https://en.wikipedia.org/wiki/Category_mistake

ordinarily 5mo ago

Doesn't seem that surprising or terrifying to me. Humans come equipped with a lot more internal biases (learned in a fairly similar fashion), and they're usually a lot more resistant to getting rid of them. The truly terrifying stuff never makes it out of the RLHF NDAs.

agnishom 5mo ago
Humans also take a lot of time in producing output, and do not feed into a crazy accelerationistic feedback loop (most of the time).
- ordinarily 5mo ago
  I mean, looking at the state of things right now in the US, I'd have to strongly disagree with you.
Terr_ 5mo ago
We ought to be terrified, when one adjusts for ll the use-cases people are talking about using these algorithms in. (Even if they ultimately back off, it's a lot of frothy bubble opportunity cost.) There a great many things people do which are not acceptable in our machines. Ex: I would not be comfortable flying on any airplane where the autopilot "just zones-out sometimes", even though it's a dysfunction also seen in people.
- famouswaffles 5mo ago
  >Ex: I would not be comfortable flying on any airplane where the autopilot "just zones-out sometimes", even though it's a dysfunction also seen in people. You might if that was the best auto-pilot could be. Have you never used a bus or taken a taxi ? The vast majority of things people are using LLMs for isn't stuff deterministic logic machines did great at, but stuff those same machines did poorly at or straight up stuff previously relegated to the domains of humans only. If your competition also "just zones out sometimes" then it's not something you're going to focus on.

tptacek 5mo ago

I think it's extraordinarily telling that people are capable of being reflexively pessimistic in response to the goblin plague. It's like something Zitron would do. This story is wonderful.

bitexploder 5mo ago
I feel at least partially responsible. I would often instruct agents to "stop being a goblin". I really enjoyed this story too, though.

bitexploder 5mo ago

We do not have the complete picture.

snakebiteagain 5mo ago

Mandatory reading on that topic: www.anthropic.com/research/small-samples-poison We're probably not noticing a LOT of malicious attempts at poisoning major AI's only because we don't know what keywords to ask (but the scammers do and will abuse it).