4 ms·

it raises several questions, my first question (besides the obvious question of why is this a thing at all) is does the chatbot actually have the information ab

it raises several questions, my first question (besides the obvious question of why is this a thing at all) is does the chatbot actually have the information about the racial background of its dev team or is it hallucinating up an answer when it lacks the data. if it does have that sort of knowledge why personally identifiable information about individual devs accessable to a public facing chatbot and how much of their HR file can be exfitraited via chatbot to anyone that asks it questions, if it doesn't have this information and it just assumes its devs are all white what does that say about the information its trained on

xncuvyv6 2y ago
[dead]
ncr100 2y ago
Hallucination. An issue we care about and it just answers regardless of accuracy. Boo.
- brookst 2y ago
  So you fact checked this comment before posting, right? Because I am deeply interested if this was hallucination or used RAG and web search to provide the answer, but I can’t tell if you just chose to hallucinate without grounding.
  - throwaway314155 2y ago
    > hallucination or used RAG and web search to provide the answer Occam's razor - it was an hallucination.
  - Rastonbury 2y ago
    Come on why would anyone put the team's races into RAG or anywhere under the hood lol
    - brookst 2y ago
      Does it seem outlandish that there might have been a press article about the team?
      - Rastonbury 2y ago
        When's the last time you saw an article with entire team responsible with gender and race for a non-flagship product at a company the size of Meta? Even if the article existed, getting the right article (instead of simply using the first Bing result, which is what the other Meta AI does) and the reasoning in a one shot instant response is better COT models like o1 If you have Messenger, try asking it to list the team behind them or their Tom Brady bot, it will not get far
noirbot 2y ago
It's Meta, so this is almost definitely some Llama variant under the hood, especially since these bots have been out for a year, evidently. It seems these were just insanely poorly trained and gated - getting a bot to comment on its dev team, including in some cases specific names of lead designers, is something that feels crazy to let a bot just do. Even if all the data it would respond with is made up slop, this exact debacle is why you do your best to make a bot not even try to answer this sort of stuff. As for what information it was trained on that it said its dev team had no black devs, that seems fairly reasonable based on general statistical information and scholarly articles it was trained on. It may still be incorrect, but that's a pretty statistically likely outcome, so it's not surprising an LLM built to parrot the statistically likely outcome would say it.

xncuvyv6 2y ago

[dead]

ncr100 2y ago

Hallucination. An issue we care about and it just answers regardless of accuracy. Boo.

brookst 2y ago
So you fact checked this comment before posting, right? Because I am deeply interested if this was hallucination or used RAG and web search to provide the answer, but I can’t tell if you just chose to hallucinate without grounding.
- throwaway314155 2y ago
  > hallucination or used RAG and web search to provide the answer Occam's razor - it was an hallucination.
- Rastonbury 2y ago
  Come on why would anyone put the team's races into RAG or anywhere under the hood lol
  - brookst 2y ago
    Does it seem outlandish that there might have been a press article about the team?
    - Rastonbury 2y ago
      When's the last time you saw an article with entire team responsible with gender and race for a non-flagship product at a company the size of Meta? Even if the article existed, getting the right article (instead of simply using the first Bing result, which is what the other Meta AI does) and the reasoning in a one shot instant response is better COT models like o1 If you have Messenger, try asking it to list the team behind them or their Tom Brady bot, it will not get far

noirbot 2y ago

It's Meta, so this is almost definitely some Llama variant under the hood, especially since these bots have been out for a year, evidently. It seems these were just insanely poorly trained and gated - getting a bot to comment on its dev team, including in some cases specific names of lead designers, is something that feels crazy to let a bot just do. Even if all the data it would respond with is made up slop, this exact debacle is why you do your best to make a bot not even try to answer this sort of stuff. As for what information it was trained on that it said its dev team had no black devs, that seems fairly reasonable based on general statistical information and scholarly articles it was trained on. It may still be incorrect, but that's a pretty statistically likely outcome, so it's not surprising an LLM built to parrot the statistically likely outcome would say it.