3 ms·

Unfortunately, I think the current strategies for RLHF are a huge contributor to hallucination / confabulation. In short, they're paying contract workers for q

Unfortunately, I think the current strategies for RLHF are a huge contributor to hallucination / confabulation. In short, they're paying contract workers for quantity, not quality; they don't have time to do independent research or follow up on citations. Unsurprisingly, the LLM optimizes for superficially convincing bullshit.

quickthrower2 3y ago
> Among those who labeled demonstration data for InstructGPT, ~90% have at least a college degree and more than one-third have a master’s degree. Source: https://huyenchip.com/2023/05/02/rlhf.html#demonstration_data https://huyenchip.com/2023/05/02/rlhf.html#demonstration_dat...
- yeck 3y ago
  Controlling for academic experience probably raises the average accuracy of labelling, but by how much? Clearly having a degree will not make you omniscient in your major, let alone other subjects.
- mdekkers 3y ago
  Are they getting paid on quantity or quality?
  - robbrown451 3y ago
    Most jobs I've known factor in both. I would assume they have processes in place that incentivize quality. Some it is as simple as you have a manager that will fire you if you produce crap. With billions in funding, and bad results causing bad press etc, you think that OpenAI would not have given this a bit of consideration?
    - mdekkers 3y ago
      > I would assume they have processes in place that incentivize quality. > you think that OpenAI would not have given this a bit of consideration? Those are just assumptions though. The issue is not “this was labelled as a shoe, but it’s a car”, the issue is about depth vs superficially, which is harder to verify. See also https://www.theverge.com/features/23764584/ai-artificial-intelligence-data-notation-labor-scale-surge-remotasks-openai-chatbots https://www.theverge.com/features/23764584/ai-artificial-int... for a well-sourced article on the subject.
robbrown451 3y ago
"In short, they're paying contract workers for quantity, not quality;" How do you know this? Just taking a wild guess, but I'd think a company with billions of funding, and a ton of people trying to find flaws in what they are producing, would have some processes in place to incentivize quality as well as quality. What you are suggesting is that a company that produces a product that is based on balancing trillions of floating point numbers makes core business decisions in the most simplistic black and white terms. "Hey, lets just go with a one and a zero on this." Bizarre assumption. Maybe I'm just good at prompting, and I'm not trying to trick it, but I don't see this "superficially convincing bullshit." Can you show me a chat where you have sincerely prompted it and gotten something that matches that description? I often see responses that are better than I could have given even if given hours to research and compose them, and I'm a pretty good writer and researcher. Here, since I'm asking you to share one where it fails as you say it does by creating "superficially convincing bullshit", I'll share several where it succeeds. https://chat.openai.com/share/523d0fec-34d3-40c4-b5a1-81c77f1529cf https://chat.openai.com/share/523d0fec-34d3-40c4-b5a1-81c77f... https://chat.openai.com/share/e09c4491-fd66-4519-92d6-d34645812c09 https://chat.openai.com/share/e09c4491-fd66-4519-92d6-d34645... https://chat.openai.com/share/233a5ae2-c726-4ddc-8452-20248e2db83f https://chat.openai.com/share/233a5ae2-c726-4ddc-8452-20248e... https://chat.openai.com/share/53e6bda1-fe97-41ce-8f5c-89d639e00dde https://chat.openai.com/share/53e6bda1-fe97-41ce-8f5c-89d639... https://chat.openai.com/share/19f80ea9-e6be-4ac3-9dd4-7ea15c9c3353 https://chat.openai.com/share/19f80ea9-e6be-4ac3-9dd4-7ea15c...
- ooterness 3y ago
  Never forget that RLHF is driven largely by sweatshop labor: https://www.washingtonpost.com/world/2023/08/28/scale-ai-remotasks-philippines-artificial-intelligence/ https://www.washingtonpost.com/world/2023/08/28/scale-ai-rem... These jobs are overwhelmingly paid by task, which puts a lot of pressure to go fast. I assert the entire "hallucination" phenomenon is a side effect of these practices. When ChatGPT makes up a fake fact with fake sources to back it up, it's largely because such lies are rated very highly by the underpaid humans who aren't incentivized to follow up on sources.
  - 3y ago

quickthrower2 3y ago

> Among those who labeled demonstration data for InstructGPT, ~90% have at least a college degree and more than one-third have a master’s degree. Source: https://huyenchip.com/2023/05/02/rlhf.html#demonstration_data https://huyenchip.com/2023/05/02/rlhf.html#demonstration_dat...

yeck 3y ago
Controlling for academic experience probably raises the average accuracy of labelling, but by how much? Clearly having a degree will not make you omniscient in your major, let alone other subjects.
mdekkers 3y ago
Are they getting paid on quantity or quality?
- robbrown451 3y ago
  Most jobs I've known factor in both. I would assume they have processes in place that incentivize quality. Some it is as simple as you have a manager that will fire you if you produce crap. With billions in funding, and bad results causing bad press etc, you think that OpenAI would not have given this a bit of consideration?
  - mdekkers 3y ago
    > I would assume they have processes in place that incentivize quality. > you think that OpenAI would not have given this a bit of consideration? Those are just assumptions though. The issue is not “this was labelled as a shoe, but it’s a car”, the issue is about depth vs superficially, which is harder to verify. See also https://www.theverge.com/features/23764584/ai-artificial-intelligence-data-notation-labor-scale-surge-remotasks-openai-chatbots https://www.theverge.com/features/23764584/ai-artificial-int... for a well-sourced article on the subject.

robbrown451 3y ago

"In short, they're paying contract workers for quantity, not quality;" How do you know this? Just taking a wild guess, but I'd think a company with billions of funding, and a ton of people trying to find flaws in what they are producing, would have some processes in place to incentivize quality as well as quality. What you are suggesting is that a company that produces a product that is based on balancing trillions of floating point numbers makes core business decisions in the most simplistic black and white terms. "Hey, lets just go with a one and a zero on this." Bizarre assumption. Maybe I'm just good at prompting, and I'm not trying to trick it, but I don't see this "superficially convincing bullshit." Can you show me a chat where you have sincerely prompted it and gotten something that matches that description? I often see responses that are better than I could have given even if given hours to research and compose them, and I'm a pretty good writer and researcher. Here, since I'm asking you to share one where it fails as you say it does by creating "superficially convincing bullshit", I'll share several where it succeeds. https://chat.openai.com/share/523d0fec-34d3-40c4-b5a1-81c77f1529cf https://chat.openai.com/share/523d0fec-34d3-40c4-b5a1-81c77f... https://chat.openai.com/share/e09c4491-fd66-4519-92d6-d34645812c09 https://chat.openai.com/share/e09c4491-fd66-4519-92d6-d34645... https://chat.openai.com/share/233a5ae2-c726-4ddc-8452-20248e2db83f https://chat.openai.com/share/233a5ae2-c726-4ddc-8452-20248e... https://chat.openai.com/share/53e6bda1-fe97-41ce-8f5c-89d639e00dde https://chat.openai.com/share/53e6bda1-fe97-41ce-8f5c-89d639... https://chat.openai.com/share/19f80ea9-e6be-4ac3-9dd4-7ea15c9c3353 https://chat.openai.com/share/19f80ea9-e6be-4ac3-9dd4-7ea15c...

ooterness 3y ago
Never forget that RLHF is driven largely by sweatshop labor: https://www.washingtonpost.com/world/2023/08/28/scale-ai-remotasks-philippines-artificial-intelligence/ https://www.washingtonpost.com/world/2023/08/28/scale-ai-rem... These jobs are overwhelmingly paid by task, which puts a lot of pressure to go fast. I assert the entire "hallucination" phenomenon is a side effect of these practices. When ChatGPT makes up a fake fact with fake sources to back it up, it's largely because such lies are rated very highly by the underpaid humans who aren't incentivized to follow up on sources.
- 3y ago