4 ms·

> Among those who labeled demonstration data for InstructGPT, ~90% have at least a college degree and more than one-third have a master’s degree. Source: https

> Among those who labeled demonstration data for InstructGPT, ~90% have at least a college degree and more than one-third have a master’s degree. Source: https://huyenchip.com/2023/05/02/rlhf.html#demonstration_data https://huyenchip.com/2023/05/02/rlhf.html#demonstration_dat...

yeck 3y ago
Controlling for academic experience probably raises the average accuracy of labelling, but by how much? Clearly having a degree will not make you omniscient in your major, let alone other subjects.
mdekkers 3y ago
Are they getting paid on quantity or quality?
- robbrown451 3y ago
  Most jobs I've known factor in both. I would assume they have processes in place that incentivize quality. Some it is as simple as you have a manager that will fire you if you produce crap. With billions in funding, and bad results causing bad press etc, you think that OpenAI would not have given this a bit of consideration?
  - mdekkers 3y ago
    > I would assume they have processes in place that incentivize quality. > you think that OpenAI would not have given this a bit of consideration? Those are just assumptions though. The issue is not “this was labelled as a shoe, but it’s a car”, the issue is about depth vs superficially, which is harder to verify. See also https://www.theverge.com/features/23764584/ai-artificial-intelligence-data-notation-labor-scale-surge-remotasks-openai-chatbots https://www.theverge.com/features/23764584/ai-artificial-int... for a well-sourced article on the subject.