3 ms·
This is what they are even admitting to: Under "3.4 Limitations of our methodology" - "3.4.1 Subjective human judgments" > A fundamental limitation of our app
by geraneum 4y ago
This is what they are even admitting to:
Under "3.4 Limitations of our methodology" - "3.4.1 Subjective human judgments"
> A fundamental limitation of our approach lies in the subjectivity of the labeling. In our study, we employ annotators who are familiar with the GPT models’ capabilities. However, this group is not occupationally diverse, potentially leading to biased judgments regarding GPTs’ reliability and effectiveness in performing tasks within unfamiliar occupations. We acknowledge that obtaining high-quality labels for each task in an occupation requires workers engaged in those occupations or, at a minimum, possessing in-depth knowledge of the diverse tasks within those occupations. This represents an important area for future work in validating these results.
- light_hue_1 4y agoFor sure. I had to read the paper to discover it's trash. But if you read the abstract, it looks like they thoroughly assessed how GPT will impact many professions. The sentence "Using a new rubric, we assess occupations based on their correspondence with GPT capabilities, incorporating both human expertise and classifications from GPT-4." does not scream to me "We asked 5 random people with no expertise in either these professions or GPT-4 what they thought and report those results". This is borderline dishonest.
- geraneum 4y agoI agree with you. This is not scientifically sound research. Reads more like a brochure to be honest.