4 ms·
Nice benchmark but the human comparison is a little lacking. They claim to have surveyed 19 experts, though the vast majority of them have only a master's degre
by pu_pe 1y ago
Nice benchmark but the human comparison is a little lacking. They claim to have surveyed 19 experts, though the vast majority of them have only a master's degree. This would be akin to comparing LLM programming expertise to a sample of programmers with less than 5 years of experience.
I'm also not sure it's a fair comparison to average human results like that. If you quiz physicians on a broad variety of topics, you shouldn't expect cardiologists to know that much about neurology and vice-versa. This is what they did here, it seems.
- KSteffensen 1y agoI'll get some downvotes for this but PhD vs master's degree difference is mostly work experience, an element of workload hazing and snobbery. Somebody with a masters degree and 5 years of work experience will likely know more than a freshly graduated PhD
- eesmith 1y agoSure, but all we know is that these "13 have a master’s degree (and are currently enroled in Ph.D. studies)". We only know they have at least "2 years of experience in chemistry after their first university-level course in chemistry." How does that qualify them as "domain experts"? What domain is their expertise? All of chemistry?
- 698969 1y agoI think the breadth vs depth thing applies here as well, the PhD will know more about the topic they're researching of course.