5 ms·

The term "PhD Knowledge" makes me cringe. PhD is about contribution to a field, not a standardized test score or a weird status-bearing substitute for IQ.

glial 2y ago
Also - at least in my experience - doing a PhD makes a person more aware of the boundaries of knowledge, and therefore generally pretty skeptical, and capable of being usefully critical, about developments in their field.
enum 2y ago
There is a trend of building hard benchmarks for LLMs, such as GPQA or Humanity's Last Exam, with questions crafted by PhDs. Ergo, most people cannot understand what these benchmarks are testing. If you're an expert in something, and if you're on HN you may be an expert in some sub-field of CS, I recommend trying to solve the questions from your field on Humanity's Last Exam. I took a crack at the Programming Languages questions myself.
- charlieyu1 2y ago
  - wants PhD knowledge - pays like $100-500 per question
levocardia 2y ago
You cannot get a PhD (in the US) without also passing an examination that, nominally at least, demonstrates you have broad and deep knowledge about your field. The qualifying exam is not the most important part, but it is a necessary part.
- enum 2y ago
  Some still exist, but this is less and less true over time. Even when a requirement is called an "exam", it isn't what a high-schooler would recognize as one. Many of the exams described on this page are personalized and only require mastery of a handful of papers that really matter to the individual student, and knowledge of related work: https://www.cs.stanford.edu/phd-program-requirements-qualifying-exams https://www.cs.stanford.edu/phd-program-requirements-qualify... I've been through something that sounds similar to these, and I would not have called it an exam.
krisoft 2y ago
In this case it is a shorthand for specialist level knowledge. Thre kind of information you would expect a person pursuing a PhD in a specific field to know but someone not doing that to not know. For example how in the “Humanity's Last Exam” AI evaluation dataset they have this question: “Hummingbirds within Apodiformes uniquely have a bilaterally paired oval bone, a sesamoid embedded in the caudolateral portion of the expanded, cruciate aponeurosis of insertion of m. depressor caudae. How many paired tendons are supported by this sesamoid bone? Answer with a number.” That’s the kind of “PhD knowledge” which this evaluation does not require.
- mkoubaa 2y ago
  That's a jargon and search problem at best. I'm not convinced at all.
  - krisoft 2y ago
    > I'm not convinced at all. What are you not convinced about? Are you not convinced this is what they mean?
    - mkoubaa 2y ago
      I'm not convinced it's even a marginally useful metric for measuring LLM performance.
      - krisoft 2y ago
        The question i quoted about the humming bird anatomy? That is the point. That is why this evaluation explicitly decides to not go down that route. Nobody (well, not me anyway) wants to convince you that it is usefull. That is the kind of question the authors of this evaluation looked at, they also felt what you are feeling, and decided to do something which doesn’t require that kind of deep and specialist knowledge. And that is what they describe in the paper’s title as “PhD knowledge not required”.
buescher 2y ago
It means you have taken graduate level courses and have completed an apprenticeship in doing publishable academic research. Now that a master's is the new bachelor's there's something of a trend to think of a PhD as the new master's but it's really qualitatively different.

glial 2y ago

Also - at least in my experience - doing a PhD makes a person more aware of the boundaries of knowledge, and therefore generally pretty skeptical, and capable of being usefully critical, about developments in their field.

enum 2y ago

There is a trend of building hard benchmarks for LLMs, such as GPQA or Humanity's Last Exam, with questions crafted by PhDs. Ergo, most people cannot understand what these benchmarks are testing. If you're an expert in something, and if you're on HN you may be an expert in some sub-field of CS, I recommend trying to solve the questions from your field on Humanity's Last Exam. I took a crack at the Programming Languages questions myself.

charlieyu1 2y ago
- wants PhD knowledge - pays like $100-500 per question

levocardia 2y ago

You cannot get a PhD (in the US) without also passing an examination that, nominally at least, demonstrates you have broad and deep knowledge about your field. The qualifying exam is not the most important part, but it is a necessary part.

enum 2y ago
Some still exist, but this is less and less true over time. Even when a requirement is called an "exam", it isn't what a high-schooler would recognize as one. Many of the exams described on this page are personalized and only require mastery of a handful of papers that really matter to the individual student, and knowledge of related work: https://www.cs.stanford.edu/phd-program-requirements-qualifying-exams https://www.cs.stanford.edu/phd-program-requirements-qualify... I've been through something that sounds similar to these, and I would not have called it an exam.

krisoft 2y ago

In this case it is a shorthand for specialist level knowledge. Thre kind of information you would expect a person pursuing a PhD in a specific field to know but someone not doing that to not know. For example how in the “Humanity's Last Exam” AI evaluation dataset they have this question: “Hummingbirds within Apodiformes uniquely have a bilaterally paired oval bone, a sesamoid embedded in the caudolateral portion of the expanded, cruciate aponeurosis of insertion of m. depressor caudae. How many paired tendons are supported by this sesamoid bone? Answer with a number.” That’s the kind of “PhD knowledge” which this evaluation does not require.

mkoubaa 2y ago
That's a jargon and search problem at best. I'm not convinced at all.
- krisoft 2y ago
  > I'm not convinced at all. What are you not convinced about? Are you not convinced this is what they mean?
  - mkoubaa 2y ago
    I'm not convinced it's even a marginally useful metric for measuring LLM performance.
    - krisoft 2y ago
      The question i quoted about the humming bird anatomy? That is the point. That is why this evaluation explicitly decides to not go down that route. Nobody (well, not me anyway) wants to convince you that it is usefull. That is the kind of question the authors of this evaluation looked at, they also felt what you are feeling, and decided to do something which doesn’t require that kind of deep and specialist knowledge. And that is what they describe in the paper’s title as “PhD knowledge not required”.

buescher 2y ago

It means you have taken graduate level courses and have completed an apprenticeship in doing publishable academic research. Now that a master's is the new bachelor's there's something of a trend to think of a PhD as the new master's but it's really qualitatively different.