3 ms·

This critique in no way invalidates the conclusions of the paper.

krona 3y ago
As I point out, the critique is simply that the paper isn't very good. In fact I'd go as far as to say its egregiously bad, even if the conclusion is 'correct' in as far as something was measured and they reported it.
- arp242 3y ago
  The conclusion of "GPT-4 can't even compete against low-quality human effort like that found on MTurk" seems alright. The article would be better if it didn't claim to compare "humans", because it doesn't – or at least, not in the common sense definition of "comparing humans vs GPT-4". But to be honest I think that's a bit pedantic. It would be different if it had found that GPT-4 outperformed MTurk, but if GPT-4 can't compete against MTurk on these tasks, then I don't think it will fare better against you or me with our full attention.
avs733 3y ago
If it’s making a claim about humans vs llms but shittified capitalism means that the humans are just using llms to make money churning on mechanical turn then you can’t make the underlying claim. That ignores random sample problems and the simplistic statistical analysis…so yeah it unfortunately does. It doesn’t prove they are wrong, but it’s not credible to say they are right
- Gare 3y ago
  It's only important that humans (or "humans") were about 90% correct. Whereas GPT-4 was only 33% correct. If those turks were using AI, that would mean there is AI out there much more capable than GPT-4.
  - avs733 3y ago
    It doesn’t… The problem is that unless the data are credible and reliable, any interpretation of that data is meaningless. For or against one’s hypotheses doesn’t matter if the data is junk. If the data is junk you can’t move past that, it’s the end of the conversation because the data has no meaning.
- LogicalRisk 3y ago
  From reading the paper and the original paper that the data for the MTurk/Prolific samples are drawn from, this is a convenience sample of 415 humans on two platforms. Each worker received a random sample of the ConceptARC problems, and the average score correct is assigned the "Human" benchmark. Perhaps by "random sample problems" you mean that the study is not representative of all of humanity? If so we can still take the paper as evaluating these 415 humans who speak English against the two models. If as you say, the workers are actually just using LLMs then this implies there is some LLM that your average MTurk worker has access to that out-performs GPT 4 and GPT 4V. That seems *extremely* unlikely to say the least. There is no need for any complex statistical analysis here since the question is simply comparing the scores on a test. It's a simple difference in means. Arguably, the main place that could benefit from additional statistical procedures would be weighting the sample to be representative of a target population, but that in no way affects the results of the study at hand.

As I point out, the critique is simply that the paper isn't very good. In fact I'd go as far as to say its egregiously bad, even if the conclusion is 'correct' in as far as something was measured and they reported it.

arp242 3y ago
The conclusion of "GPT-4 can't even compete against low-quality human effort like that found on MTurk" seems alright. The article would be better if it didn't claim to compare "humans", because it doesn't – or at least, not in the common sense definition of "comparing humans vs GPT-4". But to be honest I think that's a bit pedantic. It would be different if it had found that GPT-4 outperformed MTurk, but if GPT-4 can't compete against MTurk on these tasks, then I don't think it will fare better against you or me with our full attention.

avs733 3y ago

If it’s making a claim about humans vs llms but shittified capitalism means that the humans are just using llms to make money churning on mechanical turn then you can’t make the underlying claim. That ignores random sample problems and the simplistic statistical analysis…so yeah it unfortunately does. It doesn’t prove they are wrong, but it’s not credible to say they are right

Gare 3y ago
It's only important that humans (or "humans") were about 90% correct. Whereas GPT-4 was only 33% correct. If those turks were using AI, that would mean there is AI out there much more capable than GPT-4.
- avs733 3y ago
  It doesn’t… The problem is that unless the data are credible and reliable, any interpretation of that data is meaningless. For or against one’s hypotheses doesn’t matter if the data is junk. If the data is junk you can’t move past that, it’s the end of the conversation because the data has no meaning.
LogicalRisk 3y ago
From reading the paper and the original paper that the data for the MTurk/Prolific samples are drawn from, this is a convenience sample of 415 humans on two platforms. Each worker received a random sample of the ConceptARC problems, and the average score correct is assigned the "Human" benchmark. Perhaps by "random sample problems" you mean that the study is not representative of all of humanity? If so we can still take the paper as evaluating these 415 humans who speak English against the two models. If as you say, the workers are actually just using LLMs then this implies there is some LLM that your average MTurk worker has access to that out-performs GPT 4 and GPT 4V. That seems *extremely* unlikely to say the least. There is no need for any complex statistical analysis here since the question is simply comparing the scores on a test. It's a simple difference in means. Arguably, the main place that could benefit from additional statistical procedures would be weighting the sample to be representative of a target population, but that in no way affects the results of the study at hand.