3 ms·
ImageNet has five orders of magnitude more answers, which I would assume makes QA a completely different category of problem. The authors could probably have c
by iudqnolq 3y ago
ImageNet has five orders of magnitude more answers, which I would assume makes QA a completely different category of problem.
The authors could probably have carefully review all ~300 of their questions. If they couldn't they could have just reduced their sample size to say 50.
- muds 3y agoI admit that Imagenet isn't the best analogy here. But I'm pretty confident that this data cleaning issue would be caught in peer review. The biggest issue which I still don't understand was the removal of the test set. That was bad practice on the authors' part.
- iudqnolq 3y agoIn general with evaluations LLVMs I keep seeing issues that would be caught by a human carefully looking at the results. To a nonexpert who occasionally peeks at things it seems like having a small dataset that's just slightly too big for you to manually review is a bad practice. It also seems like 100% accuracy should have raised red flags, especially if you know your dataset isn't perfectly cleaned.