4 ms·

>> According to the paper, Computer Vision research is notably more affected by the syndrome it outlines than other sectors, with the authors noting that Natura

>> According to the paper, Computer Vision research is notably more affected by the syndrome it outlines than other sectors, with the authors noting that Natural Language Processing (NLP) research is far less affected. The authors suggest that this could be because NLP communities are ‘more coherent’ and larger in size, and because NLP datasets are more accessible and easier to curate, as well as being smaller and less resource-intensive in terms of data-gathering. I'll have to read the paper to see what exactly it says on this but my knowledge of NLP benchmark datasets is exactly the opposite: the majority are simply no use for measuring the capabilities they're supposed to be measuring. For example, natural language _understanding_ datasets are typically created as multiple-choice questionnaires, so that, a) there is already a baseline accuracy that a system can "hit" just by chance (but which is almost never noted, or compared against, in papers) and b) a system that's good at classification can beat the benchmarks black and blue without doing any "understanding". And this is, indeed, what's been going on for quite a while now with large language models that take all the trophies and are still dumb as bricks. To make matters worse, NLP also doesn't have any good metrcis of performance. Stuff like the BLEU scores are just laughably inadequate. NLP is all bad metrics over bad benchmarks. And NLP results are much harder to just "eyball" than machine vision results (and the models are much harder to interpret than machine vision models where you can at least visualise the activations and see... something). I think NLP is much, much worse than machine vision.

xiaolingxiao 5y ago
Your suspicion is correct. I worked on such a dataset paper and worked to "beat" other methods on well-accepted benchmarks with dubious accuracy scores. One fundamental issue is that outside of POS tagging, there isn't a notion of empirical truth to measure against, only a small sample of what a "normal person would think." This stands in contrast to computer vision, whereby in a task such as monocular depth perception from a single frame, you can always measure against a Lidar-scanned depth map. The system can still overfit on the benchmark and they do, but at least the baseline truth itself is not in dispute. But questions such as: is this the "appropriate" response to a query is too open to interpretation.

xiaolingxiao 5y ago

Your suspicion is correct. I worked on such a dataset paper and worked to "beat" other methods on well-accepted benchmarks with dubious accuracy scores. One fundamental issue is that outside of POS tagging, there isn't a notion of empirical truth to measure against, only a small sample of what a "normal person would think." This stands in contrast to computer vision, whereby in a task such as monocular depth perception from a single frame, you can always measure against a Lidar-scanned depth map. The system can still overfit on the benchmark and they do, but at least the baseline truth itself is not in dispute. But questions such as: is this the "appropriate" response to a query is too open to interpretation.