3 ms·

A model doesn't necessarily need to be an improvement on existing understanding to be useful though. It could have other attributes which are valuable. The map

A model doesn't necessarily need to be an improvement on existing understanding to be useful though. It could have other attributes which are valuable. The map of London (where I live) that I use to navigate on my phone is not in any sense the best map of London that has ever been created. It is definitely not an improvement on existing understanding, but it's a great improvement on not having a map. Say you need to classify all of arxiv for some reason. A model like this might give you the classification performance you need while also being easy to build and not too computationally expensive. It might give better out of sample classification performance than a model that attempts to more realistically model the problem domain. Or it might have a distribution of errors that is more tolerable for a particular use case (like say you're less worried about misclassifications of extreme cases but you really really need to be accurate in the middle of the distribution or vice versa for some reason). Or it might be more robust to over fitting or whatever. Having a wide analytical toolbox is very useful for data science and this type of NLP has its place for some use cases. While it may seem fun to poke holes in the approach taken in this case I don't think the approach is necessarily intrinsically bad in any way. It's obviously (like all tools) bad if misused in the same way that a screwdriver can be used as a hammer, but it will suck.

notahacker 3y ago
> The map of London (where I live) that I use to navigate on my phone is not in any sense the best map of London that has ever been created. It is definitely not an improvement on existing understanding, but it's a great improvement on not having a map. But if the "map" is actually composed by a ML process parsing texts about London and arranging the place names based on proximity of mentions and cardinal direction references (this might be a fun exercise, might actually result in something that looks surprisingly like a map with the right data sources, and sufficient tuning, and absolutely won't have any real world use), people are going to ask why you trusted it. To be fair to the authors of the original paper, their actual claims appear to fall some way short of the blog's insinuation that they were proposing that academics should actually use their model to decide to spend hundreds of hours of work replicating research papers (which would be the perfect example of misuse of ML). But what they've actually done is a very basic statistical ranking of possibly misclassified subfields without any ability to address systematic error.

notahacker 3y ago

> The map of London (where I live) that I use to navigate on my phone is not in any sense the best map of London that has ever been created. It is definitely not an improvement on existing understanding, but it's a great improvement on not having a map. But if the "map" is actually composed by a ML process parsing texts about London and arranging the place names based on proximity of mentions and cardinal direction references (this might be a fun exercise, might actually result in something that looks surprisingly like a map with the right data sources, and sufficient tuning, and absolutely won't have any real world use), people are going to ask why you trusted it. To be fair to the authors of the original paper, their actual claims appear to fall some way short of the blog's insinuation that they were proposing that academics should actually use their model to decide to spend hundreds of hours of work replicating research papers (which would be the perfect example of misuse of ML). But what they've actually done is a very basic statistical ranking of possibly misclassified subfields without any ability to address systematic error.