4 ms·

> It's entirely possible for this to be true and their methodology to still be effective and valid. What that would mean is that the ordering of words adds litt

> It's entirely possible for this to be true and their methodology to still be effective and valid. What that would mean is that the ordering of words adds little predictive power to the classification (whether or not the paper is replicable). I can easily imagine that to be the case. For example, it is a commonly-held belief that it would be a struggle to replicate many papers in the social sciences. If that was the case, this "bag of words" approach would work for those papers, because the mere existence of social science vocabulary would raise the likelihood of failure to replicate. What is this is true, it would also mean that the model was of zero or negative value for actual real world use. It is not new insight - least of all to academics - that papers on subjects like cancer cures, consumer prices, or perceptions of well-being are more difficult to replicate than papers on subjects like molecular bonds or the behaviour of steel under torsion. On the other hand, academics also have an understanding of study power, study construction, how [un]expected the findings are, and the actual significance of the results to the field, all hugely significant information the ML organism does not incorporate. So the model is less useful than the human judgements it purports to replace. Just because a curve fits doesn't mean the model is an improvement on existing understanding Indeed, as the model described appears to be based on naive keyword associations across a very small number of papers, it's likely less useful even than a Reddit contrarian saying "bah economics" to every economics paper, as at least the Reddit contrarian might occasionally be able to highlight conspicuously suspect-sounding elements of some abstracts, and won't classify a bad economics paper on the steel industry as better than its peers because it contains terms associated with engineering papers and the name of the prestigious institution the author is an undergraduate at

seanhunter 3y ago
A model doesn't necessarily need to be an improvement on existing understanding to be useful though. It could have other attributes which are valuable. The map of London (where I live) that I use to navigate on my phone is not in any sense the best map of London that has ever been created. It is definitely not an improvement on existing understanding, but it's a great improvement on not having a map. Say you need to classify all of arxiv for some reason. A model like this might give you the classification performance you need while also being easy to build and not too computationally expensive. It might give better out of sample classification performance than a model that attempts to more realistically model the problem domain. Or it might have a distribution of errors that is more tolerable for a particular use case (like say you're less worried about misclassifications of extreme cases but you really really need to be accurate in the middle of the distribution or vice versa for some reason). Or it might be more robust to over fitting or whatever. Having a wide analytical toolbox is very useful for data science and this type of NLP has its place for some use cases. While it may seem fun to poke holes in the approach taken in this case I don't think the approach is necessarily intrinsically bad in any way. It's obviously (like all tools) bad if misused in the same way that a screwdriver can be used as a hammer, but it will suck.
- notahacker 3y ago
  > The map of London (where I live) that I use to navigate on my phone is not in any sense the best map of London that has ever been created. It is definitely not an improvement on existing understanding, but it's a great improvement on not having a map. But if the "map" is actually composed by a ML process parsing texts about London and arranging the place names based on proximity of mentions and cardinal direction references (this might be a fun exercise, might actually result in something that looks surprisingly like a map with the right data sources, and sufficient tuning, and absolutely won't have any real world use), people are going to ask why you trusted it. To be fair to the authors of the original paper, their actual claims appear to fall some way short of the blog's insinuation that they were proposing that academics should actually use their model to decide to spend hundreds of hours of work replicating research papers (which would be the perfect example of misuse of ML). But what they've actually done is a very basic statistical ranking of possibly misclassified subfields without any ability to address systematic error.

seanhunter 3y ago

A model doesn't necessarily need to be an improvement on existing understanding to be useful though. It could have other attributes which are valuable. The map of London (where I live) that I use to navigate on my phone is not in any sense the best map of London that has ever been created. It is definitely not an improvement on existing understanding, but it's a great improvement on not having a map. Say you need to classify all of arxiv for some reason. A model like this might give you the classification performance you need while also being easy to build and not too computationally expensive. It might give better out of sample classification performance than a model that attempts to more realistically model the problem domain. Or it might have a distribution of errors that is more tolerable for a particular use case (like say you're less worried about misclassifications of extreme cases but you really really need to be accurate in the middle of the distribution or vice versa for some reason). Or it might be more robust to over fitting or whatever. Having a wide analytical toolbox is very useful for data science and this type of NLP has its place for some use cases. While it may seem fun to poke holes in the approach taken in this case I don't think the approach is necessarily intrinsically bad in any way. It's obviously (like all tools) bad if misused in the same way that a screwdriver can be used as a hammer, but it will suck.

notahacker 3y ago
> The map of London (where I live) that I use to navigate on my phone is not in any sense the best map of London that has ever been created. It is definitely not an improvement on existing understanding, but it's a great improvement on not having a map. But if the "map" is actually composed by a ML process parsing texts about London and arranging the place names based on proximity of mentions and cardinal direction references (this might be a fun exercise, might actually result in something that looks surprisingly like a map with the right data sources, and sufficient tuning, and absolutely won't have any real world use), people are going to ask why you trusted it. To be fair to the authors of the original paper, their actual claims appear to fall some way short of the blog's insinuation that they were proposing that academics should actually use their model to decide to spend hundreds of hours of work replicating research papers (which would be the perfect example of misuse of ML). But what they've actually done is a very basic statistical ranking of possibly misclassified subfields without any ability to address systematic error.