4 ms·
And yet, this is a kind of figure that possibly could be proven, if you either assign a lot of people to read / analyze the papers and mark them "cargo cult" an
by Cthulhu_ 3y ago
And yet, this is a kind of figure that possibly could be proven, if you either assign a lot of people to read / analyze the papers and mark them "cargo cult" and "not cargo cult", or automate it with modern day text analysis tooling. Getting all the papers in one place will be a logistic challenge though.
- arp242 3y agoEven just 100 or 200 random papers would be okay for a rough indication. There must be someone who has done this? That some papers are bollocks is not a controversial notion, but I genuinely have no idea if it's 10%, 50%, or 90% that's bollocks. Even just a rough indication of "about 20%" or "about 80%" would be hugely helpful. It's also unclear that non-STEM is more bollocky than STEM – Oxman's thesis is STEM, and also bollocks. STEM word salad papers don't make the news because the conclusions are rarely interesting, whereas sociology and psychology papers do, because the conclusions tend to be more interesting.
- AlotOfReading 3y agoLeaving aside the vast semantic Gulf between what Feynman was talking about and things being "bollocks", how do you evaluate the truthiness of novel arguments? The typical solution here is domain experts, aka peer review. The author's point is that they don't trust peer review, so you need some other method that's both practical and universal. I can't imagine what that could possibly look like.
- programjames 3y agoI've probably read 100 ML papers. They're pretty bad. Even the most cited ones like Adam, Attention, and Latent Diffusion are messy, unclear, or just straight up wrong at times. Adam - Their algorithm is inefficient. Their paper can be summarized as "use the signal-to-noise ratio," but the key idea is hidden in a page-long paragraph halfway through the paper. Attention - I probably read "keys, query, values" a dozen times before I realized the key and query matrices multiply with the same vector (where the "self" comes in). Latent Diffusion - Their theoretical grounding was just completely wrong. It's not some stochastic diffeq, or Markov process, or w/e terms they threw around. It's just finding the gradient of log-likelihood using finite differences (the error they add).
- deleted 3y ago[deleted]
- jrgoff 3y agoYou could also do a random sampling.
- code_biologist 3y agoSciHub provides an index of their DOIs [1] as well as torrents for the actual content. I presume one could cross reference citation counts and pull the top few hundred papers in different fields to get a pretty good sample over time and research fields. [1] https://sci-hub.se/database#download https://sci-hub.se/database#download