4 ms·
Even just 100 or 200 random papers would be okay for a rough indication. There must be someone who has done this? That some papers are bollocks is not a contro
by arp242 3y ago
Even just 100 or 200 random papers would be okay for a rough indication. There must be someone who has done this?
That some papers are bollocks is not a controversial notion, but I genuinely have no idea if it's 10%, 50%, or 90% that's bollocks. Even just a rough indication of "about 20%" or "about 80%" would be hugely helpful. It's also unclear that non-STEM is more bollocky than STEM – Oxman's thesis is STEM, and also bollocks. STEM word salad papers don't make the news because the conclusions are rarely interesting, whereas sociology and psychology papers do, because the conclusions tend to be more interesting.
- AlotOfReading 3y agoLeaving aside the vast semantic Gulf between what Feynman was talking about and things being "bollocks", how do you evaluate the truthiness of novel arguments? The typical solution here is domain experts, aka peer review. The author's point is that they don't trust peer review, so you need some other method that's both practical and universal. I can't imagine what that could possibly look like.
- programjames 3y agoI've probably read 100 ML papers. They're pretty bad. Even the most cited ones like Adam, Attention, and Latent Diffusion are messy, unclear, or just straight up wrong at times. Adam - Their algorithm is inefficient. Their paper can be summarized as "use the signal-to-noise ratio," but the key idea is hidden in a page-long paragraph halfway through the paper. Attention - I probably read "keys, query, values" a dozen times before I realized the key and query matrices multiply with the same vector (where the "self" comes in). Latent Diffusion - Their theoretical grounding was just completely wrong. It's not some stochastic diffeq, or Markov process, or w/e terms they threw around. It's just finding the gradient of log-likelihood using finite differences (the error they add).