3 ms·
> The infuriating part was that several people sent me links to the paper and (pointedly) asked me what I think about it. Written up as a Published Science Pape
by mcqueenjordan 4y ago
> The infuriating part was that several people sent me links to the paper and (pointedly) asked me what I think about it. Written up as a Published Science Paper, these speed comparisons seemed like a Fact About The Universe. And not what they really were - implementation details of some java code, written by a probably overstretched grad student. One of a whole bunch of implementations that they needed to code up.
> "Nooo! The peer reviewed science isn't right everybody! Please believe me!". But I didn't have a published paper justifying my claims. I had working code but it felt like none of the smart computer science people cared about that. Who was I? I was nobody.
Although I to some degree understand /why/ this is the way it is, it's quite frustrating the amount of ~unfounded belief people place in "published studies". I'm not advocating to denigrate science, but I think folks should still be skeptical of results (esp. low N or non-reproduced) when they're challenged, and not treat them as Facts About the Universe.
c.f. "The code was peer-reviewed, so the application must be bug-free and 100% correct, right?" /s
- cldellow 4y agoIn 2012, I was doing NLP stuff with the Stanford NLP POS tagger. They claimed it was highly optimized, and hey, who was I to doubt them? But man, it sure _felt_ slow. So I did some _very minimal_ profiling and, as a complete outsider, found small, simple changes that increased throughput by 4x without affecting maintainability. I think this fits your "low N" example -- I'm much more likely to believe something is optimal if you can tell me that multiple (experienced!) people have tried to optimize something and failed. If it's just 1 person, that 1 person better have amazing domain expertise for me to believe it.