11 ms·

No Paper Is That Good

162 points by zt 8y ago

B1FF_PSUVM 8y ago
'We are all in a mud moat, but some of us are looking at the stars', as comrade Oscar Wilde almost said, then?
deleted 8y ago
[deleted]
User23 8y ago
What happens when you apply this author’s standard to this work?
- tzs 8y ago
  He's asking for two papers to demonstrate that it is worth delving into the literature of an entire field. It's not clear that it is even meaningful to ask how it applies to the work of any particular individual author within the field.
- bpchaps 8y ago
  Heh. From the article: Does all of this hold for my papers, too? Of course. The most I can claim is that I am hyper-aware of my own epistemic frailty, and have a litany of self-imposed safeguards. But I totally understand why my critics would look at my best papers and say, “Meh, doesn’t really prove anything.”
  - User23 8y ago
    I stopped reading before I got to that point thanks to the author’s argument on account of I took his advice. Thanks for the datum.
will4274 8y ago
Allow me to disagree. Chris Okasaki's thesis, "Purely Functional Data Structures" is That Good for the field of functional programming. There are other exemplary papers, like Godel's incompleteness theorem, Cantor's diagonal argument, or Einstein's statement of special relativity. Or Satoshi's paper on Bitcoin. These papers are not exhaustive summaries of a field. But a reader comes away understanding the type of problems a field is devoted to solving and many of the existing ideas. And I believe that each is a paragon of their field.
- xevb3k 8y ago
  The article limits itself to discussing “empirical paper”s not theoretical papers. In a theoretical paper, it’s possible to make statements that stand on their own merits. In empirical science, a single paper is never really enough to support an entire field. Empirical sciences generally rely on the development of scientific consensus.
  - antt 8y ago
    That seems like an artificial distinction made up for economics. In physics there are plenty of theory papers that make very clear and concise predictions. If these papers hold then they are very much both. E.g. the original special relativity paper goes out of its way to make predictions and calculations at the state of the art of the time, which are quite jarring today.
    - eesmith 8y ago
      Dirac's early papers are, from what I have heard, amazing pieces of work. One commentary about the quality is given at https://michaelberryphysics.files.wordpress.com/2013/07/berry291.pdf https://michaelberryphysics.files.wordpress.com/2013/07/berr... .
  - will4274 8y ago
    Hmm. I guess Mendal wrote a journal, not a paper per se. Millikan's oildrop experiment?
- deleted 8y ago
  [deleted]
anonytrary 8y ago
> If you want me to read the vast literature, cite me two papers that are exemplars of that literature. I will read them. If these two papers are full of mistakes and bad reasoning, I will feel free to skip the rest of the vast literature. Because if that’s the best you can do, I’ve seen enough. This is one of worst ideas I've ever read. Wouldn't you just be cheating yourself? How is tying your own learning to someone else's ability to find the best papers in any way a smart thing to do? It would be much better to doubt the person who gave you the papers (perhaps he miscalculated which two papers were the best), than to dismiss the entire field.
- jldugger 8y ago
  It's not about dismissing the field, as much as the claim that you must have read the literature to constructively participate in a debate.
  - skywhopper 8y ago
    I think it’s reasonable to expect familiarity with “the literature” beyond “two exemplary papers” to be given any respect in an academic debate. If you aren’t reading the lit reviews and broadly in the topic how can you possibly contribute to the discussion productively? Almost certainly any points you want to make will have been examined and discussed already. This rule is about FOMO for wannabe rennaisssance-man know-it-alls who think their gut reactions are God’s own truth and demand a response from actual experts. If you don’t know the topic and don’t want to learn about it then please find something else to do.
    - xamuel 8y ago
      Here's an example. You stumble on a bunch of psychology papers in which psychologists noticed that whenever 6+ kids get together, there are either three mutual friends, or three mutual non-friends. The psychologists think this has something to do with child psychology and have written 10,000 pages on it. You immediately recognize it has nothing to do with psychology and is just basic Ramsey Theory [1]. Should you have to review thousands of pages before being allowed to chip in? [1] https://en.wikipedia.org/wiki/Ramsey%27s_theorem#Example:_R(3,_3)_=_6 https://en.wikipedia.org/wiki/Ramsey%27s_theorem#Example:_R(... A less hypothetical example: You're Bertrand Russel and you discover Russel's paradox, a one-liner which negates thousands of pages of Frege's not-yet-published logic textbook. Do you have to wait for him to publish it, and then you read it, before raising your voice?
nerdponx 8y ago
One conclusion you could draw from this: economics might actually be a giant mud moat.
- BurningFrog 8y ago
  Economics is several separate sciences, awkwardly placed in the same department.
  - allenz 8y ago
    Most departments are heterogeneous, with mutually unintelligible subfields. CS encompasses both information theory and robotics. Cultural anthropology has little in common with medical anthropology.
  - traverseda 8y ago
    Just like cognitive science has a much higher replication rate than social psychology, you will get different results with different subfields.
- barry-cotter 8y ago
  You’d be wrong. I’m not all that impressed with macroeconomics and it’s far from a mud moat. Microeconomics is making more progress and is on much sounder footing, hardly surprising given how many more empirical observations there are. If you’re only going to read one microeconomics textbook read Varian’s Intermediate Microeconomics if you can do calculus. If you can’t Cowen and Tabarrok’s Principles is pretty good and has a load of wonderful accompanying videos. Comparative Advantage https://www.mruniversity.com/courses/development-economics/comparative-advantage https://www.mruniversity.com/courses/development-economics/c...
  - nerdponx 8y ago
    Here's Smith's definition of a mud moat: And in the world of intellectual debate, this vast literature can function as a mud moat. That is a term I just made up, sticking with the metaphor of political arguments as medieval castles requiring a defense. A mud moat is just a big pit of mud surrounding your castle, causing an attacking army to get trapped in the mud while you pepper them with arrows. https://leaders.economicblogs.org/noah-smith/2017/smith-literatures-mud-moats/ https://leaders.economicblogs.org/noah-smith/2017/smith-lite... I was only half-serious in my original comment. What mud moat exists today is maintained less by economists and more by politicians and businesspeople, who find certain "classical" economic conclusions convenient. "Free markets are good - just read the economics literature!"
spruciefic399 8y ago
I was sort of on board with the two-paper rule until I got to the arguments about the reviews. Maybe in his mind integrating literature into a cohesive summary (ala meta-analysis) is a novel contribution, but if not, the attitude behind the two-paper rule is part of why science is in such crisis. The author is right, that the interpretation of the literature should be based on an accumulated read, and not one or two papers (unless they're reviews or meta-analyses).
hzhou321 8y ago
I would agree that "no paper is that good" -- on the first try or even the second try. Just as we need constantly refactor code, just as we need constantly re-edit a book, a paper that explains a scientific idea also needs constant re-write to become good. The sad reality is that scholars simply do not do that. It is understandable though. Once the paper is published, then it is no-longer novel, and there is obviously lack of reward and motivation to write the same idea again -- just to write it better. ... unless you are actually writing a review paper, but then, it appears there is little value unless the review is "complete". A good illustration of the idea need be able to high-light the key idea while avoid having minor ideas obscure the key. Therefore, a "complete" review paper rarely provide a good read. Compared to papers, a good text-book often is a much better read than all the original papers. It is not really because of the size, rather it is because a good text-book is written from the reader's point of view and focuses on conveying the idea itself (vs. selling the idea). And a good text-book takes many rounds to develop. There is no reason papers cannot be developed in a similar way as text books: once a ground-breaking paper is published, it shall be constantly updated, each new edition reflects what the author has newly learned and incorporating new development including the entire community... Alas, that is not the culture, and there is no motivation to do such.
- allenz 8y ago
  In addition to the incentive problem, you would also need to get people to peer review any changes to a paper. Most of the changes would be minor and annoying to peer review. I think that courses remain the best way of distilling and communicating knowledge. The only problem is that they're not always available to a lay audience, since there's little incentive to make them available. The other alternative for the lay audience is science/economics journalism (Economist, Scientific American, Discover, etc). This works to some extent but mostly just scratches the surface, since even regular journalism is struggling to be profitable these days. With deep technical topics there are too few readers and too few qualified, willing writers.
- jacques_chester 8y ago
  I think it's more useful for papers to be what they are: diffs. A textbook is a snapshot view. Both kinds of embodiment are necessary in their native form for different purposes.
nazgulnarsil 8y ago
Is there any cross disciplinary repository of excellent research reviews?
6ak74rfy 8y ago
On a slightly tangential note: how do you guys read research papers? Do you go through them word by word or you only skim them to get a general understanding? I try to do the former but the work seems so boring that I am hardly motivated to do this more.
- MereInterest 8y ago
  Most of the time, skimming through, picking out the parts that are different from others in the field. For physics: "Okay, this paper is from so and so, yup, same facility and equipment as the previous one. Theory section, same basic overview, skip to the end for what makes this experiment worthwhile? Anything interesting in their analysis, and how good are their statistics? Did they show the usual set of diagnostics plots for the type of analysis they did, and if not, what are they trying to downplay?"
- rflrob 8y ago
  For the most part, the way to read papers in biology is to go through the figures. When I was quickly trying to figure out if a project I was about to start up just got scooped (it did, and good for them for saving me the effort), I did it by glancing at figure 1, then reading the title of figure 1, then closely examining figure 1 while cross-referencing with the detailed caption. Then move on to 2, 3, etc. By figure 5, I had a hard time telling what was going on, so I found the part of the text that referenced it, and read it carefully. I've heard arguments that one should leave the abstract until the end, since it sometimes pushes a story harder than the data necessarily warrants, and so if you come to your own conclusion about what's going on in the paper, then check to make sure the authors claim what you feel is supported, you're less likely to be misled. Probably this makes sense for reviewing a paper, but for most things, I'd rather find out quickly if the paper is even relevant to what I'm working on.
  - dekhn 8y ago
    I've found looking at figures and captions to be really misleading. I find that I have to read the methods section first to convince myself that the data collection was correct before I can trust any figures. You have to read the abstract. It's the intro and conclusion you can skip.
dalbasal 8y ago
I assume this relates mostly to economics or the humanities (social science, if you insist) in general. These apply to sciences too, but are less debilitating in the long run. Fifth, most researchers’ priors are heavily influenced by some extremely suspicious factors. Academics in the humanities identify themselves "I am an X." X can be post structuralist, rational materialist, classical liberal or some other broad, hairy, intellectual identity. This is bad news for objectivity. Everyone has a dog in the fight.
- ci5er 8y ago
  It's interesting. As an engineering type (EE, primarily, but I had several other undergrad degrees), everything was about what is true. How do electrons move with their respective related fields across geometries. This other stuff seems to be about "what is useful to us as humans in a group matrix (culture/community". That seems like it should be studied - but I don't think we all have the same objectives. Also - some people keep trying to make it "true" vs. "useful". What say ye all? Do you think that "True" has any place in the social sciences vs. the observational "I observe this to be useful in this context" or the almost psycho-analytical "I notice that when people believe X together, society does better"? I don't know - it just seems (to me) that we spend too much time on the truth as opposed to (subjectively) communally useful. i.e. What should we decide to believe in as a community to hang together (because otherwise, we shall surely hang separately)?
  - westoncb 8y ago
    > What say ye all? Do you think that "True" has any place in the social sciences vs. the observational "I observe this to be useful in this context" or the almost psycho-analytical "I notice that when people believe X together, society does better"? The problem with that is whoever takes it upon themselves to discard truth and consider 'what will have the best effect on people/society if they were to believe it' —is that you have to manipulate other people into believing it's true on the hope that your assessment of how it's going to affect people is accurate. It's almost certainly not. It's interesting considering the human brain's criteria for accepting beliefs. It works pragmatically, not with strict adherence to truth—but, from what I can tell (and I've seen some research supporting this), the amount it's willing to deviate from (what it estimates to be) truth is proportional to the immediate demands of present circumstances. If you are in a crisis, it is willing to compromise and adopt a belief that conflicts with evidence, as long as it solves some immediate problem you're facing. But when that happens, you are accumulating something like technical debt. You become sort of 'out of harmony' with your circumstances, and problems will arise given enough time. Why would it evolve to act in that way? It seems clear the answer is: because it's generally most beneficial to survival to adopt beliefs which best fit the observed data. It's a fair assumption that society holding true beliefs is also going to be most effective for survival. So when someone manipulates someone else's beliefs because they think it will be beneficial, they are forcing that technical debt on them, and that's a decision only they should make for themselves. The same is true but amplified when considering spreading these ideas to wider audiences. That said, much of contemporary social science and the humanities are rife with the opposite sort of thinking. This is heavily fueled by the fact that much of it doesn't consider 'correspondent' truth to be a real thing, so in their framework it is meaningless to say they are compromising the truth. Which truth? You can see this in Pragmatism's approach of literally redefining 'truth' to refer to something like 'the most effective belief to adopt'; and even more pervasive are all the schools of thought influenced by a kind of radical relativism which denies the possibility of truth. (It also clarifies a lot of philosophy once you realize the philosophers are actually thinking about the best mind-programs to put into people, rather than seeking truth as a typical person would recognize it. If only the sneaky bastards would say so to their readers' faces. But of course it would not be pragmatic to do so.) It's a bad idea, and I sincerely hope the fields presently employing it fall into ruin, or are otherwise convinced to abandon it.
salty_biscuits 8y ago
Papers aren't meant to be this crystallized nugget of truth, they are a progress report on an ongoing piece of living research. They aren't meant to be infallible.
traverseda 8y ago
On the psychology replication crisis >Overall, 36% of the replications yielded significant findings (p value below 0.05) compared to 97% of the original studies that had significant effects. The mean effect size in the replications was approximately half the magnitude of the effects reported in the original studies. https://en.wikipedia.org/wiki/Replication_crisis#Psychology_replication_rates https://en.wikipedia.org/wiki/Replication_crisis#Psychology_... Of course if you point out that psychology research is strongly biased, the immediate response is that you must be an anti-intellectual. As if the system that produced such misleading results is a representation of all intellectual pursuits, and not just a mistake. The same way you get called an anti-intellectual for questioning the post-modern-analyses of whatever. I'm really sick of people claiming that criticizing obviously bad science is "anti-intellectual". When you produce that many useless papers you're obviously, as a field, lacking an understanding of why science is good and useful in the first place. I'm presuming that economics has similar problems with replication, and that you can only trust the most basic and obvious of their findings.
- sarbaz 8y ago
  A lot of this is simply that p < 0.05 is a very low bar for research. If we want to understand how the world works, we need to be able to construct mechanisms that explain why the results we measure in experiments come about. Without this thing, it's not science. A good mechanism that accurately explains some part of the world can be validated with a much better result than p < 0.05, because it can be properly isolated from other factors and its effect size can be strengthened. Without a mechanism to explain how things work, the experimental results that get published are probably just noise. And because p < 0.05 is such a low bar to pass, lots of noise gets published. Trying to reproduce an experiment whose results were drawn from /dev/urandom is pointless. In fact I'm surprised the replication was so high. I was expecting it to be around 5% (i.e. results are totally random). So maybe there is some hidden merit to all of this
  - nicoburns 8y ago
    > If we want to understand how the world works, we need to be able to construct mechanisms that explain why the results we measure in experiments come about. This, 1000 times over. Unfortunately we seem to teach that statistical analysis is sufficient to infer causation in a lot our undergraduate science programs.
dekhn 8y ago
Well, Watson & Crick (Nature 1953) is that good. It does contain an error (the actual DNA structure is slightly wrong) And Avery (JEM 1944) http://jem.rupress.org/content/79/2/137 http://jem.rupress.org/content/79/2/137) which "proved" that DNA is the molecule of heredity, is also there. But to be qualified to read these papers and appreciate why they are points of quality within a sea of crap? That's hard.
acoye 8y ago
Bitcoin's paper was that good. Simple. Easy to read. Easy to replicate (a poc in python is easy) It is a recent key innovation (the blockchain tech).
coldtea 8y ago
>The best papers get up to around .20. Again, No Paper Is That Good. If you demur, consider this: In twenty years, will you still hold up the best papers of today as “paragons” or “exemplars” of compelling empirical work? If the field is not totally vague, then yes. We can still consider certain papers in physics, or chemistry, or medicine, computer science etc. as exemplary decades, or even centuries, later, even when they deal with empirical work. Soft sciences need not apply.
- AnimalMuppet 8y ago
  Absolutely. Others have cited Shannon's paper on information theory. I would add Einstein's paper on Special Relativity. I read a translation of it on the 100th anniversary of its release, and I learned something, even though I already knew Special Relativity. (What did I learn? I don't remember precisely - it's been 15 years - but I think it was that something on the electromagnetic side was caused by time dilation.)
- UncleMeat 8y ago
  > Soft sciences need not apply Why not? Historians still revere Braudel and The Mediterranean was published nearly 70 years ago. Are you certain that there are no such exemplary works outside of the fields you like?
  - coldtea 8y ago
    No, but I'm certain that whether revered or not, Braudel's work is still "full of mistakes and bad reasoning" -- it's just that in history those are harder to tell.
- Jweb_Guru 8y ago
  Nonsense. Plenty of old computer science papers don't hold up well to scrutiny or are largely irrelevant nowadays, such as those that were trying to optimize for bottlenecks in hardware that no longer exist or have shifted to different places. Even in pure mathematics, something that was considered a great insight a long time ago might be considered fairly trivial now, and not just because the other paper came out first. Just because a paper is correct doesn't make it interesting or worthwhile--a lot of the best papers have errors in them, but propose something that's a genuinely new contribution to the field.
kabouseng 8y ago
Claude Shannon's paper on information theory [1] is possibly an example of such a paragon of work. No citations, 50 pages of pure awesome, spawned a whole research field of its own and 70 years on just as valid. [1] A Mathematical Theory of Communication - http://math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf http://math.harvard.edu/~ctm/home/text/others/shannon/entrop...
- eboyjr 8y ago
  You said it. He proposed the model for digital AND analog communication still in use today. Although being a digital software person I admit I've never been motivated to read the part about continuous channels. I was motivated however to install an archway for my house from the exact shape in Figure 7 (binary entropy function)
- thidr0 8y ago
  First paper I thought of too.
- xurukefi 8y ago
  > No citations The paper contains citations. But they are hidden in the footnotes - see page 1 or page 5 for example.
- beagle3 8y ago
  And it's easily readable with basic math/engineering knowledge. Highly recommended. Erdos has a few papers that are "that good" as long as you know all the theory leading to the new insight, but none (that I read) are as clear and freestanding as Shannon's AMToC
- joker3 8y ago
  Math is different from empirical work. Yes, Shannon's paper is great, but that's not really relevant here because it's in a completely different paradigm.
turc1656 8y ago
I suspect this is limited to economics and the social sciences. There is absolutely no way this holds up in the natural sciences like math, physics, etc. People forget the original name for what we call economics was "political economy". That alone should tell you all you need to know about the dangers of treating that field like a science. If you ever want to know why expert economists can't seem to agree on things that happened 50-100 years ago or make accurate predictions for the future, the original name is very telling. Wouldn't it be ridiculous if we were still debating the validity of f=ma or e=mc^2? Wouldn't it be crazy for someone to claim general relativity is just flat out wrong, even though GPS systems would not work properly without humans accepting it? Why is nothing remotely approaching reasonable standards used in the social sciences before acceptance?
nitwit005 8y ago
Following the link through to the original proposal, this seems like a misinterpretation of the suggestion. The problem this was supposed to "fix" was people using the existence of their being "vast literature" to shut down arguments. From that perspective, the suggestion seems fine. You should be able to dig out two examples that show your field isn't nonsense. I don't think it was meant to be a high bar: > There are actual examples of vast literatures that contain zero knowledge: Astrology, for instance. People have written so much about astrology that I bet you could spend decades reading what they've written and not even come close to the end. But at the end of the day, the only thing you'd know more about is the mindset of people who write about astrology. Because astrology is total and utter bunk.
gjm11 8y ago
It seems like a lot of commenters here are under the impression that the "two paper rule" (if someone says you should read the literature in field X, ask them for the two best papers in that field they can think of, and if those aren't impressive then don't bother looking further) is a proposal of Bryan Caplan, who wrote the OP here. That is incorrect. The two-paper rule is Noah Smith's, and Bryan Caplan is disagreeing with it. (It's scarcely possible to read any of Caplan's post without realising that; I conclude that many commenters here have not bothered to read the OP before commenting.)

162 points by zt 8y ago

B1FF_PSUVM 8y ago

'We are all in a mud moat, but some of us are looking at the stars', as comrade Oscar Wilde almost said, then?

deleted 8y ago

[deleted]

User23 8y ago

What happens when you apply this author’s standard to this work?

tzs 8y ago
He's asking for two papers to demonstrate that it is worth delving into the literature of an entire field. It's not clear that it is even meaningful to ask how it applies to the work of any particular individual author within the field.
bpchaps 8y ago
Heh. From the article: Does all of this hold for my papers, too? Of course. The most I can claim is that I am hyper-aware of my own epistemic frailty, and have a litany of self-imposed safeguards. But I totally understand why my critics would look at my best papers and say, “Meh, doesn’t really prove anything.”
- User23 8y ago
  I stopped reading before I got to that point thanks to the author’s argument on account of I took his advice. Thanks for the datum.

will4274 8y ago

Allow me to disagree. Chris Okasaki's thesis, "Purely Functional Data Structures" is That Good for the field of functional programming. There are other exemplary papers, like Godel's incompleteness theorem, Cantor's diagonal argument, or Einstein's statement of special relativity. Or Satoshi's paper on Bitcoin. These papers are not exhaustive summaries of a field. But a reader comes away understanding the type of problems a field is devoted to solving and many of the existing ideas. And I believe that each is a paragon of their field.

xevb3k 8y ago
The article limits itself to discussing “empirical paper”s not theoretical papers. In a theoretical paper, it’s possible to make statements that stand on their own merits. In empirical science, a single paper is never really enough to support an entire field. Empirical sciences generally rely on the development of scientific consensus.
- antt 8y ago
  That seems like an artificial distinction made up for economics. In physics there are plenty of theory papers that make very clear and concise predictions. If these papers hold then they are very much both. E.g. the original special relativity paper goes out of its way to make predictions and calculations at the state of the art of the time, which are quite jarring today.
  - eesmith 8y ago
    Dirac's early papers are, from what I have heard, amazing pieces of work. One commentary about the quality is given at https://michaelberryphysics.files.wordpress.com/2013/07/berry291.pdf https://michaelberryphysics.files.wordpress.com/2013/07/berr... .
- will4274 8y ago
  Hmm. I guess Mendal wrote a journal, not a paper per se. Millikan's oildrop experiment?
deleted 8y ago
[deleted]

anonytrary 8y ago

> If you want me to read the vast literature, cite me two papers that are exemplars of that literature. I will read them. If these two papers are full of mistakes and bad reasoning, I will feel free to skip the rest of the vast literature. Because if that’s the best you can do, I’ve seen enough. This is one of worst ideas I've ever read. Wouldn't you just be cheating yourself? How is tying your own learning to someone else's ability to find the best papers in any way a smart thing to do? It would be much better to doubt the person who gave you the papers (perhaps he miscalculated which two papers were the best), than to dismiss the entire field.

jldugger 8y ago
It's not about dismissing the field, as much as the claim that you must have read the literature to constructively participate in a debate.
- skywhopper 8y ago
  I think it’s reasonable to expect familiarity with “the literature” beyond “two exemplary papers” to be given any respect in an academic debate. If you aren’t reading the lit reviews and broadly in the topic how can you possibly contribute to the discussion productively? Almost certainly any points you want to make will have been examined and discussed already. This rule is about FOMO for wannabe rennaisssance-man know-it-alls who think their gut reactions are God’s own truth and demand a response from actual experts. If you don’t know the topic and don’t want to learn about it then please find something else to do.
  - xamuel 8y ago
    Here's an example. You stumble on a bunch of psychology papers in which psychologists noticed that whenever 6+ kids get together, there are either three mutual friends, or three mutual non-friends. The psychologists think this has something to do with child psychology and have written 10,000 pages on it. You immediately recognize it has nothing to do with psychology and is just basic Ramsey Theory [1]. Should you have to review thousands of pages before being allowed to chip in? [1] https://en.wikipedia.org/wiki/Ramsey%27s_theorem#Example:_R(3,_3)_=_6 https://en.wikipedia.org/wiki/Ramsey%27s_theorem#Example:_R(... A less hypothetical example: You're Bertrand Russel and you discover Russel's paradox, a one-liner which negates thousands of pages of Frege's not-yet-published logic textbook. Do you have to wait for him to publish it, and then you read it, before raising your voice?

nerdponx 8y ago

One conclusion you could draw from this: economics might actually be a giant mud moat.

BurningFrog 8y ago
Economics is several separate sciences, awkwardly placed in the same department.
- allenz 8y ago
  Most departments are heterogeneous, with mutually unintelligible subfields. CS encompasses both information theory and robotics. Cultural anthropology has little in common with medical anthropology.
- traverseda 8y ago
  Just like cognitive science has a much higher replication rate than social psychology, you will get different results with different subfields.
barry-cotter 8y ago
You’d be wrong. I’m not all that impressed with macroeconomics and it’s far from a mud moat. Microeconomics is making more progress and is on much sounder footing, hardly surprising given how many more empirical observations there are. If you’re only going to read one microeconomics textbook read Varian’s Intermediate Microeconomics if you can do calculus. If you can’t Cowen and Tabarrok’s Principles is pretty good and has a load of wonderful accompanying videos. Comparative Advantage https://www.mruniversity.com/courses/development-economics/comparative-advantage https://www.mruniversity.com/courses/development-economics/c...
- nerdponx 8y ago
  Here's Smith's definition of a mud moat: And in the world of intellectual debate, this vast literature can function as a mud moat. That is a term I just made up, sticking with the metaphor of political arguments as medieval castles requiring a defense. A mud moat is just a big pit of mud surrounding your castle, causing an attacking army to get trapped in the mud while you pepper them with arrows. https://leaders.economicblogs.org/noah-smith/2017/smith-literatures-mud-moats/ https://leaders.economicblogs.org/noah-smith/2017/smith-lite... I was only half-serious in my original comment. What mud moat exists today is maintained less by economists and more by politicians and businesspeople, who find certain "classical" economic conclusions convenient. "Free markets are good - just read the economics literature!"

spruciefic399 8y ago

I was sort of on board with the two-paper rule until I got to the arguments about the reviews. Maybe in his mind integrating literature into a cohesive summary (ala meta-analysis) is a novel contribution, but if not, the attitude behind the two-paper rule is part of why science is in such crisis. The author is right, that the interpretation of the literature should be based on an accumulated read, and not one or two papers (unless they're reviews or meta-analyses).

hzhou321 8y ago

I would agree that "no paper is that good" -- on the first try or even the second try. Just as we need constantly refactor code, just as we need constantly re-edit a book, a paper that explains a scientific idea also needs constant re-write to become good. The sad reality is that scholars simply do not do that. It is understandable though. Once the paper is published, then it is no-longer novel, and there is obviously lack of reward and motivation to write the same idea again -- just to write it better. ... unless you are actually writing a review paper, but then, it appears there is little value unless the review is "complete". A good illustration of the idea need be able to high-light the key idea while avoid having minor ideas obscure the key. Therefore, a "complete" review paper rarely provide a good read. Compared to papers, a good text-book often is a much better read than all the original papers. It is not really because of the size, rather it is because a good text-book is written from the reader's point of view and focuses on conveying the idea itself (vs. selling the idea). And a good text-book takes many rounds to develop. There is no reason papers cannot be developed in a similar way as text books: once a ground-breaking paper is published, it shall be constantly updated, each new edition reflects what the author has newly learned and incorporating new development including the entire community... Alas, that is not the culture, and there is no motivation to do such.

allenz 8y ago
In addition to the incentive problem, you would also need to get people to peer review any changes to a paper. Most of the changes would be minor and annoying to peer review. I think that courses remain the best way of distilling and communicating knowledge. The only problem is that they're not always available to a lay audience, since there's little incentive to make them available. The other alternative for the lay audience is science/economics journalism (Economist, Scientific American, Discover, etc). This works to some extent but mostly just scratches the surface, since even regular journalism is struggling to be profitable these days. With deep technical topics there are too few readers and too few qualified, willing writers.
jacques_chester 8y ago
I think it's more useful for papers to be what they are: diffs. A textbook is a snapshot view. Both kinds of embodiment are necessary in their native form for different purposes.

nazgulnarsil 8y ago

Is there any cross disciplinary repository of excellent research reviews?

6ak74rfy 8y ago

On a slightly tangential note: how do you guys read research papers? Do you go through them word by word or you only skim them to get a general understanding? I try to do the former but the work seems so boring that I am hardly motivated to do this more.

MereInterest 8y ago
Most of the time, skimming through, picking out the parts that are different from others in the field. For physics: "Okay, this paper is from so and so, yup, same facility and equipment as the previous one. Theory section, same basic overview, skip to the end for what makes this experiment worthwhile? Anything interesting in their analysis, and how good are their statistics? Did they show the usual set of diagnostics plots for the type of analysis they did, and if not, what are they trying to downplay?"
rflrob 8y ago
For the most part, the way to read papers in biology is to go through the figures. When I was quickly trying to figure out if a project I was about to start up just got scooped (it did, and good for them for saving me the effort), I did it by glancing at figure 1, then reading the title of figure 1, then closely examining figure 1 while cross-referencing with the detailed caption. Then move on to 2, 3, etc. By figure 5, I had a hard time telling what was going on, so I found the part of the text that referenced it, and read it carefully. I've heard arguments that one should leave the abstract until the end, since it sometimes pushes a story harder than the data necessarily warrants, and so if you come to your own conclusion about what's going on in the paper, then check to make sure the authors claim what you feel is supported, you're less likely to be misled. Probably this makes sense for reviewing a paper, but for most things, I'd rather find out quickly if the paper is even relevant to what I'm working on.
- dekhn 8y ago
  I've found looking at figures and captions to be really misleading. I find that I have to read the methods section first to convince myself that the data collection was correct before I can trust any figures. You have to read the abstract. It's the intro and conclusion you can skip.

dalbasal 8y ago

I assume this relates mostly to economics or the humanities (social science, if you insist) in general. These apply to sciences too, but are less debilitating in the long run. Fifth, most researchers’ priors are heavily influenced by some extremely suspicious factors. Academics in the humanities identify themselves "I am an X." X can be post structuralist, rational materialist, classical liberal or some other broad, hairy, intellectual identity. This is bad news for objectivity. Everyone has a dog in the fight.

ci5er 8y ago
It's interesting. As an engineering type (EE, primarily, but I had several other undergrad degrees), everything was about what is true. How do electrons move with their respective related fields across geometries. This other stuff seems to be about "what is useful to us as humans in a group matrix (culture/community". That seems like it should be studied - but I don't think we all have the same objectives. Also - some people keep trying to make it "true" vs. "useful". What say ye all? Do you think that "True" has any place in the social sciences vs. the observational "I observe this to be useful in this context" or the almost psycho-analytical "I notice that when people believe X together, society does better"? I don't know - it just seems (to me) that we spend too much time on the truth as opposed to (subjectively) communally useful. i.e. What should we decide to believe in as a community to hang together (because otherwise, we shall surely hang separately)?
- westoncb 8y ago
  > What say ye all? Do you think that "True" has any place in the social sciences vs. the observational "I observe this to be useful in this context" or the almost psycho-analytical "I notice that when people believe X together, society does better"? The problem with that is whoever takes it upon themselves to discard truth and consider 'what will have the best effect on people/society if they were to believe it' —is that you have to manipulate other people into believing it's true on the hope that your assessment of how it's going to affect people is accurate. It's almost certainly not. It's interesting considering the human brain's criteria for accepting beliefs. It works pragmatically, not with strict adherence to truth—but, from what I can tell (and I've seen some research supporting this), the amount it's willing to deviate from (what it estimates to be) truth is proportional to the immediate demands of present circumstances. If you are in a crisis, it is willing to compromise and adopt a belief that conflicts with evidence, as long as it solves some immediate problem you're facing. But when that happens, you are accumulating something like technical debt. You become sort of 'out of harmony' with your circumstances, and problems will arise given enough time. Why would it evolve to act in that way? It seems clear the answer is: because it's generally most beneficial to survival to adopt beliefs which best fit the observed data. It's a fair assumption that society holding true beliefs is also going to be most effective for survival. So when someone manipulates someone else's beliefs because they think it will be beneficial, they are forcing that technical debt on them, and that's a decision only they should make for themselves. The same is true but amplified when considering spreading these ideas to wider audiences. That said, much of contemporary social science and the humanities are rife with the opposite sort of thinking. This is heavily fueled by the fact that much of it doesn't consider 'correspondent' truth to be a real thing, so in their framework it is meaningless to say they are compromising the truth. Which truth? You can see this in Pragmatism's approach of literally redefining 'truth' to refer to something like 'the most effective belief to adopt'; and even more pervasive are all the schools of thought influenced by a kind of radical relativism which denies the possibility of truth. (It also clarifies a lot of philosophy once you realize the philosophers are actually thinking about the best mind-programs to put into people, rather than seeking truth as a typical person would recognize it. If only the sneaky bastards would say so to their readers' faces. But of course it would not be pragmatic to do so.) It's a bad idea, and I sincerely hope the fields presently employing it fall into ruin, or are otherwise convinced to abandon it.

salty_biscuits 8y ago

Papers aren't meant to be this crystallized nugget of truth, they are a progress report on an ongoing piece of living research. They aren't meant to be infallible.

traverseda 8y ago

On the psychology replication crisis >Overall, 36% of the replications yielded significant findings (p value below 0.05) compared to 97% of the original studies that had significant effects. The mean effect size in the replications was approximately half the magnitude of the effects reported in the original studies. https://en.wikipedia.org/wiki/Replication_crisis#Psychology_replication_rates https://en.wikipedia.org/wiki/Replication_crisis#Psychology_... Of course if you point out that psychology research is strongly biased, the immediate response is that you must be an anti-intellectual. As if the system that produced such misleading results is a representation of all intellectual pursuits, and not just a mistake. The same way you get called an anti-intellectual for questioning the post-modern-analyses of whatever. I'm really sick of people claiming that criticizing obviously bad science is "anti-intellectual". When you produce that many useless papers you're obviously, as a field, lacking an understanding of why science is good and useful in the first place. I'm presuming that economics has similar problems with replication, and that you can only trust the most basic and obvious of their findings.

sarbaz 8y ago
A lot of this is simply that p < 0.05 is a very low bar for research. If we want to understand how the world works, we need to be able to construct mechanisms that explain why the results we measure in experiments come about. Without this thing, it's not science. A good mechanism that accurately explains some part of the world can be validated with a much better result than p < 0.05, because it can be properly isolated from other factors and its effect size can be strengthened. Without a mechanism to explain how things work, the experimental results that get published are probably just noise. And because p < 0.05 is such a low bar to pass, lots of noise gets published. Trying to reproduce an experiment whose results were drawn from /dev/urandom is pointless. In fact I'm surprised the replication was so high. I was expecting it to be around 5% (i.e. results are totally random). So maybe there is some hidden merit to all of this
- nicoburns 8y ago
  > If we want to understand how the world works, we need to be able to construct mechanisms that explain why the results we measure in experiments come about. This, 1000 times over. Unfortunately we seem to teach that statistical analysis is sufficient to infer causation in a lot our undergraduate science programs.

dekhn 8y ago

Well, Watson & Crick (Nature 1953) is that good. It does contain an error (the actual DNA structure is slightly wrong) And Avery (JEM 1944) http://jem.rupress.org/content/79/2/137 http://jem.rupress.org/content/79/2/137) which "proved" that DNA is the molecule of heredity, is also there. But to be qualified to read these papers and appreciate why they are points of quality within a sea of crap? That's hard.

acoye 8y ago

Bitcoin's paper was that good. Simple. Easy to read. Easy to replicate (a poc in python is easy) It is a recent key innovation (the blockchain tech).

coldtea 8y ago

>The best papers get up to around .20. Again, No Paper Is That Good. If you demur, consider this: In twenty years, will you still hold up the best papers of today as “paragons” or “exemplars” of compelling empirical work? If the field is not totally vague, then yes. We can still consider certain papers in physics, or chemistry, or medicine, computer science etc. as exemplary decades, or even centuries, later, even when they deal with empirical work. Soft sciences need not apply.

AnimalMuppet 8y ago
Absolutely. Others have cited Shannon's paper on information theory. I would add Einstein's paper on Special Relativity. I read a translation of it on the 100th anniversary of its release, and I learned something, even though I already knew Special Relativity. (What did I learn? I don't remember precisely - it's been 15 years - but I think it was that something on the electromagnetic side was caused by time dilation.)
UncleMeat 8y ago
> Soft sciences need not apply Why not? Historians still revere Braudel and The Mediterranean was published nearly 70 years ago. Are you certain that there are no such exemplary works outside of the fields you like?
- coldtea 8y ago
  No, but I'm certain that whether revered or not, Braudel's work is still "full of mistakes and bad reasoning" -- it's just that in history those are harder to tell.
Jweb_Guru 8y ago
Nonsense. Plenty of old computer science papers don't hold up well to scrutiny or are largely irrelevant nowadays, such as those that were trying to optimize for bottlenecks in hardware that no longer exist or have shifted to different places. Even in pure mathematics, something that was considered a great insight a long time ago might be considered fairly trivial now, and not just because the other paper came out first. Just because a paper is correct doesn't make it interesting or worthwhile--a lot of the best papers have errors in them, but propose something that's a genuinely new contribution to the field.

kabouseng 8y ago

Claude Shannon's paper on information theory [1] is possibly an example of such a paragon of work. No citations, 50 pages of pure awesome, spawned a whole research field of its own and 70 years on just as valid. [1] A Mathematical Theory of Communication - http://math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf http://math.harvard.edu/~ctm/home/text/others/shannon/entrop...

eboyjr 8y ago
You said it. He proposed the model for digital AND analog communication still in use today. Although being a digital software person I admit I've never been motivated to read the part about continuous channels. I was motivated however to install an archway for my house from the exact shape in Figure 7 (binary entropy function)
thidr0 8y ago
First paper I thought of too.
xurukefi 8y ago
> No citations The paper contains citations. But they are hidden in the footnotes - see page 1 or page 5 for example.
beagle3 8y ago
And it's easily readable with basic math/engineering knowledge. Highly recommended. Erdos has a few papers that are "that good" as long as you know all the theory leading to the new insight, but none (that I read) are as clear and freestanding as Shannon's AMToC
joker3 8y ago
Math is different from empirical work. Yes, Shannon's paper is great, but that's not really relevant here because it's in a completely different paradigm.

turc1656 8y ago

I suspect this is limited to economics and the social sciences. There is absolutely no way this holds up in the natural sciences like math, physics, etc. People forget the original name for what we call economics was "political economy". That alone should tell you all you need to know about the dangers of treating that field like a science. If you ever want to know why expert economists can't seem to agree on things that happened 50-100 years ago or make accurate predictions for the future, the original name is very telling. Wouldn't it be ridiculous if we were still debating the validity of f=ma or e=mc^2? Wouldn't it be crazy for someone to claim general relativity is just flat out wrong, even though GPS systems would not work properly without humans accepting it? Why is nothing remotely approaching reasonable standards used in the social sciences before acceptance?

nitwit005 8y ago

Following the link through to the original proposal, this seems like a misinterpretation of the suggestion. The problem this was supposed to "fix" was people using the existence of their being "vast literature" to shut down arguments. From that perspective, the suggestion seems fine. You should be able to dig out two examples that show your field isn't nonsense. I don't think it was meant to be a high bar: > There are actual examples of vast literatures that contain zero knowledge: Astrology, for instance. People have written so much about astrology that I bet you could spend decades reading what they've written and not even come close to the end. But at the end of the day, the only thing you'd know more about is the mindset of people who write about astrology. Because astrology is total and utter bunk.

gjm11 8y ago

It seems like a lot of commenters here are under the impression that the "two paper rule" (if someone says you should read the literature in field X, ask them for the two best papers in that field they can think of, and if those aren't impressive then don't bother looking further) is a proposal of Bryan Caplan, who wrote the OP here. That is incorrect. The two-paper rule is Noah Smith's, and Bryan Caplan is disagreeing with it. (It's scarcely possible to read any of Caplan's post without realising that; I conclude that many commenters here have not bothered to read the OP before commenting.)