6 ms·

How to Read a Paper (2016) [pdf]

372 points by jdale27 9y ago

rocket_woman 9y ago
On the writing side of things, I really enjoyed this talk: https://www.youtube.com/watch?v=L_6xoMjFr70 https://www.youtube.com/watch?v=L_6xoMjFr70 "How to Write Papers So People Can Read Them"
- DanAndersen 9y ago
  My advisor (Voicu Popescu) wrote a couple of nice materials about "How to Write a CS Paper". The emphasis is mostly on how to do it in the computer graphics field, but perhaps it's applicable to other CS domains too: In prose form: http://webarchiv.ethz.ch/digitalartweeks/web/uploads/DAWIntra/test.pdf http://webarchiv.ethz.ch/digitalartweeks/web/uploads/DAWIntr... In slide form: http://spaf.cerias.purdue.edu/classes/Popescu.pdf http://spaf.cerias.purdue.edu/classes/Popescu.pdf
deleted 9y ago
[deleted]
tpetricek 9y ago
This has some nice hints on how to read the text of a paper, but I think it misses all the important things that you need to be aware when you want to understand a paper. In particular, things like the research paradigm or research programme in which the paper fits, its historical context etc. I wrote a post about this recently: http://tomasp.net/blog/2017/papers-we-scrutinize/ http://tomasp.net/blog/2017/papers-we-scrutinize/
mkhalil 9y ago
I first-passed this paper, and realized I needed only a first minimal second pass. Check what to do on the pass on that I on. Very helpful for new readers.
mad44 9y ago
Another advice on the topic. http://muratbuffalo.blogspot.com/2013/07/how-i-read-research-paper.html http://muratbuffalo.blogspot.com/2013/07/how-i-read-research...
AndrewOMartin 9y ago
I was all ready to mock this by saying how most papers get only 1) Read the abstract, 2) Read the conclusion, 3) Look at the graphics, from me. Turns out that it's basically what the paper says, but then goes into more detail about going into more detail. Worth at least a second pass :)
- nickpsecurity 9y ago
  It's how I got through over 10,000 papers on IT and INFOSEC. When they're well-written, I could look at the abstract, a select bit of details, related work, and conclusion. Would take me about a minute or so a paper once I got good at it. Sometimes less. Some papers were more complex or just not well written where it took significantly more time to evaluate them. All in all, though, the speed reading methods save a ton of time.
  - deleted 9y ago
    [deleted]
  - fizixer 9y ago
    I assume you're talking about a time frame of many years.
    - nickpsecurity 9y ago
      At least six. Maybe more. Obsessively, too, as it was my grand challenge problem to solve all of it with a holistic solution. Got pretty close conceptually from a number of angles. End result was software or especially hardware options will need to be developed by specialists at significant cost. Now, I still collect the science while looking at marketing or development strategies a bit more.
  - Cyph0n 9y ago
    It typically takes much longer to actually understand the contribution of a paper in engineering fields (EE, for example). Of course, it depends on how much background you have prior to reading the paper.
f_allwein 9y ago
I sometimes do an exercise with my students where I give them a paper and ask them to tell me in 5 minutes what is the research question and the answer. Usually works well (if the paper is well written), and shows hem that they can get a good grasp of papers without spending hours reading them.
- nonbel 9y ago
  My experience from journal clubs is that strategies like this just encourages parroting whatever is claimed without thinking critically.
RichardHeart 9y ago
If papers were written better, you might not need to read them over 3 times. There should be a paper called "how to write useful headlines for your paper." "first pass" "second pass" "third pass" should be replaced with unique, useful, preferably memorable headlines. For instance: "Quick scan" "Deeper but ignore details" and "Challenge every assumption in every statement" Then you haven't wasted the attention those big bold headlines get.
- jonsterling 9y ago
  This is not true. Some of the best papers ever, I have benefited from reading many times, more than three.
mrcactu5 9y ago
these "how to read articles" make me feel so illiterate -- having completed my formal education 4 years ago. of course, the truth is we do not read closely, and there are always new signals to look into
squaredpants 9y ago
This went way too meta for my procrastinating brain.
rectang 9y ago
I'd like to present a fourth option, the "0-pass": Don't read most papers. Don't feel bad about not reading them, because in general they are terribly written. Instead, read follow-on work which resynthesizes the ideas in these papers for a popular audience.
- DonbunEf7 9y ago
  You have chosen the path of the laity.
- Kepler-239b 9y ago
  Agreed. Most research papers are a headache to read, even in a field you are familiar with. The full list of reasons is here: http://www.wikipaper.org/p/Why_So_Difficult http://www.wikipaper.org/p/Why_So_Difficult The site above ^^ has not launched yet, but should help alleviate the problem.
  - a3n 9y ago
    The site has launched far enough to attract spammers. It looks like you've got some cleaning and monitoring to do: http://www.wikipaper.org/p/Special:AllPages http://www.wikipaper.org/p/Special:AllPages Bookmarked your site, looks interesting for this non-academic.
- CPUstring 9y ago
  One of the key questions after the first pass is "is this well-written enough to keep reading?"
c0achmcguirk 9y ago
This part was most interesting to me: "Incidentally, when you write a paper, you can expect most reviewers (and readers) to make only one pass over it." I understand reviewers are busy, but we depend on peer review to filter out bad or poorly-researched material. I don't think one pass is enough. Obviously, so does the author.
- seanmcdirmid 9y ago
  Depending on the conference or journal, most papers can be rejected in one pass. Heck, the reviewer might abort after the intro if the paper is particularly bad. The papers that make it past one pass get more scrutiny.
  - UncleMeat 9y ago
    I've gotten reviews where it was obvious the reviewer didn't read the paper. "You didn't address X at all" when I had a section heading under Discussion labeled X in big bold font. The review process is stochastic and leans towards reject by default.
- axlprose 9y ago
  Indeed, but then this is somewhat to be expected when the peer review process itself doesn't have much in the way of incentives for the reviewers. Journals ask reviewers to voluntarily put their own work on hold for a potentially non-trivial period of time to grok a non-stop fire hose of papers, just so their and the original authors' efforts end up buried under a mountain of other research behind some absurdly expensive paywall later. And if you regularly slip up on deadlines or opt not to play along with these requests for some reason or another, good luck maintaining good relations with that journal in the future for your own papers. Morals and civic duty aside, it's hard to blame reviewers for skimming or 'phoning it in' under such conditions. With all these subtle politics and screwy incentive structures, I'm unfortunately not surprised at all that science is having a replication crisis.
- ISL 9y ago
  More practically, one should not expect that a peer-reviewed paper is correct. For a YC-compatible analogy: think of it as code review. Pure junk gets filtered out, but bugs inevitably remain.
sangd 9y ago
I only got to 2nd pass for this paper.
- tedmiston 9y ago
  I found it interesting in a meta sort of way that the paper's first pass ctechnique can't be applied to itself (there's no conclusion).
sytelus 9y ago
For people in the field who are active authors and reviewers in CS/AI related areas: - How many papers do you read per week? - How many hours you spent per week?
spookyuser 9y ago
Obviously a different set of circumstances, but I would be curious to hear what people think about Cal Newport's Question; Evidence; Conclusion - method of reading. From the grade A students guide. I recently switched to it and found myself understanding reading assignments much better. It seems like this method is more geared for Researchers though.
haffi112 9y ago
> The first version of this document was drafted by my students: Hossein Falaki, Earl Oliver, and Sumair Ur Rahman. The paper is two pages. Why aren't these people included as authors then...?
bfirsh 9y ago
I thought that was going to be Trisha Greenhalgh's How to Read a Paper: https://www.amazon.com/How-Read-Paper-Evidence-Based-Medicine/dp/1118800966/ref=sr_1_1?ie=UTF8&qid=1493543273&sr=8-1 https://www.amazon.com/How-Read-Paper-Evidence-Based-Medicin... Superficially the same idea, but it is for advising medical practitioners on how to apply research to their work. It goes about this by showing you how to find research, critique it, analyse it, use meta-research, and so on. For somebody not in medicine, it had some transferable advice on how to use research in practice, but was mainly a detailed insight into how evidence-based medicine works. Highly recommended.
darkmorning 9y ago
I see a recursion in this post :p , Before learning to read other papers, could you please explain how to read this paper :D
unheroic 9y ago
Well this would have been very useful 6 months ago, when I started working on my undergraduate dissertation.

372 points by jdale27 9y ago

On the writing side of things, I really enjoyed this talk: https://www.youtube.com/watch?v=L_6xoMjFr70 https://www.youtube.com/watch?v=L_6xoMjFr70 "How to Write Papers So People Can Read Them"

DanAndersen 9y ago
My advisor (Voicu Popescu) wrote a couple of nice materials about "How to Write a CS Paper". The emphasis is mostly on how to do it in the computer graphics field, but perhaps it's applicable to other CS domains too: In prose form: http://webarchiv.ethz.ch/digitalartweeks/web/uploads/DAWIntra/test.pdf http://webarchiv.ethz.ch/digitalartweeks/web/uploads/DAWIntr... In slide form: http://spaf.cerias.purdue.edu/classes/Popescu.pdf http://spaf.cerias.purdue.edu/classes/Popescu.pdf

deleted 9y ago

[deleted]

tpetricek 9y ago

This has some nice hints on how to read the text of a paper, but I think it misses all the important things that you need to be aware when you want to understand a paper. In particular, things like the research paradigm or research programme in which the paper fits, its historical context etc. I wrote a post about this recently: http://tomasp.net/blog/2017/papers-we-scrutinize/ http://tomasp.net/blog/2017/papers-we-scrutinize/

mkhalil 9y ago

I first-passed this paper, and realized I needed only a first minimal second pass. Check what to do on the pass on that I on. Very helpful for new readers.

mad44 9y ago

Another advice on the topic. http://muratbuffalo.blogspot.com/2013/07/how-i-read-research-paper.html http://muratbuffalo.blogspot.com/2013/07/how-i-read-research...

AndrewOMartin 9y ago

I was all ready to mock this by saying how most papers get only 1) Read the abstract, 2) Read the conclusion, 3) Look at the graphics, from me. Turns out that it's basically what the paper says, but then goes into more detail about going into more detail. Worth at least a second pass :)

nickpsecurity 9y ago
It's how I got through over 10,000 papers on IT and INFOSEC. When they're well-written, I could look at the abstract, a select bit of details, related work, and conclusion. Would take me about a minute or so a paper once I got good at it. Sometimes less. Some papers were more complex or just not well written where it took significantly more time to evaluate them. All in all, though, the speed reading methods save a ton of time.
- deleted 9y ago
  [deleted]
- fizixer 9y ago
  I assume you're talking about a time frame of many years.
  - nickpsecurity 9y ago
    At least six. Maybe more. Obsessively, too, as it was my grand challenge problem to solve all of it with a holistic solution. Got pretty close conceptually from a number of angles. End result was software or especially hardware options will need to be developed by specialists at significant cost. Now, I still collect the science while looking at marketing or development strategies a bit more.
- Cyph0n 9y ago
  It typically takes much longer to actually understand the contribution of a paper in engineering fields (EE, for example). Of course, it depends on how much background you have prior to reading the paper.

f_allwein 9y ago

I sometimes do an exercise with my students where I give them a paper and ask them to tell me in 5 minutes what is the research question and the answer. Usually works well (if the paper is well written), and shows hem that they can get a good grasp of papers without spending hours reading them.

nonbel 9y ago
My experience from journal clubs is that strategies like this just encourages parroting whatever is claimed without thinking critically.

RichardHeart 9y ago

If papers were written better, you might not need to read them over 3 times. There should be a paper called "how to write useful headlines for your paper." "first pass" "second pass" "third pass" should be replaced with unique, useful, preferably memorable headlines. For instance: "Quick scan" "Deeper but ignore details" and "Challenge every assumption in every statement" Then you haven't wasted the attention those big bold headlines get.

jonsterling 9y ago
This is not true. Some of the best papers ever, I have benefited from reading many times, more than three.

mrcactu5 9y ago

these "how to read articles" make me feel so illiterate -- having completed my formal education 4 years ago. of course, the truth is we do not read closely, and there are always new signals to look into

squaredpants 9y ago

This went way too meta for my procrastinating brain.

rectang 9y ago

I'd like to present a fourth option, the "0-pass": Don't read most papers. Don't feel bad about not reading them, because in general they are terribly written. Instead, read follow-on work which resynthesizes the ideas in these papers for a popular audience.

DonbunEf7 9y ago
You have chosen the path of the laity.
Kepler-239b 9y ago
Agreed. Most research papers are a headache to read, even in a field you are familiar with. The full list of reasons is here: http://www.wikipaper.org/p/Why_So_Difficult http://www.wikipaper.org/p/Why_So_Difficult The site above ^^ has not launched yet, but should help alleviate the problem.
- a3n 9y ago
  The site has launched far enough to attract spammers. It looks like you've got some cleaning and monitoring to do: http://www.wikipaper.org/p/Special:AllPages http://www.wikipaper.org/p/Special:AllPages Bookmarked your site, looks interesting for this non-academic.
CPUstring 9y ago
One of the key questions after the first pass is "is this well-written enough to keep reading?"

c0achmcguirk 9y ago

This part was most interesting to me: "Incidentally, when you write a paper, you can expect most reviewers (and readers) to make only one pass over it." I understand reviewers are busy, but we depend on peer review to filter out bad or poorly-researched material. I don't think one pass is enough. Obviously, so does the author.

seanmcdirmid 9y ago
Depending on the conference or journal, most papers can be rejected in one pass. Heck, the reviewer might abort after the intro if the paper is particularly bad. The papers that make it past one pass get more scrutiny.
- UncleMeat 9y ago
  I've gotten reviews where it was obvious the reviewer didn't read the paper. "You didn't address X at all" when I had a section heading under Discussion labeled X in big bold font. The review process is stochastic and leans towards reject by default.
axlprose 9y ago
Indeed, but then this is somewhat to be expected when the peer review process itself doesn't have much in the way of incentives for the reviewers. Journals ask reviewers to voluntarily put their own work on hold for a potentially non-trivial period of time to grok a non-stop fire hose of papers, just so their and the original authors' efforts end up buried under a mountain of other research behind some absurdly expensive paywall later. And if you regularly slip up on deadlines or opt not to play along with these requests for some reason or another, good luck maintaining good relations with that journal in the future for your own papers. Morals and civic duty aside, it's hard to blame reviewers for skimming or 'phoning it in' under such conditions. With all these subtle politics and screwy incentive structures, I'm unfortunately not surprised at all that science is having a replication crisis.
ISL 9y ago
More practically, one should not expect that a peer-reviewed paper is correct. For a YC-compatible analogy: think of it as code review. Pure junk gets filtered out, but bugs inevitably remain.

sangd 9y ago

I only got to 2nd pass for this paper.

tedmiston 9y ago
I found it interesting in a meta sort of way that the paper's first pass ctechnique can't be applied to itself (there's no conclusion).

sytelus 9y ago

For people in the field who are active authors and reviewers in CS/AI related areas: - How many papers do you read per week? - How many hours you spent per week?

spookyuser 9y ago

Obviously a different set of circumstances, but I would be curious to hear what people think about Cal Newport's Question; Evidence; Conclusion - method of reading. From the grade A students guide. I recently switched to it and found myself understanding reading assignments much better. It seems like this method is more geared for Researchers though.

haffi112 9y ago

> The first version of this document was drafted by my students: Hossein Falaki, Earl Oliver, and Sumair Ur Rahman. The paper is two pages. Why aren't these people included as authors then...?

bfirsh 9y ago

I thought that was going to be Trisha Greenhalgh's How to Read a Paper: https://www.amazon.com/How-Read-Paper-Evidence-Based-Medicine/dp/1118800966/ref=sr_1_1?ie=UTF8&qid=1493543273&sr=8-1 https://www.amazon.com/How-Read-Paper-Evidence-Based-Medicin... Superficially the same idea, but it is for advising medical practitioners on how to apply research to their work. It goes about this by showing you how to find research, critique it, analyse it, use meta-research, and so on. For somebody not in medicine, it had some transferable advice on how to use research in practice, but was mainly a detailed insight into how evidence-based medicine works. Highly recommended.

darkmorning 9y ago

I see a recursion in this post :p , Before learning to read other papers, could you please explain how to read this paper :D

unheroic 9y ago

Well this would have been very useful 6 months ago, when I started working on my undergraduate dissertation.