6 ms·

TLDR: Extreme Summarization of Scientific Documents

154 points by lopespm 6y ago

behnamoh 6y ago
While there are lots of TLDR websites out there, I want to know how this one is different from them. I get it; many scientific papers are to some extent bs, and many are just wrong. For PhDs, it's a hassle to go through all of that bs to find something that is actually true. I feel like PhDs basically have to spend hundreds of hours reading papers that don't really benefit them. Tools like this could probably help with that, but as long as scientific success is measured by how may papers you've published and/or how long your papers are, I don't see any hope of actually doing science in the coming years when the academia will be essentially "saturated" with papers.
- justarandomq 6y ago
  You could link other TLDR tools and then we could read the paper to learn how they're different.
- tyingq 6y ago
  It's Apache licensed and on github, which is substantially different from a lot of systems I see described in papers.
- visarga 6y ago
  Never saw a paper lauded for its long length. Maybe the authors would have written shorter papers but they didn't have enough time.
  - bjornsing 6y ago
    No? Turing 1936 is well known as a lengthy paper. (Just kidding.)
DrNuke 6y ago
I am afraid we will still need humans actually going through papers to get nuances, original contributions, if any, and wider narratives? (plug: the content project in my profile)
Der_Einzige 6y ago
This is very neat work, and I would never have predicted that abstractive summarization would end up advancing so much more quickly in general than extractive summarization did from transformers being introduced. Makes me wish that simple highlighting of a document at the word-level was actually a sorta "solved" (gives compelling output more often then not) problem like condensed abstractive summarization is...
deleted 6y ago
[deleted]
justarandomq 6y ago
What you get from applying TLDR to their paper: We introduce SCITLDR, a new multi-target data set of 5.4KTLDRs over 3.2Kpapers. Keeping pdf's copy-paste artifacts: We introduceTLDRgeneration, a new formof extreme extreme summarization, for scientific pa-pers. Adding intro and conclusion (optional): We introduce SCITLDR, a new data set of 5.4KTLDRs over 3.2Kpapers. [0] https://scitldr.apps.allenai.org/ https://scitldr.apps.allenai.org/
- m1sta_ 6y ago
  Yikes
- austinjp 6y ago
  Yeah. I happen to have been looking at this problem in my spare time recently. I tried a bunch of abstractive AIs and approaches, and none produce consistently usable results. I'm sticking with extractive approaches plus a bunch of hard-coded general and domain-specific rules for now.
pcrh 6y ago
This kind of effort serves the function of helping people to _approximately_ "know what is known", but it's really not very useful to the more important part of research efforts, which is to know what is not known.
- s4n1ty 6y ago
  It could potentially be used with a novelty search algorithm, finding papers that are plausible but sufficiently dissimilar to anything in the training set.
- thereisnospork 6y ago
  A large part of research is spent on the understanding of what is known; parsing papers is part and parcel for professors, grad students, and corporate R+D alike. No idea if their approach is useful, but they are tackling a worthwhile problem.
  - visarga 6y ago
    > No idea if their approach is useful, but they are tackling a worthwhile problem. When there are 1000+ papers every week in your field you need some advanced tools. It's hard to read everything, it's O(N).
petercooper 6y ago
Oh! I listened to Scott Hanselman interview one of the authors of this paper on his podcast the other day. It might interest some of you as she explains it all in a very accessible way: https://www.hanselminutes.com/763/tldr-extreme-summarization-as-a-service-with-isabel-cachola https://www.hanselminutes.com/763/tldr-extreme-summarization...
- swyx 6y ago
  (hey peter!) I did get very excited about it when I heard about it, but cooled significantly when I learned that it was trained only on cs papers and required the full abstract plus paper text. nice proof of concept but going to need significant work to generalize for, say, a newsletter business like yours
  - petercooper 6y ago
    Ah yeah, I definitely wouldn't want to use this sort of thing on newsletters. I'm against automated curation generally. It's not what we're about :-)
etaioinshrdlu 6y ago
I played around with this on the demo page and found that while the generated "TLDR" are pretty good, they tend to generate sentences composed of fragments of existing sentences. Basically, it seems vaguely extractive in nature. Never did I see it summarize a concept in new words, or try to dumb down a complicated concept further than the original paper did. Given the results of GPT3 I would think that it should be possible to do much better by now, at least with enough data and compute time.
- arolihas 6y ago
  I think you’re underestimating how hard what you’re describing is. GPT-3 can mimic the language of reasoning but that doesn’t mean it’s capable of higher order reasoning.
  - FL33TW00D 6y ago
    After reading through https://www.gwern.net/GPT-3 https://www.gwern.net/GPT-3 , I suspect that GPT-3 is capable of higher order reasoning, given the right motivation (prompt).
    - arolihas 6y ago
      It’s impressive but doesn’t the need for a “good” prompt kind of show it doesn’t have the strong reasoning required to do the task you’re describing? Also there’s some interesting critique here https://www.lesswrong.com/posts/ZHrpjDc3CepSeeBuE/gpt-3-a-disappointing-paper https://www.lesswrong.com/posts/ZHrpjDc3CepSeeBuE/gpt-3-a-di...
- msamwald 6y ago
  Being very efficient at mostly extractive summarization and abstaining from abstractive summarization does seem a better bet though, because fewer things can go wrong and it is easier to check the summaries against the full text.
ultrastito 6y ago
That's what abstract are for
- eghad 6y ago
  Abstracts are important (and clearly key in generating these TLDRs), but when it comes to ranking and recommending other papers (not to mention noting whether a new paper has content that can actually push a field forward) an abstract just isn't enough.
- justarandomq 6y ago
  After some thought I agree with you that this is the wrong problem to solve. I took a narrative detour I wanted to share: Suppose we make the analogue of a scientific paper to a piece of mineral ore (in terms of their raw content, and without written symbols in them for the sake of the analogy) extracted from some mine or quarry. This ore is somehow useful to someone, even if its value is structural: the shingles on an academic roof or a heavyweight desk. What a summarizer attempts to do is use a generic refinement process that will grind up the ore and then separate the components of interest such as Iron, Uranium, or Gold. Anyone thinking that all of metallurgy reduces to simply throwing the slab into a machine and have it spew out the precious metals will find, instead, more complexity than they bargained for, and have more questions on machines or methods to resolve. Gold, Iron, Uranium, all have different extraction process. I believe this approach may give some insight in what problems to solve instead with AI: focus on those discoveries that have helped advance "metallurgy", those of discovering and understanding the structure of the mineral ore and contents (scientific papers) and their relation with current technologies at the time, not on the philosopher's stone of 'summarizing' process more akin to a hammer that makes everything seem like a nail.
  - kreeben 6y ago
    >> this is the wrong problem to solve Highly intelligent human beings have a natural ability to summarize big ideas into TLDRs. Are humans basically a bunch of "summarizers"? Probably not. Is this ability to summarize or compress big ideas into smaller, more condensed pieces of information, important to the human race? Yes, I would say that they are. So to me, this is certainly one of those problems that we correctly attempt to solve.
keyle 6y ago
Not to be mistaken with tldr, the command line utility that makes man pages readable and fun ;)
pixiemaster 6y ago
English only, as usual for NLG
- qayxc 6y ago
  In this particular case it's excusable as English is the Lingua Franca of science today. Used to be Latin, then French and German, now it's English. No big deal IMO. In fact I kind of like the way this is going since it represents a fantastic opportunity for NL researchers to stand out simply by publishing research and corpora focused exclusively on low-resource languages and non-English/Mandarin in general. It is also important to note that most of the ML research in the field is pretty much language agnostic and is concerned with general concept such as efficient en-/decoding [1], training methods [2], and even stealing pre-trained weights from APIs (like GPT-2 or even 3) without paying for training [3] :) It's just easier to get your hands on and verify English corpora, results and pre-trained models for reproducibility than say Mongolian or Gaelic so that's a factor, too. [1] https://arxiv.org/pdf/1904.09751.pdf https://arxiv.org/pdf/1904.09751.pdf [2] https://arxiv.org/pdf/2003.10555.pdf https://arxiv.org/pdf/2003.10555.pdf [3] https://arxiv.org/pdf/1910.12366.pdf https://arxiv.org/pdf/1910.12366.pdf
Thorncorona 6y ago
Very cool. Consider the paper's abstract: "We introduce TLDR generation, a new form of extreme summarization, for scientific papers. TLDR generation involves high source compression and requires expert background knowledge and understanding of complex domain-specific language. To facilitate study on this task, we introduce SciTLDR, a new multi-target dataset of 5.4K TLDRs over 3.2K papers. SciTLDR contains both author-written and expert-derived TLDRs, where the latter are collected using a novel annotation protocol that produces high-quality summaries while minimizing annotation burden. We propose CATTS, a simple yet effective learning strategy for generating TLDRs that exploits titles as an auxiliary training signal. CATTS improves upon strong baselines under both automated metrics and human evaluations. Data and code are publicly available at this https URL." The algorithm summarizes it as: “We introduce TLDR generation, a new form of extreme summarization, for scientific papers that produces high-quality summaries while minimizing annotation burden.”
anonymousDan 6y ago
This is potentially a godsend for me - I was faced with having to write a 1 paragraph summary of 70 student dissertations for an accreditation process next week. Abstracts are too long. Fingers crossed it works as advertised!
johnnujler 6y ago
Not sure what I was expecting. It gave me back the first line of the abstract as response. (For anyone wondering, the paper I tried was: "A Heterarchy of values determined by the topology of nervous nets" by Warren S. McCulloch.)
newintellectual 6y ago
Try it on a few real abstracts. It appears to pull a somewhat random sentence and clips it. It really sucks.
schuke 6y ago
“Why use lot word when few word do trick.”

154 points by lopespm 6y ago

While there are lots of TLDR websites out there, I want to know how this one is different from them. I get it; many scientific papers are to some extent bs, and many are just wrong. For PhDs, it's a hassle to go through all of that bs to find something that is actually true. I feel like PhDs basically have to spend hundreds of hours reading papers that don't really benefit them. Tools like this could probably help with that, but as long as scientific success is measured by how may papers you've published and/or how long your papers are, I don't see any hope of actually doing science in the coming years when the academia will be essentially "saturated" with papers.

justarandomq 6y ago
You could link other TLDR tools and then we could read the paper to learn how they're different.
tyingq 6y ago
It's Apache licensed and on github, which is substantially different from a lot of systems I see described in papers.
visarga 6y ago
Never saw a paper lauded for its long length. Maybe the authors would have written shorter papers but they didn't have enough time.
- bjornsing 6y ago
  No? Turing 1936 is well known as a lengthy paper. (Just kidding.)

DrNuke 6y ago

I am afraid we will still need humans actually going through papers to get nuances, original contributions, if any, and wider narratives? (plug: the content project in my profile)

Der_Einzige 6y ago

This is very neat work, and I would never have predicted that abstractive summarization would end up advancing so much more quickly in general than extractive summarization did from transformers being introduced. Makes me wish that simple highlighting of a document at the word-level was actually a sorta "solved" (gives compelling output more often then not) problem like condensed abstractive summarization is...

deleted 6y ago

[deleted]

justarandomq 6y ago

What you get from applying TLDR to their paper: We introduce SCITLDR, a new multi-target data set of 5.4KTLDRs over 3.2Kpapers. Keeping pdf's copy-paste artifacts: We introduceTLDRgeneration, a new formof extreme extreme summarization, for scientific pa-pers. Adding intro and conclusion (optional): We introduce SCITLDR, a new data set of 5.4KTLDRs over 3.2Kpapers. [0] https://scitldr.apps.allenai.org/ https://scitldr.apps.allenai.org/

m1sta_ 6y ago
Yikes
austinjp 6y ago
Yeah. I happen to have been looking at this problem in my spare time recently. I tried a bunch of abstractive AIs and approaches, and none produce consistently usable results. I'm sticking with extractive approaches plus a bunch of hard-coded general and domain-specific rules for now.

pcrh 6y ago

This kind of effort serves the function of helping people to _approximately_ "know what is known", but it's really not very useful to the more important part of research efforts, which is to know what is not known.

s4n1ty 6y ago
It could potentially be used with a novelty search algorithm, finding papers that are plausible but sufficiently dissimilar to anything in the training set.
thereisnospork 6y ago
A large part of research is spent on the understanding of what is known; parsing papers is part and parcel for professors, grad students, and corporate R+D alike. No idea if their approach is useful, but they are tackling a worthwhile problem.
- visarga 6y ago
  > No idea if their approach is useful, but they are tackling a worthwhile problem. When there are 1000+ papers every week in your field you need some advanced tools. It's hard to read everything, it's O(N).

petercooper 6y ago

Oh! I listened to Scott Hanselman interview one of the authors of this paper on his podcast the other day. It might interest some of you as she explains it all in a very accessible way: https://www.hanselminutes.com/763/tldr-extreme-summarization-as-a-service-with-isabel-cachola https://www.hanselminutes.com/763/tldr-extreme-summarization...

swyx 6y ago
(hey peter!) I did get very excited about it when I heard about it, but cooled significantly when I learned that it was trained only on cs papers and required the full abstract plus paper text. nice proof of concept but going to need significant work to generalize for, say, a newsletter business like yours
- petercooper 6y ago
  Ah yeah, I definitely wouldn't want to use this sort of thing on newsletters. I'm against automated curation generally. It's not what we're about :-)

etaioinshrdlu 6y ago

I played around with this on the demo page and found that while the generated "TLDR" are pretty good, they tend to generate sentences composed of fragments of existing sentences. Basically, it seems vaguely extractive in nature. Never did I see it summarize a concept in new words, or try to dumb down a complicated concept further than the original paper did. Given the results of GPT3 I would think that it should be possible to do much better by now, at least with enough data and compute time.

arolihas 6y ago
I think you’re underestimating how hard what you’re describing is. GPT-3 can mimic the language of reasoning but that doesn’t mean it’s capable of higher order reasoning.
- FL33TW00D 6y ago
  After reading through https://www.gwern.net/GPT-3 https://www.gwern.net/GPT-3 , I suspect that GPT-3 is capable of higher order reasoning, given the right motivation (prompt).
  - arolihas 6y ago
    It’s impressive but doesn’t the need for a “good” prompt kind of show it doesn’t have the strong reasoning required to do the task you’re describing? Also there’s some interesting critique here https://www.lesswrong.com/posts/ZHrpjDc3CepSeeBuE/gpt-3-a-disappointing-paper https://www.lesswrong.com/posts/ZHrpjDc3CepSeeBuE/gpt-3-a-di...
msamwald 6y ago
Being very efficient at mostly extractive summarization and abstaining from abstractive summarization does seem a better bet though, because fewer things can go wrong and it is easier to check the summaries against the full text.

ultrastito 6y ago

That's what abstract are for

eghad 6y ago
Abstracts are important (and clearly key in generating these TLDRs), but when it comes to ranking and recommending other papers (not to mention noting whether a new paper has content that can actually push a field forward) an abstract just isn't enough.
justarandomq 6y ago
After some thought I agree with you that this is the wrong problem to solve. I took a narrative detour I wanted to share: Suppose we make the analogue of a scientific paper to a piece of mineral ore (in terms of their raw content, and without written symbols in them for the sake of the analogy) extracted from some mine or quarry. This ore is somehow useful to someone, even if its value is structural: the shingles on an academic roof or a heavyweight desk. What a summarizer attempts to do is use a generic refinement process that will grind up the ore and then separate the components of interest such as Iron, Uranium, or Gold. Anyone thinking that all of metallurgy reduces to simply throwing the slab into a machine and have it spew out the precious metals will find, instead, more complexity than they bargained for, and have more questions on machines or methods to resolve. Gold, Iron, Uranium, all have different extraction process. I believe this approach may give some insight in what problems to solve instead with AI: focus on those discoveries that have helped advance "metallurgy", those of discovering and understanding the structure of the mineral ore and contents (scientific papers) and their relation with current technologies at the time, not on the philosopher's stone of 'summarizing' process more akin to a hammer that makes everything seem like a nail.
- kreeben 6y ago
  >> this is the wrong problem to solve Highly intelligent human beings have a natural ability to summarize big ideas into TLDRs. Are humans basically a bunch of "summarizers"? Probably not. Is this ability to summarize or compress big ideas into smaller, more condensed pieces of information, important to the human race? Yes, I would say that they are. So to me, this is certainly one of those problems that we correctly attempt to solve.

keyle 6y ago

Not to be mistaken with tldr, the command line utility that makes man pages readable and fun ;)

pixiemaster 6y ago

English only, as usual for NLG

qayxc 6y ago
In this particular case it's excusable as English is the Lingua Franca of science today. Used to be Latin, then French and German, now it's English. No big deal IMO. In fact I kind of like the way this is going since it represents a fantastic opportunity for NL researchers to stand out simply by publishing research and corpora focused exclusively on low-resource languages and non-English/Mandarin in general. It is also important to note that most of the ML research in the field is pretty much language agnostic and is concerned with general concept such as efficient en-/decoding [1], training methods [2], and even stealing pre-trained weights from APIs (like GPT-2 or even 3) without paying for training [3] :) It's just easier to get your hands on and verify English corpora, results and pre-trained models for reproducibility than say Mongolian or Gaelic so that's a factor, too. [1] https://arxiv.org/pdf/1904.09751.pdf https://arxiv.org/pdf/1904.09751.pdf [2] https://arxiv.org/pdf/2003.10555.pdf https://arxiv.org/pdf/2003.10555.pdf [3] https://arxiv.org/pdf/1910.12366.pdf https://arxiv.org/pdf/1910.12366.pdf

Thorncorona 6y ago

Very cool. Consider the paper's abstract: "We introduce TLDR generation, a new form of extreme summarization, for scientific papers. TLDR generation involves high source compression and requires expert background knowledge and understanding of complex domain-specific language. To facilitate study on this task, we introduce SciTLDR, a new multi-target dataset of 5.4K TLDRs over 3.2K papers. SciTLDR contains both author-written and expert-derived TLDRs, where the latter are collected using a novel annotation protocol that produces high-quality summaries while minimizing annotation burden. We propose CATTS, a simple yet effective learning strategy for generating TLDRs that exploits titles as an auxiliary training signal. CATTS improves upon strong baselines under both automated metrics and human evaluations. Data and code are publicly available at this https URL." The algorithm summarizes it as: “We introduce TLDR generation, a new form of extreme summarization, for scientific papers that produces high-quality summaries while minimizing annotation burden.”

anonymousDan 6y ago

This is potentially a godsend for me - I was faced with having to write a 1 paragraph summary of 70 student dissertations for an accreditation process next week. Abstracts are too long. Fingers crossed it works as advertised!

johnnujler 6y ago

Not sure what I was expecting. It gave me back the first line of the abstract as response. (For anyone wondering, the paper I tried was: "A Heterarchy of values determined by the topology of nervous nets" by Warren S. McCulloch.)

newintellectual 6y ago

Try it on a few real abstracts. It appears to pull a somewhat random sentence and clips it. It really sucks.

schuke 6y ago

“Why use lot word when few word do trick.”