8 ms·
Text Summarization with TensorFlow
- ProxCoques 10y agoThat reminds me. Whatever happened to http://summly.com http://summly.com?
- kornish 10y agoAcquired by Yahoo a few years ago. https://www.crunchbase.com/organization/summly#/entity https://www.crunchbase.com/organization/summly#/entity
- andreasklinger 10y agoalso: wasn't it just a frontend to a external summarize service (that's what i recall)
- kornish 10y agoI don't believe so, no. They did most of the original NLP through a contracted team, according to Wikipedia: https://en.wikipedia.org/wiki/Nick_D%27Aloisio#Summly https://en.wikipedia.org/wiki/Nick_D%27Aloisio#Summly.
- andreasklinger 10y agogot it! remembered it wrong - thx!
- 8ig8 10y agoThere's SUMMRY (http://smmry.com http://smmry.com). I don't recall it being very smart -- that is, it doesn't rewrite sentences. It extracts the most important/relevant ones to basically shorten an article. It's definitely useful.
- mikebotazzo 10y agoWow, I like this. As an example, here is the Google article resumed by SUMMRY. === Research Blog: Text summarization with TensorFlow Being able to develop Machine Learning models that can automatically deliver accurate summaries of longer text can be useful for digesting such large amounts of information in a compressed form, and is a long-term goal of the Google Brain team. One approach to summarization is to extract parts of the document that are deemed interesting by some metric and join them to form a summary. Above we extract the words bolded in the original text and concatenate them to form a summary. It turns out for shorter texts, summarization can be learned end-to-end with a deep learning technique called sequence-to-sequence learning, similar to what makes Smart Reply for Inbox possible. In this case, the model reads the article text and writes a suitable headline. In those tasks training from scratch with this model architecture does not do as well as some other techniques we're researching, but it serves as a baseline. We hope this release can also serve as a baseline for others in their summarization research.
- iLoch 10y agoFYI, their about section also spells out their algorithm: http://smmry.com/about http://smmry.com/about
- kilroy123 10y agoThis is what the tldr; bot on reddit uses. It's pretty darn good most of the time, and useful.
- hyperbovine 10y agoThe table is nice, but I'd like to see examples where it performs poorly as well.
- pierrec 10y agoAgreed, it seems they really hand-picked some shining examples for this post, and it would have been more interesting to see the full spectrum of when it works and when it doesn't. Perhaps the README in the Github repo is a bit more honest in terms of representativeness, though it only has 4 examples, one of them is an interesting failure: article: novell inc. chief executive officer eric schmidt has been named chairman of the internet search-engine company google . human: novell ceo named google chairman machine: novell chief executive named to head internet company https://github.com/tensorflow/models/tree/master/textsum https://github.com/tensorflow/models/tree/master/textsum (It's still great that it beats all competition on the ROUGE score, of course.)
- hisham_hm 10y agoI don't see that as a failure. It did produce a sentence that is shorter, grammatical (though "named to head" is a bit weird) and essentially true — calling Google an "internet company" would make sense in its early days (back when Google would be prefixed by "internet search-engine company").
- spicyj 10y agoMost humans would interpret "novell chief executive named to head internet company" to mean "novell is an internet company and its chief executive just became its head" which is incorrect (and a little nonsensical since the CEO already is in change).
- nl 10y agoThat's pretty interesting. It's taken me 5 readings of that sentence, including once out-loud to get your reading of it. I thought the generated summary was really, really good. But I knew that Novell wasn't considered an internet company, so it wasn't until I made myself ignore that before I could see the other reading.
- Cozumel 10y agoI wonder what it'd be like on a novel, say something like 'pride and prejudice', would it be able to essentially summarise the plot or would it end up like 'movie plots explained badly' Either way, this is great research with a ton of real world applications!
- peterjlee 10y agoIt'll be more remarkable if a machine can read a play script and summarize the plot.
- duaneb 10y agoI doubt most humans can even do this well.
- asib 10y ago> Although this task serves as a nice proof-of-concept, we started looking at more difficult datasets where reading the entire document is necessary to produce good summaries. In those tasks training from scratch with this model architecture does not do as well as some other techniques we’re researching, but it serves as a baseline. That would suggest that this method doesn't work well for long documents.
- polm23 10y agoA student in a course I TA'd in college used Microsoft Word's built-in summarization to reduce books to ten words. The results were hilarious. http://jason-huff.com/projects/autosummarize/ http://jason-huff.com/projects/autosummarize/
- nl 10y agoby classifying the emotional arcs for a filtered subset of 1,737 stories from Project Gutenberg's fiction collection, we find a set of six core trajectories which form the building blocks of complex narratives. We strengthen our findings by separately applying optimization, linear decomposition, supervised learning, and unsupervised learning http://arxiv.org/abs/1606.07772 http://arxiv.org/abs/1606.07772
- trengrj 10y ago$6,000 for the Gigaword dataset they used to train... https://catalog.ldc.upenn.edu/LDC2012T21 https://catalog.ldc.upenn.edu/LDC2012T21
- emmab 10y agoThere's not even a way to buy it as an individual who's not part of an organization! They don't even state this as a possibility.
- schmit 10y agoFunny how Google is paying for data
- peterjliu 10y agoThe code can be used to train on other data. All you really need is a collection of news articles. I think there are some free ones available. This dataset was only used to benchmark against other published results. It was first proposed in https://arxiv.org/abs/1509.00685 https://arxiv.org/abs/1509.00685.
- gravypod 10y agoI remember there was a hackernews comment I read once where the user created an emacs plugin to do just this. It would generate a single sentence summery about whatever text was input into it. Edit: Hey, found it! https://github.com/mck-/oneliner https://github.com/mck-/oneliner Probably not as sophisticated but does the business. Nice bit of work done on this.
- deleted 10y ago[deleted]
- fiatjaf 10y agoThis is way too difficult to get right, even for humans. I dream of a not-so-smart news summarization engine that will not try to rewrite the news, but only pickup all the numbers and quotations, then present them in a table of who-said-what and how-many-what, along with the title. This would put an end to filler-based journalism.
- hyperbovine 10y agoI wish you wouldn't be so dismissive of journalism and journalists. What they provide is not filler. Controversial though this opinion may be around here, there is serious value in having an actual carbon-based life form -- one who has spent years or decades covering whatever beat -- provide context and insight for the quotes and data. That they have become a dying breed spells real trouble for our civic life.
- hueving 10y agoThe journalists you describe are so few and far between that there needs to be a different term for them. The vast majority of the 'news' out there isn't anything close to what you described.
- collyw 10y agoAgreed in an age where twitter storms are worthy of front page news.
- ramblerman 10y agoThe system is not set up to reward them, if there was a genuine demand we might see more of them. The unfortunate truth is that your clickbait and byte sized arguments are what people want. Trying to solve it from the top down is a lost cause.
- evgen 10y agoThe HN crowd does not need context and insight, they can just google a few keywords and then skim a wikipedia article to achieve the expertise necessary to argue with others on a public forum...
- paulsutter 10y agoNext one could turn this into a comment generator, generating comments in the style and personality of any HN commenters with an adequate corpus.
- polm23 10y agoThat would only be accurate if you assume most people read the article.
- deleted 10y ago[deleted]
- deleted 10y ago[deleted]
- blackkettle 10y agoThe subredditsimulator subreddit does this for both articles and comments and is restricted to bots. It is still mostly inane garbage but the content and gems have improved steadily over the past year or so. Interesting experiment in any case.
- yalogin 10y agoMay be the wrong context here, but if I am starting out learning Machine Learning using the Stanford course or some other one, is Tensorflow a good candidate to look into? Or does it only contain the advanced algorithms?
- nl 10y agoTF is great once you start doing neural networks. It does have support for other things too, but there are better documented frameworks. Start with Scikit-Learn or R. Once you start doing neural network stuff, start with Keras on top of TF.
- vtempest 10y agoHere's my approach when I built my text-summary app with TensorFlow's SyntaxNet. SyntaxNet (Parsey) gives part-of-speech for each word and a parse tree showing which less-important words/phrases describe higher-level words. "She sells sea shells, down by the sea shore" => (down by the sea shore) is tagged by SyntaxNet as lower describing "sells" so it can be removed from the sentence. Removing adjectives and prepositional phrases gives us simpler sentences easily. Next, we find key words (central to sentences) for news article based on n-grams, and then score key sentences in which they appear. Use MIT ConceptNet for common-sense linking of nouns and most likely relations between them and similar words based on vectors. Generate article summary from the grammatically simple sentences. My question is how well the trained models interpret human meaning in joined sentences. I discovered that by simplifying sentences you lose the original meaning when that grammatically-low-importance word is central to the meaning. "Clinton may be the historically first nominee, who is a woman, from the Dem or GOP party to win presidency" is way different meaning than that if you remove the "who is a woman". I am also interested in how it makes sence to join-up nouns/entities across sentences. This will cause the wrong meaning unless you are building the human meaning structures like in ConceptNet by learning from the article itself, as opposed to pretrained models based on grammar or word vector in Gigaword. My work for the future, is using tf–idf style approach for deciding the key words in a sentence, which I would recommend over relative grammar/vectors. In the example in your blog post ("australian wine exports hit record high in september") you left out that it's 52.1 million liters; but if the article went on to mention or relate importance to that number, by comparing it to past records or giving it the price and so on, you can see this "52.1 million liters" phrase in this one sentence has a higher score relative to the collection of all sentences. As opposed to probabilistic word cherry picking based on prior data, this approach will enable you to extract named entities and phrases and build sentences from phrases in any sentence that grammatically refer to it.
- nl 10y agoa parse tree showing which less-important words/phrases describe higher-level words Things lower on a parse tree aren't less important than things higher. It just represents a dependency relationship.
- nl 10y agoAllegedly (according to HP labs anyway) my 15 year old software[0] was state-of-the-art at this on the CNN dataset[1]. By my thing was very heuristic based, and couldn't generate new sentences like this can. I'm pretty impressed - I'd say some of the machine generated summaries are better than the human ones. [0] http://classifier4j.sourceforge.net/ http://classifier4j.sourceforge.net/ (yes, Sourceforge! Shows how old it is!!) [1] http://dl.acm.org/citation.cfm?id=2797081 http://dl.acm.org/citation.cfm?id=2797081
- josephcooney 10y agoI seem to recall someone used Microsoft's AutoSummarize feature to reduce and reduce classical works of literature to a few lines. The results were pretty hilarious, but I can't find it now.
- GordonS 10y agoI think you mean this one: http://jason-huff.com/projects/autosummarize/ http://jason-huff.com/projects/autosummarize/
- josephcooney 10y agoYep. That's the one. Thanks.
- deepGem 10y agoI would love to use this kind of tool and look for gotchas or key hidden elements in legal documents. It's not exactly summarisation, but very useful. Something like bubbling up the important elements of the fine print.
- VierScar 10y agoDoes this include the state of the network in its already-trained state? It looks like we need to train it with the $6000 dataset if we want to get good results like those mentioned. Is it possible for its state to be saved to disk and restored (so we don't all need a copy of the dataset)?
- greenspot 10y ago> we started looking at more difficult datasets where reading the entire document is necessary to produce good summaries Was hoping to get rather some more insights on this. Because when looking at the examples given, I wonder if we really need machine learning to summarize single sentences? Just by cutting all adjectives, replacing words by abbreviations and multiple words by potential category terms, we should face similar results. Maybe it's just a start or did I miss anything?
- deleted 10y ago[deleted]
- mastazi 10y agoThe Hainan example in the article is especially impressive, the generated summary uses completely different expressions compared to the source text, yet it is spot-on. Of course those are probably cherry-picked results, but still. As a side note, it would be interesting to see how the algorithm performs with longer sources.
- wrp 10y agoWe've observed that due to the nature of news headlines, the model can generate good headlines from reading just a few sentences from the beginning of the article. This illustrates the importance of taking the trouble to understand a domain before trying to model it. I was taught in journalism class that the first paragraph of a newspaper article should summarize the story, the next 3-5 paragraphs summarize again, then the rest of the article fill in the details. Not only do the authors spend time discovering what should have been known from the outset, they reverse cause and effect. The model can generate good headlines due to the nature of newpaper writing, not due to the nature of headlines.
- kharms 10y agoThe following appears in the README.md # Run the eval. Try to avoid running on the same matchine as training. Why should one avoid that? https://github.com/tensorflow/models/tree/master/textsum https://github.com/tensorflow/models/tree/master/textsum