4 ms·
This is the approach used in the Python NLTK. That algorithm was adapted from a Java library called Classifier4J that I wrote in the early 2000's[1]. I'd never
by nl 15y ago
This is the approach used in the Python NLTK. That algorithm was adapted from a Java library called Classifier4J that I wrote in the early 2000's[1].
I'd never seen that technique before, but (like a lot of algorithms) it is quite obvious once you've seen it.
Edit: actually, it's slightly different to my technique because I just used liner scoring (ie, counting popular words). I'm not sure which technique would work best.
[1] https://groups.google.com/d/topic/nltk-dev/qV9e5TsCBHg/discussion https://groups.google.com/d/topic/nltk-dev/qV9e5TsCBHg/discu...
- Dn_Ab 15y agoHey I hadn't seen the technique above either. But I've certainly heard of your work. Unfortunately I am not able to share in the bounty that is NLTK. Anyways, it is really hard to judge these things (statistical recommenders) since the metric is inherently subjective and there really is no wrong or right answer. But the way I like to defend it is: if you are going to just skim you should at least use a statistical based approach. Better than just jumping about randomly. These days I'm more interested in abstract summarization without cheating (no templates).
- nl 15y agoUnfortunately I am not able to share in the bounty that is NLTK Why is that?
- Dn_Ab 15y agoI am much stronger in F# than Python and have already invested time in building a decent codebase in it. Also I personally think better in statically typed functional languages. I did not write the SO post which also is based on just word frequencies. I have found at least that in terms of picking the most relevant words with respect to the topic, the method I wrote which was inspired by ideas of entropy gives what I deem to be better results. Its robust against stop words and commonly repeated words that are not part of the topic. The summaries though, I cannot say are better or worse.
- joshu 15y agoAny chance you would consider explaining it not in (haskell?)
- nl 15y agoSure: My algorithm is something I made up, and from memory it works like this: 1) Remove HTML, stem, remove stopwords etc 2) Sort unique words by popularity in the text 3) Split the original text on sentence boundaries. 4) Include each sentence that first mentions the next most popular word, until the summary is the maximum length requested. http://news.ycombinator.com/item?id=1803020 http://news.ycombinator.com/item?id=1803020 Googling turns up http://sujitpal.blogspot.com.au/2009/02/summarization-with-lucene.html http://sujitpal.blogspot.com.au/2009/02/summarization-with-l... which compares a few approaches. Edit: Also http://stackoverflow.com/questions/2829303/given-a-document-select-a-relevant-snippet http://stackoverflow.com/questions/2829303/given-a-document-..., which I think is by Dn_Ab who wrote the OP.