4 ms·
Here is a simple recipe to do similar that works decently well as a start off point: -------------------------------------------------------- Count how many t
by Dn_Ab 15y ago
Here is a simple recipe to do similar that works decently well as a start off point:
--------------------------------------------------------
Count how many times each word appears in the document into a dictionary or map structure
Also make sure you track the total words.
document |> splitBySpace |> if dictionary has word then +1 else 1; totalwords++
Then split the document into sentences.
Okay, now for each sentence
==========================================
score = 0.
split sentence by space and
for each word score+= -(dictionary[word]/sum) * log(dictionary[word]/sum)
dictionaryScore.Add(sentence, score)
==========================================
So now each sentence has a score. You can sort by best and lose order. Or if you want to limit (0 - 1) based on score:
findbestScore and filter each sentence by if limit < docscore / bestscore.
As I said this is only a start off point and is susceptible to list of random words (guess why) there are many ways to make it better. Here is a portion of code I dug up from a while ago:
let inline sumMap m = m |> Map.fold (curryfst (+)) 0.
let inline internal countsAndSum n doc =
let counts = splitstr [|" "|] doc |> filterStop n |> Array.fold mapAdd Map.empty
counts, sumMap counts
let ent m sum k =
let p = (mapGet m k 0.)/sum
if p = 0. then 0. else -p * log2 (p)
let eScore doc =
let counts , sum = countsAndSum 0 doc
splitSentenceRegEx doc |> Array.map (fun str -> str, splitstr [|" "|] str |> Array.fold (flip ((+) << (ent counts sum))) 0.)
- scoot 15y agoDoes this approach have a name? It looks familiar, but I can't place it.
- pkghost 15y agoReminds me of term frequency-inverse document frequency (http://en.wikipedia.org/wiki/Tf*idf http://en.wikipedia.org/wiki/Tf*idf) which can be used with cosine similarity to compute something that approximates the "semantic" relatedness of documents. Example and more algo details here: http://www.exsupera.com/sandbox/DCM/html/document.py?id=3 http://www.exsupera.com/sandbox/DCM/html/document.py?id=3
- Dn_Ab 15y agoAlthough it sometimes gives similar results, its not tf-idf. It is simpler so faster for this task. If it has a name I don't know it, its just an idea I had: Sentences with the most uncertainty associated with them with respect to the document should be the most interesting/relevant.
- nl 15y agoThis is the approach used in the Python NLTK. That algorithm was adapted from a Java library called Classifier4J that I wrote in the early 2000's[1]. I'd never seen that technique before, but (like a lot of algorithms) it is quite obvious once you've seen it. Edit: actually, it's slightly different to my technique because I just used liner scoring (ie, counting popular words). I'm not sure which technique would work best. [1] https://groups.google.com/d/topic/nltk-dev/qV9e5TsCBHg/discussion https://groups.google.com/d/topic/nltk-dev/qV9e5TsCBHg/discu...
- Dn_Ab 15y agoHey I hadn't seen the technique above either. But I've certainly heard of your work. Unfortunately I am not able to share in the bounty that is NLTK. Anyways, it is really hard to judge these things (statistical recommenders) since the metric is inherently subjective and there really is no wrong or right answer. But the way I like to defend it is: if you are going to just skim you should at least use a statistical based approach. Better than just jumping about randomly. These days I'm more interested in abstract summarization without cheating (no templates).
- nl 15y agoUnfortunately I am not able to share in the bounty that is NLTK Why is that?
- Dn_Ab 15y agoI am much stronger in F# than Python and have already invested time in building a decent codebase in it. Also I personally think better in statically typed functional languages. I did not write the SO post which also is based on just word frequencies. I have found at least that in terms of picking the most relevant words with respect to the topic, the method I wrote which was inspired by ideas of entropy gives what I deem to be better results. Its robust against stop words and commonly repeated words that are not part of the topic. The summaries though, I cannot say are better or worse.
- joshu 15y ago