Nice work. This looks similar to an algorithm developed in the 1950s at IBM by H.P. Luhn. Luhn's algorithm basically finds the most "important" words (mostly vi

Nice work. This looks similar to an algorithm developed in the 1950s at IBM by H.P. Luhn. Luhn's algorithm basically finds the most "important" words (mostly via frequency) and then ranks the "importance" of sentences according to the co-occurrences of these "important" words. The summary of the document then becomes the "top n" sentences in chronological order. An implementation of Luhn's algorithm (which accepts a URL or HTML input and also handles relatively clean text extraction with boilerpipe) is available here: http://nbviewer.ipython.org/github/ptwobrussell/Mining-the-Social-Web-2nd-Edition/blob/master/ipynb/Chapter%209%20-%20Twitter%20Cookbook.ipynb#Example-24.-Summarizing-link-targets http://nbviewer.ipython.org/github/ptwobrussell/Mining-the-S... I haven't kept up much with "state of the art" summarization techniques, but this approach is pretty intuitive and works about as well as anything I can imagine (without getting into machine-driven language generation, which is a whole different topic.)