3 ms·
I think the arguments against summly have more to do with the fact that they didn't have any tech or in-house expertise. They licensed tech from SRI and even ha
by tensor 14y ago
I think the arguments against summly have more to do with the fact that they didn't have any tech or in-house expertise. They licensed tech from SRI and even had another group build the app itself.
That aside, summarization is a relatively old field of study (in CS) and there is tons of good information to read and even many free libraries. My suggestion would be to try out an unsupervised learning algorithm such as LDA. You still need training data, but you don't need to label it with categories. The downside is that you will have no control over what it learns. Still, classic examples of LDA involve classifying news sources.
For standard linear classification, understanding Bayes is important, but for actual implementation look at something like liblinear and use logistic regression with regularization. The difference between Bayes and LR is that Bayes optimizes learning the underlying probability distribution while LR optimizes "getting the classification right", otherwise known as optimizing the expectation. Regularization controls for overfitting and there can be a big difference between the type you use (L1 vs L2) and the settings. Dont' make the mistake of treating it as a minor tweak.
- samsnelling 14y ago>> Dont' make the mistake of treating it as a minor tweak. Yes. Exactly. With how rapidly I have been developing this (2 weeks in the making), I think I could pivot it relatively easily. Really however, classification is just an added layer on top of what I already have. Thank you for your comment. I will look into the stuff you mentioned! This is all relatively new to me!
- jmduke 14y agoI've read a lot of literature on classification, but nothing about summarization. Do you have any recommended literature/libraries?
- samsnelling 14y agojmduke, I started with (http://en.wikipedia.org/wiki/Automatic_summarization http://en.wikipedia.org/wiki/Automatic_summarization) specifically the unsupervised keyphrase extraction. In terms of libraries, there aren't a huge amount of small packages out there outside of the monolithic Stanford NLP package (http://nlp.stanford.edu/software/lex-parser.shtml http://nlp.stanford.edu/software/lex-parser.shtml) and such. When I get back to the house I would be glad to share my bookmarks with you if you are interested.
- jmduke 14y agoThat'd be great! My email is in my profile if you'd prefer that method.
- Homunculiheaded 14y agoTo make the choice of L1 vs L2 a little more flexible there's also Elastic net regularization [0] which lets you combine both L1 and L2 Also agree with LDA, I've recently been using it for some unsupervised text classification and the topics it generates are impressively sane. 0.http://en.wikipedia.org/wiki/Elastic_net_regularization http://en.wikipedia.org/wiki/Elastic_net_regularization