3 ms·
It's a fun exercise for a single tweet feed, but it is unfortunately not very useful when applied to larger scale learning, since you start hitting the model's
by Daishiman 12y ago
It's a fun exercise for a single tweet feed, but it is unfortunately not very useful when applied to larger scale learning, since you start hitting the model's limitations of a fixed number of topics and, its potentially long runtime, and the fact that it's not an online method (although there are online variants of LDA).
- xtacy 12y agoWould you like to point us to some progress on how to overcome these limitations with LDA?
- tansey 12y agoThe gensim [0] package has a nice implementation of online LDA that can handle massive streaming datasets. If you want to avoid specifying the number of topics, you can use HDP-LDA. David Blei (inventor of LDA) has a reference implementation on his website [1], along with many other variants of LDA. [0] http://radimrehurek.com/gensim/ http://radimrehurek.com/gensim/ [1] http://www.cs.princeton.edu/~blei/topicmodeling.html http://www.cs.princeton.edu/~blei/topicmodeling.html
- datahipster 12y agoAlso check out Vowpal Wabbit [0]. Although this is a tool that provides several online versions of classification, regression, etc, there's an online LDA module included as well. It takes advantage of the "hashing trick", meaning that you can build an LDA model with a fixed memory footprint. I've had a lot of success using this tool. [0] https://github.com/JohnLangford/vowpal_wabbit https://github.com/JohnLangford/vowpal_wabbit