4 ms·
Would you like to point us to some progress on how to overcome these limitations with LDA?
by xtacy 12y ago
Would you like to point us to some progress on how to overcome these limitations with LDA?
- tansey 12y agoThe gensim [0] package has a nice implementation of online LDA that can handle massive streaming datasets. If you want to avoid specifying the number of topics, you can use HDP-LDA. David Blei (inventor of LDA) has a reference implementation on his website [1], along with many other variants of LDA. [0] http://radimrehurek.com/gensim/ http://radimrehurek.com/gensim/ [1] http://www.cs.princeton.edu/~blei/topicmodeling.html http://www.cs.princeton.edu/~blei/topicmodeling.html
- datahipster 12y agoAlso check out Vowpal Wabbit [0]. Although this is a tool that provides several online versions of classification, regression, etc, there's an online LDA module included as well. It takes advantage of the "hashing trick", meaning that you can build an LDA model with a fixed memory footprint. I've had a lot of success using this tool. [0] https://github.com/JohnLangford/vowpal_wabbit https://github.com/JohnLangford/vowpal_wabbit