6 ms·

Topic mining with LDA and Kmeans and interactive clustering in Python

81 points by ahmedbesbes 10y ago

stuartaxelowen 10y ago
I'm still waiting to see interesting topic modeling results on non-news data. The news has this tendency to follow a relatively low amount of topics, making it easy to discover meaningful word distributions, but reviews, messages, and comments written by "normal humans" almost never seem to.
- mathgenius 10y ago
  > interesting topic modeling results on non-news data. this is easy enough to do, I'm wondering what exactly is your definition of "interesting" ?
  - stuartaxelowen 10y ago
    Interesting in that it provides some real insight - confirming that "crash" and "frozen", etc are similar in app reviews is not interesting. Interesting topics are more precise: people talking about the video being frozen or not playing is a singular topic, but given that video is used in so many other reviews, would not be detected as a topic using LDA.
- autokad 10y ago
  I have done countless LDA on tweets and it worked great. here is an example of taking tweets that have 'diabetes' and correlating topics with counties. https://www.linkedin.com/in/karl-dailey-02557b65/treasury/position:755225215/?entityUrn=urn%3Ali%3Afs_treasuryMedia%3A(ACoAAA3JzV0BT9zF9Db_PxdLfE28gQQfK_PzmhM%2C1486424963566) https://www.linkedin.com/in/karl-dailey-02557b65/treasury/po... The original topics were actually better, but I was asked to adjust the language to wipe out common language across all tweets. You can still see interesting things going on though. North East: Charity, Hospitals, Research. South: Koolaid, sweat tea. etc. I had also done topic modeling on customer surveys for Comcast (I cant show them), but the topics identified 3 key features that lead to low customer satisfaction. I have also used LDA on grocery purchases (for mere fun), and it worked out really great as well.
  - stuartaxelowen 10y ago
    But what real insights do the topics give? The only LDA results I've seen are "fairly obvious" or "not understandable". Seeing a "car, road, light, drive, trip, ..." topic is not insightful - this is a topic that is obvious to most humans. A more interesting topic would be one that is understandable but was not obvious, even to the people intimately involved in data's subject matter. These are harder to discover, but they do exist - and I have never seen LDA surface them.
    - autokad 10y ago
      I see quantifiable and visualizible insights. Its also good for generating features, and I can also create interaction features with important n-grams. If it cant give you what you are looking for, maybe you are asking a fish to climb a tree
    - nl 10y ago
      The purpose of LDA/topic visualisations isn't to do something humans can't do. It's to do it extremely rapidly on large amounts of data.
- computerex 10y ago
  positive reviews: http://pastebin.com/enEZzKmC http://pastebin.com/enEZzKmC negative reviews: http://pastebin.com/vkGn7ZLH http://pastebin.com/vkGn7ZLH Amazon Echo, roughly ~34k reviews. https://en.wikipedia.org/wiki/Non-negative_matrix_factorization https://en.wikipedia.org/wiki/Non-negative_matrix_factorizat... It's surprisingly useful for finding interesting trends in reviews.
- thecity2 10y ago
  "20 newsgroups" is a pretty ubiquitous human-generated data set for testing LDA and other NLP techniques. I've run LDA on it myself and it recovers topics fairly well. The thing is, reviews, messages, comments, etc, in general, do tend to revolve around some central topic(s). For example, this comment right now is about LDA. It's not totally random.
- greglindahl 10y ago
  blekko did some work with LDA against our 2,000 "slashtags", almost all of which were non-news-oriented. Sorry that this is not a free link: http://ieeexplore.ieee.org/document/7050791/ http://ieeexplore.ieee.org/document/7050791/
- jacquesm 10y ago
  I did a project around this (AutoTagger): search google for all the subjects in the library of congress, spider the first 10 pages, strip off anything not body text, build a vocabulary with frequency, merge vocabularies to get a subject vocabulary, reject words common across all vocabularies. Then when presented with a new article on any subject create the vocabulary and frequencies for that article, strip out the words stripped previously and then do a match against all subjects. Very easy to parallelize, very good results (surprisingly good, for such a simple algorithm in fact).
- refrigerator 10y ago
  There are a lot of interesting applications of topic modelling (including more interesting + powerful techniques than LDA) in recommender systems - worth looking into if you're curious.
- flgr 10y ago
  In a previous life we used LDA (together with some other techniques) to find clusters of similar Android applications (based on their descriptions). We then looked for apps within a cluster that used vastly different permission-gated APIs to identify malware and grayware. It worked pretty well. See http://dl.acm.org/citation.cfm?id=2568276 http://dl.acm.org/citation.cfm?id=2568276
RockyMcNuts 10y ago
Good stuff ... if you're interested in this stuff, you might be interested in Richard Socher's Stanford NLP class - http://cs224d.stanford.edu/ http://cs224d.stanford.edu/ I have a dumb question about Bokeh ... if I use RStudio I can create some markdown, export to PDF, send around to team/clients. Typically I can't do anything interactive without hosting on a shiny server or referencing some server-side magic. If I use Bokeh, is it as simple as creating a Jupyter notebook, saving to HTML, and then anyone can see all the D3 magic? Or are there any drawbacks like file size, compatibility, browsers not letting you open due to security or whatnot? If any pointers, tutorials, examples anyone has would love to hear them, look me up if it's not deemed on-topic.
- Tarq0n 10y ago
  If you only need basic filtering and brushing for interactivity there's a new library called crosstalk that can do that. Some htmlwidgets like Plotly also support some interactivity without the need for a server. http://rstudio.github.io/crosstalk/ http://rstudio.github.io/crosstalk/ Edit: Note that you can knit rMarkdown to HTML as well, which is what enables most of this stuff.
  - RockyMcNuts 10y ago
    hmmh, thanks, might check that one out. for next project maybe I'll try both ways, with ggplot and rmarkdown to HTML, and with Bokeh in python. The ggplot will just knit to PNGs, would be interesting if ggplot could output D3 with intelligent rollovers, maybe eventually some scriptable animations. One could use ggplotly but I don't want to post anything to, or host anything on a server, has to work 100% offline.
    - Tarq0n 10y ago
      The R version of plotly has been fully offline ever since they open sourced it. Alternatively you could also try ggvis, but it's not as polished as ggplot yet.
      - 10y ago
kmonad 10y ago
Hey Ahmed, nice write up! One thought I had was that I would be careful using KMeans on tSNE transformed data. From my understanding, while tSNE preserves local relationships between data point, it (heavily) distorts long-distance relationships. What this means is that (if I get this right), you have almost arbitrary clustering results if your k is chosen such that it is not equal to the number of local clusters (a thing you don't know in advance). So clustering on distances between data points in tSNE coordinates seems to be a risky way to draw conclusions. I am not saying this is wrong, but if what I am saying is correct (please point out if I got something wrong), then your analysis may perhaps work better if you did KMeans on say the final 5-dimensional PCA space or something like that, and leave tsne to visualize the data (and perhaps label the data points by the clustering on PCAs?). Do you agree / does that make sense?
- HIP_HOP 10y ago
  He fits and transforms Kmeans on the output of TfIdfVectorizer and not tSNE. tSNE is only used for 2D visualisation. kmeans = kmeans_model.fit(vz) kmeans_clusters = kmeans.predict(vz)
  - kmonad 10y ago
    You're right. In that case, I would worry about the high dimensionality of that array. Does MiniBatchKMeans do some internal dimensionality reduction?

81 points by ahmedbesbes 10y ago

I'm still waiting to see interesting topic modeling results on non-news data. The news has this tendency to follow a relatively low amount of topics, making it easy to discover meaningful word distributions, but reviews, messages, and comments written by "normal humans" almost never seem to.

mathgenius 10y ago
> interesting topic modeling results on non-news data. this is easy enough to do, I'm wondering what exactly is your definition of "interesting" ?
- stuartaxelowen 10y ago
  Interesting in that it provides some real insight - confirming that "crash" and "frozen", etc are similar in app reviews is not interesting. Interesting topics are more precise: people talking about the video being frozen or not playing is a singular topic, but given that video is used in so many other reviews, would not be detected as a topic using LDA.
autokad 10y ago
I have done countless LDA on tweets and it worked great. here is an example of taking tweets that have 'diabetes' and correlating topics with counties. https://www.linkedin.com/in/karl-dailey-02557b65/treasury/position:755225215/?entityUrn=urn%3Ali%3Afs_treasuryMedia%3A(ACoAAA3JzV0BT9zF9Db_PxdLfE28gQQfK_PzmhM%2C1486424963566) https://www.linkedin.com/in/karl-dailey-02557b65/treasury/po... The original topics were actually better, but I was asked to adjust the language to wipe out common language across all tweets. You can still see interesting things going on though. North East: Charity, Hospitals, Research. South: Koolaid, sweat tea. etc. I had also done topic modeling on customer surveys for Comcast (I cant show them), but the topics identified 3 key features that lead to low customer satisfaction. I have also used LDA on grocery purchases (for mere fun), and it worked out really great as well.
- stuartaxelowen 10y ago
  But what real insights do the topics give? The only LDA results I've seen are "fairly obvious" or "not understandable". Seeing a "car, road, light, drive, trip, ..." topic is not insightful - this is a topic that is obvious to most humans. A more interesting topic would be one that is understandable but was not obvious, even to the people intimately involved in data's subject matter. These are harder to discover, but they do exist - and I have never seen LDA surface them.
  - autokad 10y ago
    I see quantifiable and visualizible insights. Its also good for generating features, and I can also create interaction features with important n-grams. If it cant give you what you are looking for, maybe you are asking a fish to climb a tree
  - nl 10y ago
    The purpose of LDA/topic visualisations isn't to do something humans can't do. It's to do it extremely rapidly on large amounts of data.
computerex 10y ago
positive reviews: http://pastebin.com/enEZzKmC http://pastebin.com/enEZzKmC negative reviews: http://pastebin.com/vkGn7ZLH http://pastebin.com/vkGn7ZLH Amazon Echo, roughly ~34k reviews. https://en.wikipedia.org/wiki/Non-negative_matrix_factorization https://en.wikipedia.org/wiki/Non-negative_matrix_factorizat... It's surprisingly useful for finding interesting trends in reviews.
thecity2 10y ago
"20 newsgroups" is a pretty ubiquitous human-generated data set for testing LDA and other NLP techniques. I've run LDA on it myself and it recovers topics fairly well. The thing is, reviews, messages, comments, etc, in general, do tend to revolve around some central topic(s). For example, this comment right now is about LDA. It's not totally random.
greglindahl 10y ago
blekko did some work with LDA against our 2,000 "slashtags", almost all of which were non-news-oriented. Sorry that this is not a free link: http://ieeexplore.ieee.org/document/7050791/ http://ieeexplore.ieee.org/document/7050791/
jacquesm 10y ago
I did a project around this (AutoTagger): search google for all the subjects in the library of congress, spider the first 10 pages, strip off anything not body text, build a vocabulary with frequency, merge vocabularies to get a subject vocabulary, reject words common across all vocabularies. Then when presented with a new article on any subject create the vocabulary and frequencies for that article, strip out the words stripped previously and then do a match against all subjects. Very easy to parallelize, very good results (surprisingly good, for such a simple algorithm in fact).
refrigerator 10y ago
There are a lot of interesting applications of topic modelling (including more interesting + powerful techniques than LDA) in recommender systems - worth looking into if you're curious.
flgr 10y ago
In a previous life we used LDA (together with some other techniques) to find clusters of similar Android applications (based on their descriptions). We then looked for apps within a cluster that used vastly different permission-gated APIs to identify malware and grayware. It worked pretty well. See http://dl.acm.org/citation.cfm?id=2568276 http://dl.acm.org/citation.cfm?id=2568276

RockyMcNuts 10y ago

Good stuff ... if you're interested in this stuff, you might be interested in Richard Socher's Stanford NLP class - http://cs224d.stanford.edu/ http://cs224d.stanford.edu/ I have a dumb question about Bokeh ... if I use RStudio I can create some markdown, export to PDF, send around to team/clients. Typically I can't do anything interactive without hosting on a shiny server or referencing some server-side magic. If I use Bokeh, is it as simple as creating a Jupyter notebook, saving to HTML, and then anyone can see all the D3 magic? Or are there any drawbacks like file size, compatibility, browsers not letting you open due to security or whatnot? If any pointers, tutorials, examples anyone has would love to hear them, look me up if it's not deemed on-topic.

Tarq0n 10y ago
If you only need basic filtering and brushing for interactivity there's a new library called crosstalk that can do that. Some htmlwidgets like Plotly also support some interactivity without the need for a server. http://rstudio.github.io/crosstalk/ http://rstudio.github.io/crosstalk/ Edit: Note that you can knit rMarkdown to HTML as well, which is what enables most of this stuff.
- RockyMcNuts 10y ago
  hmmh, thanks, might check that one out. for next project maybe I'll try both ways, with ggplot and rmarkdown to HTML, and with Bokeh in python. The ggplot will just knit to PNGs, would be interesting if ggplot could output D3 with intelligent rollovers, maybe eventually some scriptable animations. One could use ggplotly but I don't want to post anything to, or host anything on a server, has to work 100% offline.
  - Tarq0n 10y ago
    The R version of plotly has been fully offline ever since they open sourced it. Alternatively you could also try ggvis, but it's not as polished as ggplot yet.
    - 10y ago

kmonad 10y ago

Hey Ahmed, nice write up! One thought I had was that I would be careful using KMeans on tSNE transformed data. From my understanding, while tSNE preserves local relationships between data point, it (heavily) distorts long-distance relationships. What this means is that (if I get this right), you have almost arbitrary clustering results if your k is chosen such that it is not equal to the number of local clusters (a thing you don't know in advance). So clustering on distances between data points in tSNE coordinates seems to be a risky way to draw conclusions. I am not saying this is wrong, but if what I am saying is correct (please point out if I got something wrong), then your analysis may perhaps work better if you did KMeans on say the final 5-dimensional PCA space or something like that, and leave tsne to visualize the data (and perhaps label the data points by the clustering on PCAs?). Do you agree / does that make sense?

HIP_HOP 10y ago
He fits and transforms Kmeans on the output of TfIdfVectorizer and not tSNE. tSNE is only used for 2D visualisation. kmeans = kmeans_model.fit(vz) kmeans_clusters = kmeans.predict(vz)
- kmonad 10y ago
  You're right. In that case, I would worry about the high dimensionality of that array. Does MiniBatchKMeans do some internal dimensionality reduction?