3 ms·

i've actually found the performance of gensim (the topic modeling python module i use here) to be pretty great. we're not at a scale where CPU performance is ma

by msalahi 13y ago

i've actually found the performance of gensim (the topic modeling python module i use here) to be pretty great. we're not at a scale where CPU performance is make or break just yet, so i haven't done any comprehensive testing of performance. but i've definitely not run into any performance issues worth complaining about. however, gensim is 100% based on lazy evaluation where it can be, so it's relatively light on the CPU. i love NLTK as well, but it did lack in the dimensionality reduction/topic modeling department which gensim did so beautifully. LDA + SVM seemed like an interesting approach to go with, and it didn't disappoint.

drakaal 13y ago
The issue with Genism is you have to know what you are trying to analyze before you analyze it. It doesn't do well if you use the wrong corpus or if like you mention start with a million word corpus. If you were analyzing emails in a single organization all day you could probably sort out topics really well. Doing all of the web it breaks down because it gets less accurate the larger the variety of content.
- msalahi 13y ago
  "doing all of the web" will cause pretty much any approach to AI/machine learning/NLP to break down. i'm a big believer in it being the responsibility of the engineer employing these techniques to take stock of the problem at hand and find out what constraints you can take advantage of to achieve better performance/accuracy/prettiness of code. there's not really a silver bullet that you can just release on the internet with the task of bringing back incredibly useful information without "knowing what you're trying to analyze before you analyze it."
  - drakaal 13y ago
    Wait an hour. We decided to push that bit of code live in Alpha. :-)
    - msalahi 13y ago
      SCIENCE!
  - agibsonccc 13y ago
    Web Developers are a neat bunch. It's amazing what kind of inference you can do with exploiting document structure to make different kind of inferences alongside more traditional approaches like word frequency analysis, LDA, or even deep learning/word distributional inference. NLP on the web especially question answering and search, can still be greatly expanded upon.

by msalahi 13y ago

drakaal 13y ago

The issue with Genism is you have to know what you are trying to analyze before you analyze it. It doesn't do well if you use the wrong corpus or if like you mention start with a million word corpus. If you were analyzing emails in a single organization all day you could probably sort out topics really well. Doing all of the web it breaks down because it gets less accurate the larger the variety of content.

msalahi 13y ago
"doing all of the web" will cause pretty much any approach to AI/machine learning/NLP to break down. i'm a big believer in it being the responsibility of the engineer employing these techniques to take stock of the problem at hand and find out what constraints you can take advantage of to achieve better performance/accuracy/prettiness of code. there's not really a silver bullet that you can just release on the internet with the task of bringing back incredibly useful information without "knowing what you're trying to analyze before you analyze it."
- drakaal 13y ago
  Wait an hour. We decided to push that bit of code live in Alpha. :-)
  - msalahi 13y ago
    SCIENCE!
- agibsonccc 13y ago
  Web Developers are a neat bunch. It's amazing what kind of inference you can do with exploiting document structure to make different kind of inferences alongside more traditional approaches like word frequency analysis, LDA, or even deep learning/word distributional inference. NLP on the web especially question answering and search, can still be greatly expanded upon.