3 ms·

not using raw word frequencies but http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation. Didnt kno

not using raw word frequencies but http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation. Didnt know about tf-idf thanks for the tip.

hnriot 12y ago
it's hard to imagine someone knowing an LDA without also knowing about TF.IDF (it's a dot product, not a hyphen)
- sprobertson 12y ago
  That and the questionable use of stopwords makes it sound like they're just slapping some marketing on an out-of-the box LDA implementation (not that I blame them, it's a dense algorithm).
- GFK_of_xmaspast 12y ago
  The OP is pushing "Algorithmica" whose manifesto is here: http://blog.algorithmia.com/post/75680476188/algorithm-development-is-broken http://blog.algorithmia.com/post/75680476188/algorithm-devel... and this doesn't really strike me as much of a victory for the idea that it's just the implementation of an algorithm being the sticking point in practice.
  - doppenhe 12y ago
    We are just showing the versatility of the platform through a real world use case. LDA is hard to implement/scale for the untrained same as many other machine learning, optimization, graph traversing,etc algorithms. What we are building is crowd-sourced and generalized API where all these algorithms can be combined and used together to really make any application smarter. The demo we show here is a version of how we used our platform to generate tags for all entries in our API by combining algorithms that existed already in Algorithmia. (modified for performance over quality due to the volume that HN would bring). Cheers.
    - s0x 12y ago
      It's only hard because most libriaries I've seen have so little documentation available. It's simple once you understand the library. We need people picking these libraries up, implementing them on weekends on fun projects, documenting their work and code, and publishing it for everyone to learn from.

not using raw word frequencies but http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation. Didnt know about tf-idf thanks for the tip.

hnriot 12y ago

it's hard to imagine someone knowing an LDA without also knowing about TF.IDF (it's a dot product, not a hyphen)

sprobertson 12y ago
That and the questionable use of stopwords makes it sound like they're just slapping some marketing on an out-of-the box LDA implementation (not that I blame them, it's a dense algorithm).
GFK_of_xmaspast 12y ago
The OP is pushing "Algorithmica" whose manifesto is here: http://blog.algorithmia.com/post/75680476188/algorithm-development-is-broken http://blog.algorithmia.com/post/75680476188/algorithm-devel... and this doesn't really strike me as much of a victory for the idea that it's just the implementation of an algorithm being the sticking point in practice.
- doppenhe 12y ago
  We are just showing the versatility of the platform through a real world use case. LDA is hard to implement/scale for the untrained same as many other machine learning, optimization, graph traversing,etc algorithms. What we are building is crowd-sourced and generalized API where all these algorithms can be combined and used together to really make any application smarter. The demo we show here is a version of how we used our platform to generate tags for all entries in our API by combining algorithms that existed already in Algorithmia. (modified for performance over quality due to the volume that HN would bring). Cheers.
  - s0x 12y ago
    It's only hard because most libriaries I've seen have so little documentation available. It's simple once you understand the library. We need people picking these libraries up, implementing them on weekends on fun projects, documenting their work and code, and publishing it for everyone to learn from.