3 ms·

Thanks, shoten looks wonderful and exciting. You're welcome to reference/reuse words2map as you wish, while the authors of word2vec, t-SNE, HDBSCAN, and others

by legel 10y ago

Thanks, shoten looks wonderful and exciting. You're welcome to reference/reuse words2map as you wish, while the authors of word2vec, t-SNE, HDBSCAN, and others deserve all the credit. :)

ganeshkrishnan 10y ago
Thanks for the kind words. We have bit of known unknowns and I am sure plenty of unknown unknowns. What we are trying to implement is a rank based on "personality cluster" of the person querying. So in your example Pablo Picasso searching would get slightly different results compared to Kanye West (because they are different personality clusters)
- legel 10y ago
  That sounds awesome, and technically achievable in various capacities. Something a colleague thought of recently that may be of interest is hierarchical mapping, particularly if the number of personalities is great - so, e.g., having a map for all users, and then having unique maps for each users, and programmatically transitioning across maps like this to the extent that "zooming in" is useful. Kind of fun to think about. The other thing that comes to mind is the ironic dependency of words2map right on the elephant in your room: Google. In particular, when you dive into the code, you'll see that the first searches are free, but then become $5 / 1000 at scale, which is probably not good for your objective. Therefore you may wish to replace the Google search functionality of words2map with your own search engine. If you explore doing so, I'd love to see the results, and would be happy to incorporate as an option for words2map users looking for a completely free implementation. Just have a look at the research_keywords function if this of interest. And best of luck!
  - ganeshkrishnan 10y ago
    Pretty close to what I am trying to do, hierarchical "soft/overlapping" clustering is what I am investigating. I have the design ready for matching user cluster to corpus cluster. Then I use okapiBM25 to get primary results from this cluster and rank them (with an undecided algorithm). Then, second step, I augment the rank (increase or decrease) subset of top results with predetermined queries that match these results. For example among two equal documents & search query "Who is Tommen" if the second document has more people clicking then I increase the pagerank for second document (by a function of how many more people prefer the second document)
    - legel 10y ago
      This is an interesting approach. The idea of overlapping clusters is appealing, you're probably familiar with the LDA family of algorithms that provide nice data structures for this.

by legel 10y ago

Thanks, shoten looks wonderful and exciting. You're welcome to reference/reuse words2map as you wish, while the authors of word2vec, t-SNE, HDBSCAN, and others deserve all the credit. :)

ganeshkrishnan 10y ago

Thanks for the kind words. We have bit of known unknowns and I am sure plenty of unknown unknowns. What we are trying to implement is a rank based on "personality cluster" of the person querying. So in your example Pablo Picasso searching would get slightly different results compared to Kanye West (because they are different personality clusters)

legel 10y ago
That sounds awesome, and technically achievable in various capacities. Something a colleague thought of recently that may be of interest is hierarchical mapping, particularly if the number of personalities is great - so, e.g., having a map for all users, and then having unique maps for each users, and programmatically transitioning across maps like this to the extent that "zooming in" is useful. Kind of fun to think about. The other thing that comes to mind is the ironic dependency of words2map right on the elephant in your room: Google. In particular, when you dive into the code, you'll see that the first searches are free, but then become $5 / 1000 at scale, which is probably not good for your objective. Therefore you may wish to replace the Google search functionality of words2map with your own search engine. If you explore doing so, I'd love to see the results, and would be happy to incorporate as an option for words2map users looking for a completely free implementation. Just have a look at the research_keywords function if this of interest. And best of luck!
- ganeshkrishnan 10y ago
  Pretty close to what I am trying to do, hierarchical "soft/overlapping" clustering is what I am investigating. I have the design ready for matching user cluster to corpus cluster. Then I use okapiBM25 to get primary results from this cluster and rank them (with an undecided algorithm). Then, second step, I augment the rank (increase or decrease) subset of top results with predetermined queries that match these results. For example among two equal documents & search query "Who is Tommen" if the second document has more people clicking then I increase the pagerank for second document (by a function of how many more people prefer the second document)
  - legel 10y ago
    This is an interesting approach. The idea of overlapping clusters is appealing, you're probably familiar with the LDA family of algorithms that provide nice data structures for this.