4 ms·
Ok, I'm double dipping. Another problem I'm trying to solve is: I've got a database of several hundred interesting conversation questions that I've collected ov
by yosito 5y ago
Ok, I'm double dipping. Another problem I'm trying to solve is: I've got a database of several hundred interesting conversation questions that I've collected over the years. Essentially just strings, though I've attempted to categorize them, rank them, and add other metadata. I'd like to figure out a way to sort them or dedupe them based on semantic similarity, but I'm not sure how to determine semantic similarity without painstakingly going through and manually looking for similar questions. Any suggestions on how to solve this would be welcome.
- fshbbdssbbgdd 5y agoPut the questions in some semantic embedding space. Now you’ll have a vector representing each question. Then for each question, you can sort all the questions by how far the Euclidean distance is between their vectors. Or use some clustering algorithm like k-means to find clusters. By web search I found this to tutorial to put sentences in an embedding space: https://github.com/BramVanroy/bert-for-inference/blob/master/introduction-to-bert.ipynb https://github.com/BramVanroy/bert-for-inference/blob/master... I did not read this and am not endorsing it, but it looks like it’s doing roughly what I’m suggesting.
- johnmoberg 5y agoYep, exactly this. Check out sentence-transformers https://pypi.org/project/sentence-transformers/0.3.0/ https://pypi.org/project/sentence-transformers/0.3.0/, they have some great pre-trained models. Once you have the embeddings you can just compute the cosine similarity.
- phenkdo 5y agoYou could use BERT or Word2Vec or GLOVE. They are very simple to use with HuggingFace's library.