4 ms·

Would this be able to find near matches within substrings, e.g. a search for "house" in "this is my hiuse typo" would bring back a near match result?

afandian 11y ago
No. The character distribution of those two strings is massively different. You'd have to tokenize the long string into words first. EDIT: If I understood correctly and the vectors are normalised, it would match 'house' to 'hiusehiusehiuse'. Which may or may not be desirable.
- dalke 11y ago
  The vectors are not normalized. That is: from __future__ import print_function from collections import Counter def cosine_similarity_unweighted(a, b): a = Counter(a) b = Counter(b) dotProduct = sum((c in a) * (c in b) for c in a) magnitudeA = sum(1 for c in a) magnitudeB = sum(1 for c in b) return dotProduct / (magnitudeA * magnitudeB)**0.5 def cosine_similarity_weighted(a, b): a = Counter(a) b = Counter(b) dotProduct = sum(a.get(c, 0) * b.get(c, 0) for c in a) magnitudeA = sum(value**2 for value in a.values()) magnitudeB = sum(value**2 for value in b.values()) return dotProduct / (magnitudeA * magnitudeB)**0.5 print("Scores for 'hello' and 'holl'") print("unweighted", cosine_similarity_unweighted("hello", "holl")) print("weighted", cosine_similarity_weighted("hello", "holl")) gives the output: Scores for 'hello' and 'holl' unweighted 0.866025403784 weighted 0.925820099773 The article gives: holl index: 0.9258200997725514 which means the article used letter weights. That said, the scores for your example are the same, because each of the letters in 'house' is used only once (the 3 in the numerator cancels out the 3 in the denominator): Scores for 'house' and 'hiusehiusehiuse' unweighted 0.8 weighted 0.8 If the query had instead been 'houss', you would see a difference in the similarities: Scores for 'houss' and 'hiusehiusehiuse' unweighted 0.67082039325 weighted 0.676123403783

afandian 11y ago

No. The character distribution of those two strings is massively different. You'd have to tokenize the long string into words first. EDIT: If I understood correctly and the vectors are normalised, it would match 'house' to 'hiusehiusehiuse'. Which may or may not be desirable.

dalke 11y ago
The vectors are not normalized. That is: from __future__ import print_function from collections import Counter def cosine_similarity_unweighted(a, b): a = Counter(a) b = Counter(b) dotProduct = sum((c in a) * (c in b) for c in a) magnitudeA = sum(1 for c in a) magnitudeB = sum(1 for c in b) return dotProduct / (magnitudeA * magnitudeB)**0.5 def cosine_similarity_weighted(a, b): a = Counter(a) b = Counter(b) dotProduct = sum(a.get(c, 0) * b.get(c, 0) for c in a) magnitudeA = sum(value**2 for value in a.values()) magnitudeB = sum(value**2 for value in b.values()) return dotProduct / (magnitudeA * magnitudeB)**0.5 print("Scores for 'hello' and 'holl'") print("unweighted", cosine_similarity_unweighted("hello", "holl")) print("weighted", cosine_similarity_weighted("hello", "holl")) gives the output: Scores for 'hello' and 'holl' unweighted 0.866025403784 weighted 0.925820099773 The article gives: holl index: 0.9258200997725514 which means the article used letter weights. That said, the scores for your example are the same, because each of the letters in 'house' is used only once (the 3 in the numerator cancels out the 3 in the denominator): Scores for 'house' and 'hiusehiusehiuse' unweighted 0.8 weighted 0.8 If the query had instead been 'houss', you would see a difference in the similarities: Scores for 'houss' and 'hiusehiusehiuse' unweighted 0.67082039325 weighted 0.676123403783