3 ms·

It actually seems to imply the opposite: “ As document length increases, the complexity of the text also increases, rendering cosine similarity less effective i

It actually seems to imply the opposite: “ As document length increases, the complexity of the text also increases, rendering cosine similarity less effective in capturing nuanced semantic relationships” I think its true that using TfIdf to calculate the vectors doesnt scale, but we’ve seen that the OpenAi embedding models are pretty capable of encoding semantics of a longer document.

gcanyon 2y ago
The dimensionality of cosine similarity refers to how you describe a given chunk of text, not how long or short the text itself is. It's easy for humans to visualize cosine similarity between two-dimensional vectors: draw two angles on graph paper, starting from the origin and going at some angle for some distance (normally 0-1) from the origin. You can visually understand that a 45-degree angle is far from a 270-degree angle, unless both happen to be very very short. But that is almost of negative help in describing what's going on here: the vectors in question have hundreds (or more!) dimensions, and are (except very abstractly) impossible for humans to visualize. Cosine similarity only becomes useful, and interestingly becomes counterintuitively useful, when used at these higher dimensions. My point was that the article doesn't describe the above.
didgeoridoo 2y ago
I think this is referring to using a single cosine similarity metric for a given piece of text. The similarity between the strings “dog” and “cat” is qualitatively different from (and substantially more meaningful than) the similarity between the full contents of War and Peace and Anna Karenina, despite both being described by a single value. In practice, you would never try to calculate the similarity between such long texts. You would use a chunking strategy to break them into pieces more likely to yield meaningful similarity values.
marginalia_nu 2y ago
TF-IDF for long documents is in general not very informative. TF-IDF works well for shorter documents that are largely about one topic. It will most likely not catch onto the fact that the Iliad is about the Trojan war. (TF-IDF vectors are largely viewed as a cute trick but ultimately a bit of a dead end in classic information retrieval.) That said, cosine similarity is definitely still useful for large vectors of things that aside from tf-idf.

gcanyon 2y ago

The dimensionality of cosine similarity refers to how you describe a given chunk of text, not how long or short the text itself is. It's easy for humans to visualize cosine similarity between two-dimensional vectors: draw two angles on graph paper, starting from the origin and going at some angle for some distance (normally 0-1) from the origin. You can visually understand that a 45-degree angle is far from a 270-degree angle, unless both happen to be very very short. But that is almost of negative help in describing what's going on here: the vectors in question have hundreds (or more!) dimensions, and are (except very abstractly) impossible for humans to visualize. Cosine similarity only becomes useful, and interestingly becomes counterintuitively useful, when used at these higher dimensions. My point was that the article doesn't describe the above.

didgeoridoo 2y ago

I think this is referring to using a single cosine similarity metric for a given piece of text. The similarity between the strings “dog” and “cat” is qualitatively different from (and substantially more meaningful than) the similarity between the full contents of War and Peace and Anna Karenina, despite both being described by a single value. In practice, you would never try to calculate the similarity between such long texts. You would use a chunking strategy to break them into pieces more likely to yield meaningful similarity values.

marginalia_nu 2y ago

TF-IDF for long documents is in general not very informative. TF-IDF works well for shorter documents that are largely about one topic. It will most likely not catch onto the fact that the Iliad is about the Trojan war. (TF-IDF vectors are largely viewed as a cute trick but ultimately a bit of a dead end in classic information retrieval.) That said, cosine similarity is definitely still useful for large vectors of things that aside from tf-idf.