4 ms·
Am I understanding correctly that the Numpy one doesn't actually use Numpy to do the calculation, but just to store the data? If so that's a super disingenuous
by ladberg 3y ago
Am I understanding correctly that the Numpy one doesn't actually use Numpy to do the calculation, but just to store the data? If so that's a super disingenuous example. The speed of Numpy comes from being able to stay in C for the entire hot loop.
EDIT: I tested it myself, also using ChatGPT to code it for me.
>>> can you write a cosine distance function using numpy?
def cosine_distance(vector1, vector2):
dot_product = np.dot(vector1, vector2)
norm_vector1 = np.linalg.norm(vector1)
norm_vector2 = np.linalg.norm(vector2)
cosine_similarity = dot_product / (norm_vector1 * norm_vector2)
cosine_distance = 1 - cosine_similarity
return cosine_distance
The original list one takes 161us on my machine, passing a Numpy array to the original function takes 750us, and using the native Numpy function takes 6.24us.
Using Numpy correctly makes it >100x faster!
- ks2048 3y agoYes, the Numpy example is bad. He should try something like, 1 - np.dot(a,b) / (np.linalg.norm(a) * np.linalg.norm(b))
- arthurcolle 3y agoI implemented cosine similarity too in the last week or two for something and the 1- threw me for a loop until I looked at the math for another 15 seconds, realized that it didn't make much sense for "Cat" and "Cat" to have 0 similarity haha
- contravariant 3y agoYou'll probably want to use a function to normalize vectors if you don't want it to barf when you accidentally put something close to a 0 vector in there.
- deleted 3y ago[deleted]
- rdedev 3y agoSo following the numpy example the author gives one using scipy. This should be the same if not faster than a pure numpy implementation