6 ms·
So text embeddings are kind of random like LLMs?
When I was messing around with random words in Cohere's playground for embedding, I noticed that the way words related to each other seemed to change every time I ran it. So, I tried the same thing with OpenAI's text embeddings and got different results for the same text, even though they were pretty similar. I always thought text embeddings were like a pre-calculated, static thing. Guess I was wrong?
- fithisux 3y agoThey appear random because in reality they are hash values. Hashing is like "deterministically" randomizing.
- gandan 3y agoI stumbled upon a discussion in the OpenAI community about why OpenAI's embeddings sometimes yield different vectors for the same text. https://community.openai.com/t/why-openai-embedding-return-different-vectors-for-the-same-text-input/144495 https://community.openai.com/t/why-openai-embedding-return-d... Skimming through, it seems the issue might be due to random timing in the GPU, as highlighted in one of the posts. https://community.openai.com/t/why-openai-embedding-return-different-vectors-for-the-same-text-input/144495/9?u=simpleusername96 https://community.openai.com/t/why-openai-embedding-return-d... I'm quite new to this field, but it appears that this issue is different from hashing. I'd appreciate your thoughts on this!
- fithisux 3y agoAh ok, you talk about something very specific. Not why embeddings for parts of a body of text appear random. But, why computing embeddings in the same way for a specific body yield different results. The link is revealing. If f is approximately an associative function and you compute in parallel a,b,c with the goal of computing X = f(f(a,b) ,c) ~ f(a, f(b, c)) = Y in some case you may end up a,b finishing first and then c of b,c finishing first and then a. So, you end up computing X in some cases and in other Y. The more values you compute in parallel, the move variations. Some computations like multiplication, addition fall into this category ....
- sk11001 3y ago> got different results for the same text, even though they were pretty similar. Exactly the same text or similar text? There's a big difference between the two. The same text should produce identical results every time.
- gandan 3y agoI actually got different embeddings from the exact same text. I should have been clearer about that.