2 ms·
I think OpenAI could fix this by having one extra loss function that token embedding vectors must not be too close to one another by euclidean distance. Or th
by londons_explore 3y ago
I think OpenAI could fix this by having one extra loss function that token embedding vectors must not be too close to one another by euclidean distance.
Or they could require the mean and standard deviation of embedding vectors to be mean=0, stddev=1.0 not just for the whole set of vectors, but for each individual vector. In fact, their documentation says they do this anyway[1], although it appears they do not in fact...
[1]: https://help.openai.com/en/articles/6824809-embeddings-frequently-asked-questions#:~:text=OpenAI%20embeddings%20are%20normalized%20to%20length%201%2C https://help.openai.com/en/articles/6824809-embeddings-frequ...
- jxmorris12 3y agoThis comment is misguided for several reasons. Most notably, you’re linking to the sentence embeddings API, which is totally separate from the token embeddings that are inputted into GPT-style models. Also, a vector with std 1 doesn’t necessarily have norm of 1, which is what the link says (but point is moot because they’re talking about a different type of embeddings)