3 ms·
I haven't noticed this behavior in such extreme ways as you. I've worked with many different embeddings using this library and the OpenAI API for data normaliza
by pietz 3y ago
I haven't noticed this behavior in such extreme ways as you. I've worked with many different embeddings using this library and the OpenAI API for data normalization, extraction and structuring tasks.
On one hand it's super impressive what you can get out of these embeddings. On the other I'm with you that there seems to be a missing piece to make them work great.
An additional MLP or Siamese Network works really well but it seems like there should be something easier and unsupervised given your set of vectorized samples.
Thinking out loud, could PCA help with this by centering the data and removing dimensions that mostly model noise?
- anon373839 3y ago> I haven't noticed this behavior in such extreme ways as you. I should have clarified that it really depends on the model. I have had this problem to a greater extent with the GTE and BGE embeddings. But I use them anyway, because they’re so strong overall. PCA is an interesting idea, and worth looking at.
- whakim 3y agoI've tried PCA (and perhaps more usefully, UMAP) and while can be great for stuff like unsupervised learning, I think the trick is to choose a classifier (as per my comment above) that can do a good job with high-dimensional data.