2 ms·
I quickly read this article. Two things stood out: the big-O complexity discussion and the fact that embeddings get really complicated for large datasets like 1
by xrd 4y ago
I quickly read this article. Two things stood out: the big-O complexity discussion and the fact that embeddings get really complicated for large datasets like 100M Spotify songs.
I'm more curious about embeddings for small datasets. For example, if you have a very small set of information, say the transcript from a meeting, can you then run a very context specific set of generated embeddings against it and generally not worry about the performance issues? It seems like these things are important only when doing massive scale embeddings and massive scale of data.
Am I wrong about the implications of all this? And the takeaway is that postgres is probably fine for all usage like that, rather than a specialist database.
- thomasahle 4y agoIf you don't have 100M vectors, but more like 100K, you can just brute force search them. Is that what you mean?