4 ms·

This is an interesting feature. But you don't need to use just OpenAI's embeddings. You can generate your own embeddings with open source SOTA transformer model

This is an interesting feature. But you don't need to use just OpenAI's embeddings. You can generate your own embeddings with open source SOTA transformer models which would probably work just the same. You could generate a couple hundred thousand embeddings with a rented A100 for less than 2 dollars. And the point of converting text or other objects (like images) into embeddings is to compare a large number of documents to a source document very fast. It's more useful to put the embeddings in something like Redis. This pgvector data type would be good for an offline backup of vectors.

mattnewton 4y ago
Does redis let you do some kind of similarity based lookup? Is that new?
freeqaz 4y ago
Do you have any examples that come to mind for this? I'd love to understand what other models exist and what Redis extensions exist to compare embeddings.
- qigtyofhp 4y ago
  This will load the transformer models from Hugging Face (their models have a similar architecture to OpenAI embedding models): https://www.sbert.net/docs/pretrained_models.html https://www.sbert.net/docs/pretrained_models.html Redis has approximate nearest-neighbors vector similarity search: https://redis-py.readthedocs.io/en/stable/examples/search_vector_similarity_examples.html https://redis-py.readthedocs.io/en/stable/examples/search_ve... Generate the embeddings on a rented GPU, push to Redis then do a similarity search. Store vectors in Redis using ndarray.tobytes()
e12e 4y ago
> And the point of converting text or other objects (like images) into embeddings is to compare a large number of documents to a source document very fast. But a "source document" could also be a natural language query that you need to convert to an embedding - if you want to enable natural language search on your corpus? (maybe along with handling queries in French getting good semantic hits in English etc?)?

mattnewton 4y ago

Does redis let you do some kind of similarity based lookup? Is that new?

freeqaz 4y ago

Do you have any examples that come to mind for this? I'd love to understand what other models exist and what Redis extensions exist to compare embeddings.

qigtyofhp 4y ago
This will load the transformer models from Hugging Face (their models have a similar architecture to OpenAI embedding models): https://www.sbert.net/docs/pretrained_models.html https://www.sbert.net/docs/pretrained_models.html Redis has approximate nearest-neighbors vector similarity search: https://redis-py.readthedocs.io/en/stable/examples/search_vector_similarity_examples.html https://redis-py.readthedocs.io/en/stable/examples/search_ve... Generate the embeddings on a rented GPU, push to Redis then do a similarity search. Store vectors in Redis using ndarray.tobytes()

e12e 4y ago

> And the point of converting text or other objects (like images) into embeddings is to compare a large number of documents to a source document very fast. But a "source document" could also be a natural language query that you need to convert to an embedding - if you want to enable natural language search on your corpus? (maybe along with handling queries in French getting good semantic hits in English etc?)?