6 ms·
> My script read through each of the products we had responses for, called OpenAI's embedding api and loaded it into Pinecone - with a reference to the Supabase
by pbourke 4y ago
> My script read through each of the products we had responses for, called OpenAI's embedding api and loaded it into Pinecone - with a reference to the Supabase response entry.
OpenAI and the Pinecone database are not really needed for this task. A simple SBERT encoding of the product texts, followed by storing the vectors in a dense numpy array or faiss index would be more than sufficient. Especially if one is operating in batch mode, the locality and simplicity can’t be beat and you can easily scale to 100k-1M texts in your corpus on commodity hardware/VPS (though NVME disk will see a nice performance gain over regular SSD)
- vimota 4y agoYep that's true! I'd probably do something like that if I were starting again, but the ease of calling a few APIs is pretty nice. I feel like that alone will drive a lot of adoption of some of these platforms even if it can just be done locally.
- rvnx 4y agoGreat result!
- freediver 4y agoI'd argue that using something like SBERT + Faiss is easier and would take less time (you do not have two account creations + one billing setup), plus a working example of SBERT + Faiss is probably total less than 10 lines of code.
- rattray 4y agoWhat does it look like? I've never heard of either.
- deleted 4y ago[deleted]
- pbourke 4y ago# pip install faiss-cpu sentence-transformers from sentence_transformers import SentenceTransformer import faiss # replace with own texts - this is a bad example since it contains only single words with open("/usr/share/dict/words", mode="r") as infile: corpus = { num: s.strip() for num, s in enumerate(infile.readlines()) } # encode the corpus using a good sentence transformer model - will be slow if no GPU model = SentenceTransformer("all-mpnet-base-v2") corpus_vectors = model.encode(sentences=list(corpus.values())) # construct a faiss kNN index num_vectors, num_dimensions = corpus_vectors.shape index = faiss.index_factory(num_dimensions, "L2norm,Flat") index.add(corpus_vectors) # optional: save index for reuse faiss.write_index(index, "/tmp/corpus_index.bin") # index = faiss.read_index("/tmp/corpus_index.bin") # encode target text and find 10 nearest neighbors in index target_vector = model.encode(sentences=["apples"]) distances, nearest_indexes = index.search(target_vector, 10) print(list(zip([corpus[i] for i in nearest_indexes[0]], distances[0]))) # [('apples', 4.382169e-13), ('fruits', 0.47413948), ('fruit', 0.57227534), ...
- vimota 4y agoThis is great, thanks!
- KRAKRISMOTT 4y agoPerhaps the Faiss dependency can be dropped? Supabase supports the pg_vector extension now.
- bilater 4y agomost HN comment ever
- sjcsjc 4y agoYes. Very Dropbox.
- Ozzie_osman 4y agoNot really. That famous Dropbox comment was questioning the value of the product while offering a more difficult way to get the same result. Commenter here is not questioning the value of any product, but offering a potentially simpler way of achieving the same thing.
- EGreg 4y agoVery HN thread, I dig it
- mrwnmonm 4y agoWhat is the famous Dropbox comment?
- Ozzie_osman 4y agohttps://news.ycombinator.com/item?id=9224 https://news.ycombinator.com/item?id=9224
- mrwnmonm 4y agoThanks <3
- theturtletalks 4y agoThe DropBox comment was also “offering” a simpler way of storing data using rsync. The issue is that these guys solved a problem and are seeing the monetary benefit, but the first comment is explaining in tech jargon how to solve this problem “better”. Can’t that user just accept the end result is the same, but OP got there without in-depth knowledge of technology that can solve this faster?
- deleted 4y ago[deleted]
- cardosof 4y agoI come to HN for posts like this, thank you.