3 ms·

> And it pretty much worked! Using prompts to find matches is not really ideal, but we want to use GPT's semantic understanding. That's where Embeddings come in

by swyx 4y ago

> And it pretty much worked! Using prompts to find matches is not really ideal, but we want to use GPT's semantic understanding. That's where Embeddings come in. sounds like you ended up not using GPT3 in the end which is probably wise. i'm curious if you might see further savings using other cheaper embeddings that are available on huggingface. but its probably not material at this point. did you also consider using pgvector instead of pinecone? https://news.ycombinator.com/item?id=34684593 https://news.ycombinator.com/item?id=34684593 any painpoints with pinecone you can recall?

roseway4 4y ago
I've seen really good results using BERT and other open-source models for matching symptoms / healthcare service names to CPT/HCPCS code descriptions. Even for specialized domains, some of the freely available models perform well for ANN search. While BERT may not be viewed as state-of-the-art, versions of it have relatively low memory requirements versus newer models which is nice if you're self-hosting and/or care about scalability.
- joshgel 4y ago
  I’m curious what problems you’ve applied this to. Would love to chat if you are open. (My email is in my profile)
devxpy 4y ago
Creating the index on pinecone takes about a minute. Creating a table on postgres should take a few milliseconds!
vimota 4y ago
I didn't use the GPT3 autocomplete API in the end (though I did play around with it) but did use the embeddings API (which I believe is still considered part of the "GPT3" model, but I could be wrong!). I totally could! I think each use case should dictate which model you should use, in my case I was not super cost or latency sensitive since it was a small dataset and I cared more about accuracy. But I'm planning on using something like https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 https://huggingface.co/sentence-transformers/all-MiniLM-L6-v... for my next project where latency and cost will matter more :) I have a lot of thoughts around that last question! The Supabase article came out way after I implemented this (August of last year) so I didn't even think to do that, not sure if it was even supported back then, but I'd probably reach for that if I was re-doing the project to reduce the number of systems I needed. I think the power of having the vector search done in the same DB as the rest of the data is that sometimes you may want to have structured filtering before the semantic/vector ranking (ie. only select user N's items and rank by similarity to <query>) which is trickier to do in Pinecone. They support metadata filtering but it feels like an after thought. For the project I'm working on now (https://pinched.io https://pinched.io) , we'd like to filter on certain parameters as well as rank by relevance, so I'm going to explore combining structured querying with semantic search (ie. pgvector or something similar on DuckDB if it adds support for this).
- swyx 4y ago
  > https://pinched.io https://pinched.io requested invite! i have a moderately large twitter so could be a good test heheh. i use https://www.flock.network/ https://www.flock.network/ for this stuff normally but the UX isnt that great so hoping for better.
itake 4y ago
My understanding is pinecone is much faster, but for this small search space, I doubt pgvectors would be noticably worse. I tested an early version of pgvector against faiss and found faiss had much better performance https://github.com/pgvector/pgvector/issues/3 https://github.com/pgvector/pgvector/issues/3

by swyx 4y ago

roseway4 4y ago

I've seen really good results using BERT and other open-source models for matching symptoms / healthcare service names to CPT/HCPCS code descriptions. Even for specialized domains, some of the freely available models perform well for ANN search. While BERT may not be viewed as state-of-the-art, versions of it have relatively low memory requirements versus newer models which is nice if you're self-hosting and/or care about scalability.

joshgel 4y ago
I’m curious what problems you’ve applied this to. Would love to chat if you are open. (My email is in my profile)

devxpy 4y ago

Creating the index on pinecone takes about a minute. Creating a table on postgres should take a few milliseconds!

vimota 4y ago

I didn't use the GPT3 autocomplete API in the end (though I did play around with it) but did use the embeddings API (which I believe is still considered part of the "GPT3" model, but I could be wrong!). I totally could! I think each use case should dictate which model you should use, in my case I was not super cost or latency sensitive since it was a small dataset and I cared more about accuracy. But I'm planning on using something like https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 https://huggingface.co/sentence-transformers/all-MiniLM-L6-v... for my next project where latency and cost will matter more :) I have a lot of thoughts around that last question! The Supabase article came out way after I implemented this (August of last year) so I didn't even think to do that, not sure if it was even supported back then, but I'd probably reach for that if I was re-doing the project to reduce the number of systems I needed. I think the power of having the vector search done in the same DB as the rest of the data is that sometimes you may want to have structured filtering before the semantic/vector ranking (ie. only select user N's items and rank by similarity to <query>) which is trickier to do in Pinecone. They support metadata filtering but it feels like an after thought. For the project I'm working on now (https://pinched.io https://pinched.io) , we'd like to filter on certain parameters as well as rank by relevance, so I'm going to explore combining structured querying with semantic search (ie. pgvector or something similar on DuckDB if it adds support for this).

swyx 4y ago
> https://pinched.io https://pinched.io requested invite! i have a moderately large twitter so could be a good test heheh. i use https://www.flock.network/ https://www.flock.network/ for this stuff normally but the UX isnt that great so hoping for better.

itake 4y ago

My understanding is pinecone is much faster, but for this small search space, I doubt pgvectors would be noticably worse. I tested an early version of pgvector against faiss and found faiss had much better performance https://github.com/pgvector/pgvector/issues/3 https://github.com/pgvector/pgvector/issues/3