7 ms·

Using GPT3, Supabase and Pinecone to automate a personalized marketing campaign

252 points by vimota 4y ago

mdorazio 4y ago
Are you saving the match pairs somewhere? I imagine 1) there are a finite number of them, 2) doing an exact lookup in a DB first will be faster and easier than calling GPT3 and Pinecone every time, and 3) eventually GPT3 APIs will get pricey enough to make you think twice unless you're running your own instance on a cluster.
swyx 4y ago
> And it pretty much worked! Using prompts to find matches is not really ideal, but we want to use GPT's semantic understanding. That's where Embeddings come in. sounds like you ended up not using GPT3 in the end which is probably wise. i'm curious if you might see further savings using other cheaper embeddings that are available on huggingface. but its probably not material at this point. did you also consider using pgvector instead of pinecone? https://news.ycombinator.com/item?id=34684593 https://news.ycombinator.com/item?id=34684593 any painpoints with pinecone you can recall?
- roseway4 4y ago
  I've seen really good results using BERT and other open-source models for matching symptoms / healthcare service names to CPT/HCPCS code descriptions. Even for specialized domains, some of the freely available models perform well for ANN search. While BERT may not be viewed as state-of-the-art, versions of it have relatively low memory requirements versus newer models which is nice if you're self-hosting and/or care about scalability.
  - joshgel 4y ago
    I’m curious what problems you’ve applied this to. Would love to chat if you are open. (My email is in my profile)
- devxpy 4y ago
  Creating the index on pinecone takes about a minute. Creating a table on postgres should take a few milliseconds!
- vimota 4y ago
  I didn't use the GPT3 autocomplete API in the end (though I did play around with it) but did use the embeddings API (which I believe is still considered part of the "GPT3" model, but I could be wrong!). I totally could! I think each use case should dictate which model you should use, in my case I was not super cost or latency sensitive since it was a small dataset and I cared more about accuracy. But I'm planning on using something like https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 https://huggingface.co/sentence-transformers/all-MiniLM-L6-v... for my next project where latency and cost will matter more :) I have a lot of thoughts around that last question! The Supabase article came out way after I implemented this (August of last year) so I didn't even think to do that, not sure if it was even supported back then, but I'd probably reach for that if I was re-doing the project to reduce the number of systems I needed. I think the power of having the vector search done in the same DB as the rest of the data is that sometimes you may want to have structured filtering before the semantic/vector ranking (ie. only select user N's items and rank by similarity to <query>) which is trickier to do in Pinecone. They support metadata filtering but it feels like an after thought. For the project I'm working on now (https://pinched.io https://pinched.io) , we'd like to filter on certain parameters as well as rank by relevance, so I'm going to explore combining structured querying with semantic search (ie. pgvector or something similar on DuckDB if it adds support for this).
  - swyx 4y ago
    > https://pinched.io https://pinched.io requested invite! i have a moderately large twitter so could be a good test heheh. i use https://www.flock.network/ https://www.flock.network/ for this stuff normally but the UX isnt that great so hoping for better.
- itake 4y ago
  My understanding is pinecone is much faster, but for this small search space, I doubt pgvectors would be noticably worse. I tested an early version of pgvector against faiss and found faiss had much better performance https://github.com/pgvector/pgvector/issues/3 https://github.com/pgvector/pgvector/issues/3
deleted 4y ago
[deleted]
Hyption 4y ago
I don't like the unscientific ad for his gf company. 'which helped launch the movement of those opposed to endocrine disruptors, was retracted and its author found to have committed scientific misconduct'
- oluwie 4y ago
  mehh … you might be overthinking it it’s his blog either way.
jamesblonde 4y ago
I have see a lot of people write about how important the interaction between vector DBs and chat-GPT3 (and GPT3) is. I am still not much wiser after this article. Is it that it makes it easier to go from: user query -> GPT3 response -> Lookup in VectorDB -> send response based on closest embedding in VectorDB ?
- pablo24602 4y ago
  All the embedding-enabled GPT-3 apps I've seen do the following: User query -> Retrieve closest embedding's plaintext counterpart -> feed plaintext as context to GPT-3 prompt.
  - jamesblonde 4y ago
    Is this a form of prompt engineering then? Your vector DB has well formed prompts - users write random stuff, map it to the closest well formed prompt?
    - roseway4 4y ago
      In this context, it is for semantic matching similar to: "My daily face cream is BrandX's low-sheen formulation" -> "BrandX Matte Face Moisturizer"
    - deleted 4y ago
      [deleted]
    - Ozzie_osman 4y ago
      It's more like "prompt augmentation" or "prompt orchestration". Classic example is doing Q&A over a corpus. You can't feed the entire corpus into a GPT3 prompt. So you embed snippets of the corpus on vector space, then when you get a query, you vectorize that and find the nearest neighbor snippets, then send the question and snippets into GPT3 to answer the question (with those snippets as context). OP's example is a little different, because he's not even using Gpt3 completions, he's just using their embeddings API to vectorize product names, then when he gets a new product name, he maps it into the space to find the nearest product names.
pjakubowski 4y ago
Awesome to see the integration between Klaviyo automation and GPT-3 AI and using it to streamline your girlfriends processes. Keep up the fantastic work!
- hummus_bae 4y ago
  [dead]
pbourke 4y ago
> My script read through each of the products we had responses for, called OpenAI's embedding api and loaded it into Pinecone - with a reference to the Supabase response entry. OpenAI and the Pinecone database are not really needed for this task. A simple SBERT encoding of the product texts, followed by storing the vectors in a dense numpy array or faiss index would be more than sufficient. Especially if one is operating in batch mode, the locality and simplicity can’t be beat and you can easily scale to 100k-1M texts in your corpus on commodity hardware/VPS (though NVME disk will see a nice performance gain over regular SSD)
- vimota 4y ago
  Yep that's true! I'd probably do something like that if I were starting again, but the ease of calling a few APIs is pretty nice. I feel like that alone will drive a lot of adoption of some of these platforms even if it can just be done locally.
  - rvnx 4y ago
    Great result!
  - freediver 4y ago
    I'd argue that using something like SBERT + Faiss is easier and would take less time (you do not have two account creations + one billing setup), plus a working example of SBERT + Faiss is probably total less than 10 lines of code.
    - rattray 4y ago
      What does it look like? I've never heard of either.
      - deleted 4y ago
        [deleted]
      - pbourke 4y ago
        # pip install faiss-cpu sentence-transformers from sentence_transformers import SentenceTransformer import faiss # replace with own texts - this is a bad example since it contains only single words with open("/usr/share/dict/words", mode="r") as infile: corpus = { num: s.strip() for num, s in enumerate(infile.readlines()) } # encode the corpus using a good sentence transformer model - will be slow if no GPU model = SentenceTransformer("all-mpnet-base-v2") corpus_vectors = model.encode(sentences=list(corpus.values())) # construct a faiss kNN index num_vectors, num_dimensions = corpus_vectors.shape index = faiss.index_factory(num_dimensions, "L2norm,Flat") index.add(corpus_vectors) # optional: save index for reuse faiss.write_index(index, "/tmp/corpus_index.bin") # index = faiss.read_index("/tmp/corpus_index.bin") # encode target text and find 10 nearest neighbors in index target_vector = model.encode(sentences=["apples"]) distances, nearest_indexes = index.search(target_vector, 10) print(list(zip([corpus[i] for i in nearest_indexes[0]], distances[0]))) # [('apples', 4.382169e-13), ('fruits', 0.47413948), ('fruit', 0.57227534), ...
throwthere 4y ago
This looks incredible and magical to me. How do you learn to create things like this as a mostly web programmer? Vectorization, etc I had no idea could integrate with gpt etc but honestly it looks kind of obvious/effortless to the author.
- vimota 4y ago
  Appreciate the kind words, but I'm sure you could pick it up pretty quickly too :) The OpenAI docs are pretty helpful as a starting point: https://platform.openai.com/docs/guides/embeddings/use-cases https://platform.openai.com/docs/guides/embeddings/use-cases
djoldman 4y ago
How long did this take? Did you consider something like openrefine or fuzzy matching / levenshtein distance? Seems like a common data cleaning ask with a small amount of data.
- vimota 4y ago
  I played around with that and pgtrgm (https://www.postgresql.org/docs/current/pgtrgm.html https://www.postgresql.org/docs/current/pgtrgm.html) a bit but unfortunately didn't have great results. I did do a bunch of manual data cleaning though, and also had some overriding logic if certain keywords match it would avoid semantic search and default to a result (for common ones).
sexangel 4y ago
> 100s of human hours saved wait till its thousands, millions, billions . . .
EGreg 4y ago
Why not just use GPT-3 or even GPT-2 classifier API? No generative AI needed
- EForEndeavour 4y ago
  Looks like the Classifications API is deprecated: https://platform.openai.com/docs/guides/classifications https://platform.openai.com/docs/guides/classifications
- apienx 4y ago
  Embeddings are superior (classifiers are deprecated).
wyem 4y ago
Loved reading it. Will feature this in my newsletter on AI Tools and learning resources, AI Brews https://aibrews.com https://aibrews.com
NotYourLawyer 4y ago
This is pure spam.
mattfrommars 4y ago
Pardon my ignorance here. I started to play around with text generation today and came around plenty of resource but hard to make any sense of it. I had this working https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui and instead of it being able to answer question, it revolves around the concept of generating text. In your case and ChatGPT3, does is it provide output based on the data you feed it? If that is the case, is there anything related to training the model to use your data? I am trying to gauge a sense of what is going on.
espe 4y ago
i fail to see how this data cleaning could not be solved with proper tokenization and some distance measure. the amount of power used for those api calls is slighty obscene. edit: don't want to rant. it's not a bad post and i'm sure there is many and far more wasteful examples than this.
ipv6ipv4 4y ago
How do you know if the output that was sent to customers (who believe they are getting accurate results from a knowledgable human being, BTW) is correct?
1f60c 4y ago
I'm disappointed that the article doesn’t explain what they ended up doing.
fswd 4y ago
What is pinecone and is there a link to a website?
- madmax108 4y ago
  Pinecone is a vector database: https://www.pinecone.io/ https://www.pinecone.io/

252 points by vimota 4y ago

Are you saving the match pairs somewhere? I imagine 1) there are a finite number of them, 2) doing an exact lookup in a DB first will be faster and easier than calling GPT3 and Pinecone every time, and 3) eventually GPT3 APIs will get pricey enough to make you think twice unless you're running your own instance on a cluster.

swyx 4y ago

> And it pretty much worked! Using prompts to find matches is not really ideal, but we want to use GPT's semantic understanding. That's where Embeddings come in. sounds like you ended up not using GPT3 in the end which is probably wise. i'm curious if you might see further savings using other cheaper embeddings that are available on huggingface. but its probably not material at this point. did you also consider using pgvector instead of pinecone? https://news.ycombinator.com/item?id=34684593 https://news.ycombinator.com/item?id=34684593 any painpoints with pinecone you can recall?

roseway4 4y ago
I've seen really good results using BERT and other open-source models for matching symptoms / healthcare service names to CPT/HCPCS code descriptions. Even for specialized domains, some of the freely available models perform well for ANN search. While BERT may not be viewed as state-of-the-art, versions of it have relatively low memory requirements versus newer models which is nice if you're self-hosting and/or care about scalability.
- joshgel 4y ago
  I’m curious what problems you’ve applied this to. Would love to chat if you are open. (My email is in my profile)
devxpy 4y ago
Creating the index on pinecone takes about a minute. Creating a table on postgres should take a few milliseconds!
vimota 4y ago
I didn't use the GPT3 autocomplete API in the end (though I did play around with it) but did use the embeddings API (which I believe is still considered part of the "GPT3" model, but I could be wrong!). I totally could! I think each use case should dictate which model you should use, in my case I was not super cost or latency sensitive since it was a small dataset and I cared more about accuracy. But I'm planning on using something like https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 https://huggingface.co/sentence-transformers/all-MiniLM-L6-v... for my next project where latency and cost will matter more :) I have a lot of thoughts around that last question! The Supabase article came out way after I implemented this (August of last year) so I didn't even think to do that, not sure if it was even supported back then, but I'd probably reach for that if I was re-doing the project to reduce the number of systems I needed. I think the power of having the vector search done in the same DB as the rest of the data is that sometimes you may want to have structured filtering before the semantic/vector ranking (ie. only select user N's items and rank by similarity to <query>) which is trickier to do in Pinecone. They support metadata filtering but it feels like an after thought. For the project I'm working on now (https://pinched.io https://pinched.io) , we'd like to filter on certain parameters as well as rank by relevance, so I'm going to explore combining structured querying with semantic search (ie. pgvector or something similar on DuckDB if it adds support for this).
- swyx 4y ago
  > https://pinched.io https://pinched.io requested invite! i have a moderately large twitter so could be a good test heheh. i use https://www.flock.network/ https://www.flock.network/ for this stuff normally but the UX isnt that great so hoping for better.
itake 4y ago
My understanding is pinecone is much faster, but for this small search space, I doubt pgvectors would be noticably worse. I tested an early version of pgvector against faiss and found faiss had much better performance https://github.com/pgvector/pgvector/issues/3 https://github.com/pgvector/pgvector/issues/3

deleted 4y ago

[deleted]

Hyption 4y ago

I don't like the unscientific ad for his gf company. 'which helped launch the movement of those opposed to endocrine disruptors, was retracted and its author found to have committed scientific misconduct'

oluwie 4y ago
mehh … you might be overthinking it it’s his blog either way.

jamesblonde 4y ago

I have see a lot of people write about how important the interaction between vector DBs and chat-GPT3 (and GPT3) is. I am still not much wiser after this article. Is it that it makes it easier to go from: user query -> GPT3 response -> Lookup in VectorDB -> send response based on closest embedding in VectorDB ?

pablo24602 4y ago
All the embedding-enabled GPT-3 apps I've seen do the following: User query -> Retrieve closest embedding's plaintext counterpart -> feed plaintext as context to GPT-3 prompt.
- jamesblonde 4y ago
  Is this a form of prompt engineering then? Your vector DB has well formed prompts - users write random stuff, map it to the closest well formed prompt?
  - roseway4 4y ago
    In this context, it is for semantic matching similar to: "My daily face cream is BrandX's low-sheen formulation" -> "BrandX Matte Face Moisturizer"
  - deleted 4y ago
    [deleted]
  - Ozzie_osman 4y ago
    It's more like "prompt augmentation" or "prompt orchestration". Classic example is doing Q&A over a corpus. You can't feed the entire corpus into a GPT3 prompt. So you embed snippets of the corpus on vector space, then when you get a query, you vectorize that and find the nearest neighbor snippets, then send the question and snippets into GPT3 to answer the question (with those snippets as context). OP's example is a little different, because he's not even using Gpt3 completions, he's just using their embeddings API to vectorize product names, then when he gets a new product name, he maps it into the space to find the nearest product names.

pjakubowski 4y ago

Awesome to see the integration between Klaviyo automation and GPT-3 AI and using it to streamline your girlfriends processes. Keep up the fantastic work!

hummus_bae 4y ago
[dead]

pbourke 4y ago

> My script read through each of the products we had responses for, called OpenAI's embedding api and loaded it into Pinecone - with a reference to the Supabase response entry. OpenAI and the Pinecone database are not really needed for this task. A simple SBERT encoding of the product texts, followed by storing the vectors in a dense numpy array or faiss index would be more than sufficient. Especially if one is operating in batch mode, the locality and simplicity can’t be beat and you can easily scale to 100k-1M texts in your corpus on commodity hardware/VPS (though NVME disk will see a nice performance gain over regular SSD)

vimota 4y ago
Yep that's true! I'd probably do something like that if I were starting again, but the ease of calling a few APIs is pretty nice. I feel like that alone will drive a lot of adoption of some of these platforms even if it can just be done locally.
- rvnx 4y ago
  Great result!
- freediver 4y ago
  I'd argue that using something like SBERT + Faiss is easier and would take less time (you do not have two account creations + one billing setup), plus a working example of SBERT + Faiss is probably total less than 10 lines of code.
  - rattray 4y ago
    What does it look like? I've never heard of either.
    - deleted 4y ago
      [deleted]
    - pbourke 4y ago
      # pip install faiss-cpu sentence-transformers from sentence_transformers import SentenceTransformer import faiss # replace with own texts - this is a bad example since it contains only single words with open("/usr/share/dict/words", mode="r") as infile: corpus = { num: s.strip() for num, s in enumerate(infile.readlines()) } # encode the corpus using a good sentence transformer model - will be slow if no GPU model = SentenceTransformer("all-mpnet-base-v2") corpus_vectors = model.encode(sentences=list(corpus.values())) # construct a faiss kNN index num_vectors, num_dimensions = corpus_vectors.shape index = faiss.index_factory(num_dimensions, "L2norm,Flat") index.add(corpus_vectors) # optional: save index for reuse faiss.write_index(index, "/tmp/corpus_index.bin") # index = faiss.read_index("/tmp/corpus_index.bin") # encode target text and find 10 nearest neighbors in index target_vector = model.encode(sentences=["apples"]) distances, nearest_indexes = index.search(target_vector, 10) print(list(zip([corpus[i] for i in nearest_indexes[0]], distances[0]))) # [('apples', 4.382169e-13), ('fruits', 0.47413948), ('fruit', 0.57227534), ...

throwthere 4y ago

This looks incredible and magical to me. How do you learn to create things like this as a mostly web programmer? Vectorization, etc I had no idea could integrate with gpt etc but honestly it looks kind of obvious/effortless to the author.

vimota 4y ago
Appreciate the kind words, but I'm sure you could pick it up pretty quickly too :) The OpenAI docs are pretty helpful as a starting point: https://platform.openai.com/docs/guides/embeddings/use-cases https://platform.openai.com/docs/guides/embeddings/use-cases

djoldman 4y ago

How long did this take? Did you consider something like openrefine or fuzzy matching / levenshtein distance? Seems like a common data cleaning ask with a small amount of data.

vimota 4y ago
I played around with that and pgtrgm (https://www.postgresql.org/docs/current/pgtrgm.html https://www.postgresql.org/docs/current/pgtrgm.html) a bit but unfortunately didn't have great results. I did do a bunch of manual data cleaning though, and also had some overriding logic if certain keywords match it would avoid semantic search and default to a result (for common ones).

sexangel 4y ago

> 100s of human hours saved wait till its thousands, millions, billions . . .

EGreg 4y ago

Why not just use GPT-3 or even GPT-2 classifier API? No generative AI needed

EForEndeavour 4y ago
Looks like the Classifications API is deprecated: https://platform.openai.com/docs/guides/classifications https://platform.openai.com/docs/guides/classifications
apienx 4y ago
Embeddings are superior (classifiers are deprecated).

wyem 4y ago

Loved reading it. Will feature this in my newsletter on AI Tools and learning resources, AI Brews https://aibrews.com https://aibrews.com

NotYourLawyer 4y ago

This is pure spam.

mattfrommars 4y ago

Pardon my ignorance here. I started to play around with text generation today and came around plenty of resource but hard to make any sense of it. I had this working https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui and instead of it being able to answer question, it revolves around the concept of generating text. In your case and ChatGPT3, does is it provide output based on the data you feed it? If that is the case, is there anything related to training the model to use your data? I am trying to gauge a sense of what is going on.

espe 4y ago

i fail to see how this data cleaning could not be solved with proper tokenization and some distance measure. the amount of power used for those api calls is slighty obscene. edit: don't want to rant. it's not a bad post and i'm sure there is many and far more wasteful examples than this.

ipv6ipv4 4y ago

How do you know if the output that was sent to customers (who believe they are getting accurate results from a knowledgable human being, BTW) is correct?

1f60c 4y ago

I'm disappointed that the article doesn’t explain what they ended up doing.

fswd 4y ago

What is pinecone and is there a link to a website?

madmax108 4y ago
Pinecone is a vector database: https://www.pinecone.io/ https://www.pinecone.io/