3 ms·
Credit goes to Chris Lee: https://www.chrisleeportfolio.com https://www.chrisleeportfolio.com Explanation when you click “Wat this?”: > This is a Streamlit ap
by gk1 4y ago
Credit goes to Chris Lee: https://www.chrisleeportfolio.com https://www.chrisleeportfolio.com
Explanation when you click “Wat this?”:
> This is a Streamlit app I prototyped for performing semantic search on the King James Bible. It conducts full text search as well as semantic search, which is useful for surfacing passages that are similar in meaning to the query, even if the passages don't explicitly contain the query keyword(s). Suppose you wanted to bring up all verses that reference the infamous snake that tempted Eve. In a traditional keyword search system, searching for 'snake' wouldn't yield any results because the KJV uses the term 'serpent'. A semantic search system would take that 'snake' query and retrieve the relevant verses that contain 'serpent' as well as similar verses like ones about reptiles.
> Under the hood, I've generated vector embeddings of every verse in the Bible using SBERT (https://www.sbert.net/ https://www.sbert.net/), and stored those embeddings in a vector database called Pinecone (https://www.pinecone.io https://www.pinecone.io). Every time you submit a query, it's converted to its vector representation using SBERT. That query vector is then sent to Pinecone, which performs an Approximate Nearest Neighbor (https://www.pinecone.io/learn/what-is-similarity-search/ https://www.pinecone.io/learn/what-is-similarity-search/) search, retrieving the top n verses that are the most semantically similar to our query. The verses returned are ranked in order of most to least similar.
(Full disclosure: I work for Pinecone, but I have no connection to this demo.)
- angrygoat 4y agoHow does pinecone compare to pgvector? https://github.com/pgvector/pgvector https://github.com/pgvector/pgvector
- michaelmior 4y agoMilvus[0] would be another interesting comparison. [0] https://milvus.io/ https://milvus.io/
- gk1 4y agoSame general difference: Pinecone is SaaS that's ready to go with a few API calls, while Milvus an open-source tool that requires work to set up, scale, and maintain.
- michaelmior 4y agoI meant in terms of performance, but all those points are certainly true.
- chrislee973 4y agoIn terms of performance, I'm not sure, although I'm willing to bet on Pinecone as they have their own proprietary indexing algorithm. I can speak in terms of developer experience though, as I did use Pinecone to build this app. One obvious but important difference is that with pgvector, you need to spin up and self host a postgres server on your own. Pinecone provides a batteries-included, fully managed experience. Also, it seems that a lot of important operations in pgvector are conducted with SQL statements. With Pinecone, you don't need to deal with SQL or with any ORMs. In terms of this project, I would not have chosen pgvector. I don't want to deal with the PITA that comes with manually setting up and self hosting a vector database. When I'm building a demo or a prototype, I care about speed of development, which lets me more effectively explore the possibility space of whatever I'm building. I'm not a database admin, so when I deploy the project, I don't want to do database admin tasks. The ease of use of Pinecone's API lets me move fast, and it was very intuitive to learn. One downside of Pinecone is that although they have a generous free tier, their next highest tier is $50/month (and that's the low end of that tier). This is unfortunate for solo devs like myself who are likely to graduate from the free tier and would be willing to pay a bit more for a higher tier, but find the $50/month plan to be overkill. I think they're trying to target startups with that $50/month plan.
- deleted 4y ago[deleted]