13 ms·

What is a Vector Database? (2021)

• soft_camel 3y ago
  An alternative to Pinecone is MyScale DB. It is SQL-based vector database and has generous free tier now. https://myscale.com/ https://myscale.com/ 1. Manages both structured and vectorized data in a single database and can perform joint queries and analytics on both types of data. 2. Cloud-native OLAP database architecture enables operations on vectorized data to be executed with astounding speed. 3. Complete and extended SQL support for all data operations, accessible via developer tools such as Python SDK.
• etiam 3y ago
  Salakhutdinov R. R, and Hinton, G. E. (2007) "Semantic Hashing" Proceedings of the SIGIR Workshop on Information Retrieval and Applications of Graphical Models https://www.cs.toronto.edu/~hinton/absps/sh.pdf https://www.cs.toronto.edu/~hinton/absps/sh.pdf
• defrost 3y ago
  I'm assuming, at its core, a vector database is a collection of vectors all of the same dimension N, for some N, and is a set of points in R^N ? ( aside from all the per vector associated meta-data, various dimensional reduction, nearest neighbour, etc, operations describe in the article ).
  • modernpink 3y ago
    Not sure what this comment is driving at. "At it's core" a database is a collection of data, sure.
    • defrost 3y ago
      and a "vector database" is a collection of vectors, sure. The question comes from is it the CompSci | HN | AI domain nomenclature to assume a vector database is made up of vectors that are all of the same dimension over a continuum (eg. N real numbers for fixed N) or are vector databases made up of mixed vectors (no fixed dimension) and discrete values, etc. I ask as the linked article doesn't specify but does appear to imply.
      • seanhunter 3y ago
        As per my comment earlier, the dimension isn't fixed. The usual use case (storing embeddings) is instructive as to the range of values. For token embeddings, often the embedding is generated via a lookup in a fixed vocabulary of tokens token to a token ID. So say your vocab is words, the value is a word id which would obviously be an integer not a real. Here's an intro to word embeddings https://wiki.pathmind.com/word2vec https://wiki.pathmind.com/word2vec and here's one for positional embeddings (the new hotness given how zeitgeisty GPTs are) https://theaisummer.com/positional-embeddings/ https://theaisummer.com/positional-embeddings/
      • modernpink 3y ago
        An embedding model will map a string of text (of variable length) to R^D. Each model has its own fixed D, yes. (Typically the vector is a unit vector for performance reasons.) The main function of vector database is similarity lookup so you would calculate approximate nearest neighbours of a vector with a scalar vector distance metric (e.g cosine similarity). These similarity metrics operate on two vectors of the same dimension. You would not mix and match embedding models (e.g. with differing dimensions) at look-up time. The target vector table assumes you will look it up with a vector created from the exact same embedding model and version that was used to backfill it. The API documentation for a look up operation may be more illuminating here: >vector (array of floats) >The query vector. This should be the same length as the dimension of the index being queried. Each query() request can contain only one of the parameters id or vector. https://docs.pinecone.io/reference/query https://docs.pinecone.io/reference/query
        • seanhunter 3y ago
          D is fixed for the model but not for the database. You don't need a seperate database for each model.
          • modernpink 3y ago
            Yes that's imprecision on my part. For a table D is fixed, but not necessarily across tables in the vector database index
  • seanhunter 3y ago
    Not really. For starters not all the vectors in the database are of the same dimension (at least in ones I've used). You can for example in pgvector have multiple tables with a column of vector type and those tables can have different cardinality. In any case your description is strangely reductionist. The important thing about a vector db is that is typically designed to store embeddings used in various ML applications. So say you are doing NLP you can tokenize some input and then store the token and positional embeddings in a vector db and then use it for similarity search, training etc.
    • defrost 3y ago
      Mixed tables each with reduced cardinalities makes sense. > your description is strangely reductionist. Sure - pure | applied math background, old enough to have used Postgres when it was known as Ingres, to have patched in Spatial relations before it had the GIS functions it has now, and to have written libraries for GIS linked { 256 | 1024 | 2048 } D vector databases for signal aquisition | processing. I'm late to the 'modern' discussions & just checking my read - I can think of applications for mixed dimensions and discrete space vectors and there are analogs to trad R^N ops for those cases.
      • tudorw 3y ago
        i think it might be a n-dimensional topological manifold with a nth dimensional geodesic supplying the shortest path, anyone actually done topology?
        • tudorw 3y ago
          if you take a 2d piece of paper, draw the numbers 1 to 5 horizontally, then look at the distance between 1 and 5, or 1 and 3, they vary, now roll it into as cylinder, it's a manifold, and now the distance between 1 and 5 and 1 and 1 and 3 are the same, no matter how many numbers you had written on the paper, they would be connected by the shortest path, something like a helix I believe, the helix being the geodesic, now, increase the dimension, but keep rolling it up into that 'cylinder' and using a new geodesic to connect the shortest path, this is my layman's take on it, if you are #math please jump in and put me right!
• mritchie712 3y ago
  If you want to play with a vector database and already use postgres, there's pgvector[0]. It's easy to add as an extension (supports Postgres 11+). Supabase wrote a solid tutorial[1] (you don't need to run it on Supabase). 0 - https://github.com/pgvector/pgvector https://github.com/pgvector/pgvector 1 - https://supabase.com/blog/openai-embeddings-postgres-vector https://supabase.com/blog/openai-embeddings-postgres-vector
  • imaurer 3y ago
    I am bullish Pgvector because I am “postgres for everything guy”. Current concerns are the scaling and recall performance. The author is looking at product quantization along with other ideas: https://github.com/pgvector/pgvector/issues/27 https://github.com/pgvector/pgvector/issues/27 More details on product quantization: https://mccormickml.com/2017/10/13/product-quantizer-tutorial-part-1/ https://mccormickml.com/2017/10/13/product-quantizer-tutoria... A nice repo that tracks the ANN relative performance of different indexes: https://mccormickml.com/2017/10/13/product-quantizer-tutorial-part-1/ https://mccormickml.com/2017/10/13/product-quantizer-tutoria... Also shoutout to Weaviate because they have great docs, are open source and have very informative YouTube channel. https://weaviate.io/ https://weaviate.io/
    • mritchie712 3y ago
      Yep, we're (https://www.definite.app/ https://www.definite.app/) using pgvector and I was initially concerned about scaling, but it doesn't seem it will be a problem for our use case. I definitely wouldn't use it if I was building a feature for Slack, but works for us!
      • overview 3y ago
        At a glance, your product seems like a fit for my team. However, your landing page doesn’t give specifics. What exactly does it do?
        • mritchie712 3y ago
• shanghaikid 3y ago
  Why not choose an open-source solution https://github.com/milvus-io/milvus https://github.com/milvus-io/milvus, free!
  • Alifatisk 3y ago
    Or Redis Vector similarity?
  • sir_eliah 3y ago
    Does anyone have some real, production experience with milvus? I'm interested this database performs in larger scale. Let's say, you have millions of vectors and traffic reaching thousands requests/s.
    • bluecoconut 3y ago
      Not production, but yes to scale: I pushed milvus to ~140 million vectors (768 dimension) (though only a handful of requests per second (~10)), and it faired alright once everything was up and running and relatively static on the document side. Rebuilding indexes and stability were a bit of a hassle at times (I was live adding more documents to it ~1 million per 30 minutes) and it would occasionally fall over and need to rebuild, subsequently causing a lot more load, rejecting new documents, etc.). Probably lots of tuning I probably could have done to eek out more performance and stability though. Ended up being hours of effort on the rebuilds and lots of careful management of RAM (on a 300 GB RAM machine) for the scale you are saying "larger scale": At the few million documents scale I would just suggest using just any libary, eg. `hnsw` in `nmslib` or `faiss`. I just did some benchmarks with 1M docs, `cosinesimil_sparse` on `78628` dimensional binary vectors (nmslib `hnsw`) -> 30 seconds to build the index, and can process a batch of 100 document query in 3ms (Each with 100 KNN). Based on this question, i just put a loop over it and it handled 1000 random queries (non batched) in 1.11 seconds. (~1 GB peak RAM usage, and using 24 threads) All in all, my personal opinion is: even up to few "millions" scale, i'm finding using the underlying libraries (`faiss` and `nmslib`) significantly easier than using the wrapper tools / databases (milvus and pinecone). I don't really get the point of a separate piece of infra for something that is essentially ~15 lines of python at most scales that matter (~few millions). (Note, in the ~10k-100k scale or less, simple numpy and sort seems to be fast enough (and exact) or just exact NN w/ sklearn.neighbors)... And when you push to scales that it does start breaking (100 million+), then the database versions seem to break as well (and require fiddling with lots of bespoke config)
• deleted 3y ago
  [deleted]
• zcbenz 3y ago
  Suppose one is writing a desktop app doing searches of embeddings, is there a lightweight vector database that runs on all major desktop platforms? I'm aware of redis and sqlite-vss, but none runs natively on Windows.
  • jstx1 3y ago
    You can use libraries like faiss or ann for the search, you don't need a vector database when you work at small-to-medium scale.
  • killthebuddha 3y ago
    I would run postgres with pgvector https://github.com/pgvector/pgvector https://github.com/pgvector/pgvector. I wouldn't call postgres lightweight of course, but it's definitely lightweight in the sense that it doesn't add a whole bunch of new garbage to an otherwise traditional application.
  • dtrailin 3y ago
    Chroma runs on Windows since I believe it's just a python package: https://github.com/chroma-core/chroma https://github.com/chroma-core/chroma
• carom 3y ago
  I looked at the concepts in FAISS and it seems fairly straightforward. In non-jargon you have dimensionality reduction and neighborhoods. DR is taking a long embedding and doing something to make it shorter. An easy to follow method for this is minhash. Neighborhoods is representing a cluster of embeddings with a single representative to speed up comparisons. For example, find me the two closest representatives then doing a deeper comparison on all the residents. Now the feature I haven't seem that will probably cause me to build instead of buy. Most seem designed for a single organization and a single use. For example, Spotify song recommender. I would like to store embedding from multiple models and be able to search per model. I would also like fine grain user access control, so users could search their embeddings and grant access to others.
  • gk1 3y ago
    If the different models use the same dimensionality, you can keep their embeddings within different namespaces inside the same index. See: https://docs.pinecone.io/docs/namespaces https://docs.pinecone.io/docs/namespaces If you mean "user access control" within your company, there are basic access controls within Pinecone. See: https://docs.pinecone.io/docs/add-users-to-projects-and-organizations https://docs.pinecone.io/docs/add-users-to-projects-and-orga... If you mean for your end-users, you can use namespaces again to separate embeddings for different users inside one index. See: https://docs.pinecone.io/docs/multitenancy https://docs.pinecone.io/docs/multitenancy There isn't yet a combination of the two, where you provide Pinecone API access to end-users.
    • carom 3y ago
      Thank you, I'll definitely play with pinecone before I build. The dimensionality might vary between models or versions of models. Additionally, the end goal would be to expose it to users and not have to post filter. So probably an index per user. Not sure how expensive that is to recalculate regularly.
• Alifatisk 3y ago
  There is a lot of talks about Vector Databases nowadays, is there a trend I am missing out on?
  • WolfOliver 3y ago
    ChatGPT / OpenAI
  • Longwelwind 3y ago
    AI trend due to ChatGPT. For NLP use-cases, you can use a vector database to index the embeddings of your texts. For exemple, if you implement a document retrieval (a search engine like Google), you train a transformer model that takes a text as input (the content of your document) and produces a vector of number as output (the embedding). You then index your documents by transorming them to their embeddings and storing them inside your vector database. When you want to perform a query using keywords, you transform your keywords into a vector, and then ask your vector database to send you the most similar documents, using a similary function such as the cosine function.
    • politician 3y ago
      It seems like it would be better to index a document multiple times by generating embeddings for every paragraph rather than once per document. What am I missing?
• jn2clark 3y ago
  If anyone is looking for a vector search engine, see here https://github.com/marqo-ai/marqo https://github.com/marqo-ai/marqo. Has additional functionality to make vector search much easier.
• dcl 3y ago
  I don't understand how so much money has been poured in to these companies? I get the why the techniques are suitable, but I just assumed who ever wants to do this kind of retrieval can probably implement a suitable Approx. NN library themselves? Especially so, because getting good embeddings is the hard part, not the search?
  • jstx1 3y ago
    Anyone who wants this can implement their own library themselves? When has this worked for any problem ever? Searching efficiently is a problem, and there's several open source and proprietary solutions but I don't get how you can put it in the "everyone should roll their own" category.
  • softwaredoug 3y ago
    At billion vector scale, doing this yourself is pretty impossible
    • deleted 3y ago
      [deleted]
    • dmezzetti 3y ago
      Faiss has long discussed strategies for scaling to 1B - 1T records here - https://github.com/facebookresearch/faiss/wiki/Indexing-1G-vectors https://github.com/facebookresearch/faiss/wiki/Indexing-1G-v... There are plenty of options available to run your own local vector database, txtai is one of them. Ultimately depends if you have a sizable development team or not. But saying it is impossible is a step too far.
      • deleted 3y ago
        [deleted]
      • hiyou102 3y ago
        Even in that article with much smaller vectors than what GPT puts out (1536 dimensions) QPS drops below 100 if recall@1 is more than 0.4. That's to say nothing of cost of regenerating this index using incremental updates. I don't get why people on HN are so adamant on the idea that no one needs scale beyond 1 machine ever.
• wejick 3y ago
  For prototyping purpose I found that chromadb integration in langchain is very easy to use. That's said I'm not sure about production usage. https://blog.langchain.dev/langchain-chroma/ https://blog.langchain.dev/langchain-chroma/
  • tudorw 3y ago
    Trying chromaDB out, seems to work with langchain and langflow out of the box, nice :)
    • objectivetruth 3y ago
      [dead]
• softwaredoug 3y ago
  Y not Elasticsearch? I don't see it addressed in the article, but Elastic 8 has ANN support, and every other feature you'd expect out of a ranking system. Vectors are only one piece of the puzzle for building such a system. (honest question, not trying to troll, as I truly do <3 these pinecone articles) (Similarly, Y not Solr, Vespa, etc etc) :)
  • heipei 3y ago
    Plus Elasticsearch is a breeze to operate and scale in a fault-tolerant matter.
    • b112 3y ago
      I suspect missing sarcasm tags here. The very least, from a lack of a stable, security update only release beanch.
      • heipei 3y ago
        Not really, I've been operating 10+ node Elasticsearch cluster for years, running on a workload scheduler (Nomad). I never have to perform any maintenance or housekeeping except deleting old indices, and updates are performed by bumping a container version number and then restarting nodes one-by-one with a delay in between.
  • sidi 3y ago
    There currently isn't a way to filter docs alongside a KNN query, and the dimension support is limited to 1024 (a Lucene limitation) and OpenAI embeddings are 1536 dimensions - also indexing performance is not comparable. Wishing this changes, as they're a good stack for the reasons you state
    • softwaredoug 3y ago
      True though I do think 2k dims is coming it 8.8
• HarHarVeryFunny 3y ago
  Karpathy (OpenAI, ex. Tesla) recently tweeted about KISS and just using np.array instead of a vector database ! Searching for similar vectors is basically the (approximate) KNN problem, although I imagine more specialized search methods might apply depending on what you are doing.
  • dmezzetti 3y ago
    100% agree. There are so many simple use cases where people are jumping to a complex option to start. For something like 10K records, a NumPy or PyTorch matrix operation could be enough. Recently published an article discussing this: https://neuml.hashnode.dev/customize-your-own-embeddings-database https://neuml.hashnode.dev/customize-your-own-embeddings-dat...
    • javier2 3y ago
      Well, we have about 4B vectors we want to index, with a constantly changing dataset. We are actually running just fine on Elasticsearch with ~1B docs indexed right now, but the hardware costs are looking expensive.
      • dmezzetti 3y ago
        With 4B vectors, you can look at methods like quantization and compression, both detailed here for Faiss - https://github.com/facebookresearch/faiss/wiki/Indexing-1G-vectors https://github.com/facebookresearch/faiss/wiki/Indexing-1G-v... Elasticsearch uses HNSW, not sure what options they have but quantization/compression will help reduce disk storage requirements. Alternatively, you can look at dimensionality reduction algorithms and only store that output in ES. Or pick a model with a small number of dimensions. For example https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 https://huggingface.co/sentence-transformers/all-MiniLM-L6-v... only has 384 dims vs 768/1024/2048/4096.
  • 0xDEF 3y ago
    >Karpathy (OpenAI, ex. Tesla) recently tweeted about KISS and just using np.array instead of a vector database The context was a very underwhelming side project of his: A movie search engine but you had to use the exact titles of the movies to get results. It only revealed that he doesn't appreciate what similarity search actually is. It feels almost blasphemous to call a Karpathy side project underwhelming. He is a genius and it really felt unlike him to write that "just use np.array" tweet.
• phemartin 3y ago
  Listen to this article (24-min) at https://playtext.app/doc/clhahltvf0000l20fvv9rhkyb https://playtext.app/doc/clhahltvf0000l20fvv9rhkyb
• tomasreimers 3y ago
  I have a dumb question: isn't the the same set algorithms and technology we've needed to develop for geospatial search (given coordinates, find the nearest coordinates)? I remember reading about how Google maps did a very similar thing to figure out which points of interest to load based on your coordinates and zoom. Can't we repurpose that technology? Or did those bake in assumptions around being 2 dimensional (while this is highly dimensional?)
  • ninja3925 3y ago
    The main problem is that the embeddings are getting larger and larger (1,000+ dimensions). The pressure is then on reducing memory use through techniques such as Product Quantization while not losing too much accuracy. Once this is done, the search heuristics are not difficult (find the cells to explore and return nearest neighbors).
  • softwaredoug 3y ago
    Things get really weird in high dimensions. Orthogonality is expected, for example. Proximity is really rare. See: https://softwaredoug.com/blog/2023/02/28/probability-of-dot-product.html https://softwaredoug.com/blog/2023/02/28/probability-of-dot-...
  • pnathan 3y ago
    Higher dimensional algorithms get very messy - see X-trees vs R-trees. E.g., KNN is very fast, if and only if you have a high performing R-tree query. so while the 2d algos might "fall out" of the sophisticated cases, the sophisticated cases will need to go in a different optimization direction.
  • pbadams 3y ago
    Apart from the comments you've already gotten, another goal of geospatial systems is to support range queries (e.g. for the bounding box of the user's screen, what are all the businesses in that box). In higher dimensions range queries are mostly useless and the focus is on NN queries. But as the other comments have mostly said, it's mainly dimensionality and scale differences that drive the design differences (e.g. graphs end up working better than trees in high dimensions)
• quickthrower2 3y ago
  I feel like I need a Vector Database Database to choose the closest Vector Database (in cosine similarity) to the job at hand!
  • justinclift 3y ago
    Ahh, that'd be the Vector Database Broker then? ;)
  • softwaredoug 3y ago
    There is ANN Benchmarks http://ann-benchmarks.com/ http://ann-benchmarks.com/
• jn2clark 3y ago
  In the quest for ultimate speed, I started developing a vector database in assembly using gpt4 as a side project https://github.com/jn2clark/GPT4Memory https://github.com/jn2clark/GPT4Memory.
• jfengel 3y ago
  Can someone give me the quickie on what a vector embedding is? The article assumes I know it, and everything else in the article seems trivial.
  • asgeir 3y ago
    https://www.youtube.com/watch?v=gQddtTdmG_8 https://www.youtube.com/watch?v=gQddtTdmG_8
  • rocgf 3y ago
    I'm not a data scientist or anything, but as far as my understanding goes, you can think of it as a list of numbers (floats, integers, whatever), each number representing a feature of the thing you're trying to represent as a vector (an image, a video, whatever).
  • ninja3925 3y ago
    An embedding is a numerical representation of a complex object. In an image, each pixel is a dimension but it does not have any meaning in itself - you need to look at the rest of the image to understand that the pixel is part of a cat. Embeddings is a way to represent this meaning. Think about it as a “summary” of a image / document.
  • raindear 3y ago
    For example, you can create an embedding for an image using a neural net that has been trained to receive an image and output a vector of 1024 floats, which represent the content of the image. This vector is a lossy compressed version of the image.
    • jfengel 3y ago
      And (if I'm understanding correctly) vectors that are near each other (in a mathematical sense) represent inputs that are "near" each other (in a conceptual sense). So... a vector database can be organized to quickly retrieve objects with particular characteristics, without rigidly defining what those characteristics are. Have I got it?
• andre-z 3y ago
  An open-source Pinecone alternative https://github.com/qdrant/qdrant https://github.com/qdrant/qdrant With a cloud offering along https://cloud.qdrant.io https://cloud.qdrant.io with 1GB cluster for free to try out. Disclaimer: I'm part of the team.
  • mbrochh 3y ago
    Is it different/better than https://milvus.io https://milvus.io ?
  • jcmoscon 3y ago
    Hey this is pretty cool! I will try it! Thanks for sharing it.
  • echelon 3y ago
    Perfect for our next zero shot model. We'll give it a spin! Thanks for building this.
  • zcesur 3y ago
    qdrant also pays open source contributors: https://news.ycombinator.com/item?id=35828003 https://news.ycombinator.com/item?id=35828003 disclaimer: i'm a founder at algora.io, the platform that enables these paid contributions
    • ffback 3y ago
      "Algora charges a 23% fee over your rewarded bounties (20% Algora fee + 3% Stripe fee). The fee is applied when you complete your bounty payments." https://docs.algora.io/bounties/payments#compliance https://docs.algora.io/bounties/payments#compliance If algora.io didn't charge %23 of the bounty I would have tried to contribute. It felt unfair to me.
      • zcesur 3y ago
        hey ffback, contributors get 100% of the bounty award :) the organization pays the fee on top of the bounty. will update the docs to make this more clear, thank you!
• jabo 3y ago
  I get this question frequently - why not use FAISS or ANNOY directly, instead of a vector database, so glad to see this aspect covered in this article. Plug: If you're ever looking for an open source alternative to Pinecone, we recently added vector search to Typesense: https://typesense.org/docs/0.24.1/api/vector-search.html https://typesense.org/docs/0.24.1/api/vector-search.html The key thing is that it's in-memory and allows you to combine attribute-based filtering, together with HNSW-based nearest-neighbor search. We're also working on a way to automatically generate embeddings from within Typesense using any ML models of your choice. So Algolia + Pinecone + Open Source + Self-Hostable with a cloud hosted option = Typesense
• wskish 3y ago
  After working through several projects that utilized local hnswlib and different databases for text and vector persistence, I integrated open source hnswlib with sqlite to create an embedded vector search engine that can easily scale up to millions of embeddings. For self-hosted situations of under 10M embeddings and less than insane throughput I think this combo is hard to beat. https://github.com/jiggy-ai/hnsqlite https://github.com/jiggy-ai/hnsqlite
• cubefox 3y ago
  Unfortunately this piece is nebulous on what an embedding is. Apparently it is saved as an array of floats, and it has some string of text it is associated with, and the float arrays are compared by "similarity". None of these explains what an embedding really is. My best guess is that the embedding represents the meaning of the natural language string it was generated from, such that strings with "similar" embeddings have similar meaning. But that's just speculation.
  • opwieurposiu 3y ago
    An embedding is a a way to map words into a high-dimensional "concept space", so they can be processed by ML algorithms. The most popular one is word2vec https://jalammar.github.io/illustrated-word2vec/ https://jalammar.github.io/illustrated-word2vec/
    • cubefox 3y ago
      Okay, "mapping into concept space" is at least compatible with my meaning theory, but by itself it doesn't say much, since in principle anything can be mapped to anything.
    • crabbone 3y ago
      Sorry, that's even less helpful in the context of a database... but thanks for trying.
      • jmalicki 3y ago
        A vector database is used for things where you're trying query "I have this image, give me a list of the 10 closest images and metrics of how similar they are." You use a machine learning model (like word2vec, OpenAI, etc.) to produce an "embedding" that describes the image, text, video, etc., which is your "vector". For all of the other images in your database, you also run them through the same model, and store their embedding vectors in the vector database. Then, you ask the database "I have this vector, what are the most similar vectors, and what are their primary keys, so I can see what content they refer to". Think: you want to implement google "search by image". This is the basics of how you'd do that.
• nutanc 3y ago
  Do any of the vector databases have support for bit embeddings. We have created bit embeddings[1] for sentences and they save a lot of space. Currently we are just using numpy and sometimes faiss to search through these bit embeddings. Would love for one of the vector dbs to support bit embeddings natively. Then we don't have to engineer that piece :) [1] https://gpt3experiments.substack.com/p/building-a-new-embeddingbit-vector https://gpt3experiments.substack.com/p/building-a-new-embedd...
  • bckr 3y ago
    Love this idea. Do you have measurements on how it impacts performance of algorithms?
  • esafak 3y ago
    These are called binary embeddings, and they have been used successfully at pinterest (https://www.arxiv-vanity.com/papers/1908.01707/ https://www.arxiv-vanity.com/papers/1908.01707/) and Tencent (https://paperswithcode.com/paper/binary-embedding-based-retrieval-at-tencent https://paperswithcode.com/paper/binary-embedding-based-retr...) I can't speak for the competition, but weaviate seems to support them: https://weaviate.io/developers/weaviate/concepts/binary-passage-retrieval https://weaviate.io/developers/weaviate/concepts/binary-pass...
• PaulHoule 3y ago
  Talk about people who were ready with the right product at the right time.
• next_xibalba 3y ago
  I wish I could see the data on how Pinecone’s marketing materials keep making the front page of HN. It’s super sus that this keeps happening.
  • killthebuddha 3y ago
    I imagine it's because there is a *massive* scramble to incorporate language models into existing applications and most of the time you need a vector database to do that. As a proxy for understanding the scale of this scramble, ChatGPT is the fastest growing software product ever by a pretty substantial margin.
    • next_xibalba 3y ago
      Without a doubt LLMs are driving the interest in vector Dbs. More so, I am wondering how it is that Pinecone manages to land on the HN front page so frequently given the large number of alternatives (see all the other comments in this thread). It suggests to me a coordinated marketing effort (brigading, cronyism, etc.)
      • killthebuddha 3y ago
        Definitely maybe, but also I think pinecone is putting a lot of resources into devrel. For example, see their learning center [1] and examples [2]. Oh man, now I sound like I'm part of the brigading :) FWIW I recently evaluated pinecone and alternatives and decided not go with pinecone (pgvector). [1] https://www.pinecone.io/learn/ https://www.pinecone.io/learn/ [2] https://docs.pinecone.io/docs/examples https://docs.pinecone.io/docs/examples Having myself recently bootstrapped an understanding of language models et al, I would not be surprised if the pinecone learning center gets a lot of traffic.
• gk1 3y ago
  Hey all, I'm from Pinecone (shocker). Addressing common questions... What are vector DBs used for? > Storing and search through embeddings at scale, which are created and consumed by LLMs and other AI models for applications like semantic search and chatbots (eg, to avoid hallucinations). Why use a managed vector DB like Pinecone instead of [Faiss, pgvector, self-hosted thing, numpy.array, etc]? > Usually comes down to scale and convenience. If you're dealing with a small amount of embeddings, say anything less than 10M, you're probably fine just reaching for the closest and most convenient option. (We try to make Pinecone that convenient option, and our free plan holds up to ~100k 1536-dimension embeddings.) If you're dealing with larger scale -- say hundreds of millions to billions of embeddings -- and have strict performance requirements, and aren't thrilled by the thought of managing your own vector database like we are, then you should consider Pinecone. It turns out there's a sufficiently large population that falls into the latter category, just as with any other database category. If you're new to this the best place to "see for yourself" is our free plan (https://app.pinecone.io https://app.pinecone.io) and collection of examples (https://docs.pinecone.io/docs/examples https://docs.pinecone.io/docs/examples). Allow me one more plug: We're hosting a webinar next week about testing Pinecone performance with your own data and performance requirements. I have a feeling lots of folks reading this would find that useful. → https://pinecone-io.zoom.us/webinar/register/WN_z9JqLjLGTyu4YPq9R7MsCw https://pinecone-io.zoom.us/webinar/register/WN_z9JqLjLGTyu4...
  • jmole 3y ago
    I know nothing about vector databases – is this just “replace SQL with a dot product and return a ranked list (with optimizations)”?
    • kirill5pol 3y ago
      That’s nearest neighbour search which scales O(n^2) for the number of vectors in your DB, what these DBs (and libraries like FAISS) use is approximate nearest neighbour, which makes the search much, much faster.
  • znagengast 3y ago
    How are you guys thinking about the embedding generation side of things? It seems like that part has a generally hefty compute cost before it even gets into the index - I just open sourced a swift package to try to make that part as easy as possible, the example project exports directly to pinecone. https://github.com/ZachNagengast/similarity-search-kit https://github.com/ZachNagengast/similarity-search-kit
• hallqv 3y ago
  Why not use Elasticsearch/Opensearch? Way more battle-tested and has many more features for text data. Also uses the same vector indexing algos.
• tqi 3y ago
  "Since the vector database provides approximate results, the main trade-offs we consider are between accuracy and speed. The more accurate the result, the slower the query will be. However, a good system can provide ultra-fast search with near-perfect accuracy." What does "near-perfect" mean in this context, type 1 or type 2?
• vrglvrglvrgl 3y ago
  [dead]
• berkle4455 3y ago
  So, uh, how do I generate vector embeddings in the first place? What's a good ground-level area to start?
  • tudorg 3y ago
    An easy practical option for text is to use the OpenAI embeddings API.
• frabcus 3y ago
  Is there a MySQL plugin for vectors?
• singularity2001 3y ago
  May be a bit old-fashioned but is there any standalone Java Vector Database?
• zh217 3y ago
  If anyone wants to try a FOSS vector-relational-graph hybrid database for more complicated workloads than simple vector search, here it is: https://github.com/cozodb/cozo/ https://github.com/cozodb/cozo/ About the integrated vector search: https://docs.cozodb.org/en/latest/releases/v0.6.html https://docs.cozodb.org/en/latest/releases/v0.6.html It also does duplicate detection (Minhash-LSH) and full-text search within the query language itself: https://docs.cozodb.org/en/latest/releases/v0.7.html https://docs.cozodb.org/en/latest/releases/v0.7.html HN discussion a few days ago: https://news.ycombinator.com/item?id=35641164 https://news.ycombinator.com/item?id=35641164 Disclaimer: I wrote it.
  • drunkan 3y ago
    Thanks for the links and discussions, I’m keeping an eye on this one it looks really promising, at least in the hybrid area compared to the much hyped surrealDB whose graph implementation looks more like an afterthought when you get down to the technical details, functionality and performance
  • digdugdirk 3y ago
    Glad I hopped into this thread while your comment was recent enough to be at the top. This is super interesting! Apologies if you went over this in your other post (or the docs, I'll be digging into this over the weekend) but could you share a bit about why you went this route? What you tried, what the hangups were/are with other approaches, and if there are any interesting possibilities with your approach that other vector databases just wouldn't be able to do?
    • zh217 3y ago
      For me personally the most important motivations are to have recursive queries using vector search, and to integrate graphs and vectors. Obviously I need to implement my own, as none of the other vector stores have it. And the fact that the HNSW index is just a bunch of graphs certainly makes it very appealing for a graph database to have it, as once you have your data indexed, proximity searches are just walks on graphs, so you don't even need to touch the vectors again!