6 ms·
I built multiple systems using vector search, one of them demoed in a search engine for non-commercial content at http://teclis.com http://teclis.com Running v
by freediver 5y ago
I built multiple systems using vector search, one of them demoed in a search engine for non-commercial content at http://teclis.com http://teclis.com
Running vector search (also sometimes referred to as semantic search, or a part of semantic search stack) is a trivial matter with open-source libraries like Faiss https://github.com/facebookresearch/faiss https://github.com/facebookresearch/faiss
It takes 5 minutes to set up. You can search billion vectors on common hardware. For low-latency (up to couple of hundred milliseconds) use cases, it is highly unlikely that any cloud solution like this would be a better choice than something deployed on premise because of the network overhead.
(worth noting is that there are about two dozen vector search libraries, all benchmarked at http://ann-benchmarks.com/ http://ann-benchmarks.com/ and most of them open-source)
A much more interesting (and harder) problem is creating good vectors to begin with. This refers to the process of converting a text or an image to a multidimensional vector, usually done by a machine learning model such as BERT (for text) or ImageNet (for images).
Try entering a query like 'gpt3' or '2019' into the news search demo linked in the Google's PR:
https://matchit.magellanic-clouds.com/ https://matchit.magellanic-clouds.com/
The results are nonsensical. Not because the vector search didn't do its job well, but because generated vectors were suboptimal to begin with. Having good vectors is 99% of the semantic search problem.
A nice demo of what semantic search can do is Google's Talk to Books https://books.google.com/talktobooks/ https://books.google.com/talktobooks/
This area of research s fascinating. For those who want to play with this more, an interesting end-to-end (including both vector generation and search) open-source solution is Haystack https://github.com/deepset-ai/haystack https://github.com/deepset-ai/haystack
- noud 5y agoI just made a couple of searches with teclis. I have to say, it's not bad. It's clearly not complete and I get several empty searches. But the content of the results are of higher quality than what I get with Google or DDG. Nice work!
- freediver 5y agoThanks. The index is tiny and it is just a proof of concept of what a single person can do with technologies available nowadays. I felt it is better for it to return zero results than bad results.
- gk1 5y ago> The index is tiny What was the largest index you've had on Faiss? That seems to affect whether people think of it as more than adequate or terribly inadequate.
- freediver 5y agoThis demo is only about million vectors. The largest I had in Faiss was embeddings of the entire Wikipedia (scale in the neighborhood of ~30 million vectors). I know people running few billion vectors in Faiss.
- leobg 5y agoSo one vector per article? Doesn’t this skew results? A short article with 0.9 relevance score would rank higher than a long article containing one paragraph with 1.0 relevance. Am I mistaken?
- visarga 5y agoIt's a known problem. Choosing appropriately between cosine similarity or dot product depending on your use case can help.
- marginalia_nu 5y agoYeah, this is also a problem traditional search ranking algorithms needs to compensate for.
- leobg 5y agoAlso, BERT on cheap hardware? I thought that without a GPU, vectorizing millions of snippets or doing sub-second queries was basically out of the question.
- mrg3_2013 5y agoThanks for the reference to haystack. I didn't know it existed! I was looking into huggingface that seems to allow to build your own language model and train (still learning - but thats what I've learnt so far). I don't know how expensive these get (for example, if you have 100K lines?). Any thoughts on how this compares to HuggingFace and any anecdotes on time it would take to custom train ?
- m3at 5y agoNot the author but I can answer that! Haystack does not really compare to huggingface as they are directly using it as a library, it's more of a layer on top. Serving a search system for 100K lines (short document?) is trivial on a single CPU machine, so the costs will be low. For training if you stick to fine tuning and your dataset is relatively small (<10M) you could use collab with a free gpu. Training time really depends on your use case, the simplest classification may be handled in a couple of hours.
- mrg3_2013 5y agoThank you @m3at!
- leobg 5y agoSo how are you creating the embeddings for your search engine? GloVe? Sentence BERT? Are you training your own models? Are you employing any kind of normalization? There are so many variables to optimize on many levels. Which is, of course, what makes this whole area super exciting.
- hiddencost 5y agoI recommend readers take parent post with a grain of salt. (1) Google's offering returns with-in <5ms, in my experience. (2) the demo is for paragraphs, not short text. You're putting mismatched data into the input, of course it's not going to work. Try a paragraph as suggested.
- freediver 5y agoHmm.. There is no web service that returns response in <5ms, unless you are sitting at the very terminal of the hardware producing the output. The demo featured in this PR takes about 800-1000ms total to produce search results. How much of that is the actual API is not known. Typically, https request to an API in the cloud will cost you at least 50ms of network latency, more likely 100ms-200ms. If you are running vector search on premise you will obviously not have this overhead. Text embeddings typically work for short text as well as paragraphs (paragraph embeddings are usually mean/max of word embeddings anyway) simply because most commercial use cases demand handling of short text input (because nobody is inputting a paragraph into a typical search box; what use is a news search if you can not type a single word like 'Biden' or 'gpt3' into it).
- hiddencost 5y agoIt's a cloud offering, so the machines are located near each other. Using it with other cloud services is a fair comparison to running it in the same box on-prem. The offering is similarity search, not a search engine. They offer image to image as another comparison point.
- freediver 5y agoWith the caveat of having to use GCP to host your server too, I can agree with you (although 5ms still sounds incredibly low, how many vectors was that?). I was obviously talking about a general use case where a user considers using an API like this vs running Faiss, and their server can be anywhere (a use case that is more common to me personally).
- 1vuio0pswjnm7 5y agoReally like the idea of teclis, i.e., a non-commercial search engine. Is it correct that teclis is HTTP-only (via port 1333) and TLS is not an option. (NB. I am not suggesting there is anything wrong with HTTP. I am simply curious if TLS is available.)
- freediver 5y agoThere is nothing wrong with using HTTP if the data transferred is not sensitive like in this case for demo purposes (and if anything, it is also faster for the user).
- akiselev 5y agoSearch phrases are sensitive information.
- charcircuit 5y agoA MITM can inject anything into the page including ads or scripts and is able to redirect you to anywhere on the internet.
- dontreact 5y agoThere are two huge things your 5 minute setup is missing which are very hard techinically to tackle 1. Incrementally updating the search space. Not that easy to do, and becomes more important to not just do the dumb thing of retraining the entire index on every update for larger datasets. 2. Combining vector search and some database-like search in an efficient manner. I don't know if this Google post really solves that problem or if they just do the vector lookup followed by a parallelized linear scan, but this is still an open research/unsolved problem.
- moab 5y agoCan you expand more or provide a concrete example for the second point? What kind of database-like searches are you thinking about for spatial data? Things like range-queries can already be (approximately) done. Or are you thinking about relational style queries on data associated with each point?
- dontreact 5y agoYes exactly, relational style queries with each data point. Maybe you have some metadata about your images and maybe you need to join against another table to properly query them. But at the same time you want to only grab the first k nearest neighbors according to vector similarity.
- gk1 5y agoPinecone does this, at least if I’m understanding your use case right: https://www.pinecone.io/docs/metadata-filtering/ https://www.pinecone.io/docs/metadata-filtering/ And you’re right, it wasn’t easy to build.
- etiennedi 5y agoSpot on! Both of those were motivating factors when building Weaviate (Open Source Vector Search Engine). We really wanted it to feel like a full database or search engine. You should be able to do anything you would do with Elasticsearch, etc. There should be no waiting time between creating an object and searching. Incremental Updates and Deletes supported, etc. On your second point about efficient filtering, check out this article I wrote outlining how filtered vector search works in Weaviate: https://towardsdatascience.com/effects-of-filtered-hnsw-searches-on-recall-and-latency-434becf8041c https://towardsdatascience.com/effects-of-filtered-hnsw-sear... For even more details on filtering, check the documentation: https://www.semi.technology/developers/weaviate/current/architecture/prefiltering.html https://www.semi.technology/developers/weaviate/current/arch...
- GistNoesis 5y ago>Try entering a query like 'gpt3' or '2019' into the news search demo The results are bad because this is not the kind of input the engine was trained to handle. You should copy paste a part of an existing article. It will embed it into a multidimensional space and do a similarity search (the same way it was trained to (by converting a full article paragraph to a vector). If you give it just a word it can't convert it to a meaningful representation because the network wasn't trained to do this. But you can train it differently and have it able to handle a few words. For example you can summarize every article to a few sentences and keywords and use a traditional keyword search. One usual way to create good vector representation is to use encode simultaneously two different space to the same vector space. You encode 'queries' and 'answers' such that they are close for the known (query-answer) pairs. This is what CLIP did, encoding both images and their corresponding description to a same vector space. You can download the precomputed clip embeddings LAION-400-MILLION OPEN DATASET on academictorrents.com CLIP can do such thing for the problem of semantic image search because the problem of matching an image to its description is quite well defined. But quite often there is no unique apriori meaningful way of matching a query to an answer, specially as the index get big. In the case of a basic query 'gpt-3', the query is quite vague and its not obvious with respect to which direction you should do the ranking (Do you meaning you want articles generated by gpt-3 ? Articles containing gpt-3, a basic definition ?). There is no a priori good answer, and that's where you can use your additional context to refine the query. For example Siri or a NLP bot, could ask you to be more explicit in what you mean. Or it can have multiple representation of your vector space and return the top-1 for each of those representation, and hope that you give it feedback by clicking the one that was more meaningful to you as requester.
- blackcat201 5y agoI agree having a good vector is important to start with. However this is not very hard to make it work, you only need to finetune some of the clip models[1] to run it well. Disclose: I have built a vector search engine to proof this idea[2] [1]: https://github.com/openai/CLIP https://github.com/openai/CLIP [2]: https://unisearch.cc/search?q=how+to+reset+mac+ssd https://unisearch.cc/search?q=how+to+reset+mac+ssd
- leobg 5y agoI like your search demo. Have you considered adding autofocus to the input? The result accuracy is quite impressive.
- jkb79 5y ago>A much more interesting (and harder) problem is creating good vectors to begin >with. Indeed, this is the hardest problem. Vector search shines when used in-domain using deep representation learning, for example bi-encoders on top of transformer models for text domain. However, these models does not generalize well when used out of domain as the representations changes. Hence, in many cases, simple BM25 beats most of the dense vector models when used in a different domain. See https://arxiv.org/abs/2104.08663 https://arxiv.org/abs/2104.08663
- sgc 5y agoDo you have a starting point for generating useful vectors for full text search? To simplify, I have a 350k document academic database, with docs of maybe 500-5000 words. I would like to be able to provide similarity results as a form of "concordance". Do sentence, paragraph, or full doc vectors work the best? Do things work better creating vectors from sentence vectors for longer sections, or have you had better results making longer vectors directly? Do different vectorizing algorithms work better on different sized sections of text? I have not had much luck finding discussion or write ups on this type of stragetizing on chunks and algorithms to apply, beyond just people saying that yes that's how it's done.
- leobg 5y agoDepends on what you want your search results to be. Do you want a list of documents, sorted by relevancy? Or a list of sentences? Or paragraphs? Personally, I find paragraphs a good middle ground. However, if many of your documents contain very long paragraphs, it would still be tedious to go through your search results. So what might work better in this case would be to slide a window of, say, 1-3 sentences over each document. This way, you'll be able to find the most relevant 1-3 sentence snippet, both within one document, and within your entire corpus.
- sgc 5y agoThat is definitely part of the thought, and I am tending to thinking that there needs to be more than one index, to be able to compare both sentence level and larger sections on different tabs, since the comparisons are quite different in nature (avoiding hard lines, it is still the case that sentence comparison is closer to grammatical and linguistic dependency, whereas longer passages will tend to reflect deeper thematic relationships). What I was attempting to ask was more on how to effectively compare longer sections of text. It sounds like you are saying it is fine to compare them directly. So, due to very, very irregular formatting between texts that will likely never change, we would probably be stuck using only sentences and 'documents' (an article, a chapter of a book, etc). So my questions, from a technical perspective, are: 1) do you find you get better results treating these longer texts as a series or words or a series of sentences, and should we be performing a double vectorization, were we vectorize all sentences, and then vectorize the longer text as a series of sentence vectors instead of using a word vectorization technique directly? 2) Assuming we have decided we want to vectorize sentences and longer sections of texts regardless of whether we use sentence vectors for the latter, do we use the same or different vectorization techniques/algorithms for both tasks, and do you have any recommendations on specific algorithms for either case? Many thanks!