16 ms·

Find anything fast with Google's vector search technology

368 points by sshroot 5y ago

tomc1985 5y ago
So... fuzzy logic Everything old is new again! Again!
heisenbit 5y ago
Considering the number of possible keywords and comparing this with what are feasible vector lengths I wonder whether vector search is not weaker when looking at the long tail.
gk1 5y ago
It's great to see more and more talk of vector search and vector databases. We've been promoting this technology for over a year now and have several intro articles for anyone looking to learn more[1], and a generous free tier on our vector search service[2] for anyone looking to give vector search a shot. [1] https://www.pinecone.io/learn/ https://www.pinecone.io/learn/ [2] https://app.pinecone.io/ https://app.pinecone.io/ We are also actively researching the space, and just recently published a paper on improving Google's ScaNN: https://arxiv.org/abs/2112.02179 https://arxiv.org/abs/2112.02179
- dvaun 5y ago
  I've been toying with making a deckbuilder for Magic: The Gathering and could see this being potentially useful for finding fun card combinations. Thanks!
  - gk1 5y ago
    That would be a fun use case for us to promote. Let me know when it's ready! The free plan supports as many as 1 million items, more than enough for the all MTG cards in existence. Plus you can add and filter by metadata, like card type and properties.
    - dvaun 5y ago
      > Plus you can add and filter by metadata, like card type and properties. I read through your docs and figure that will be part of the approach. An idea I had was to find similar, or "next best", cards for replacement in popular decks or to achieve similar effects in order to bring down the cost of EDH, Modern, etc. formats. I'm just getting back into the hobby again, so having a tool like this would make my wife and wallet happy :)
      - 16mb 5y ago
        I’ve resorted to playing modern with high quality fakes. Otherwise wouldn’t have the budget. Checkout bootlegmtg on reddit
  - kruptos 5y ago
    I love this idea. I would pay for that service!
  - thirdtrigger 5y ago
    We are actually discussing this on the Weaviate Slack :-) https://weaviate.slack.com/archives/D02JM9D3HND/p1634731283003400 https://weaviate.slack.com/archives/D02JM9D3HND/p16347312830...
- wswope 5y ago
  That reference/learning page is a great resource! As for Pinecone itself, what are the main selling points as you see them for a simple application (e.g. comparing trigram-vectorized sets of strings) when compared to a home-rolled solution using postgres with array types? Better performance, ease of indexing, etc.?
  - gk1 5y ago
    I pinged someone more technical from our team to chime in. In the meantime I can say moving to the dense vector + ANN search combo turns regular searches into semantic searches, which means more relevant results. If that's the case for you, then you can use Pinecone to go further and make those results fast (<100ms), fresh (CRUD + live index updates), and filtered (apply single-stage metadata- filtering). All on a fully managed system that you can scale up/down with one API call.
  - jamesbriggs 5y ago
    It will depend on your use-case, but primarily: (1) Pinecone uses dense vectors which can encode much more meaningful info, eg the actual 'semantic meaning' behind a sentence as we (people) would understand it, or the context in an image. Because of this, we can enable much richer, human-like interaction/search in your applications (2) Performance wise, before joining Pinecone I was spending a lot of time with other dense vectors search tools like Faiss, and it isn't easy to get good or even reasonable accuracy and latency, particularly for large datasets. When I first used Pinecone, it took me maybe 10 minutes to figure everything out and start querying a reasonable dataset, search times were very fast and the accuracy incredible. Pinecone's tech is built by people that live and breath vector search, and what they've built outperforms anything I can build, even if I spend months trying to build it. I got better performance with Pinecone in 10 mins. (3) Everything is production ready, no need to worry about deployment, security, maintenance etc, Pinecone deal with it and you can even use the service for free up to 1M vectors.
- indeed30 5y ago
  Does Pinecone have any position on the status of document embeddings and whether they would be considered PII? One of the challenges of using a fully managed service is the headache of adding yet another data subprocessor and all of the legal and compliance questions that raises.
  - gk1 5y ago
    That depends on the document. We do not see the original document, only the embedding. You can argue that is sufficiently obfuscated to not count as PII. The good news is we are SOC2 compliant and GDPR-friendly and do a bunch of other stuff to help you meet security compliance requirements: https://www.pinecone.io/security/ https://www.pinecone.io/security/
    - indeed30 5y ago
      No, I understand that. I guess my question is actually around your experience with "You can argue that is sufficiently obfuscated to not count as PII" and whether your customers are actually successful with this argument.
      - gk1 5y ago
        Those who need more assurance just look at our SOC2 compliance, or have us go through a security review, or opt for the dedicated-environment deployment option.
- nop_slide 5y ago
  I just want to chime in and say that the resources on your website look amazing. I spent 5 minutes poking around and it looks really high quality. I'm dabbling in Postgres's full text search (ts_vector) for a small website, I know that is extremely simple compared to the offerings you provide, but your site has me quite interested in this space now. Eager to learn more about this tech!
  - gk1 5y ago
    Glad you think so. Makes me want to expand it even more! What would you like to see covered?
    - nop_slide 5y ago
      Maybe this is too simple, but a comparison to Postgres' ts_vector and how to do something similar in your service?
tomcooks 5y ago
I would be happy with "find anything with Google search"
- eliseumds 5y ago
  It exists: https://developers.google.com/custom-search/v1/overview https://developers.google.com/custom-search/v1/overview. Might not be cheap for search-as-you-type though.
  - sanxiyn 5y ago
    GP is complaining Google's search quality is poor, not inquiring about Google's search API.
deleted 5y ago
[deleted]
monkeybutton 5y ago
If you are interested in how ScaNN compares to other approximation algorithms, there are some benchmarks here: http://ann-benchmarks.com/ http://ann-benchmarks.com/
CoolGuySteve 5y ago
Is this more or less a k-d tree as a service? Where any distance function can be used to index the data? Or is it something different?
- monkeybutton 5y ago
  A k-d tree gives you exact answers to nearest neighbour queries.
  - srean 5y ago
    A k-d tree is a data structure. Whether you use that for exact nearest neighbor query or approximate is up to the algorithm used. K-d trees work well for a handful of dimension beyond that it becomes quite expensive.
- hamilyon2 5y ago
  I thought k-d trees were useless in high-dimentional case. So, it must be something else.
  - contravariant 5y ago
    I'd say they're about as useful as euclidean distance is.
- ahurmazda 5y ago
  More or less but as always the devil is in the detail. Here is a paper[1] that summarizes issues with naive approaches. Incidentally. the proposed solution (Hierarchical NSW) in this paper performed fairly well in the industry benchmarks [1] https://arxiv.org/ftp/arxiv/papers/1603/1603.09320.pdf https://arxiv.org/ftp/arxiv/papers/1603/1603.09320.pdf
eob 5y ago
My 2022 wish list is a Postgres plugin that adds vector + AKNN support that plays well with relational queries. There are so many use cases of that. I believe Ant Financial has published an open source one but iirc the English language documentation is sparse.
- ccleve 5y ago
  Do you have a link? I'd like to see it. I googled and did not find much pertaining to "ant financial" and "postgres". Perhaps your google-fu is better than mine...
- etiennedi 5y ago
  Checkout the open source vector search engine Weaviate: https://github.com/semi-technologies/weaviate https://github.com/semi-technologies/weaviate It’s not a relational db, but it supports Graph-like connections between objects, which makes it really easy to model your relations.
  - thirdtrigger 5y ago
    Jup – this is an example from the demo dataset in the docs: https://link.semi.technology/3DPcphe https://link.semi.technology/3DPcphe
- akane 5y ago
  Check out pgvector: https://github.com/ankane/pgvector https://github.com/ankane/pgvector (disclosure: am author) It uses IVFFlat indexing, but could be extended to support product quantization / ScaNN.
amelius 5y ago
Does anyone know of a good benchmark suite for search technology? (And how well does the technique of the article work wrt it?)
- GhettoComputers 5y ago
  Use the one that gives you the best results, bing is too good for porn so it shows me porn when I am not even looking for it.
- jkb79 5y ago
  MS Marco is the largest relevancy collection in the open, see https://microsoft.github.io/msmarco/ https://microsoft.github.io/msmarco/
323 5y ago
People say google search is terrible these days, but I find the opposite. I can vaguely describe in a sentence the gist of an article I've read, or an image, and the proper result will usually be in the first page. Of course, it doesn't always work, sometimes there are "hash collisions" so to speak, but I don't think the old algorithm would have been more successfully either, since if I knew the exact keywords to use, I wouldn't need to start with a vague description in the first place.
- Jemaclus 5y ago
  I've literally gone to Google and typed something very similar to "That guy in that thing with the dog" and the correct answer shows up as the first result. It's quite brilliant and magical how they do that. But sometimes it's a total miss when I want something very specific, and it just shows me other things I didn't ask for.
- bitcharmer 5y ago
  I have the exact opposite experience. Search results are nowadays ridden with crap from companies that learned how to game SEO. If not that, you get scam websites or other ad/malware infested trash.
  - radicaldreamer 5y ago
    Wish there was an easy way to remove forbes.com results
    - greybeardgeek 5y ago
      you can append -site:forbes.* to your query string
    - jiveturkey 5y ago
      https://news.ycombinator.com/item?id=29546433 https://news.ycombinator.com/item?id=29546433
    - hidden-spyder 5y ago
dorianmariefr 5y ago
Will probably be available as Postgres extension at some point. Seems like only special indexing of vectors is needed
ahurmazda 5y ago
For a similar ANN/vector search capabilities, https://vespa.ai/ https://vespa.ai/ is a great open-source solution. Elasticsearch may offer some form of ANN too but need to double check
- sanxiyn 5y ago
  I don't think Elasticsearch has one yet, but OpenSearch does: https://opensearch.org/docs/latest/search-plugins/knn/index/ https://opensearch.org/docs/latest/search-plugins/knn/index/
  - m_ke 5y ago
    Lucene 9.0 just shipped with hnsw support, should make it into ES at some point (https://twitter.com/msokolov/status/1468395332531003393 https://twitter.com/msokolov/status/1468395332531003393) EDIT: ES integration PR: https://github.com/elastic/elasticsearch/issues/78473 https://github.com/elastic/elasticsearch/issues/78473
  - ahurmazda 5y ago
    Ah! Great to know. ANN searches are becoming table stakes at this point. Hopefully, we will see more and more platforms adding it to their repertoire.
- jkb79 5y ago
  Yes, Vespa.ai can index billion scale vector datasets https://blog.vespa.ai/billion-scale-knn/ https://blog.vespa.ai/billion-scale-knn/ Vespa also allows expressing hybrid sparse and dense retrieval (WAND for sparse, ANN via HNSW for dense). It's also easy to express multi stage retrieval ranking phases, as vector search alone is not achieving state-of-the-art ranking results, see https://blog.vespa.ai/pretrained-transformer-language-models-for-search-part-4/ https://blog.vespa.ai/pretrained-transformer-language-models...
freediver 5y ago
I built multiple systems using vector search, one of them demoed in a search engine for non-commercial content at http://teclis.com http://teclis.com Running vector search (also sometimes referred to as semantic search, or a part of semantic search stack) is a trivial matter with open-source libraries like Faiss https://github.com/facebookresearch/faiss https://github.com/facebookresearch/faiss It takes 5 minutes to set up. You can search billion vectors on common hardware. For low-latency (up to couple of hundred milliseconds) use cases, it is highly unlikely that any cloud solution like this would be a better choice than something deployed on premise because of the network overhead. (worth noting is that there are about two dozen vector search libraries, all benchmarked at http://ann-benchmarks.com/ http://ann-benchmarks.com/ and most of them open-source) A much more interesting (and harder) problem is creating good vectors to begin with. This refers to the process of converting a text or an image to a multidimensional vector, usually done by a machine learning model such as BERT (for text) or ImageNet (for images). Try entering a query like 'gpt3' or '2019' into the news search demo linked in the Google's PR: https://matchit.magellanic-clouds.com/ https://matchit.magellanic-clouds.com/ The results are nonsensical. Not because the vector search didn't do its job well, but because generated vectors were suboptimal to begin with. Having good vectors is 99% of the semantic search problem. A nice demo of what semantic search can do is Google's Talk to Books https://books.google.com/talktobooks/ https://books.google.com/talktobooks/ This area of research s fascinating. For those who want to play with this more, an interesting end-to-end (including both vector generation and search) open-source solution is Haystack https://github.com/deepset-ai/haystack https://github.com/deepset-ai/haystack
- noud 5y ago
  I just made a couple of searches with teclis. I have to say, it's not bad. It's clearly not complete and I get several empty searches. But the content of the results are of higher quality than what I get with Google or DDG. Nice work!
  - freediver 5y ago
    Thanks. The index is tiny and it is just a proof of concept of what a single person can do with technologies available nowadays. I felt it is better for it to return zero results than bad results.
ShamelessC 5y ago
This gh repo makes it pretty easy to create similar tech by first embedding any images you have using the released "CLIP" model from Open AI and then creating a Faiss index over these embeds for quick retrieval/decode. You can then do text->image, and image->image semantic search. https://github.com/rom1504/clip-retrieval https://github.com/rom1504/clip-retrieval
thirdtrigger 5y ago
Interesting – we are working on an open source vector search engine called Weaviate and did the same for the complete Wikipedia and Wikidata. [1] Docs: https://www.semi.technology/developers/weaviate/current/ https://www.semi.technology/developers/weaviate/current/ [2] Github: https://github.com/semi-technologies/weaviate https://github.com/semi-technologies/weaviate [3] Wikipedia demo dataset: https://github.com/semi-technologies/semantic-search-through-Wikipedia-with-Weaviate https://github.com/semi-technologies/semantic-search-through... [4] Wikidata dataset: https://github.com/semi-technologies/biggraph-wikidata-search-with-weaviate https://github.com/semi-technologies/biggraph-wikidata-searc... Last week there was also a feature on Techcrunch about vector search and Weaviate: https://techcrunch.com/2021/12/11/2246180/ https://techcrunch.com/2021/12/11/2246180/
- CShorten 5y ago
  I've made some videos on Weaviate as well (Henry AI Labs) if interested: [1] Wikipedia Vector Search Demo with Weaviate: https://www.youtube.com/watch?v=IGB8vjCuay0 https://www.youtube.com/watch?v=IGB8vjCuay0 [2] Vector Search through Wikidata with Weaviate: https://www.youtube.com/watch?v=T4zlvknSbGc https://www.youtube.com/watch?v=T4zlvknSbGc [3] Demonstrations of Deep Learning: https://www.youtube.com/watch?v=5jbneytoKi0 https://www.youtube.com/watch?v=5jbneytoKi0 [4] Weaviate's GraphQL API for Neurosymbolic Search: https://www.youtube.com/watch?v=K_2X48Tln9U https://www.youtube.com/watch?v=K_2X48Tln9U [5] Introducing the Weaviate Vector Search Engine: https://www.youtube.com/watch?v=AS_2U_INpKk https://www.youtube.com/watch?v=AS_2U_INpKk
  - thirdtrigger 5y ago
    These are all great! There is also this video about modern search engines and Weaviate on the AI Coffee Break YT channel: https://www.youtube.com/watch?v=YkK5IKgxp-c https://www.youtube.com/watch?v=YkK5IKgxp-c
    - visarga 5y ago
      Well done promo video.
Hokusai 5y ago
It's not very good. I tried different pictures and the results are almost random. A picture from a cartoon returns from logos to any type of drawing. A picture of a battery returns cars and shops. A picture of food worked as expected and I got more food pictures.
Kydlaw 5y ago
There is a lot done vector search technology right now. I was less fortunate when looking at vector storage. I already looked at Pinecone or Weaviate but they are all paid products. Is there some people having feedback on this?
- thirdtrigger 5y ago
  Not true – Weaviate is open source: https://github.com/semi-technologies/weaviate https://github.com/semi-technologies/weaviate
- Xenoamorphous 5y ago
  ElasticSearch supports vectors (dense ones, they supported sparse ones at some point but they removed support I think), and has things like cosine similarity functions built in. Not sure how “free” it is though.
- gk1 5y ago
  Pinecone has a free tier that’s quite generous, fits around 1M items and will fit even more soon. Not sure this helps but just mentioning in case.
slig 5y ago
Let's say I have a content website with about 20k content pages. I want to automatically cluster the pages so that the each page has the related content linked. Right now I'm using a hacked together tf–idf using sklearn and Python2, and it just works. The downsides are that I have to compute everything offline whenever I add new content, and that it's one more thing to maintain/upgrade. I'm wondering if anyone has a suggestion of a SaaS or another alternative for my use case? Thanks!
- ShamelessC 5y ago
  Python2?
  - slig 5y ago
    Yes, it's been running for about 10 years.
- nickmancol 5y ago
  I think this PostgreSQL is your friend in this case https://github.com/ankane/pgvector https://github.com/ankane/pgvector
  - slig 5y ago
    Thank you, will have a look!
bserge 5y ago
> This is how Google services find ~~valuable content~~ absolute garbage for a wide variety of users worldwide in milliseconds. Leave it to the fucking user. I don't search my notes with vague phrases hoping the (dumb) "AI" will somehow find what I'm looking for. I search very specific keywords that I've associated with the content when I saved them. Same for a "wide area" search like the Internet, using Google. Perhaps it will improve with time. Right now I'd opt out if I could.
Lamad123 5y ago
Now I cannot even find a song on google or youtube even though I search several lines of the song's lyrics!!
pfd1986 5y ago
More "Find _something_ fast with vector search". I was not successful in finding anything relevant. PageRank works because it _ranks_ pages by, among other features, number and quality of visitors. E.g. searching for Huxley quote gives me silly blog posts about saving money. Query: "The function of the brain and nervous system is to protect us from being overwhelmed and confused by this mass of largely useless and irrelevant knowledge, by shutting out most of what we should otherwise perceive or remember at any moment, and leaving only that very small and special selection which is likely to be practically useful." Answer: "How to trick your brain into saving money"
___q 5y ago
So how do you game this? "Googlebomb" this? I assume it's harder than keyword-based search? As a search engine, what efforts do I take to stop someone from gaming vector-based search engines?
shanghaikid 5y ago
If you are not using GCP or you want to have an open-source alternative, Please check my project Milvus vector database (https://milvus.io https://milvus.io). We've published a bunch of demo cases powered by vector database on GitHub. https://github.com/milvus-io/bootcamp https://github.com/milvus-io/bootcamp We have built Milvus vector database upon ANN libraries like faiss, annoy, nsmlib, etc. We are aiming to create a cloud-scalable vector database. So Milvus comes to the crossroad of vector search and cloud database. There are many interesting system design topics in the development of Milvus 2.0. We will continue to share our experiences and thoughts on this topic.
currentsapi 5y ago
If anyone is interested, I maintain a list of open source vector search engine services[1]. Feel free to submit a new issues or merge request if you wish for new library added [1] https://github.com/currentsapi/awesome-vector-search https://github.com/currentsapi/awesome-vector-search
andre-z 5y ago
We are developing open-source vector search technology. https://github.com/qdrant/qdrant https://github.com/qdrant/qdrant It is a neural search engine with extended filtering support that implements a custom modification of the HNSW algorithm for Approximate Nearest Neighbour search. It allows applying search filters, including geolocation, without compromising on results. Developed entirely in Rust language. You can find some demos and documentation here https://qdrant.tech https://qdrant.tech
- generall 5y ago
  I wounder if there is any alternative implementations of ScaNN in languages like Rust.
visarga 5y ago
What if we had local vector search on our web browser history (the content as it was displayed)? That would be radical. I'm wondering why browser vendors don't scramble to create the personal vector database. It could be integrated through a browser extension to insert local results when doing regular web searches, or provide context for a speech based personal assistant. Having a neural net at hand could also prove useful in semantic filtering of webpages (hide or highlight content) and curating your news feeds.
- jamesbriggs 5y ago
  Agree that this would be super helpful
est 5y ago
What's the sqlite equivilant of vector search engine?
yboris 5y ago
I'm curious about Gensim Doc2Vec Model. I used it 3 years ago and got decent results in vectorizing articles and then finding articles that were similar based on input text (half-written article for example). What is new here? https://radimrehurek.com/gensim/auto_examples/tutorials/run_doc2vec_lee.html#sphx-glr-auto-examples-tutorials-run-doc2vec-lee-py https://radimrehurek.com/gensim/auto_examples/tutorials/run_...

368 points by sshroot 5y ago

tomc1985 5y ago

So... fuzzy logic Everything old is new again! Again!

Considering the number of possible keywords and comparing this with what are feasible vector lengths I wonder whether vector search is not weaker when looking at the long tail.

gk1 5y ago

It's great to see more and more talk of vector search and vector databases. We've been promoting this technology for over a year now and have several intro articles for anyone looking to learn more[1], and a generous free tier on our vector search service[2] for anyone looking to give vector search a shot. [1] https://www.pinecone.io/learn/ https://www.pinecone.io/learn/ [2] https://app.pinecone.io/ https://app.pinecone.io/ We are also actively researching the space, and just recently published a paper on improving Google's ScaNN: https://arxiv.org/abs/2112.02179 https://arxiv.org/abs/2112.02179

dvaun 5y ago
I've been toying with making a deckbuilder for Magic: The Gathering and could see this being potentially useful for finding fun card combinations. Thanks!
- gk1 5y ago
  That would be a fun use case for us to promote. Let me know when it's ready! The free plan supports as many as 1 million items, more than enough for the all MTG cards in existence. Plus you can add and filter by metadata, like card type and properties.
  - dvaun 5y ago
    > Plus you can add and filter by metadata, like card type and properties. I read through your docs and figure that will be part of the approach. An idea I had was to find similar, or "next best", cards for replacement in popular decks or to achieve similar effects in order to bring down the cost of EDH, Modern, etc. formats. I'm just getting back into the hobby again, so having a tool like this would make my wife and wallet happy :)
    - 16mb 5y ago
      I’ve resorted to playing modern with high quality fakes. Otherwise wouldn’t have the budget. Checkout bootlegmtg on reddit
- kruptos 5y ago
  I love this idea. I would pay for that service!
- thirdtrigger 5y ago
  We are actually discussing this on the Weaviate Slack :-) https://weaviate.slack.com/archives/D02JM9D3HND/p1634731283003400 https://weaviate.slack.com/archives/D02JM9D3HND/p16347312830...
wswope 5y ago
That reference/learning page is a great resource! As for Pinecone itself, what are the main selling points as you see them for a simple application (e.g. comparing trigram-vectorized sets of strings) when compared to a home-rolled solution using postgres with array types? Better performance, ease of indexing, etc.?
- gk1 5y ago
  I pinged someone more technical from our team to chime in. In the meantime I can say moving to the dense vector + ANN search combo turns regular searches into semantic searches, which means more relevant results. If that's the case for you, then you can use Pinecone to go further and make those results fast (<100ms), fresh (CRUD + live index updates), and filtered (apply single-stage metadata- filtering). All on a fully managed system that you can scale up/down with one API call.
- jamesbriggs 5y ago
  It will depend on your use-case, but primarily: (1) Pinecone uses dense vectors which can encode much more meaningful info, eg the actual 'semantic meaning' behind a sentence as we (people) would understand it, or the context in an image. Because of this, we can enable much richer, human-like interaction/search in your applications (2) Performance wise, before joining Pinecone I was spending a lot of time with other dense vectors search tools like Faiss, and it isn't easy to get good or even reasonable accuracy and latency, particularly for large datasets. When I first used Pinecone, it took me maybe 10 minutes to figure everything out and start querying a reasonable dataset, search times were very fast and the accuracy incredible. Pinecone's tech is built by people that live and breath vector search, and what they've built outperforms anything I can build, even if I spend months trying to build it. I got better performance with Pinecone in 10 mins. (3) Everything is production ready, no need to worry about deployment, security, maintenance etc, Pinecone deal with it and you can even use the service for free up to 1M vectors.
indeed30 5y ago
Does Pinecone have any position on the status of document embeddings and whether they would be considered PII? One of the challenges of using a fully managed service is the headache of adding yet another data subprocessor and all of the legal and compliance questions that raises.
- gk1 5y ago
  That depends on the document. We do not see the original document, only the embedding. You can argue that is sufficiently obfuscated to not count as PII. The good news is we are SOC2 compliant and GDPR-friendly and do a bunch of other stuff to help you meet security compliance requirements: https://www.pinecone.io/security/ https://www.pinecone.io/security/
  - indeed30 5y ago
    No, I understand that. I guess my question is actually around your experience with "You can argue that is sufficiently obfuscated to not count as PII" and whether your customers are actually successful with this argument.
    - gk1 5y ago
      Those who need more assurance just look at our SOC2 compliance, or have us go through a security review, or opt for the dedicated-environment deployment option.
nop_slide 5y ago
I just want to chime in and say that the resources on your website look amazing. I spent 5 minutes poking around and it looks really high quality. I'm dabbling in Postgres's full text search (ts_vector) for a small website, I know that is extremely simple compared to the offerings you provide, but your site has me quite interested in this space now. Eager to learn more about this tech!
- gk1 5y ago
  Glad you think so. Makes me want to expand it even more! What would you like to see covered?
  - nop_slide 5y ago
    Maybe this is too simple, but a comparison to Postgres' ts_vector and how to do something similar in your service?

tomcooks 5y ago

I would be happy with "find anything with Google search"

eliseumds 5y ago
It exists: https://developers.google.com/custom-search/v1/overview https://developers.google.com/custom-search/v1/overview. Might not be cheap for search-as-you-type though.
- sanxiyn 5y ago
  GP is complaining Google's search quality is poor, not inquiring about Google's search API.

deleted 5y ago

[deleted]

monkeybutton 5y ago

If you are interested in how ScaNN compares to other approximation algorithms, there are some benchmarks here: http://ann-benchmarks.com/ http://ann-benchmarks.com/

CoolGuySteve 5y ago

Is this more or less a k-d tree as a service? Where any distance function can be used to index the data? Or is it something different?

monkeybutton 5y ago
A k-d tree gives you exact answers to nearest neighbour queries.
- srean 5y ago
  A k-d tree is a data structure. Whether you use that for exact nearest neighbor query or approximate is up to the algorithm used. K-d trees work well for a handful of dimension beyond that it becomes quite expensive.
hamilyon2 5y ago
I thought k-d trees were useless in high-dimentional case. So, it must be something else.
- contravariant 5y ago
  I'd say they're about as useful as euclidean distance is.
ahurmazda 5y ago
More or less but as always the devil is in the detail. Here is a paper[1] that summarizes issues with naive approaches. Incidentally. the proposed solution (Hierarchical NSW) in this paper performed fairly well in the industry benchmarks [1] https://arxiv.org/ftp/arxiv/papers/1603/1603.09320.pdf https://arxiv.org/ftp/arxiv/papers/1603/1603.09320.pdf

eob 5y ago

My 2022 wish list is a Postgres plugin that adds vector + AKNN support that plays well with relational queries. There are so many use cases of that. I believe Ant Financial has published an open source one but iirc the English language documentation is sparse.

ccleve 5y ago
Do you have a link? I'd like to see it. I googled and did not find much pertaining to "ant financial" and "postgres". Perhaps your google-fu is better than mine...
etiennedi 5y ago
Checkout the open source vector search engine Weaviate: https://github.com/semi-technologies/weaviate https://github.com/semi-technologies/weaviate It’s not a relational db, but it supports Graph-like connections between objects, which makes it really easy to model your relations.
- thirdtrigger 5y ago
  Jup – this is an example from the demo dataset in the docs: https://link.semi.technology/3DPcphe https://link.semi.technology/3DPcphe
akane 5y ago
Check out pgvector: https://github.com/ankane/pgvector https://github.com/ankane/pgvector (disclosure: am author) It uses IVFFlat indexing, but could be extended to support product quantization / ScaNN.

amelius 5y ago

Does anyone know of a good benchmark suite for search technology? (And how well does the technique of the article work wrt it?)

GhettoComputers 5y ago
Use the one that gives you the best results, bing is too good for porn so it shows me porn when I am not even looking for it.
jkb79 5y ago
MS Marco is the largest relevancy collection in the open, see https://microsoft.github.io/msmarco/ https://microsoft.github.io/msmarco/

323 5y ago

People say google search is terrible these days, but I find the opposite. I can vaguely describe in a sentence the gist of an article I've read, or an image, and the proper result will usually be in the first page. Of course, it doesn't always work, sometimes there are "hash collisions" so to speak, but I don't think the old algorithm would have been more successfully either, since if I knew the exact keywords to use, I wouldn't need to start with a vague description in the first place.

Jemaclus 5y ago
I've literally gone to Google and typed something very similar to "That guy in that thing with the dog" and the correct answer shows up as the first result. It's quite brilliant and magical how they do that. But sometimes it's a total miss when I want something very specific, and it just shows me other things I didn't ask for.
bitcharmer 5y ago
I have the exact opposite experience. Search results are nowadays ridden with crap from companies that learned how to game SEO. If not that, you get scam websites or other ad/malware infested trash.
- radicaldreamer 5y ago
  Wish there was an easy way to remove forbes.com results
  - greybeardgeek 5y ago
    you can append -site:forbes.* to your query string
  - jiveturkey 5y ago
    https://news.ycombinator.com/item?id=29546433 https://news.ycombinator.com/item?id=29546433
  - hidden-spyder 5y ago

dorianmariefr 5y ago

Will probably be available as Postgres extension at some point. Seems like only special indexing of vectors is needed

ahurmazda 5y ago

For a similar ANN/vector search capabilities, https://vespa.ai/ https://vespa.ai/ is a great open-source solution. Elasticsearch may offer some form of ANN too but need to double check

sanxiyn 5y ago
I don't think Elasticsearch has one yet, but OpenSearch does: https://opensearch.org/docs/latest/search-plugins/knn/index/ https://opensearch.org/docs/latest/search-plugins/knn/index/
- m_ke 5y ago
  Lucene 9.0 just shipped with hnsw support, should make it into ES at some point (https://twitter.com/msokolov/status/1468395332531003393 https://twitter.com/msokolov/status/1468395332531003393) EDIT: ES integration PR: https://github.com/elastic/elasticsearch/issues/78473 https://github.com/elastic/elasticsearch/issues/78473
- ahurmazda 5y ago
  Ah! Great to know. ANN searches are becoming table stakes at this point. Hopefully, we will see more and more platforms adding it to their repertoire.
jkb79 5y ago
Yes, Vespa.ai can index billion scale vector datasets https://blog.vespa.ai/billion-scale-knn/ https://blog.vespa.ai/billion-scale-knn/ Vespa also allows expressing hybrid sparse and dense retrieval (WAND for sparse, ANN via HNSW for dense). It's also easy to express multi stage retrieval ranking phases, as vector search alone is not achieving state-of-the-art ranking results, see https://blog.vespa.ai/pretrained-transformer-language-models-for-search-part-4/ https://blog.vespa.ai/pretrained-transformer-language-models...

freediver 5y ago

I built multiple systems using vector search, one of them demoed in a search engine for non-commercial content at http://teclis.com http://teclis.com Running vector search (also sometimes referred to as semantic search, or a part of semantic search stack) is a trivial matter with open-source libraries like Faiss https://github.com/facebookresearch/faiss https://github.com/facebookresearch/faiss It takes 5 minutes to set up. You can search billion vectors on common hardware. For low-latency (up to couple of hundred milliseconds) use cases, it is highly unlikely that any cloud solution like this would be a better choice than something deployed on premise because of the network overhead. (worth noting is that there are about two dozen vector search libraries, all benchmarked at http://ann-benchmarks.com/ http://ann-benchmarks.com/ and most of them open-source) A much more interesting (and harder) problem is creating good vectors to begin with. This refers to the process of converting a text or an image to a multidimensional vector, usually done by a machine learning model such as BERT (for text) or ImageNet (for images). Try entering a query like 'gpt3' or '2019' into the news search demo linked in the Google's PR: https://matchit.magellanic-clouds.com/ https://matchit.magellanic-clouds.com/ The results are nonsensical. Not because the vector search didn't do its job well, but because generated vectors were suboptimal to begin with. Having good vectors is 99% of the semantic search problem. A nice demo of what semantic search can do is Google's Talk to Books https://books.google.com/talktobooks/ https://books.google.com/talktobooks/ This area of research s fascinating. For those who want to play with this more, an interesting end-to-end (including both vector generation and search) open-source solution is Haystack https://github.com/deepset-ai/haystack https://github.com/deepset-ai/haystack

noud 5y ago
I just made a couple of searches with teclis. I have to say, it's not bad. It's clearly not complete and I get several empty searches. But the content of the results are of higher quality than what I get with Google or DDG. Nice work!
- freediver 5y ago
  Thanks. The index is tiny and it is just a proof of concept of what a single person can do with technologies available nowadays. I felt it is better for it to return zero results than bad results.

ShamelessC 5y ago

This gh repo makes it pretty easy to create similar tech by first embedding any images you have using the released "CLIP" model from Open AI and then creating a Faiss index over these embeds for quick retrieval/decode. You can then do text->image, and image->image semantic search. https://github.com/rom1504/clip-retrieval https://github.com/rom1504/clip-retrieval

thirdtrigger 5y ago

Interesting – we are working on an open source vector search engine called Weaviate and did the same for the complete Wikipedia and Wikidata. [1] Docs: https://www.semi.technology/developers/weaviate/current/ https://www.semi.technology/developers/weaviate/current/ [2] Github: https://github.com/semi-technologies/weaviate https://github.com/semi-technologies/weaviate [3] Wikipedia demo dataset: https://github.com/semi-technologies/semantic-search-through-Wikipedia-with-Weaviate https://github.com/semi-technologies/semantic-search-through... [4] Wikidata dataset: https://github.com/semi-technologies/biggraph-wikidata-search-with-weaviate https://github.com/semi-technologies/biggraph-wikidata-searc... Last week there was also a feature on Techcrunch about vector search and Weaviate: https://techcrunch.com/2021/12/11/2246180/ https://techcrunch.com/2021/12/11/2246180/

CShorten 5y ago
I've made some videos on Weaviate as well (Henry AI Labs) if interested: [1] Wikipedia Vector Search Demo with Weaviate: https://www.youtube.com/watch?v=IGB8vjCuay0 https://www.youtube.com/watch?v=IGB8vjCuay0 [2] Vector Search through Wikidata with Weaviate: https://www.youtube.com/watch?v=T4zlvknSbGc https://www.youtube.com/watch?v=T4zlvknSbGc [3] Demonstrations of Deep Learning: https://www.youtube.com/watch?v=5jbneytoKi0 https://www.youtube.com/watch?v=5jbneytoKi0 [4] Weaviate's GraphQL API for Neurosymbolic Search: https://www.youtube.com/watch?v=K_2X48Tln9U https://www.youtube.com/watch?v=K_2X48Tln9U [5] Introducing the Weaviate Vector Search Engine: https://www.youtube.com/watch?v=AS_2U_INpKk https://www.youtube.com/watch?v=AS_2U_INpKk
- thirdtrigger 5y ago
  These are all great! There is also this video about modern search engines and Weaviate on the AI Coffee Break YT channel: https://www.youtube.com/watch?v=YkK5IKgxp-c https://www.youtube.com/watch?v=YkK5IKgxp-c
  - visarga 5y ago
    Well done promo video.

Hokusai 5y ago

It's not very good. I tried different pictures and the results are almost random. A picture from a cartoon returns from logos to any type of drawing. A picture of a battery returns cars and shops. A picture of food worked as expected and I got more food pictures.

Kydlaw 5y ago

There is a lot done vector search technology right now. I was less fortunate when looking at vector storage. I already looked at Pinecone or Weaviate but they are all paid products. Is there some people having feedback on this?

thirdtrigger 5y ago
Not true – Weaviate is open source: https://github.com/semi-technologies/weaviate https://github.com/semi-technologies/weaviate
Xenoamorphous 5y ago
ElasticSearch supports vectors (dense ones, they supported sparse ones at some point but they removed support I think), and has things like cosine similarity functions built in. Not sure how “free” it is though.
gk1 5y ago
Pinecone has a free tier that’s quite generous, fits around 1M items and will fit even more soon. Not sure this helps but just mentioning in case.

slig 5y ago

Let's say I have a content website with about 20k content pages. I want to automatically cluster the pages so that the each page has the related content linked. Right now I'm using a hacked together tf–idf using sklearn and Python2, and it just works. The downsides are that I have to compute everything offline whenever I add new content, and that it's one more thing to maintain/upgrade. I'm wondering if anyone has a suggestion of a SaaS or another alternative for my use case? Thanks!

ShamelessC 5y ago
Python2?
- slig 5y ago
  Yes, it's been running for about 10 years.
nickmancol 5y ago
I think this PostgreSQL is your friend in this case https://github.com/ankane/pgvector https://github.com/ankane/pgvector
- slig 5y ago
  Thank you, will have a look!

bserge 5y ago

> This is how Google services find ~~valuable content~~ absolute garbage for a wide variety of users worldwide in milliseconds. Leave it to the fucking user. I don't search my notes with vague phrases hoping the (dumb) "AI" will somehow find what I'm looking for. I search very specific keywords that I've associated with the content when I saved them. Same for a "wide area" search like the Internet, using Google. Perhaps it will improve with time. Right now I'd opt out if I could.

Lamad123 5y ago

Now I cannot even find a song on google or youtube even though I search several lines of the song's lyrics!!

pfd1986 5y ago

More "Find _something_ fast with vector search". I was not successful in finding anything relevant. PageRank works because it _ranks_ pages by, among other features, number and quality of visitors. E.g. searching for Huxley quote gives me silly blog posts about saving money. Query: "The function of the brain and nervous system is to protect us from being overwhelmed and confused by this mass of largely useless and irrelevant knowledge, by shutting out most of what we should otherwise perceive or remember at any moment, and leaving only that very small and special selection which is likely to be practically useful." Answer: "How to trick your brain into saving money"

___q 5y ago

So how do you game this? "Googlebomb" this? I assume it's harder than keyword-based search? As a search engine, what efforts do I take to stop someone from gaming vector-based search engines?

shanghaikid 5y ago

If you are not using GCP or you want to have an open-source alternative, Please check my project Milvus vector database (https://milvus.io https://milvus.io). We've published a bunch of demo cases powered by vector database on GitHub. https://github.com/milvus-io/bootcamp https://github.com/milvus-io/bootcamp We have built Milvus vector database upon ANN libraries like faiss, annoy, nsmlib, etc. We are aiming to create a cloud-scalable vector database. So Milvus comes to the crossroad of vector search and cloud database. There are many interesting system design topics in the development of Milvus 2.0. We will continue to share our experiences and thoughts on this topic.

currentsapi 5y ago

If anyone is interested, I maintain a list of open source vector search engine services[1]. Feel free to submit a new issues or merge request if you wish for new library added [1] https://github.com/currentsapi/awesome-vector-search https://github.com/currentsapi/awesome-vector-search

andre-z 5y ago

We are developing open-source vector search technology. https://github.com/qdrant/qdrant https://github.com/qdrant/qdrant It is a neural search engine with extended filtering support that implements a custom modification of the HNSW algorithm for Approximate Nearest Neighbour search. It allows applying search filters, including geolocation, without compromising on results. Developed entirely in Rust language. You can find some demos and documentation here https://qdrant.tech https://qdrant.tech

generall 5y ago
I wounder if there is any alternative implementations of ScaNN in languages like Rust.

visarga 5y ago

What if we had local vector search on our web browser history (the content as it was displayed)? That would be radical. I'm wondering why browser vendors don't scramble to create the personal vector database. It could be integrated through a browser extension to insert local results when doing regular web searches, or provide context for a speech based personal assistant. Having a neural net at hand could also prove useful in semantic filtering of webpages (hide or highlight content) and curating your news feeds.

jamesbriggs 5y ago
Agree that this would be super helpful

est 5y ago

What's the sqlite equivilant of vector search engine?

yboris 5y ago

I'm curious about Gensim Doc2Vec Model. I used it 3 years ago and got decent results in vectorizing articles and then finding articles that were similar based on input text (half-written article for example). What is new here? https://radimrehurek.com/gensim/auto_examples/tutorials/run_doc2vec_lee.html#sphx-glr-auto-examples-tutorials-run-doc2vec-lee-py https://radimrehurek.com/gensim/auto_examples/tutorials/run_...