6 ms·

Interesting – we are working on an open source vector search engine called Weaviate and did the same for the complete Wikipedia and Wikidata. [1] Docs: https:/

Interesting – we are working on an open source vector search engine called Weaviate and did the same for the complete Wikipedia and Wikidata. [1] Docs: https://www.semi.technology/developers/weaviate/current/ https://www.semi.technology/developers/weaviate/current/ [2] Github: https://github.com/semi-technologies/weaviate https://github.com/semi-technologies/weaviate [3] Wikipedia demo dataset: https://github.com/semi-technologies/semantic-search-through-Wikipedia-with-Weaviate https://github.com/semi-technologies/semantic-search-through... [4] Wikidata dataset: https://github.com/semi-technologies/biggraph-wikidata-search-with-weaviate https://github.com/semi-technologies/biggraph-wikidata-searc... Last week there was also a feature on Techcrunch about vector search and Weaviate: https://techcrunch.com/2021/12/11/2246180/ https://techcrunch.com/2021/12/11/2246180/

CShorten 5y ago
I've made some videos on Weaviate as well (Henry AI Labs) if interested: [1] Wikipedia Vector Search Demo with Weaviate: https://www.youtube.com/watch?v=IGB8vjCuay0 https://www.youtube.com/watch?v=IGB8vjCuay0 [2] Vector Search through Wikidata with Weaviate: https://www.youtube.com/watch?v=T4zlvknSbGc https://www.youtube.com/watch?v=T4zlvknSbGc [3] Demonstrations of Deep Learning: https://www.youtube.com/watch?v=5jbneytoKi0 https://www.youtube.com/watch?v=5jbneytoKi0 [4] Weaviate's GraphQL API for Neurosymbolic Search: https://www.youtube.com/watch?v=K_2X48Tln9U https://www.youtube.com/watch?v=K_2X48Tln9U [5] Introducing the Weaviate Vector Search Engine: https://www.youtube.com/watch?v=AS_2U_INpKk https://www.youtube.com/watch?v=AS_2U_INpKk
- thirdtrigger 5y ago
  These are all great! There is also this video about modern search engines and Weaviate on the AI Coffee Break YT channel: https://www.youtube.com/watch?v=YkK5IKgxp-c https://www.youtube.com/watch?v=YkK5IKgxp-c
  - visarga 5y ago
    Well done promo video.
mravl 5y ago
Real eyeopener. this will change the search industry completely
- detaro 5y ago
  why?
  - thirdtrigger 5y ago
    That's a fair question – but I'm going to assume the open-source nature is being meant with this.
  - bserge 5y ago
    Everyone's gonna start saving bookmarks again because search will become useless for about a decade. Thereby turning Pinboard into a multibillion dollar service still run by one person and revolutionizing small business :D
mariushn 5y ago
Looks great! Dumb question: How does Weaviate know that "Scandinavian" is close to "Finnish" ? The source not having "Scandinavian" at all. If their vectors are close, then the "vectorization" is quite standard for any text, and also per language?
- etiennedi 5y ago
  The models that create the vector embeddings are trained on either general or domain specific knowledge. So, to oversimplify it a bit: The model has learned - based on the training data it was presented with - that "Scandinavian" has a relationship to "Finnish". Since the vector space is high-dimensional you can think of each language concept having a distinct place in that space. In this case the concept for "Scandinavian" and "Finnish" were close enough that you got a matching result. To simplifiy it even more: The vectors do not represent the words but the meaning behind them. So, the two sentences "I like wine" and "The fermented juice of grapes is my favorite beverage" have zero keywords overlapping, but are semantically identical. So a good model would give them very similar vectors even though a traditional search engine would find zero resemblence between them. EDIT: Just realized I didn't answer the second part of your question. Yes, the models are language-specific, but there are also multilingual models that work across a large no. of different languages.
  - mariushn 5y ago
    Thank you! Now I understand why others mentioned that generating vectors is the hard part.
    - etiennedi 5y ago
      I agree, but at the same time now is the easiest it's ever been to create great vectors. Sentence-Bert [1] by Nils Reimers is a collection of pre-trained models specficially trained to create good vectors. You can use them out of the box with Weaviate. All you have to do is select your desired model [2] and your text (or images, etc.) will be translated into vectors at import time. As I mentioned in another comment, with Weaviate the goal is to make it as easy to use as any existing search engine or database while still providing you the benefits of Deep Learning & Vector Search. [1] Sentence-BERT: https://sbert.net https://sbert.net [2] Weaviate Customizer with Out-of-the-box models: https://www.semi.technology/developers/weaviate/current/getting-started/installation.html#customize-your-weaviate-setup https://www.semi.technology/developers/weaviate/current/gett...

CShorten 5y ago

I've made some videos on Weaviate as well (Henry AI Labs) if interested: [1] Wikipedia Vector Search Demo with Weaviate: https://www.youtube.com/watch?v=IGB8vjCuay0 https://www.youtube.com/watch?v=IGB8vjCuay0 [2] Vector Search through Wikidata with Weaviate: https://www.youtube.com/watch?v=T4zlvknSbGc https://www.youtube.com/watch?v=T4zlvknSbGc [3] Demonstrations of Deep Learning: https://www.youtube.com/watch?v=5jbneytoKi0 https://www.youtube.com/watch?v=5jbneytoKi0 [4] Weaviate's GraphQL API for Neurosymbolic Search: https://www.youtube.com/watch?v=K_2X48Tln9U https://www.youtube.com/watch?v=K_2X48Tln9U [5] Introducing the Weaviate Vector Search Engine: https://www.youtube.com/watch?v=AS_2U_INpKk https://www.youtube.com/watch?v=AS_2U_INpKk

thirdtrigger 5y ago
These are all great! There is also this video about modern search engines and Weaviate on the AI Coffee Break YT channel: https://www.youtube.com/watch?v=YkK5IKgxp-c https://www.youtube.com/watch?v=YkK5IKgxp-c
- visarga 5y ago
  Well done promo video.

mravl 5y ago

Real eyeopener. this will change the search industry completely

detaro 5y ago
why?
- thirdtrigger 5y ago
  That's a fair question – but I'm going to assume the open-source nature is being meant with this.
- bserge 5y ago
  Everyone's gonna start saving bookmarks again because search will become useless for about a decade. Thereby turning Pinboard into a multibillion dollar service still run by one person and revolutionizing small business :D

mariushn 5y ago

Looks great! Dumb question: How does Weaviate know that "Scandinavian" is close to "Finnish" ? The source not having "Scandinavian" at all. If their vectors are close, then the "vectorization" is quite standard for any text, and also per language?

etiennedi 5y ago
The models that create the vector embeddings are trained on either general or domain specific knowledge. So, to oversimplify it a bit: The model has learned - based on the training data it was presented with - that "Scandinavian" has a relationship to "Finnish". Since the vector space is high-dimensional you can think of each language concept having a distinct place in that space. In this case the concept for "Scandinavian" and "Finnish" were close enough that you got a matching result. To simplifiy it even more: The vectors do not represent the words but the meaning behind them. So, the two sentences "I like wine" and "The fermented juice of grapes is my favorite beverage" have zero keywords overlapping, but are semantically identical. So a good model would give them very similar vectors even though a traditional search engine would find zero resemblence between them. EDIT: Just realized I didn't answer the second part of your question. Yes, the models are language-specific, but there are also multilingual models that work across a large no. of different languages.
- mariushn 5y ago
  Thank you! Now I understand why others mentioned that generating vectors is the hard part.
  - etiennedi 5y ago
    I agree, but at the same time now is the easiest it's ever been to create great vectors. Sentence-Bert [1] by Nils Reimers is a collection of pre-trained models specficially trained to create good vectors. You can use them out of the box with Weaviate. All you have to do is select your desired model [2] and your text (or images, etc.) will be translated into vectors at import time. As I mentioned in another comment, with Weaviate the goal is to make it as easy to use as any existing search engine or database while still providing you the benefits of Deep Learning & Vector Search. [1] Sentence-BERT: https://sbert.net https://sbert.net [2] Weaviate Customizer with Out-of-the-box models: https://www.semi.technology/developers/weaviate/current/getting-started/installation.html#customize-your-weaviate-setup https://www.semi.technology/developers/weaviate/current/gett...