4 ms·

Good point about MySQL/MariaDB. I think this is different though because search engines are at a big pivot point to include approximate-nearest-neighbor dense

Good point about MySQL/MariaDB. I think this is different though because search engines are at a big pivot point to include approximate-nearest-neighbor dense vector search (it has before been only sparse vector search for Lucene based platforms). Specifically, this feature https://github.com/elastic/elasticsearch/issues/42326#issuecomment-910719424 https://github.com/elastic/elasticsearch/issues/42326#issuec... will be a big change for Elasticsearch. OpenSearch might try to mimic the API, but implementation details here will matter a lot, since this type of search is really picky when it comes to performance/recall balance. OpenDistro has already been working on their own version: https://opendistro.github.io/for-elasticsearch/features/knn.html https://opendistro.github.io/for-elasticsearch/features/knn.... ...so will they switch their API? Perhaps - but the results are going to be very different.

gk1 5y ago
AFAIK when it comes to nearest-neighbor searches Elasticsearch is catching up with OpenDistro k-NN, and not the other way around.
- binarymax 5y ago
  Perhaps, but the point about divergence stands! Elastic won’t be copying the OpenDistro KNN API.
  - _msw_ 5y ago
    It is pretty clear to me that Elastic is planning to build their ANN features differently than OpenDistro's k-NN implementation, or other plugins modules that extend Easticsearch in similar ways. They now will build on the Apache Lucene capabilities that were collaboratively built "upstream" by a number of individuals, some that work for Amazon and some that work for Elastic. From the linked issue, it seemed that they were originally planning to develop this as a proprietary feature of Elasticsearch, without contributing the functionality to Apache Lucene, but then changed direction when the Apache Lucene developers (some of which are currently employed to do such work by Amazon) started to build its approximate nearest neighbor (ANN) vector search capabilities. [1] It's great to see folks that work for Elastic collaborating and building on what is in Apache Lucene to extend the utility of ANN with Hierarchical Navigable Small World Graphs (HNSW) [2]! From this, I think it should be possible to implement an Open Source version of the functionality with a compatible API, if that is something that OpenSearch users seek. [1] https://issues.apache.org/jira/browse/LUCENE-9004 https://issues.apache.org/jira/browse/LUCENE-9004 [2] https://github.com/apache/lucene/pull/250 https://github.com/apache/lucene/pull/250
    - binarymax 5y ago
      "it should be possible to implement an Open Source version of the functionality". Elasticsearch is still open source...just not in a license that Amazon can take advantage of.

gk1 5y ago

AFAIK when it comes to nearest-neighbor searches Elasticsearch is catching up with OpenDistro k-NN, and not the other way around.

binarymax 5y ago
Perhaps, but the point about divergence stands! Elastic won’t be copying the OpenDistro KNN API.
- _msw_ 5y ago
  It is pretty clear to me that Elastic is planning to build their ANN features differently than OpenDistro's k-NN implementation, or other plugins modules that extend Easticsearch in similar ways. They now will build on the Apache Lucene capabilities that were collaboratively built "upstream" by a number of individuals, some that work for Amazon and some that work for Elastic. From the linked issue, it seemed that they were originally planning to develop this as a proprietary feature of Elasticsearch, without contributing the functionality to Apache Lucene, but then changed direction when the Apache Lucene developers (some of which are currently employed to do such work by Amazon) started to build its approximate nearest neighbor (ANN) vector search capabilities. [1] It's great to see folks that work for Elastic collaborating and building on what is in Apache Lucene to extend the utility of ANN with Hierarchical Navigable Small World Graphs (HNSW) [2]! From this, I think it should be possible to implement an Open Source version of the functionality with a compatible API, if that is something that OpenSearch users seek. [1] https://issues.apache.org/jira/browse/LUCENE-9004 https://issues.apache.org/jira/browse/LUCENE-9004 [2] https://github.com/apache/lucene/pull/250 https://github.com/apache/lucene/pull/250
  - binarymax 5y ago
    "it should be possible to implement an Open Source version of the functionality". Elasticsearch is still open source...just not in a license that Amazon can take advantage of.