5 ms·

Whenever I see quotes like “182x faster than MySQL! 29x faster than Elasticsearch!” as the intro to a project, I’m immediately sceptical of the quality of the e

Whenever I see quotes like “182x faster than MySQL! 29x faster than Elasticsearch!” as the intro to a project, I’m immediately sceptical of the quality of the entire project.

snikolaev 4y ago
Me too! That's why it's very important when you say things like that to make sure you provide enough evidences. In this case there is https://db-benchmarks.com/ https://db-benchmarks.com/ where you can see all the details and it's 100% opensource - https://github.com/db-benchmarks/db-benchmarks https://github.com/db-benchmarks/db-benchmarks including the UI https://github.com/db-benchmarks/ui https://github.com/db-benchmarks/ui . The results themselves are opensource too - https://github.com/db-benchmarks/db-benchmarks/tree/main/results https://github.com/db-benchmarks/db-benchmarks/tree/main/res... So anyone can reproduce the results or at least look carefully into them, understand the testing methodology etc.
- tmikaeld 4y ago
  Comparing full-text search engines on queries that aren't full-text search are of course slow. It would be interesting to see queries benchmarked for their intended workload (logs/sorting/full-text etc). Fastest Avg: https://db-benchmarks.com/?cache=fast_avg&engines=elasticsearch_tuned_32%2Cmanticoresearch_columnar&tests=taxi&memory=110000&queries=0%2C1%2C2%2C3%2C4%2C5%2C6%2C7%2C8%2C9%2C10%2C11%2C12%2C13%2C14%2C15%2C16 https://db-benchmarks.com/?cache=fast_avg&engines=elasticsea... Slowest: https://db-benchmarks.com/?cache=slowest&engines=elasticsearch_tuned_32%2Cmanticoresearch_columnar&tests=taxi&memory=110000&queries=0%2C1%2C2%2C3%2C4%2C5%2C6%2C7%2C8%2C9%2C10%2C11%2C12%2C13%2C14%2C15%2C16 https://db-benchmarks.com/?cache=slowest&engines=elasticsear...
  - snikolaev 4y ago
    Elasticsearch is used not only for full-text search. It's widely used for analytics (aggregations) and filtering too, e.g. when you do log analytics. Comparing Elasticsearch / Manticore with MySQL may be not the fairest thing since they are too different, but comparing them one with another and using not only full-text queries seems fine to me.
    - blowski 4y ago
      It all seems too good to be true, so I'd like understand more about the limitations.
      - thayne 4y ago
        Well for one it's written in c++, which means it is more likely to have memory safety bugs, which could potentially be security vulnerabilities.
        nasretdinov 4y ago
        While generally true, I would argue that for the use cases where full-text search is mostly used (e.g. either search through a public database, or, quite the opposite, an internal system that does search through logs collected from various sources), in practice security vulnerabilities are less of a concern because usually even if you can expose some data stored in the full text index using that vulnerability, it would still only expose data you could already find in that search engine and that's already accessible to you :).
        thayne 4y ago
        That might be true in some cases. But for the public data case, you probably still need to worry about DoS or data corruption. In the logs case, a malicious actor can probably control at least part of the logs, so if a bug leads to arbitrary code execution, a bad actor could possibly get all kinds of valuable data. Also, just to be clear, the language doesn't necessarily mean there are significant security bugs. A well written c++ app is probably better than a poorly written java app. It's just harder to avoid memory bugs in c++ than java.
    - suction 4y ago
      I'd say (fulltext) search is one of the least interesting features of ES, and the aggregations are its USP. E.g. moving averages on (biggish) datasets can be calculated on very cheap hardware.
marginalia_nu 4y ago
I'm not even entirely sure what such a comparison means for a DBMS to be faster than another. In practice, claims boil down to this query is 182x faster than that query in MySQL. Using the same type of comparison, you can come to the conclusion that MySQL is 10x faster than MySQL because one query was constructed one way and the other another. (Applying induction, we can thus prove that MySQL is infinitely fast ;-)

Whenever I see quotes like “182x faster than MySQL! 29x faster than Elasticsearch!” as the intro to a project, I’m immediately sceptical of the quality of the entire project.

snikolaev 4y ago

Me too! That's why it's very important when you say things like that to make sure you provide enough evidences. In this case there is https://db-benchmarks.com/ https://db-benchmarks.com/ where you can see all the details and it's 100% opensource - https://github.com/db-benchmarks/db-benchmarks https://github.com/db-benchmarks/db-benchmarks including the UI https://github.com/db-benchmarks/ui https://github.com/db-benchmarks/ui . The results themselves are opensource too - https://github.com/db-benchmarks/db-benchmarks/tree/main/results https://github.com/db-benchmarks/db-benchmarks/tree/main/res... So anyone can reproduce the results or at least look carefully into them, understand the testing methodology etc.

tmikaeld 4y ago
Comparing full-text search engines on queries that aren't full-text search are of course slow. It would be interesting to see queries benchmarked for their intended workload (logs/sorting/full-text etc). Fastest Avg: https://db-benchmarks.com/?cache=fast_avg&engines=elasticsearch_tuned_32%2Cmanticoresearch_columnar&tests=taxi&memory=110000&queries=0%2C1%2C2%2C3%2C4%2C5%2C6%2C7%2C8%2C9%2C10%2C11%2C12%2C13%2C14%2C15%2C16 https://db-benchmarks.com/?cache=fast_avg&engines=elasticsea... Slowest: https://db-benchmarks.com/?cache=slowest&engines=elasticsearch_tuned_32%2Cmanticoresearch_columnar&tests=taxi&memory=110000&queries=0%2C1%2C2%2C3%2C4%2C5%2C6%2C7%2C8%2C9%2C10%2C11%2C12%2C13%2C14%2C15%2C16 https://db-benchmarks.com/?cache=slowest&engines=elasticsear...
- snikolaev 4y ago
  Elasticsearch is used not only for full-text search. It's widely used for analytics (aggregations) and filtering too, e.g. when you do log analytics. Comparing Elasticsearch / Manticore with MySQL may be not the fairest thing since they are too different, but comparing them one with another and using not only full-text queries seems fine to me.
  - blowski 4y ago
    It all seems too good to be true, so I'd like understand more about the limitations.
    - thayne 4y ago
      Well for one it's written in c++, which means it is more likely to have memory safety bugs, which could potentially be security vulnerabilities.
      - nasretdinov 4y ago
        While generally true, I would argue that for the use cases where full-text search is mostly used (e.g. either search through a public database, or, quite the opposite, an internal system that does search through logs collected from various sources), in practice security vulnerabilities are less of a concern because usually even if you can expose some data stored in the full text index using that vulnerability, it would still only expose data you could already find in that search engine and that's already accessible to you :).
        thayne 4y ago
        That might be true in some cases. But for the public data case, you probably still need to worry about DoS or data corruption. In the logs case, a malicious actor can probably control at least part of the logs, so if a bug leads to arbitrary code execution, a bad actor could possibly get all kinds of valuable data. Also, just to be clear, the language doesn't necessarily mean there are significant security bugs. A well written c++ app is probably better than a poorly written java app. It's just harder to avoid memory bugs in c++ than java.
  - suction 4y ago
    I'd say (fulltext) search is one of the least interesting features of ES, and the aggregations are its USP. E.g. moving averages on (biggish) datasets can be calculated on very cheap hardware.

marginalia_nu 4y ago

I'm not even entirely sure what such a comparison means for a DBMS to be faster than another. In practice, claims boil down to this query is 182x faster than that query in MySQL. Using the same type of comparison, you can come to the conclusion that MySQL is 10x faster than MySQL because one query was constructed one way and the other another. (Applying induction, we can thus prove that MySQL is infinitely fast ;-)