3 ms·

Benchmarks are always "It depends". And what it depends on are your data volume, how you want to query, whether you value ingestion greater than query speed an

by buro9 4y ago

Benchmarks are always "It depends". And what it depends on are your data volume, how you want to query, whether you value ingestion greater than query speed and timeliness and so forth. Elastic sweet spot is that it indexes everything, and you can query fast as a result. But it does this at the cost of ingest as it's doing the work to build indexes during ingestion and so ingest is more CPU intensive and can hit limits here. As a general purpose workhorse, Elastic shines. Loki sweet spot is that it has very few indexes, so ingestion is cheap and extremely capable for huge data volumes. It does this at the cost of query performance over very large data sets - without indexes queries brute force via mapreduce, which means you really want to specify where to look (which log streams) and when to look (a time window) and in that it excels. For logs, Loki is heaven. ClickHouse sweet spot is the indexes (columns) are very explicitly configured by engineers who know what the data looks like and how they want to query it. Now the ingest cost is balanced, and the query performance is great - but it did this at the cost of you knowing your data and how you're going to query it most of the time - it's not so good for esoteric questions that you'd never anticipated (though you can get very far through some of their column data types allowing you to be reasonably flexible on this). For BI data, ClickHouse is incredible. They all have sweet spots, and a benchmark is not going to answer the real questions - what data volume do you have, what do you value (ingest and preserve everything vs fastest query speed for ad-hoc queries vs a balanced approach), do you know how you want to query the data, etc? Other thoughts: Loki has recently moved to TSDB for the backend storage, these benchmarks don't go there. Elastic can use less disk if you configure for synthetic source (https://github.com/elastic/elasticsearch/issues/86603 https://github.com/elastic/elasticsearch/issues/86603) which discards the raw byte copy of the ingested data and only retains knowledge in the indexes, and uses the indexes to reconstruct the source should you request it. Enabling this is only supported by ES for metric type data (according to docs) but it is possible to enable it for other types of data. Nothing to add about ClickHouse, I've used all three databases and worked against all three for huge volumes of data - if I want a more OLAP style querying than OLTP and I know my data then today ClickHouse absolutely shines here with the others playing catch up. Elastic and Loki shine far more for OLTP workloads, so it's a trade-off again (though Elastic does a good job at doing pretty well for more OLAP cases than Loki does today due to it having column storage - and Loki wins on being able to ingest more cheaply - meaning being able to ingest more at the same or lower cost). What you value and what you want to do is up to you.

pranay01 4y ago
thanks for the note. makes a lot of sense. >I want a more OLAP style querying than OLTP and I know my data then today ClickHouse absolutely shines here with the others playing catch up Curious, what type of OLAP workloads did you work with? Were there cases where you tried both CH and Elastic for the same use case and decided for oner over the other?
- buro9 4y ago
  Again - "it depends". The decisions I've seen firsthand (or been a part of): CH vs ES: In petabyte sized stores of highly homogenised data (very few well structured types of data, i.e. "HTTP requests" where all logs were precisely following the same format and blended business data (i.e. the HTTP logs contained info on customer, tenant, etc)) and the primary goal was business insight and operator analytics - ClickHouse was the clear winner (capability to ingest at this scale whilst providing query performance at this scale). This was Cloudflare for the HTTP logs, L7 firewall logs and the L3 logs (sampled packet headers). They've blogged about them enough that there's no secrecy they are using CH for this. Loki vs CH: In petabyte sized stores of logs from tens to hundreds of thousands of machines where every log is differently structured across apps, OS, device where the primary goal was to store all logs and provide operator observability - Loki was the clear winner (capability to ingest, control over storage costs). There is no one-size-fits-all, and benchmarks are not useful in answering the questions that should be asked. Engineers should go back to first principles and evaluate systems and solutions according to their specific criteria. Some may even end on an entirely different solution than the ones you've outlined - PostgreSQL is pretty phenomenal too, and the full text search on that is great.

by buro9 4y ago

pranay01 4y ago

thanks for the note. makes a lot of sense. >I want a more OLAP style querying than OLTP and I know my data then today ClickHouse absolutely shines here with the others playing catch up Curious, what type of OLAP workloads did you work with? Were there cases where you tried both CH and Elastic for the same use case and decided for oner over the other?

buro9 4y ago
Again - "it depends". The decisions I've seen firsthand (or been a part of): CH vs ES: In petabyte sized stores of highly homogenised data (very few well structured types of data, i.e. "HTTP requests" where all logs were precisely following the same format and blended business data (i.e. the HTTP logs contained info on customer, tenant, etc)) and the primary goal was business insight and operator analytics - ClickHouse was the clear winner (capability to ingest at this scale whilst providing query performance at this scale). This was Cloudflare for the HTTP logs, L7 firewall logs and the L3 logs (sampled packet headers). They've blogged about them enough that there's no secrecy they are using CH for this. Loki vs CH: In petabyte sized stores of logs from tens to hundreds of thousands of machines where every log is differently structured across apps, OS, device where the primary goal was to store all logs and provide operator observability - Loki was the clear winner (capability to ingest, control over storage costs). There is no one-size-fits-all, and benchmarks are not useful in answering the questions that should be asked. Engineers should go back to first principles and evaluate systems and solutions according to their specific criteria. Some may even end on an entirely different solution than the ones you've outlined - PostgreSQL is pretty phenomenal too, and the full text search on that is great.