3 ms·

I don't have firsthand experience, but everyone I know with second- (colleague) and third- (acquaintance) -hand experience says the performance promises hold up

I don't have firsthand experience, but everyone I know with second- (colleague) and third- (acquaintance) -hand experience says the performance promises hold up.

nemo44x 5y ago
Of course it does - it’s purpose built for a narrow use case. However it’s an extremely popular use case. Clickhouse optimizes on the 2 most important things for OLAP - minimal disk space due to compression benefits of columnar storage and minimal compute for the same reason - and therefore fast. However it isn’t flexible when you want to expand the use case. You can’t do any sort of text search, no complex joins (there are no foreign keys), and you need to order you tables there way you want to sort them. For certain things it’s perfect. It was built to solve a problem Yandax had and that’s notable. But it doesn’t have anywhere near the flexibility of Elasticsearch for example. But yes it’s purpose built to be extremely fast and minimize storage for the types of use cases it is built for.
- hodgesrm 5y ago
  I don't think it's accurate to say the ClickHouse use case is narrow. ClickHouse is extremely good at loading and querying events arriving in near-real time from event streams like Kafka. It can also load very efficiently from data lakes. Like Druid it can offer low latency response even as the data set size scales into trillions of rows. ClickHouse is used for everything from log management to managing CDN delivery to real-time marketing and many other applications. It's gone far beyond the web analytics use case for which it was originally developed at Yandex. Edit: clarification
  - nemo44x 5y ago
    Can you explain more how it’s used for real time marketing and other things? Ive only used it for OLAP which although it has a lot of different things is still more or less the same thing abstractly. Would love to hear about these other uses.
    - hodgesrm 5y ago
      For example, you can use ClickHouse queries to dynamically change the shape of pages based on user behavior across multiple sites (aka retargeting). You can can also use ClickHouse to manage CDN downloads in real time. Here are a couple of talks that illustrate both use cases. We still call this OLAP but it's quite different from traditional uses. In particular the core data come event streams. https://altinity.com/presentations/2020/06/16/big-data-in-real-time-how-clickhouse-powers-admirals-visitor-relationships-for-publishers https://altinity.com/presentations/2020/06/16/big-data-in-re... https://altinity.com/webinarspage/2020/6/23/big-data-and-beautiful-video-how-clickhouse-enables-mux-to-deliver-content-at-scale https://altinity.com/webinarspage/2020/6/23/big-data-and-bea...
- DevKoala 5y ago
  > However it isn’t flexible when you want to expand the use case. You can’t do any sort of text search, no complex joins (there are no foreign keys), and you need to order you tables there way you want to sort them. That is false. I have built a large scale system that does tons of text searches, complex joins and even queries on top of JSON objects with performance that rivals BigQuery and surpasses it in terms of cost. Edit: actually much better performance when you account for BigQuery’s cold start scenario.
  - nemo44x 5y ago
    BigQuery isn’t really OLTP and if I needed real time results for things like search I don’t think Clickhouse is the solution at even limited scale. I guess everyone’s mileage varies though.

I don't have firsthand experience, but everyone I know with second- (colleague) and third- (acquaintance) -hand experience says the performance promises hold up.

nemo44x 5y ago

Of course it does - it’s purpose built for a narrow use case. However it’s an extremely popular use case. Clickhouse optimizes on the 2 most important things for OLAP - minimal disk space due to compression benefits of columnar storage and minimal compute for the same reason - and therefore fast. However it isn’t flexible when you want to expand the use case. You can’t do any sort of text search, no complex joins (there are no foreign keys), and you need to order you tables there way you want to sort them. For certain things it’s perfect. It was built to solve a problem Yandax had and that’s notable. But it doesn’t have anywhere near the flexibility of Elasticsearch for example. But yes it’s purpose built to be extremely fast and minimize storage for the types of use cases it is built for.

hodgesrm 5y ago
I don't think it's accurate to say the ClickHouse use case is narrow. ClickHouse is extremely good at loading and querying events arriving in near-real time from event streams like Kafka. It can also load very efficiently from data lakes. Like Druid it can offer low latency response even as the data set size scales into trillions of rows. ClickHouse is used for everything from log management to managing CDN delivery to real-time marketing and many other applications. It's gone far beyond the web analytics use case for which it was originally developed at Yandex. Edit: clarification
- nemo44x 5y ago
  Can you explain more how it’s used for real time marketing and other things? Ive only used it for OLAP which although it has a lot of different things is still more or less the same thing abstractly. Would love to hear about these other uses.
  - hodgesrm 5y ago
    For example, you can use ClickHouse queries to dynamically change the shape of pages based on user behavior across multiple sites (aka retargeting). You can can also use ClickHouse to manage CDN downloads in real time. Here are a couple of talks that illustrate both use cases. We still call this OLAP but it's quite different from traditional uses. In particular the core data come event streams. https://altinity.com/presentations/2020/06/16/big-data-in-real-time-how-clickhouse-powers-admirals-visitor-relationships-for-publishers https://altinity.com/presentations/2020/06/16/big-data-in-re... https://altinity.com/webinarspage/2020/6/23/big-data-and-beautiful-video-how-clickhouse-enables-mux-to-deliver-content-at-scale https://altinity.com/webinarspage/2020/6/23/big-data-and-bea...
DevKoala 5y ago
> However it isn’t flexible when you want to expand the use case. You can’t do any sort of text search, no complex joins (there are no foreign keys), and you need to order you tables there way you want to sort them. That is false. I have built a large scale system that does tons of text searches, complex joins and even queries on top of JSON objects with performance that rivals BigQuery and surpasses it in terms of cost. Edit: actually much better performance when you account for BigQuery’s cold start scenario.
- nemo44x 5y ago
  BigQuery isn’t really OLTP and if I needed real time results for things like search I don’t think Clickhouse is the solution at even limited scale. I guess everyone’s mileage varies though.