5 ms·

Has anyone used Clickhouse in production in anger? Our DevOps guy hates it and complains about how "fragile" it is. Keeping zookeeper happy seems to be a huge

by doix 5y ago

Has anyone used Clickhouse in production in anger? Our DevOps guy hates it and complains about how "fragile" it is. Keeping zookeeper happy seems to be a huge pain. Getting consistent backups seems to be difficult for some reason, and there is some problem with multiple replicas. As in, if you have 3 replicas and one dies, when it comes back up it will be out of sync and will refuse to resync itself automatically. So someone has to go and prod it to make it work again. In practice, none of this has ever been an issue and only comes up when he does his "chaos engineering" tests, but it makes him very nervous. On the otherhand, it's been orders of magnitude faster than using AWS Redshift/Apache Drill/PostgreSQL whenever we try and benchmark them on comparable hardware, so we stick with it.

ekimekim 5y ago
Hi, I ran a Clickhouse server in production at non-trivial scale for storage and query of application logs (semi-structured JSON documents). In our case we were consuming from a kafka topic which was the source of truth, so we opted not to use Clickhouse's built-in write replication and instead manually sharded and wrote to each replica independently (this meant replicas weren't always in perfect sync for brand-new data, but in practice this was acceptable). The distributed query side still worked perfectly despite this weird setup, even when we were doing weird things like re-sharding or taking some replicas down for one reason or another. I can't speak to backups as, once again, we were considering the kafka stream the source of truth and not the clickhouse datastore. We archived the data directly out of kafka, in a format more suitable to long term archival instead of live query. My experience with Clickhouse was that the core primitives were rock solid, but you were often on shaky ground when using more niche features - we encountered crashes when trying to use bloom filters for text search, for example. The performance of Clickhouse absolutely blew me away. It could read, filter and aggregate results as fast as the underlying disks could serve it data. I'll point out though that while Clickhouse is very good at what it does, it puts a lot of onus on the person designing the schema and writing the query to make sure it does it in the right way. In our case this worked well because we only had a small number of "shapes" of query we needed to serve, and we had a CLI tool that translated the user's query written in a simple DSL (really just a key-matches-value filter expression) into the SQL that would result in an efficient query on clickhouse's end. But for more flexibility-demanding workloads this could be a major issue as you need to know the system pretty well to write good queries.
vulkoingim 5y ago
In my experience there is a bit of a learning curve until you understand how each of the parts fit together and how to optimize for your use case. ZooKeeper indeed was one of those pain points, but once you get it stable there is practically no need to touch it. I found the tips in the docs on optimizing ZK really helpful [1]. Anoter mistake we did early on with ZooKeeper, that caused us a lot of pain was running it with network attached disks (we ran ClickHouse+ZK in the cloud). Once we switched to local disks, 99% problems disappeared. [1] https://clickhouse.tech/docs/en/operations/tips/#zookeeper https://clickhouse.tech/docs/en/operations/tips/#zookeeper
lnsp 5y ago
I'm quite sure Cloudflare uses ClickHouse for their web analytics stuff (see https://blog.cloudflare.com/http-analytics-for-6m-requests-per-second-using-clickhouse/ https://blog.cloudflare.com/http-analytics-for-6m-requests-p...).
- jgrahamc 5y ago
  We do. Extensively.
- wdb 5y ago
  PostHog is using Clickhouse for the Cloud version
caust1c 5y ago
Similar experience to vulkoingim: steep learning curve but quite stable once deployed properly. Schema management in zookeeper has been the biggest pain point for us. Occasionally individual clickhouse shards will get out of sync during a schema update, which can be hard to diagnose. We use a heavily modified version of clickhouse-backup[1], which works well for us. As for hands-off replica reboot: you must have an automated process to reapply the same schema which exists in zookeeper, otherwise it won't resync. If the local schema gets out of sync with that in zookeeper, then you'll have issues again. I expect a lot of these ergonomics issues will be fixed over time. It's already much easier to use than it was 3 years ago, and even if progress on usability and reducing the learning curve is slow the database performance makes it worth it. [1] https://github.com/AlexAkulov/clickhouse-backup https://github.com/AlexAkulov/clickhouse-backup
- FridgeSeal 5y ago
  I hold out perpetual hope that ClickHouse will move to a Raft or similar based consensus model and drop the external dependence on ZooKeeper on day.
  - tonyhb 5y ago
    This is happening: https://github.com/ClickHouse/ClickHouse/issues/15090 https://github.com/ClickHouse/ClickHouse/issues/15090 They're using NuRaft from ebay to drop zookeeper. As far as I remember, it's actually already merged and available to test. Performance is ~3x slower in terms of write as they're coordinating everything themselves, but it should be _much_ easier to maintain. I'm looking forward to this, too, and want to see more in terms of docs / support here :)
    - FridgeSeal 5y ago
      Woah! That’s really exciting, I’d definitely tolerate some write slow-down if it means not having to run ZK.
      - hodgesrm 5y ago
rdsubhas 5y ago
What you say is true, clustering clickhouse is hard. But we are happy in a very different way. What used to be 6+6 cluster of graphite is now served by a single clickhouse instance handling millions of unique metric datapoints everyday, that we haven't even had the need to cluster it yet. It's an order of magnitude difference in performance & cost. Scale, but just in a different way, not just horizontal but holistically. Maybe when we need to cluster, we'll face the same issues.
dominotw 5y ago
How does the setup and operational ease compared to druid. Afaik, druid also uses zookeeper. Is there any reason someone would choose druid over clickhouse?

by doix 5y ago

ekimekim 5y ago

Hi, I ran a Clickhouse server in production at non-trivial scale for storage and query of application logs (semi-structured JSON documents). In our case we were consuming from a kafka topic which was the source of truth, so we opted not to use Clickhouse's built-in write replication and instead manually sharded and wrote to each replica independently (this meant replicas weren't always in perfect sync for brand-new data, but in practice this was acceptable). The distributed query side still worked perfectly despite this weird setup, even when we were doing weird things like re-sharding or taking some replicas down for one reason or another. I can't speak to backups as, once again, we were considering the kafka stream the source of truth and not the clickhouse datastore. We archived the data directly out of kafka, in a format more suitable to long term archival instead of live query. My experience with Clickhouse was that the core primitives were rock solid, but you were often on shaky ground when using more niche features - we encountered crashes when trying to use bloom filters for text search, for example. The performance of Clickhouse absolutely blew me away. It could read, filter and aggregate results as fast as the underlying disks could serve it data. I'll point out though that while Clickhouse is very good at what it does, it puts a lot of onus on the person designing the schema and writing the query to make sure it does it in the right way. In our case this worked well because we only had a small number of "shapes" of query we needed to serve, and we had a CLI tool that translated the user's query written in a simple DSL (really just a key-matches-value filter expression) into the SQL that would result in an efficient query on clickhouse's end. But for more flexibility-demanding workloads this could be a major issue as you need to know the system pretty well to write good queries.

vulkoingim 5y ago

In my experience there is a bit of a learning curve until you understand how each of the parts fit together and how to optimize for your use case. ZooKeeper indeed was one of those pain points, but once you get it stable there is practically no need to touch it. I found the tips in the docs on optimizing ZK really helpful [1]. Anoter mistake we did early on with ZooKeeper, that caused us a lot of pain was running it with network attached disks (we ran ClickHouse+ZK in the cloud). Once we switched to local disks, 99% problems disappeared. [1] https://clickhouse.tech/docs/en/operations/tips/#zookeeper https://clickhouse.tech/docs/en/operations/tips/#zookeeper

lnsp 5y ago

I'm quite sure Cloudflare uses ClickHouse for their web analytics stuff (see https://blog.cloudflare.com/http-analytics-for-6m-requests-per-second-using-clickhouse/ https://blog.cloudflare.com/http-analytics-for-6m-requests-p...).

jgrahamc 5y ago
We do. Extensively.
wdb 5y ago
PostHog is using Clickhouse for the Cloud version

caust1c 5y ago

Similar experience to vulkoingim: steep learning curve but quite stable once deployed properly. Schema management in zookeeper has been the biggest pain point for us. Occasionally individual clickhouse shards will get out of sync during a schema update, which can be hard to diagnose. We use a heavily modified version of clickhouse-backup[1], which works well for us. As for hands-off replica reboot: you must have an automated process to reapply the same schema which exists in zookeeper, otherwise it won't resync. If the local schema gets out of sync with that in zookeeper, then you'll have issues again. I expect a lot of these ergonomics issues will be fixed over time. It's already much easier to use than it was 3 years ago, and even if progress on usability and reducing the learning curve is slow the database performance makes it worth it. [1] https://github.com/AlexAkulov/clickhouse-backup https://github.com/AlexAkulov/clickhouse-backup

FridgeSeal 5y ago
I hold out perpetual hope that ClickHouse will move to a Raft or similar based consensus model and drop the external dependence on ZooKeeper on day.
- tonyhb 5y ago
  This is happening: https://github.com/ClickHouse/ClickHouse/issues/15090 https://github.com/ClickHouse/ClickHouse/issues/15090 They're using NuRaft from ebay to drop zookeeper. As far as I remember, it's actually already merged and available to test. Performance is ~3x slower in terms of write as they're coordinating everything themselves, but it should be _much_ easier to maintain. I'm looking forward to this, too, and want to see more in terms of docs / support here :)
  - FridgeSeal 5y ago
    Woah! That’s really exciting, I’d definitely tolerate some write slow-down if it means not having to run ZK.
    - hodgesrm 5y ago

rdsubhas 5y ago

What you say is true, clustering clickhouse is hard. But we are happy in a very different way. What used to be 6+6 cluster of graphite is now served by a single clickhouse instance handling millions of unique metric datapoints everyday, that we haven't even had the need to cluster it yet. It's an order of magnitude difference in performance & cost. Scale, but just in a different way, not just horizontal but holistically. Maybe when we need to cluster, we'll face the same issues.

dominotw 5y ago

How does the setup and operational ease compared to druid. Afaik, druid also uses zookeeper. Is there any reason someone would choose druid over clickhouse?