16 ms·

ClickHouse Cloud is now in Public Beta

278 points by taubek 4y ago

zX41ZdbW 4y ago
I have (an extremely boring, but quite hands-on) video about various ways of ClickHouse optimizations on top of external storage: https://www.youtube.com/watch?v=rK2BsaaaOCA https://www.youtube.com/watch?v=rK2BsaaaOCA (starting from around 40:00).
gsanderson 4y ago
Checking the pricing, $0.0125 per write unit. It says each write "generates one or more write units". So ... $1.25 per 100 writes? That can't be right. I wondered if it meant writes per second (like how AWS DynamoDB or Azure CosmosDB work, with their unit-based billing).
- tbragin 4y ago
  With an analytical database like ClickHouse, you can write many rows with a single INSERT statement (thousands of rows, millions of rows, and more). In fact, this kind of batching is recommended. Larger inserts will consume more write units than smaller inserts. Check out our billing FAQ for some examples, and we will be enhancing it with more detail as questions from our users come in (we'll work on clarifying this specific point): https://clickhouse.com/docs/en/manage/billing/ https://clickhouse.com/docs/en/manage/billing/ We also provide a $300 credit free trial to try out your workload and see how it translates to some of the usage dimensions. Finally, this is a Beta product, so keep the feedback coming!
  - gsanderson 4y ago
    Thanks, I agree batching inserts would indeed be a good idea and it makes sense that's recommended. However that link you mention (as of now) does not specify what a write unit is. So if that could be clarified, that would be great. Since from your reply it sounds like one INSERT would indeed (at a minimum) incur one write unit. And thus 100 writes could indeed cost $1.25. Which could get expensive, fast.
    - tbragin 4y ago
      An INSERT can consume less than one write unit, depending on how many rows and bytes it writes to how many partitions, columns, materialized views, data types, etc. So, a "write unit", which corresponds to hundreds of low-level write operations, typically translates to many batched INSERTS. We are working to improve our examples in the FAQ to clarify - thank you so much for asking the question!
      - ok_dad 4y ago
        You know what would be really helpful? Some tooling around measuring "write units", like a fake local ClickHouseCloud API you could submit a write to and see how many "write units" that particular write would take. Of course, that would make pricing transparent and easy and SaaS companies don't seem like that. I challenge you folks to actually figure out a way to prevent surprise bills, until then all this "write unit" stuff is bullshit.
      - mike_d 4y ago
        > We are working to improve our examples in the FAQ to clarify Based on the comments here, and my own confusion, I think figuring out a different way of billing read/write operations is in order.
  - mrwnmonm 4y ago
    "Each write operation (INSERT, DELETE, etc) consumes write units depending on the number of rows, columns, and partitions it writes to."
  - benjaminwootton 4y ago
    How about if you are streaming in from Kafka and inserting each event as if arrives? Clickhouse is ideal for rapid analytics over event data so to introduce batching would be disappointing. Batch upload is of course more cost effective, but I would expect that to be more typical in a traditional data warehouse where we are uploading hourly or daily extracts. Clickhouse would likely show up in more real time and streaming architectures where events arrive sporadically. I am a huge fan and advocate of Clickhouse, but the concept of a write unit is strange and the likely charges if you have to insert small batches would make this unviable to use. A crypto analytics platform I built on top of Clickhouse would cost in the $millions per month vs $X00 or low $X000 with competitors or self hosting.
- zX41ZdbW 4y ago
  A single INSERT takes around 0.01 write units.
datalopers 4y ago
Clickhouse is an amazing product but this pricing looks excessive. A single node instance with a fast disk is more than sufficient for most needs: https://hub.docker.com/r/clickhouse/clickhouse-server https://hub.docker.com/r/clickhouse/clickhouse-server If you need a cluster, https://github.com/Altinity/clickhouse-operator https://github.com/Altinity/clickhouse-operator makes things easy
- beoberha 4y ago
  I’m sure there are tons of people very willing to pay for this if it means not having to run their own k8s clusters. Managed offerings like this are expensive, but worth every penny for some.
- derefr 4y ago
  We're a company that has been using the private beta of ClickHouse Cloud. IMHO, the unique value-prop of this offering is that it elastically scales reads (compute), like Redshift/Snowflake/BigQuery/other cloud data warehouses do, while also "being Clickhouse", and so giving you very traditional SQL querying capabilities (where those others all ask you to bend your SQL to fit the DB, sometimes in pretty ridiculous ways.) I would suggest not thinking of this offering as "ClickHouse, but in the cloud"; but rather, thinking of this as "a cloud data warehouse, but rather than using its own proprietary query engine, it's just ClickHouse." If you haven't evaluated other cloud data warehouses as alternatives to solving your problem (and found them wanting for one reason or another), then you're likely not in the niche that would see a positive profit margin on using ClickHouse Cloud.
thomoco 4y ago
I wanted to note that ClickHouse Cloud results are now also being reported in the public ClickBench results: https://benchmark.clickhouse.com/ https://benchmark.clickhouse.com/ Good to see transparent comparisons available now for Cloud performance vs. self-hosted or bare metal results as well as results from our peers. The ClickHouse team will continue to optimize further - as scale and performance is a relentless pursuit here at ClickHouse, and something we expect to be performed transparently and in a reproducible manner. Public benchmarking benefits all of us in the tech industry as we learn from each other in sharing the best techniques for attaining high performance within a cloud architecture Full disclosure: I do work for ClickHouse, although have also been a past member of SPEC in developing and advocating for public, standardized benchmarks
- morelisp 4y ago
  Can you clarify what a "write unit" is? Naively it sounds like it might be blocks x partitions x replicas that actually hit disk. (Which is also probably not very clear to people not already using CH, but I have at least middling knowledge of CH's i/o patterns and I have no clue what a "write unit" is from the page's description.)
  - tylerhannan 4y ago
    It's Tyler from ClickHouse. Check out the response below that has a reference to some of our billing FAQs.
    - morelisp 4y ago
      Right, that link covers read units which is also what I expected - essentially the number of files I have to touch - but I still have no clue about write units. Is one block on one non-partitioned non-distributed table one write unit? What about one insert that's two blocks on such a table? What about one block on a null engine with two MVs listening to insert into two non-partitioned non-distributed tables? What if the table is a replacing mergetree, do I incur WUs for compactions? etc. My worry is that it is essentially 1 WU = 1 new part file, which I understand makes sense to bill on but is tremendously intransparent for users - at least I have no clue how often we roll new part files, instead I'm focused on total network and disk i/o performance on one side and client query latency on the other.
kawsper 4y ago
We're currently using InfluxDB for some timeseries metrics, but the 2.0 migration path have been terrible, even for simple examples, so we're looking for something else. Have anyone migrated from InfluxDB to ClickHouse?
- monstrado 4y ago
  Disclaimer: I work at ClickHouse At a previous company, I wrote a simple TCP server to receive LineProtocol, parse it and write to ClickHouse. I was absolutely blown away by how fast I could chart data in Grafana [1]. The compression was stellar, as well...I was able to store and chart years of history data. We basically just stopped sending data to Influx and migrated everything over to the ClickHouse backend. [1] https://grafana.com/grafana/plugins/grafana-clickhouse-datasource/ https://grafana.com/grafana/plugins/grafana-clickhouse-datas...
- danielvf 4y ago
  Clickhouse is so stupidly, mindbogglingly fast at what it does that you can often replace a much more specialized databases or specialized schemas / queries with brute force Clickhouse and come out far ahead. It really depends on the size of what you are doing. I've used Clickhouse for timeseries data in the tens of millions events per day and it worked well.
base 4y ago
Is there an easy way to have ClickHouse Cloud ingest data from MySQL hosted in Amazon RDS?
- thomoco 4y ago
  There are a few options for migrating or synchronizing data from MySQL - I'd recommend starting with this page in the ClickHouse Docs - there is a nice video there that explains some of those options: https://clickhouse.com/docs/en/integrations/migration/ https://clickhouse.com/docs/en/integrations/migration/ Depending on what you are trying to achieve, you could use clickhouse-local with the MySQL engine to move data, or could use an ETL/ETL tool to migrate/sync
  - hodgesrm 4y ago
    Good point. If you just want to copy data far and away the easiest way to transfer data is using MySQLDatabaseEngine. You can even copy table definitions. Watch for issues with Decimal datatypes if you do this.
    - morelisp 4y ago
      Be careful with this engine, it's easy to accidentally expose the password as you only need table read permissions if it wasn't set up using an external credential file. https://github.com/ClickHouse/ClickHouse/issues/3311 https://github.com/ClickHouse/ClickHouse/issues/3311 I also had some pretty bad join performance (CH table joined to MySQL table), the quick solution to both of these is that we instead use the table function (https://clickhouse.com/docs/en/sql-reference/table-functions/mysql/ https://clickhouse.com/docs/en/sql-reference/table-functions...) to copy the data periodically.
      - hodgesrm 4y ago
        Use Named Collections to protect credentials. They are very handy. Here's an article that discusses use in the JDBC Bridge. https://altinity.com/blog/connecting-clickhouse-to-external-data-sources-using-the-jdbc-bridge https://altinity.com/blog/connecting-clickhouse-to-external-... Disclaime: I work for Altinity.
chrisballinger 4y ago
At first I was a bit confused about why The Onion’s spin-off site had a cloud offering, but that one is called ClickHole.
skadamat 4y ago
For context, there are multiple Clickhouse companies. This is the one that spun out of Yandex and took the Clickhouse name as their namesake for the company! Altinity has been in the space for a while offering Clickhouse services as well: https://altinity.com/ https://altinity.com/ EDIT: brazenly was the wrong word here :)
- stingraycharles 4y ago
  Wait, wasn't Clickhouse itself spun out of Yandex anyway? I.e. Yandex opensources Clickhouse, then later starts Clickhouse (the company)?
  - kelp 4y ago
    That is literally what happened. There is nothing brazen here.
- qaq 4y ago
  "brazenly" :)? Are you for real? Top 3 commiters work there including Alexey who created CH in the first place alexey-milovidov 17,522 commits 5,648,537 ++ 5,580,633 -- alesapin 4,618 commits 1,024,262 ++ 932,501 -- KochetovNicolai 3,867 commits 377,420 ++ 314,035 --
- hodgesrm 4y ago
  Hi! Thank you for mentioning my company Altinity. We love ClickHouse and have been supporting customers on it for years. That said I would like to defend ClickHouse and their use of the logo. They acquired the IP and it's their right to use it as they please. It's hard to imagine them not using the ClickHouse name. There's a crowd of companies using/supporting ClickHouse so it's the obvious way to market themselves. The interesting question is whether it's a good thing for users to have a large company controlling ClickHouse rather than having it in a foundation.
  - mrwnmonm 4y ago
    Will Altinity Cloud compete with ClickHouse Cloud now? :)
epberry 4y ago
Great stuff! I know the ClickHouse team and they are world class. If you're more familiar with a relational database some things will feel weird. For example, you should not insert 1 row at a time, there's over 1,000 built-in functions, and the default connection is often over HTTPS, but not 443.
- zX41ZdbW 4y ago
  ClickHouse Cloud provides port 443 as well as 8443. You can insert one row at a time, it's perfectly ok with "async_insert=1".
encoderer 4y ago
I’m a little sad to see them embrace magic “insert unit” pricing, instead of taking the approach Altinity uses where you are renting an instance size that you can compare apples-to-apples with running your own cluster on ec2.
- aseipp 4y ago
  Honestly that's something they can probably offer separately, but I really prefer this pricing for most use cases where I want a database that is always available but has bursty request/response patterns. This means I can have an analytical database available for all my small services, websites, etc without having to think too much about availability, support, and a constant price overhead. But ClickHouse is so fast you can get pretty far with a $10 VPS, I admit. Probably the best comparison is CockroachDB Cloud. They have a "serverless" offering based on unit pricing and a dedicated offering based on provisioned servers + support/maintenance overhead. I think that would be the ideal place to go long-term, but I'm super excited for this current one. I love ClickHouse and want to support them. ClickHouse is also an interesting case because there's lots of options to migrate clusters, use S3 as long-term storage, etc to where I don't particularly feel locked into this offering if I ever wanted to shift into my own.
- izrailev 4y ago
  Serverless "pay for usage" is different than the fixed-size dedicated cluster pricing model, and can be quite a bit cheaper, especially with spiky query traffic. It should also be more reliable, since you don't have to predict and provision capacity for your peak usage ahead of time. Disclaimer: I work for ClickHouse.
  - encoderer 4y ago
    Ok but it can also be quite a bit more expensive and your bill is less predictable. Disclaimer: I’m a customer of altinity.
    - izrailev 4y ago
      [dead]
datalopers 4y ago
Is this fully hosted or runs in my cloud? Is this running https://github.com/ClickHouse/ClickHouse https://github.com/ClickHouse/ClickHouse or a closed-source Clickhouse Cloud variant with added features?
- kbiyer 4y ago
  It is a fully hosted offering and is based on ClickHouse core v22.10
  - lsllc 4y ago
    Curious how CH are ultimately actually hosting this, on a public cloud or is this their own cloud.
    - morelisp 4y ago
      Billing page says AWS for storage, so I'd assume the same for compute. If they're really doing elastic reads as suggested in another comment I don't think this can be the standard CH server (at least for reads) - or I've missed something very exciting in recent versions.
      - datalopers 4y ago
        It feels like there's gotta be something proprietary beyond just using a s3 disk setup but maybe not? is simply having data_cache_enabled and cache_path pointing to local nvme ssd sufficient to achieve similar speeds? * https://clickhouse.com/docs/en/guides/sre/configuring-s3-for-clickhouse-use/ https://clickhouse.com/docs/en/guides/sre/configuring-s3-for... * https://clickhouse.com/docs/en/integrations/s3/s3-merge-tree/ https://clickhouse.com/docs/en/integrations/s3/s3-merge-tree...
        morelisp 4y ago
        For non-elastic writes this might be sufficient (I have no clue as to performance of AWS S3 for this kind of use case) - but supposedly[0] there's some elastic read capacity. As far as I know that's not possible with standard CH at all, and may be where some secret sauce is. I can vaguely imagine how you'd build such a thing, you'd spin up new instances that knows your write-primary's sharding/partitioning scheme - heck you could even approximate it with slightly-augmented clickhouse-clients doing the S3 pulls directly and no real additional "server" - but as far as I know there's no way out of the box. [0] https://news.ycombinator.com/item?id=33081658 https://news.ycombinator.com/item?id=33081658
MentallyRetired 4y ago
Some unsolicited feedback: 1) I had no idea what Clickhouse was for the first 30 seconds looking at the homepage. I now understand it to be a database of some sort. I shouldn't have seen the words "performance" and "cloud" and "serverless" before seeing the word database, right? I'm starting off confused. There shouldn't be an assumption that I know what you all do. 2) I have no idea what a column oriented database is. I've been a developer for 29 years (mostly frontend but I do a lot of full stack too). If I need an explainer, a lot of devs will. Aside from that, it looks like a nice offering and I wish you all the best!
- cinbun8 4y ago
  If you don't know what a column oriented DB is, that page is probably not for you.
  - bagels 4y ago
    I know what a column oriented DB is, but ClickHouse was not on my radar before. The pitch on the landing page is that ClickHouse Cloud is great if you love ClickHouse. If you don't know what ClickHouse is, you have to do some work to find out.
  - pqdbr 4y ago
    Care to elaborate? I’m a fullstack Rails dev for 12 years and I had no idea either, just like OP. Why alienate potential users from the get go?
  - ipaddr 4y ago
    Isn't it just a database in Second normal form (2NF)?
    - bfung 4y ago
      No. Column oriented storage db’s make it fast to read 10s to billions of “rows”, usually when you have data that you want to read that can be independent of other columns. Ex: (stock close) - row storage isn’t going to buy you much here, you’d rather read the whole “stock close” column as fast as possible. Whereas traditional db’s, data like [first name, last name], the columns may have way less meaning on their own and you need both columns to have the data make sense. A traditional db with B(-)tree storage is much slower when dealing with that type of usage. Storing (stock close) in a single column format makes that type of query much faster.
andrewmutz 4y ago
Clickhouse was spun out of Yandex, which is a Russian corporation. Given existing geopolitical tensions, is there anything to worry about there? Does anyone know how much of the Clickhouse team (or ownership) is still located in Russia?
- 46Bit 4y ago
  They have been extremely clear in their support of Ukraine https://clickhouse.com/blog/we-stand-with-ukraine https://clickhouse.com/blog/we-stand-with-ukraine
  - risyachka 4y ago
    It's just a bunch of text, doesn't show any support whatsoever. They can show support by donating to UA defence and showing proof (important - not some neutral org). Otherwise it is not support but a bunch of bs
    - ericb 4y ago
      In places like Russia, those words are very dangerous, and could get you jailed or sent to the front. Even calling it a "war" was/is a punishable offense. So, yes, words, but the potential consequences of these words have more significance than empty air.
    - xwowsersx 4y ago
      Why do you get to define the acceptable forms of support...?
- kelp 4y ago
  You can see on their jobs page and 'our story' page, they are mostly in the US and The Netherlands.
  - mike_d 4y ago
    Yandex is technically incorporated in The Netherlands and has an engineering presence there, but they are as Russian controlled as can be.
lambdadmitry 4y ago
It's interesting how diligently they wiped our any and all mentions of Yandex from their website, to the extent that Google struggles to find meaningful mentions apart from a few exceptions like [1]. Quite peculiar considering that just two years ago, according to that presentation, they were still considering themselves a Yandex project, even the link in that tweet is https://clickhouse.yandex https://clickhouse.yandex . For those unaware, Yandex is a Russian internet megacorp, think Google but in cahoots with the authoritarian government: cherrypicked news coverage friendly to the government, effectively a monopoly across multiple verticals (eg they bought out Uber in Russia), etc. In 2020, the year that deck seems to be from, they were already censoring their news feed [2] and tweaking search ranking to promote pro-government results [3] for years. [1]: https://presentations.clickhouse.com/meetup40/introduction/ https://presentations.clickhouse.com/meetup40/introduction/ [2]: https://meduza.io/feature/2022/05/05/my-zamuchilis-borotsya https://meduza.io/feature/2022/05/05/my-zamuchilis-borotsya in Russian, but google translate does a reasonable job [3]: https://www.svoboda.org/a/30580605.html https://www.svoboda.org/a/30580605.html same
- ceejayoz 4y ago
  > think Google but in cahoots with the authoritarian government Sure, but the split appears have been very successful in this regard. https://clickhouse.com/blog/we-stand-with-ukraine https://clickhouse.com/blog/we-stand-with-ukraine
  - lambdadmitry 4y ago
    Yes, it seems the split was triggered by the war, but the censorship/promotion/collaboration with the government [0] were not worth breaking the ties. A cynical view would be that it's only now that the association became bad for business. [0]: By the way, Yandex was showing Crimea as unqualifiedly Russian territory since 2014, until the war, when they just removed borders between countries completely Edit: qualified the remark
    - kgeist 4y ago
      >By the way, Yandex was showing Crimea as unqualifiedly Russian territory since 2014 Google and Apple used to show it as part of Russia to Russian users as well. https://www.google.com/amp/s/techcrunch.com/2019/11/27/apple-and-google-maps-accommodate-russias-annexation-of-crimea/amp/ https://www.google.com/amp/s/techcrunch.com/2019/11/27/apple...
jcuenod 4y ago
Just wanted to chime in and say that I'm using a self-hosted instance of CH for a small hobby project and the performance is awesome.
singhrac 4y ago
I'm pretty interested in running Clickhouse, but to poll the room: is it easy to run without an infra person maintaining it? We don't need any kind of persistence guarantee or backups, happy to tear it down every day and refresh the data from sources, but does it have complicated config? How well does it work on say, a single core?
- klysm 4y ago
  If you don't really care about the data then I'd say it's probably pretty easy to run without an infra person right? If you hit a problem just nuke it and try again?
- fermuch 4y ago
  I've been running clickhouse for production as a solo dev for a year now and it's been great so far. I've had to tune it a bit for the machines I have, but everything works and it's incredibly fast. Not even comparable to postgres. For analytics, it goes way beyond what you'd expect, even under heavy writing. Even join-heavy queries work great.
- danielbln 4y ago
  Much, much simpler than a lot of dbs I've worked at. It's mostly fire and forget, easily doable without an infra person, imo.
- jcuenod 4y ago
  I've been running it in a container on docker swarm (on a celeron 3205U—2x1.5GHz cores). My CH container stores read only data, and I periodically update the snapshot, which involves deploying a newer image. This is very straightforward but, because it's read only, I don't even need a persistent volume. I've found the documentation to be pretty good and there's a really active telegram group where devs offer help (if you get noticed, but that's not too hard).
- IanCal 4y ago
  Another voice to say it's really easy on a single node. I ran it for a lot of work purely because of this.
- 4y ago
Jemaclus 4y ago
We are using Clickhouse Cloud at our company and the speed at which it serves up data is mind-blowing to our team, which had been using older systems for almost 10 years. Congrats on the public beta, and we can't wait to see what comes next!
vorillaz 4y ago
Happy ClickHouse user here. This is one amazing piece of software to be honest, for anyone ever wanted to parse, analyse and query billions of time series entry points ClickHouse is the way to go. The cloud offering seems like an amazing product for companies that they could afford I am not sure if the billing is right, but for 5M inserts per month the total bill would be $62K.
- jcims 4y ago
  I hate the part of my brain that has allowed the name to interfere with my interest in even looking at it.
- teacpde 4y ago
  That's quite high price tag per insert, do you have to write large amount of data per insert?
- IanCal 4y ago
  A write unit is not the same thing as a single insert, if for that cost you've multiplied it up.
- nemo44x 4y ago
  That can’t be right, that’s insane. I would find it easy to do 5m inserts in 30 minutes.
- zX41ZdbW 4y ago
  No, write unit is not a single INSERT. A single INSERT will take around 0.01 write units.
edf13 4y ago
Slightly aside… OLAP/Column dbs generally work well for large bulk inserts with many analytical queries… How do clickhouse (or other column dbs) work with larger inserts (e.g. log data type inserts)?
- sharms 4y ago
  Uber is using it for several PB of log data at scale: https://www.uber.com/en-IN/blog/logging/ https://www.uber.com/en-IN/blog/logging/
axlee 4y ago
Unsollicited feedback: I do not receive any validation email when creating an account, so I cannot log in. It is also impossible to reset the password.
- tbragin 4y ago
  So sorry about that! Could you send us an email at support@clickhouse.com and we'll get to the bottom of it?
risyachka 4y ago
Note: the final beneficiaries are russians and there is huge probability that taxes from clickhouse cloud will go to russian budget to sponsor war. Clickhouse inc is subsidiary of Clickhouse B.V in Netherlands and is controlled by a bunch of russians.
- hunterb123 4y ago
  Under that logic all of SV sponsors war in the middle east. Can we leave the russiaphobia in the political threads at least?
- DandyDev 4y ago
  You have left multiple comments now om this post with disparaging remarks about the Clickhouse team without any evidence to backup your claims. Not super classy if you ask me.
  - risyachka 4y ago
    Russians are bombing my city right this second, so sorry if I mentioned an issue with them more than a single time.
ian-whitestone 4y ago
Outside of being open source, how does ClickHouse differ from Snowflake/BigQuery? In what scenarios would I choose ClickHouse over those existing solutions?
- 62951413 4y ago
  Druid and Pinot are more likely to be the peer group (e.g. see https://leventov.medium.com/comparison-of-the-open-source-olap-systems-for-big-data-clickhouse-druid-and-pinot-8e042a5ed1c7 https://leventov.medium.com/comparison-of-the-open-source-ol...)
- zX41ZdbW 4y ago
  ClickHouse supports ad-hoc analytics, real-time reporting, and time series workloads at the same time. It is perfectly suited for user-facing analytics services. It supports low-latency (<100ms) queries for real-time analytics as well as high query throughput (500 QPS and more) - all of this with real-time data ingestion of logs, events, and time series. Take some notable examples from the list: https://clickhouse.com/docs/en/about-us/adopters/ https://clickhouse.com/docs/en/about-us/adopters/, something around web analytics, APM, ad networks, telecom data... ClickHouse is perfectly suited for these use cases. But if you try to align these scenarios with, say, BigQuery, they will become almost impossible or prohibitively expensive or just slow. There are specialized systems for real-time analytics like Druid and Pinot, but ClickHouse does it better: https://benchmark.clickhouse.com/ https://benchmark.clickhouse.com/ There are specialized systems for time-series workloads like InfluxDB and TimescaleDB, but ClickHouse does it better: https://gitlab.com/gitlab-org/incubation-engineering/apm/apm/-/issues/4 https://gitlab.com/gitlab-org/incubation-engineering/apm/apm... https://arxiv.org/pdf/2204.09795.pdf https://arxiv.org/pdf/2204.09795.pdf http://cds.cern.ch/record/2667383/ http://cds.cern.ch/record/2667383/ There are specialized systems for logs and APM, but ClickHouse does it better: https://blog.cloudflare.com/log-analytics-using-clickhouse/ https://blog.cloudflare.com/log-analytics-using-clickhouse/ There are specialized systems for ad-hoc analytics, but ClickHouse does it better as well: https://github.com/crottyan/mgbench https://github.com/crottyan/mgbench Well, even if you want to process a text file, ClickHouse will do it better than any other tool: https://github.com/dcmoura/spyql/blob/master/notebooks/json_benchmark.ipynb https://github.com/dcmoura/spyql/blob/master/notebooks/json_... And ClickHouse looks like a normal relational database - there is no need for multiple components for different tiers (like in Druid), no need for manual partitioning into "daily", "hourly" tables (like you do in Spark and Bigquery), no need for lambda architecture... It's refreshing how something can be both simple and fast.
wizwit999 4y ago
Clickhouse is great but the ops and scaling make it notoriously difficult to self host. If you have a lot of log data and want something open source and serverless you can self host, check out Matano (https://github.com/matanolabs/matano https://github.com/matanolabs/matano).
- deleted 4y ago
  [deleted]
- benjaminwootton 4y ago
  Clickhouse is known for being super easy to run and operate vs peers. A single binary and very simple configuration. The only acknowledged downside is that it does not have automatic rebalancing of data if new nodes are added.
  - wizwit999 4y ago
    Exactly, getting started is easy but dealing with scaling and operations when it's deployed in a production environment is not simple, primarily because it doesn't separate storage and compute.
    - qoega 4y ago
      I think ClickHouse Cloud Beta addressed exactly this concern. It separates storage and compute and deals with scaling. There is no sharding so you don't need to deal with scaling.
didip 4y ago
The 2 things that makes me hesitant about ClickHouse is that: 1. Data rebalancing is not automatic. 2. It doesn't really have a concept of cold-tier pushed to S3 directly, so cluster management is not simple for a small team. Other than that, ClickHouse looks super amazing.
- zX41ZdbW 4y ago
  ClickHouse can push cold data directly to S3, so S3 will be used as cold storage and the local filesystem as hot storage. Another approach is to store all in S3 with local caching, it is much more easy and somewhat more efficient. ClickHouse Cloud covers these concerns.
  - didip 4y ago
    Ah this must be a new feature? If you could share the docs related to this, I'd really appreciate it.
    - pandemicsyn 4y ago
      Not affiliated with Clickhouse, but fwiw they've been working on this for a while (I recall coming across it like +2 years ago in an old Altinity blog post): https://clickhouse.com/docs/en/integrations/s3/s3-merge-tree/#storage-tiers https://clickhouse.com/docs/en/integrations/s3/s3-merge-tree...
mrwnmonm 4y ago
People who uses CH inserting lots of JSON objects... stay back with this kind of pricing :)
loveapdb 4y ago
I am from selectdb company. selectdb is a cloud native data warehouse developed by the founding team of apache doris. Recently we submitted ClickBench's test results and we achieved good rankings. Later, we will incorporate all performance optimizations into Apache Doris and submit the test results of Apache Doris. Regardless of selectdb or doris, we not only have outstanding performance on large wide tables, but also perform well in scenarios such as tpc-h where there are many joins. https://en.selectdb.com/blog/SelectDB%20Topped%20ClickBench%20to%20Be%20No.1%20in%20The%20World https://en.selectdb.com/blog/SelectDB%20Topped%20ClickBench%...

278 points by taubek 4y ago

I have (an extremely boring, but quite hands-on) video about various ways of ClickHouse optimizations on top of external storage: https://www.youtube.com/watch?v=rK2BsaaaOCA https://www.youtube.com/watch?v=rK2BsaaaOCA (starting from around 40:00).

gsanderson 4y ago

Checking the pricing, $0.0125 per write unit. It says each write "generates one or more write units". So ... $1.25 per 100 writes? That can't be right. I wondered if it meant writes per second (like how AWS DynamoDB or Azure CosmosDB work, with their unit-based billing).

tbragin 4y ago
With an analytical database like ClickHouse, you can write many rows with a single INSERT statement (thousands of rows, millions of rows, and more). In fact, this kind of batching is recommended. Larger inserts will consume more write units than smaller inserts. Check out our billing FAQ for some examples, and we will be enhancing it with more detail as questions from our users come in (we'll work on clarifying this specific point): https://clickhouse.com/docs/en/manage/billing/ https://clickhouse.com/docs/en/manage/billing/ We also provide a $300 credit free trial to try out your workload and see how it translates to some of the usage dimensions. Finally, this is a Beta product, so keep the feedback coming!
- gsanderson 4y ago
  Thanks, I agree batching inserts would indeed be a good idea and it makes sense that's recommended. However that link you mention (as of now) does not specify what a write unit is. So if that could be clarified, that would be great. Since from your reply it sounds like one INSERT would indeed (at a minimum) incur one write unit. And thus 100 writes could indeed cost $1.25. Which could get expensive, fast.
  - tbragin 4y ago
    An INSERT can consume less than one write unit, depending on how many rows and bytes it writes to how many partitions, columns, materialized views, data types, etc. So, a "write unit", which corresponds to hundreds of low-level write operations, typically translates to many batched INSERTS. We are working to improve our examples in the FAQ to clarify - thank you so much for asking the question!
    - ok_dad 4y ago
      You know what would be really helpful? Some tooling around measuring "write units", like a fake local ClickHouseCloud API you could submit a write to and see how many "write units" that particular write would take. Of course, that would make pricing transparent and easy and SaaS companies don't seem like that. I challenge you folks to actually figure out a way to prevent surprise bills, until then all this "write unit" stuff is bullshit.
    - mike_d 4y ago
      > We are working to improve our examples in the FAQ to clarify Based on the comments here, and my own confusion, I think figuring out a different way of billing read/write operations is in order.
- mrwnmonm 4y ago
  "Each write operation (INSERT, DELETE, etc) consumes write units depending on the number of rows, columns, and partitions it writes to."
- benjaminwootton 4y ago
  How about if you are streaming in from Kafka and inserting each event as if arrives? Clickhouse is ideal for rapid analytics over event data so to introduce batching would be disappointing. Batch upload is of course more cost effective, but I would expect that to be more typical in a traditional data warehouse where we are uploading hourly or daily extracts. Clickhouse would likely show up in more real time and streaming architectures where events arrive sporadically. I am a huge fan and advocate of Clickhouse, but the concept of a write unit is strange and the likely charges if you have to insert small batches would make this unviable to use. A crypto analytics platform I built on top of Clickhouse would cost in the $millions per month vs $X00 or low $X000 with competitors or self hosting.
zX41ZdbW 4y ago
A single INSERT takes around 0.01 write units.

datalopers 4y ago

Clickhouse is an amazing product but this pricing looks excessive. A single node instance with a fast disk is more than sufficient for most needs: https://hub.docker.com/r/clickhouse/clickhouse-server https://hub.docker.com/r/clickhouse/clickhouse-server If you need a cluster, https://github.com/Altinity/clickhouse-operator https://github.com/Altinity/clickhouse-operator makes things easy

beoberha 4y ago
I’m sure there are tons of people very willing to pay for this if it means not having to run their own k8s clusters. Managed offerings like this are expensive, but worth every penny for some.
derefr 4y ago
We're a company that has been using the private beta of ClickHouse Cloud. IMHO, the unique value-prop of this offering is that it elastically scales reads (compute), like Redshift/Snowflake/BigQuery/other cloud data warehouses do, while also "being Clickhouse", and so giving you very traditional SQL querying capabilities (where those others all ask you to bend your SQL to fit the DB, sometimes in pretty ridiculous ways.) I would suggest not thinking of this offering as "ClickHouse, but in the cloud"; but rather, thinking of this as "a cloud data warehouse, but rather than using its own proprietary query engine, it's just ClickHouse." If you haven't evaluated other cloud data warehouses as alternatives to solving your problem (and found them wanting for one reason or another), then you're likely not in the niche that would see a positive profit margin on using ClickHouse Cloud.

thomoco 4y ago

I wanted to note that ClickHouse Cloud results are now also being reported in the public ClickBench results: https://benchmark.clickhouse.com/ https://benchmark.clickhouse.com/ Good to see transparent comparisons available now for Cloud performance vs. self-hosted or bare metal results as well as results from our peers. The ClickHouse team will continue to optimize further - as scale and performance is a relentless pursuit here at ClickHouse, and something we expect to be performed transparently and in a reproducible manner. Public benchmarking benefits all of us in the tech industry as we learn from each other in sharing the best techniques for attaining high performance within a cloud architecture Full disclosure: I do work for ClickHouse, although have also been a past member of SPEC in developing and advocating for public, standardized benchmarks

morelisp 4y ago
Can you clarify what a "write unit" is? Naively it sounds like it might be blocks x partitions x replicas that actually hit disk. (Which is also probably not very clear to people not already using CH, but I have at least middling knowledge of CH's i/o patterns and I have no clue what a "write unit" is from the page's description.)
- tylerhannan 4y ago
  It's Tyler from ClickHouse. Check out the response below that has a reference to some of our billing FAQs.
  - morelisp 4y ago
    Right, that link covers read units which is also what I expected - essentially the number of files I have to touch - but I still have no clue about write units. Is one block on one non-partitioned non-distributed table one write unit? What about one insert that's two blocks on such a table? What about one block on a null engine with two MVs listening to insert into two non-partitioned non-distributed tables? What if the table is a replacing mergetree, do I incur WUs for compactions? etc. My worry is that it is essentially 1 WU = 1 new part file, which I understand makes sense to bill on but is tremendously intransparent for users - at least I have no clue how often we roll new part files, instead I'm focused on total network and disk i/o performance on one side and client query latency on the other.

kawsper 4y ago

We're currently using InfluxDB for some timeseries metrics, but the 2.0 migration path have been terrible, even for simple examples, so we're looking for something else. Have anyone migrated from InfluxDB to ClickHouse?

monstrado 4y ago
Disclaimer: I work at ClickHouse At a previous company, I wrote a simple TCP server to receive LineProtocol, parse it and write to ClickHouse. I was absolutely blown away by how fast I could chart data in Grafana [1]. The compression was stellar, as well...I was able to store and chart years of history data. We basically just stopped sending data to Influx and migrated everything over to the ClickHouse backend. [1] https://grafana.com/grafana/plugins/grafana-clickhouse-datasource/ https://grafana.com/grafana/plugins/grafana-clickhouse-datas...
danielvf 4y ago
Clickhouse is so stupidly, mindbogglingly fast at what it does that you can often replace a much more specialized databases or specialized schemas / queries with brute force Clickhouse and come out far ahead. It really depends on the size of what you are doing. I've used Clickhouse for timeseries data in the tens of millions events per day and it worked well.

base 4y ago

Is there an easy way to have ClickHouse Cloud ingest data from MySQL hosted in Amazon RDS?

thomoco 4y ago
There are a few options for migrating or synchronizing data from MySQL - I'd recommend starting with this page in the ClickHouse Docs - there is a nice video there that explains some of those options: https://clickhouse.com/docs/en/integrations/migration/ https://clickhouse.com/docs/en/integrations/migration/ Depending on what you are trying to achieve, you could use clickhouse-local with the MySQL engine to move data, or could use an ETL/ETL tool to migrate/sync
- hodgesrm 4y ago
  Good point. If you just want to copy data far and away the easiest way to transfer data is using MySQLDatabaseEngine. You can even copy table definitions. Watch for issues with Decimal datatypes if you do this.
  - morelisp 4y ago
    Be careful with this engine, it's easy to accidentally expose the password as you only need table read permissions if it wasn't set up using an external credential file. https://github.com/ClickHouse/ClickHouse/issues/3311 https://github.com/ClickHouse/ClickHouse/issues/3311 I also had some pretty bad join performance (CH table joined to MySQL table), the quick solution to both of these is that we instead use the table function (https://clickhouse.com/docs/en/sql-reference/table-functions/mysql/ https://clickhouse.com/docs/en/sql-reference/table-functions...) to copy the data periodically.
    - hodgesrm 4y ago
      Use Named Collections to protect credentials. They are very handy. Here's an article that discusses use in the JDBC Bridge. https://altinity.com/blog/connecting-clickhouse-to-external-data-sources-using-the-jdbc-bridge https://altinity.com/blog/connecting-clickhouse-to-external-... Disclaime: I work for Altinity.

chrisballinger 4y ago

At first I was a bit confused about why The Onion’s spin-off site had a cloud offering, but that one is called ClickHole.

skadamat 4y ago

For context, there are multiple Clickhouse companies. This is the one that spun out of Yandex and took the Clickhouse name as their namesake for the company! Altinity has been in the space for a while offering Clickhouse services as well: https://altinity.com/ https://altinity.com/ EDIT: brazenly was the wrong word here :)

stingraycharles 4y ago
Wait, wasn't Clickhouse itself spun out of Yandex anyway? I.e. Yandex opensources Clickhouse, then later starts Clickhouse (the company)?
- kelp 4y ago
  That is literally what happened. There is nothing brazen here.
qaq 4y ago
"brazenly" :)? Are you for real? Top 3 commiters work there including Alexey who created CH in the first place alexey-milovidov 17,522 commits 5,648,537 ++ 5,580,633 -- alesapin 4,618 commits 1,024,262 ++ 932,501 -- KochetovNicolai 3,867 commits 377,420 ++ 314,035 --
hodgesrm 4y ago
Hi! Thank you for mentioning my company Altinity. We love ClickHouse and have been supporting customers on it for years. That said I would like to defend ClickHouse and their use of the logo. They acquired the IP and it's their right to use it as they please. It's hard to imagine them not using the ClickHouse name. There's a crowd of companies using/supporting ClickHouse so it's the obvious way to market themselves. The interesting question is whether it's a good thing for users to have a large company controlling ClickHouse rather than having it in a foundation.
- mrwnmonm 4y ago
  Will Altinity Cloud compete with ClickHouse Cloud now? :)

epberry 4y ago

Great stuff! I know the ClickHouse team and they are world class. If you're more familiar with a relational database some things will feel weird. For example, you should not insert 1 row at a time, there's over 1,000 built-in functions, and the default connection is often over HTTPS, but not 443.

zX41ZdbW 4y ago
ClickHouse Cloud provides port 443 as well as 8443. You can insert one row at a time, it's perfectly ok with "async_insert=1".

encoderer 4y ago

I’m a little sad to see them embrace magic “insert unit” pricing, instead of taking the approach Altinity uses where you are renting an instance size that you can compare apples-to-apples with running your own cluster on ec2.

aseipp 4y ago
Honestly that's something they can probably offer separately, but I really prefer this pricing for most use cases where I want a database that is always available but has bursty request/response patterns. This means I can have an analytical database available for all my small services, websites, etc without having to think too much about availability, support, and a constant price overhead. But ClickHouse is so fast you can get pretty far with a $10 VPS, I admit. Probably the best comparison is CockroachDB Cloud. They have a "serverless" offering based on unit pricing and a dedicated offering based on provisioned servers + support/maintenance overhead. I think that would be the ideal place to go long-term, but I'm super excited for this current one. I love ClickHouse and want to support them. ClickHouse is also an interesting case because there's lots of options to migrate clusters, use S3 as long-term storage, etc to where I don't particularly feel locked into this offering if I ever wanted to shift into my own.
izrailev 4y ago
Serverless "pay for usage" is different than the fixed-size dedicated cluster pricing model, and can be quite a bit cheaper, especially with spiky query traffic. It should also be more reliable, since you don't have to predict and provision capacity for your peak usage ahead of time. Disclaimer: I work for ClickHouse.
- encoderer 4y ago
  Ok but it can also be quite a bit more expensive and your bill is less predictable. Disclaimer: I’m a customer of altinity.
  - izrailev 4y ago
    [dead]

datalopers 4y ago

Is this fully hosted or runs in my cloud? Is this running https://github.com/ClickHouse/ClickHouse https://github.com/ClickHouse/ClickHouse or a closed-source Clickhouse Cloud variant with added features?

kbiyer 4y ago
It is a fully hosted offering and is based on ClickHouse core v22.10
- lsllc 4y ago
  Curious how CH are ultimately actually hosting this, on a public cloud or is this their own cloud.
  - morelisp 4y ago
    Billing page says AWS for storage, so I'd assume the same for compute. If they're really doing elastic reads as suggested in another comment I don't think this can be the standard CH server (at least for reads) - or I've missed something very exciting in recent versions.
    - datalopers 4y ago
      It feels like there's gotta be something proprietary beyond just using a s3 disk setup but maybe not? is simply having data_cache_enabled and cache_path pointing to local nvme ssd sufficient to achieve similar speeds? * https://clickhouse.com/docs/en/guides/sre/configuring-s3-for-clickhouse-use/ https://clickhouse.com/docs/en/guides/sre/configuring-s3-for... * https://clickhouse.com/docs/en/integrations/s3/s3-merge-tree/ https://clickhouse.com/docs/en/integrations/s3/s3-merge-tree...
      - morelisp 4y ago
        For non-elastic writes this might be sufficient (I have no clue as to performance of AWS S3 for this kind of use case) - but supposedly[0] there's some elastic read capacity. As far as I know that's not possible with standard CH at all, and may be where some secret sauce is. I can vaguely imagine how you'd build such a thing, you'd spin up new instances that knows your write-primary's sharding/partitioning scheme - heck you could even approximate it with slightly-augmented clickhouse-clients doing the S3 pulls directly and no real additional "server" - but as far as I know there's no way out of the box. [0] https://news.ycombinator.com/item?id=33081658 https://news.ycombinator.com/item?id=33081658

MentallyRetired 4y ago

Some unsolicited feedback: 1) I had no idea what Clickhouse was for the first 30 seconds looking at the homepage. I now understand it to be a database of some sort. I shouldn't have seen the words "performance" and "cloud" and "serverless" before seeing the word database, right? I'm starting off confused. There shouldn't be an assumption that I know what you all do. 2) I have no idea what a column oriented database is. I've been a developer for 29 years (mostly frontend but I do a lot of full stack too). If I need an explainer, a lot of devs will. Aside from that, it looks like a nice offering and I wish you all the best!

cinbun8 4y ago
If you don't know what a column oriented DB is, that page is probably not for you.
- bagels 4y ago
  I know what a column oriented DB is, but ClickHouse was not on my radar before. The pitch on the landing page is that ClickHouse Cloud is great if you love ClickHouse. If you don't know what ClickHouse is, you have to do some work to find out.
- pqdbr 4y ago
  Care to elaborate? I’m a fullstack Rails dev for 12 years and I had no idea either, just like OP. Why alienate potential users from the get go?
- ipaddr 4y ago
  Isn't it just a database in Second normal form (2NF)?
  - bfung 4y ago
    No. Column oriented storage db’s make it fast to read 10s to billions of “rows”, usually when you have data that you want to read that can be independent of other columns. Ex: (stock close) - row storage isn’t going to buy you much here, you’d rather read the whole “stock close” column as fast as possible. Whereas traditional db’s, data like [first name, last name], the columns may have way less meaning on their own and you need both columns to have the data make sense. A traditional db with B(-)tree storage is much slower when dealing with that type of usage. Storing (stock close) in a single column format makes that type of query much faster.

andrewmutz 4y ago

Clickhouse was spun out of Yandex, which is a Russian corporation. Given existing geopolitical tensions, is there anything to worry about there? Does anyone know how much of the Clickhouse team (or ownership) is still located in Russia?

46Bit 4y ago
They have been extremely clear in their support of Ukraine https://clickhouse.com/blog/we-stand-with-ukraine https://clickhouse.com/blog/we-stand-with-ukraine
- risyachka 4y ago
  It's just a bunch of text, doesn't show any support whatsoever. They can show support by donating to UA defence and showing proof (important - not some neutral org). Otherwise it is not support but a bunch of bs
  - ericb 4y ago
    In places like Russia, those words are very dangerous, and could get you jailed or sent to the front. Even calling it a "war" was/is a punishable offense. So, yes, words, but the potential consequences of these words have more significance than empty air.
  - xwowsersx 4y ago
    Why do you get to define the acceptable forms of support...?
kelp 4y ago
You can see on their jobs page and 'our story' page, they are mostly in the US and The Netherlands.
- mike_d 4y ago
  Yandex is technically incorporated in The Netherlands and has an engineering presence there, but they are as Russian controlled as can be.

lambdadmitry 4y ago

It's interesting how diligently they wiped our any and all mentions of Yandex from their website, to the extent that Google struggles to find meaningful mentions apart from a few exceptions like [1]. Quite peculiar considering that just two years ago, according to that presentation, they were still considering themselves a Yandex project, even the link in that tweet is https://clickhouse.yandex https://clickhouse.yandex . For those unaware, Yandex is a Russian internet megacorp, think Google but in cahoots with the authoritarian government: cherrypicked news coverage friendly to the government, effectively a monopoly across multiple verticals (eg they bought out Uber in Russia), etc. In 2020, the year that deck seems to be from, they were already censoring their news feed [2] and tweaking search ranking to promote pro-government results [3] for years. [1]: https://presentations.clickhouse.com/meetup40/introduction/ https://presentations.clickhouse.com/meetup40/introduction/ [2]: https://meduza.io/feature/2022/05/05/my-zamuchilis-borotsya https://meduza.io/feature/2022/05/05/my-zamuchilis-borotsya in Russian, but google translate does a reasonable job [3]: https://www.svoboda.org/a/30580605.html https://www.svoboda.org/a/30580605.html same

ceejayoz 4y ago
> think Google but in cahoots with the authoritarian government Sure, but the split appears have been very successful in this regard. https://clickhouse.com/blog/we-stand-with-ukraine https://clickhouse.com/blog/we-stand-with-ukraine
- lambdadmitry 4y ago
  Yes, it seems the split was triggered by the war, but the censorship/promotion/collaboration with the government [0] were not worth breaking the ties. A cynical view would be that it's only now that the association became bad for business. [0]: By the way, Yandex was showing Crimea as unqualifiedly Russian territory since 2014, until the war, when they just removed borders between countries completely Edit: qualified the remark
  - kgeist 4y ago
    >By the way, Yandex was showing Crimea as unqualifiedly Russian territory since 2014 Google and Apple used to show it as part of Russia to Russian users as well. https://www.google.com/amp/s/techcrunch.com/2019/11/27/apple-and-google-maps-accommodate-russias-annexation-of-crimea/amp/ https://www.google.com/amp/s/techcrunch.com/2019/11/27/apple...

jcuenod 4y ago

Just wanted to chime in and say that I'm using a self-hosted instance of CH for a small hobby project and the performance is awesome.

singhrac 4y ago

I'm pretty interested in running Clickhouse, but to poll the room: is it easy to run without an infra person maintaining it? We don't need any kind of persistence guarantee or backups, happy to tear it down every day and refresh the data from sources, but does it have complicated config? How well does it work on say, a single core?

klysm 4y ago
If you don't really care about the data then I'd say it's probably pretty easy to run without an infra person right? If you hit a problem just nuke it and try again?
fermuch 4y ago
I've been running clickhouse for production as a solo dev for a year now and it's been great so far. I've had to tune it a bit for the machines I have, but everything works and it's incredibly fast. Not even comparable to postgres. For analytics, it goes way beyond what you'd expect, even under heavy writing. Even join-heavy queries work great.
danielbln 4y ago
Much, much simpler than a lot of dbs I've worked at. It's mostly fire and forget, easily doable without an infra person, imo.
jcuenod 4y ago
I've been running it in a container on docker swarm (on a celeron 3205U—2x1.5GHz cores). My CH container stores read only data, and I periodically update the snapshot, which involves deploying a newer image. This is very straightforward but, because it's read only, I don't even need a persistent volume. I've found the documentation to be pretty good and there's a really active telegram group where devs offer help (if you get noticed, but that's not too hard).
IanCal 4y ago
Another voice to say it's really easy on a single node. I ran it for a lot of work purely because of this.
4y ago

Jemaclus 4y ago

We are using Clickhouse Cloud at our company and the speed at which it serves up data is mind-blowing to our team, which had been using older systems for almost 10 years. Congrats on the public beta, and we can't wait to see what comes next!

vorillaz 4y ago

Happy ClickHouse user here. This is one amazing piece of software to be honest, for anyone ever wanted to parse, analyse and query billions of time series entry points ClickHouse is the way to go. The cloud offering seems like an amazing product for companies that they could afford I am not sure if the billing is right, but for 5M inserts per month the total bill would be $62K.

jcims 4y ago
I hate the part of my brain that has allowed the name to interfere with my interest in even looking at it.
teacpde 4y ago
That's quite high price tag per insert, do you have to write large amount of data per insert?
IanCal 4y ago
A write unit is not the same thing as a single insert, if for that cost you've multiplied it up.
nemo44x 4y ago
That can’t be right, that’s insane. I would find it easy to do 5m inserts in 30 minutes.
zX41ZdbW 4y ago
No, write unit is not a single INSERT. A single INSERT will take around 0.01 write units.

edf13 4y ago

Slightly aside… OLAP/Column dbs generally work well for large bulk inserts with many analytical queries… How do clickhouse (or other column dbs) work with larger inserts (e.g. log data type inserts)?

sharms 4y ago
Uber is using it for several PB of log data at scale: https://www.uber.com/en-IN/blog/logging/ https://www.uber.com/en-IN/blog/logging/

axlee 4y ago

Unsollicited feedback: I do not receive any validation email when creating an account, so I cannot log in. It is also impossible to reset the password.

tbragin 4y ago
So sorry about that! Could you send us an email at support@clickhouse.com and we'll get to the bottom of it?

risyachka 4y ago

Note: the final beneficiaries are russians and there is huge probability that taxes from clickhouse cloud will go to russian budget to sponsor war. Clickhouse inc is subsidiary of Clickhouse B.V in Netherlands and is controlled by a bunch of russians.

hunterb123 4y ago
Under that logic all of SV sponsors war in the middle east. Can we leave the russiaphobia in the political threads at least?
DandyDev 4y ago
You have left multiple comments now om this post with disparaging remarks about the Clickhouse team without any evidence to backup your claims. Not super classy if you ask me.
- risyachka 4y ago
  Russians are bombing my city right this second, so sorry if I mentioned an issue with them more than a single time.

ian-whitestone 4y ago

Outside of being open source, how does ClickHouse differ from Snowflake/BigQuery? In what scenarios would I choose ClickHouse over those existing solutions?

62951413 4y ago
Druid and Pinot are more likely to be the peer group (e.g. see https://leventov.medium.com/comparison-of-the-open-source-olap-systems-for-big-data-clickhouse-druid-and-pinot-8e042a5ed1c7 https://leventov.medium.com/comparison-of-the-open-source-ol...)
zX41ZdbW 4y ago
ClickHouse supports ad-hoc analytics, real-time reporting, and time series workloads at the same time. It is perfectly suited for user-facing analytics services. It supports low-latency (<100ms) queries for real-time analytics as well as high query throughput (500 QPS and more) - all of this with real-time data ingestion of logs, events, and time series. Take some notable examples from the list: https://clickhouse.com/docs/en/about-us/adopters/ https://clickhouse.com/docs/en/about-us/adopters/, something around web analytics, APM, ad networks, telecom data... ClickHouse is perfectly suited for these use cases. But if you try to align these scenarios with, say, BigQuery, they will become almost impossible or prohibitively expensive or just slow. There are specialized systems for real-time analytics like Druid and Pinot, but ClickHouse does it better: https://benchmark.clickhouse.com/ https://benchmark.clickhouse.com/ There are specialized systems for time-series workloads like InfluxDB and TimescaleDB, but ClickHouse does it better: https://gitlab.com/gitlab-org/incubation-engineering/apm/apm/-/issues/4 https://gitlab.com/gitlab-org/incubation-engineering/apm/apm... https://arxiv.org/pdf/2204.09795.pdf https://arxiv.org/pdf/2204.09795.pdf http://cds.cern.ch/record/2667383/ http://cds.cern.ch/record/2667383/ There are specialized systems for logs and APM, but ClickHouse does it better: https://blog.cloudflare.com/log-analytics-using-clickhouse/ https://blog.cloudflare.com/log-analytics-using-clickhouse/ There are specialized systems for ad-hoc analytics, but ClickHouse does it better as well: https://github.com/crottyan/mgbench https://github.com/crottyan/mgbench Well, even if you want to process a text file, ClickHouse will do it better than any other tool: https://github.com/dcmoura/spyql/blob/master/notebooks/json_benchmark.ipynb https://github.com/dcmoura/spyql/blob/master/notebooks/json_... And ClickHouse looks like a normal relational database - there is no need for multiple components for different tiers (like in Druid), no need for manual partitioning into "daily", "hourly" tables (like you do in Spark and Bigquery), no need for lambda architecture... It's refreshing how something can be both simple and fast.

wizwit999 4y ago

Clickhouse is great but the ops and scaling make it notoriously difficult to self host. If you have a lot of log data and want something open source and serverless you can self host, check out Matano (https://github.com/matanolabs/matano https://github.com/matanolabs/matano).

deleted 4y ago
[deleted]
benjaminwootton 4y ago
Clickhouse is known for being super easy to run and operate vs peers. A single binary and very simple configuration. The only acknowledged downside is that it does not have automatic rebalancing of data if new nodes are added.
- wizwit999 4y ago
  Exactly, getting started is easy but dealing with scaling and operations when it's deployed in a production environment is not simple, primarily because it doesn't separate storage and compute.
  - qoega 4y ago
    I think ClickHouse Cloud Beta addressed exactly this concern. It separates storage and compute and deals with scaling. There is no sharding so you don't need to deal with scaling.

didip 4y ago

The 2 things that makes me hesitant about ClickHouse is that: 1. Data rebalancing is not automatic. 2. It doesn't really have a concept of cold-tier pushed to S3 directly, so cluster management is not simple for a small team. Other than that, ClickHouse looks super amazing.

zX41ZdbW 4y ago
ClickHouse can push cold data directly to S3, so S3 will be used as cold storage and the local filesystem as hot storage. Another approach is to store all in S3 with local caching, it is much more easy and somewhat more efficient. ClickHouse Cloud covers these concerns.
- didip 4y ago
  Ah this must be a new feature? If you could share the docs related to this, I'd really appreciate it.
  - pandemicsyn 4y ago
    Not affiliated with Clickhouse, but fwiw they've been working on this for a while (I recall coming across it like +2 years ago in an old Altinity blog post): https://clickhouse.com/docs/en/integrations/s3/s3-merge-tree/#storage-tiers https://clickhouse.com/docs/en/integrations/s3/s3-merge-tree...

mrwnmonm 4y ago

People who uses CH inserting lots of JSON objects... stay back with this kind of pricing :)

loveapdb 4y ago

I am from selectdb company. selectdb is a cloud native data warehouse developed by the founding team of apache doris. Recently we submitted ClickBench's test results and we achieved good rankings. Later, we will incorporate all performance optimizations into Apache Doris and submit the test results of Apache Doris. Regardless of selectdb or doris, we not only have outstanding performance on large wide tables, but also perform well in scenarios such as tpc-h where there are many joins. https://en.selectdb.com/blog/SelectDB%20Topped%20ClickBench%20to%20Be%20No.1%20in%20The%20World https://en.selectdb.com/blog/SelectDB%20Topped%20ClickBench%...