4 ms·

These formats look like an attempt to get a halfway solution: you want to get something like a real MPP analytic DBMS (e.g., ClickHouse) but have to use a data

These formats look like an attempt to get a halfway solution: you want to get something like a real MPP analytic DBMS (e.g., ClickHouse) but have to use a data lake for some reason. It resembles previous trendy technologies that are mostly forgotten now, such as: - Lambda architecture (based on a wrong assumption that you cannot have a real-time and historical layers in the same system); - Multidimensional OLAP (based on a wrong assumption that you cannot do analytic queries directly on non-aggregated data); - Big data (based on a wrong assumption that map-reduce is better than relational DBMS). I'm exaggerating a little. Disclaimer: I work on ClickHouse, and I'm a follower of every technology in the data processing area.

vgt 4y ago
It would be good if you labeled your posts so as to reveal your bias. I understand why folks want options. At the end of the day, folks want an easy to use, ALWAYS CORRECT stable database, with minimal well-documented predictable knobs, correct distributed execution plan, no OOMs, separation of storage and compute, and standard SQL, and Clickhouse struggles with all of the above. (co-founder of MotherDuck)
- thomoco 4y ago
  Could you please elaborate on your comments and possible misconceptions about ClickHouse? Proven stability, massive scale, predictability, native SQL, and industry-best performance are all well-recognized characteristics of clickhouse, so your comments here seem a bit biased. I am interested to learn more about your point of view, as well as tangentially the strategic vision of MotherDuck as a company. (VP Support at ClickHouse)
  - vgt 4y ago
    Speaking from nearly a decade working on BigQuery, and a year working at Firebolt. - Stability. It OOMS, your CTO mentioned that last week. - It is not correct. I believe your team is aware of cases in which your very own benchmarks revealed Clickhouse to be incorrect. - Scale. The distributed plan is broken and I'm not sure Clickhouse even has shuffle. - SQL. It is very non-standard. - Knobs. Lots of knobs that are poorly documented. It's unclear which are mandatory. You have to restart for most. Don't get me wrong, I love open source, and I love what Clickhouse has done. I am not a fan of overselling. There are problems with Clickhouse. Trying to sell it as a superset of the modern CDW is not doing users any favors.
    - monstrado 4y ago
      As an engineer who admires the work done by DuckDB, I'm disappointed that the co-founder of its evolution is spreading FUD about competitors before its even in the competitive conversation. > Stability. It OOMS, your CTO mentioned that last week. I ran ClickHouse clusters for years with zero stability issues (even as a beginner at the time) at an extremely large volume video game studio with real-time needs. Using online materialized views, I was able to construct rollups of vital KPIs at millisecond level while maintaining multi-thousand QPS. Stability was never a concern of ours, and quite frankly, we were kind of blown away. > Scale. The distributed plan is broken and I'm not sure Clickhouse even has shuffle. First, I hate the word "broken" with zero explanation what you mean by this. Based on your language, I'm assuming you're just suggesting the distributed plans aren't as efficient as possible, a limitation that the engineers are not shy to admit. > SQL. It is very non-standard. I would argue the language is more a superset than "non-standard". Most everything for us just worked, and often I found areas of SQL that I could reduce significantly due to the "non-standard" extras they've added. For example: Did you know they have built-in aggregate functions for computing retention?! > Knobs. Lots of knobs that are poorly documented. It's unclear which are mandatory. You have to restart for most. Yes, there are a lot of knobs. ClickHouse works wonderfully out of the box with the default knobs, but you're free to tinker because that's how flexible the technology is. You worked at Google for over a decade? You should know. Google's tech is notorious for having a TON of knobs for their internal technology (e.g. BigTable). Just because the knobs are there doesn't mean they must be tuned, it just means the engineers thought ahead. Also, the vast majority of configuration changes I've made never required a restart...I'm not even sure why you pointed this out. (Disclaimer: I have been using ClickHouse successfully for several years)
    - datatrashfire 4y ago
      Funny noticing this same post pattern by Mother Duck in other threads. It's not a good look.
      - glogla 4y ago
        It is quite surreal that in addition to the usual Databricks astroturfing and various fans arguing, we now also have founders and VPs of companies arguing about who is better on HN.
- deleted 4y ago
  [deleted]
- zX41ZdbW 4y ago
  We are also interested in the comparison of Hudi, Delta Lake, and Iceberg. We have recently added support for Hudi and Delta Lake; you can check here: https://clickhouse.com/docs/en/engines/table-engines/integrations/deltalake https://clickhouse.com/docs/en/engines/table-engines/integra... It is a read-only implementation: ClickHouse can read and process the external data in the Hudi or Delta Lake format. Apache Iceberg is pending. There is no good C++ library for it. But at least the overall structure is simple, as it is not hard to implement it. The overall principle - whatever data format it is, ClickHouse should support it in a fast, stable, and ALWAYS CORRECT manner. If you have more experience to share, please do it.
  - glogla 4y ago
    > We have recently added support for Hudi and Delta Lake; you can check here: https://clickhouse.com/docs/en/engines/table-engines/integra https://clickhouse.com/docs/en/engines/table-engines/integra... Just tried it out. It seems to only support partitioned Delta tables, non-partitioned return CANNOT_EXTRACT_TABLE_STRUCTURE. Is that on purpose or is it a bug?
glogla 4y ago
The formats are kind of a halfway solution, because trying to build something with MPP semantics on objects stores is difficult. The difference between MPP and something like Databricks or Trino working with object store is that while MPP can likely get much better performance and especially latency from the same hardware, operating it is much harder. You don't "backup" Databricks - the data is stored in object storage and that is it. You don't have to plan storage sizing quarters upfront, and you never get in trouble because there is unexpected data spike. Compute resizes are trivial, there is no rebalancing. Upgrades are easy, because you're just upgrading the compute and you can't break data that way. You can give each user group (like batch one and interactive one, or each team) their dedicated compute over common data and it works. That compute can spin up and down and autoscale to save some money. You don't have to think about how to replicate my table across a cluster or anything. And so on, and so forth. Running a big data and analytics platform - place where tens of teams, tens of applications and hundreds or thousands of analysts come for data and where they build their solutions - is already enough of a challenge without all this operations work, and that is why Snowflake and Databricks are worth that crazy money. If someone could solve the challenge of having MPP that is as easy to manage as Snowflake or a Lakehouse, that would be quite the differentiator. And maybe you people already did and I just didn't notice, I don't know :)

vgt 4y ago

It would be good if you labeled your posts so as to reveal your bias. I understand why folks want options. At the end of the day, folks want an easy to use, ALWAYS CORRECT stable database, with minimal well-documented predictable knobs, correct distributed execution plan, no OOMs, separation of storage and compute, and standard SQL, and Clickhouse struggles with all of the above. (co-founder of MotherDuck)

thomoco 4y ago
Could you please elaborate on your comments and possible misconceptions about ClickHouse? Proven stability, massive scale, predictability, native SQL, and industry-best performance are all well-recognized characteristics of clickhouse, so your comments here seem a bit biased. I am interested to learn more about your point of view, as well as tangentially the strategic vision of MotherDuck as a company. (VP Support at ClickHouse)
- vgt 4y ago
  Speaking from nearly a decade working on BigQuery, and a year working at Firebolt. - Stability. It OOMS, your CTO mentioned that last week. - It is not correct. I believe your team is aware of cases in which your very own benchmarks revealed Clickhouse to be incorrect. - Scale. The distributed plan is broken and I'm not sure Clickhouse even has shuffle. - SQL. It is very non-standard. - Knobs. Lots of knobs that are poorly documented. It's unclear which are mandatory. You have to restart for most. Don't get me wrong, I love open source, and I love what Clickhouse has done. I am not a fan of overselling. There are problems with Clickhouse. Trying to sell it as a superset of the modern CDW is not doing users any favors.
  - monstrado 4y ago
    As an engineer who admires the work done by DuckDB, I'm disappointed that the co-founder of its evolution is spreading FUD about competitors before its even in the competitive conversation. > Stability. It OOMS, your CTO mentioned that last week. I ran ClickHouse clusters for years with zero stability issues (even as a beginner at the time) at an extremely large volume video game studio with real-time needs. Using online materialized views, I was able to construct rollups of vital KPIs at millisecond level while maintaining multi-thousand QPS. Stability was never a concern of ours, and quite frankly, we were kind of blown away. > Scale. The distributed plan is broken and I'm not sure Clickhouse even has shuffle. First, I hate the word "broken" with zero explanation what you mean by this. Based on your language, I'm assuming you're just suggesting the distributed plans aren't as efficient as possible, a limitation that the engineers are not shy to admit. > SQL. It is very non-standard. I would argue the language is more a superset than "non-standard". Most everything for us just worked, and often I found areas of SQL that I could reduce significantly due to the "non-standard" extras they've added. For example: Did you know they have built-in aggregate functions for computing retention?! > Knobs. Lots of knobs that are poorly documented. It's unclear which are mandatory. You have to restart for most. Yes, there are a lot of knobs. ClickHouse works wonderfully out of the box with the default knobs, but you're free to tinker because that's how flexible the technology is. You worked at Google for over a decade? You should know. Google's tech is notorious for having a TON of knobs for their internal technology (e.g. BigTable). Just because the knobs are there doesn't mean they must be tuned, it just means the engineers thought ahead. Also, the vast majority of configuration changes I've made never required a restart...I'm not even sure why you pointed this out. (Disclaimer: I have been using ClickHouse successfully for several years)
  - datatrashfire 4y ago
    Funny noticing this same post pattern by Mother Duck in other threads. It's not a good look.
    - glogla 4y ago
      It is quite surreal that in addition to the usual Databricks astroturfing and various fans arguing, we now also have founders and VPs of companies arguing about who is better on HN.
deleted 4y ago
[deleted]
zX41ZdbW 4y ago
We are also interested in the comparison of Hudi, Delta Lake, and Iceberg. We have recently added support for Hudi and Delta Lake; you can check here: https://clickhouse.com/docs/en/engines/table-engines/integrations/deltalake https://clickhouse.com/docs/en/engines/table-engines/integra... It is a read-only implementation: ClickHouse can read and process the external data in the Hudi or Delta Lake format. Apache Iceberg is pending. There is no good C++ library for it. But at least the overall structure is simple, as it is not hard to implement it. The overall principle - whatever data format it is, ClickHouse should support it in a fast, stable, and ALWAYS CORRECT manner. If you have more experience to share, please do it.
- glogla 4y ago
  > We have recently added support for Hudi and Delta Lake; you can check here: https://clickhouse.com/docs/en/engines/table-engines/integra https://clickhouse.com/docs/en/engines/table-engines/integra... Just tried it out. It seems to only support partitioned Delta tables, non-partitioned return CANNOT_EXTRACT_TABLE_STRUCTURE. Is that on purpose or is it a bug?

glogla 4y ago

The formats are kind of a halfway solution, because trying to build something with MPP semantics on objects stores is difficult. The difference between MPP and something like Databricks or Trino working with object store is that while MPP can likely get much better performance and especially latency from the same hardware, operating it is much harder. You don't "backup" Databricks - the data is stored in object storage and that is it. You don't have to plan storage sizing quarters upfront, and you never get in trouble because there is unexpected data spike. Compute resizes are trivial, there is no rebalancing. Upgrades are easy, because you're just upgrading the compute and you can't break data that way. You can give each user group (like batch one and interactive one, or each team) their dedicated compute over common data and it works. That compute can spin up and down and autoscale to save some money. You don't have to think about how to replicate my table across a cluster or anything. And so on, and so forth. Running a big data and analytics platform - place where tens of teams, tens of applications and hundreds or thousands of analysts come for data and where they build their solutions - is already enough of a challenge without all this operations work, and that is why Snowflake and Databricks are worth that crazy money. If someone could solve the challenge of having MPP that is as easy to manage as Snowflake or a Lakehouse, that would be quite the differentiator. And maybe you people already did and I just didn't notice, I don't know :)