6 ms·

Databricks broke the record by 2x) and is 10x more cost effective, in an audited benchmark. Snowflake should participate in the official, audited benchmark. Cus

by maslam 5y ago

Databricks broke the record by 2x) and is 10x more cost effective, in an audited benchmark. Snowflake should participate in the official, audited benchmark. Customers win when businesses are open and transparent…

jiggawatts 5y ago
Audited how? If you look at the Snowflake response the numbers being posted by Databricks look outright faked or otherwise false.
- maslam 5y ago
  Hey jiggawatts - TPC is the official way to audit benchmarks in the database industry. They’ve been around for a bit, but let me know if you want more info, I’m happy to share more about them.
  - lmeyerov 5y ago
    It sounds fundamentally busted if a competitor can submit benchmarks for someone else. TPC is great in general, but I didn't realize it had such a gaping flaw. TPC submissions take real time/$/energy/expertise, so I don't know anyone who has ever done it casually. Ex: It was a multi-company effort for the RAPIDS community to get enough API coverage & edge case optimization for an end-to-end GPU submission on the big data one (SQL, ...), and even there the TPC folks made them resubmit if I remember right. Also, note how the parent's response did not actually answer 'audited how'. Pushing the work to the questioner is on the shortlist of techniques studied by misinformation researchers. I'm a fan of both companies, so disappointing to see from a company rep.
    - rxin 5y ago
      Check my reply, Leo.
      - lmeyerov 5y ago
        The audit question is on Databricks marketing unaudited Snowflake TPC numbers. I do think Snowflake is big enough to run TPC, but how you guys choose to market is on you. But: I think it's cool both companies got it to $200-300. Way better than years ago. Next stop: GPUs :)
        rxin 5y ago
        Ah ok. Wasn't clear. I think some repro scripts will be available soon.
      - deleted 5y ago
        [deleted]
  - redis_mlc 5y ago
    > TPC is the official way to audit benchmarks in the database industry. TPC is a benchmark suite for a certain problem class. It says nothing about how the databases are configured or managed. In case you're thick, the above is a polite way of calling you a liar. This is why Oracle and other database vendors don't allow publishing of benchmarks. It's to protect them from incompetent or lazy authors primarily. Source: DBA.
- Spivak 5y ago
  The results are so crazy different that either Snowflake or Databricks are wrong or outright lying.
  - jiggawatts 5y ago
    This is my point also, and I'm being downvoted for it. If two people are in disagreement about the same facts, then one of them is either misinformed or lying. It's that simple. If the only recourse seems to be to sink to the level of mud-slinging, with no clear ability to point to the audit trail and say "this is where it all went wrong", then it calls into question the value of that auditing process. I'm personally unimpressed with the TPC process in general. I remember one "benchmark" that showed the performance of a 2RU server breaking some record, and it was a minor footnote that it was using a disk array with 7,500 drives in it -- dedicated to that one server for the duration of the test. That's an absurd setup that will never exist at any customer, ever. I ran that same software myself on literally the exact same server, and it couldn't even begin to approach the posted TPC numbers on typical storage. It was at least two orders of magnitude slower. The rub was that its inefficient usage of storage was the main problem, and the vendor was pulling a smoke & mirrors trick to hide this deficiency of their product. The TPC numbers were an outright fraud in this case, at least in my mind. So to me, TPC looks like a staged show where the auditors are more like the referees in a WWE wrestling competition.
    - ttmahdy 5y ago
      The TPC audit process tends to be thorough and strict. Possibly you missed a configuration that was included in the Full Disclosure Report or Supporting Files? The Databricks official, audited benchmark was executed against Databricks SQL which is a PaaS service that doesn't allow special tuning btw.
      - jiggawatts 5y ago
        I didn’t miss it. That doesn’t make it any less misleading.
      - AtlasLion 5y ago
- rxin 5y ago
  There's an official TPC process to audit and review the benchmark process. This debate can be easiest settled by everybody participating in the official benchmark, like we (Databricks) did. The official review process is significantly more complicated than just offering a static dataset that's been highly optimized for answering the exact set of queries. It includes data loading, data maintenance (insert and delete data), sequential query test, and concurrent query test. You can see the description of the official process in this 141 page document: http://tpc.org/tpc_documents_current_versions/pdf/tpc-ds_v3.2.0.pdf http://tpc.org/tpc_documents_current_versions/pdf/tpc-ds_v3.... Consider the following analogy: Professional athletes compete in the Olympics, and there are official judges and a lot of stringent rules and checks to ensure fairness. That's the real arena. That's what we (Databricks) have done with the official TPC-DS world record. For example, in data warehouse systems, data loading, ordering and updates can affect performance substantially, so it’s most useful to compare both systems on the official benchmark. But what’s really interesting to me is that even the Snowflake self-reported numbers ($267) are still more expensive than the Databricks’ numbers ($143 on spot, and $242 on demand). This is despite Databricks cost being calculated on our enterprise tier, while Snowflake used their cheapest tier without any enterprise features (e.g. disaster recovery). Edit: added link to audit process doc
  - jiggawatts 5y ago
    Please also refer to my comment below on the value of the TPC audit process: https://news.ycombinator.com/item?id=29208172 https://news.ycombinator.com/item?id=29208172
  - _dark_matter_ 5y ago
    Thanks for the additional context here. As someone who works for a company that pays for both databricks and snowflake, I will say that these results don't surprise me. Spark has always been infinitely configurable, in my experience. There are probably tens of thousands of possible configurations; everything from Java heap size to parquet block size. Snowflake is the opposite: you can't even specify partitions! There is only clustering. For a business, running snowflake is easy because engineers don't have to babysit it, and we like it because now we're free to work on more interesting problems. Everybody wins. Unless those problems are DB optimization. Then snowflake can actually get in your way.
mst 5y ago
Databricks and snowflake should pay an independent third party to re-run these. In-house benchmarks by either company don't count with results this different.
- cmhill 5y ago
  Databricks didn't run the Snowflake comparison in-house. From their article it says: "These results were corroborated by research from Barcelona Supercomputing Center, which frequently runs TPC-DS on popular data warehouses. Their latest research benchmarked Databricks and Snowflake, and found that Databricks was 2.7x faster and 12x better in terms of price performance."
  - dekhn 5y ago
    I don't trust a supercomputer center to do a good job running a TPC benchmark (I do trust them to run LINPACK benchmarks).

by maslam 5y ago

jiggawatts 5y ago

Audited how? If you look at the Snowflake response the numbers being posted by Databricks look outright faked or otherwise false.

maslam 5y ago
Hey jiggawatts - TPC is the official way to audit benchmarks in the database industry. They’ve been around for a bit, but let me know if you want more info, I’m happy to share more about them.
- lmeyerov 5y ago
  It sounds fundamentally busted if a competitor can submit benchmarks for someone else. TPC is great in general, but I didn't realize it had such a gaping flaw. TPC submissions take real time/$/energy/expertise, so I don't know anyone who has ever done it casually. Ex: It was a multi-company effort for the RAPIDS community to get enough API coverage & edge case optimization for an end-to-end GPU submission on the big data one (SQL, ...), and even there the TPC folks made them resubmit if I remember right. Also, note how the parent's response did not actually answer 'audited how'. Pushing the work to the questioner is on the shortlist of techniques studied by misinformation researchers. I'm a fan of both companies, so disappointing to see from a company rep.
  - rxin 5y ago
    Check my reply, Leo.
    - lmeyerov 5y ago
      The audit question is on Databricks marketing unaudited Snowflake TPC numbers. I do think Snowflake is big enough to run TPC, but how you guys choose to market is on you. But: I think it's cool both companies got it to $200-300. Way better than years ago. Next stop: GPUs :)
      - rxin 5y ago
        Ah ok. Wasn't clear. I think some repro scripts will be available soon.
    - deleted 5y ago
      [deleted]
- redis_mlc 5y ago
  > TPC is the official way to audit benchmarks in the database industry. TPC is a benchmark suite for a certain problem class. It says nothing about how the databases are configured or managed. In case you're thick, the above is a polite way of calling you a liar. This is why Oracle and other database vendors don't allow publishing of benchmarks. It's to protect them from incompetent or lazy authors primarily. Source: DBA.
Spivak 5y ago
The results are so crazy different that either Snowflake or Databricks are wrong or outright lying.
- jiggawatts 5y ago
  This is my point also, and I'm being downvoted for it. If two people are in disagreement about the same facts, then one of them is either misinformed or lying. It's that simple. If the only recourse seems to be to sink to the level of mud-slinging, with no clear ability to point to the audit trail and say "this is where it all went wrong", then it calls into question the value of that auditing process. I'm personally unimpressed with the TPC process in general. I remember one "benchmark" that showed the performance of a 2RU server breaking some record, and it was a minor footnote that it was using a disk array with 7,500 drives in it -- dedicated to that one server for the duration of the test. That's an absurd setup that will never exist at any customer, ever. I ran that same software myself on literally the exact same server, and it couldn't even begin to approach the posted TPC numbers on typical storage. It was at least two orders of magnitude slower. The rub was that its inefficient usage of storage was the main problem, and the vendor was pulling a smoke & mirrors trick to hide this deficiency of their product. The TPC numbers were an outright fraud in this case, at least in my mind. So to me, TPC looks like a staged show where the auditors are more like the referees in a WWE wrestling competition.
  - ttmahdy 5y ago
    The TPC audit process tends to be thorough and strict. Possibly you missed a configuration that was included in the Full Disclosure Report or Supporting Files? The Databricks official, audited benchmark was executed against Databricks SQL which is a PaaS service that doesn't allow special tuning btw.
    - jiggawatts 5y ago
      I didn’t miss it. That doesn’t make it any less misleading.
    - AtlasLion 5y ago
rxin 5y ago
There's an official TPC process to audit and review the benchmark process. This debate can be easiest settled by everybody participating in the official benchmark, like we (Databricks) did. The official review process is significantly more complicated than just offering a static dataset that's been highly optimized for answering the exact set of queries. It includes data loading, data maintenance (insert and delete data), sequential query test, and concurrent query test. You can see the description of the official process in this 141 page document: http://tpc.org/tpc_documents_current_versions/pdf/tpc-ds_v3.2.0.pdf http://tpc.org/tpc_documents_current_versions/pdf/tpc-ds_v3.... Consider the following analogy: Professional athletes compete in the Olympics, and there are official judges and a lot of stringent rules and checks to ensure fairness. That's the real arena. That's what we (Databricks) have done with the official TPC-DS world record. For example, in data warehouse systems, data loading, ordering and updates can affect performance substantially, so it’s most useful to compare both systems on the official benchmark. But what’s really interesting to me is that even the Snowflake self-reported numbers ($267) are still more expensive than the Databricks’ numbers ($143 on spot, and $242 on demand). This is despite Databricks cost being calculated on our enterprise tier, while Snowflake used their cheapest tier without any enterprise features (e.g. disaster recovery). Edit: added link to audit process doc
- jiggawatts 5y ago
  Please also refer to my comment below on the value of the TPC audit process: https://news.ycombinator.com/item?id=29208172 https://news.ycombinator.com/item?id=29208172
- _dark_matter_ 5y ago
  Thanks for the additional context here. As someone who works for a company that pays for both databricks and snowflake, I will say that these results don't surprise me. Spark has always been infinitely configurable, in my experience. There are probably tens of thousands of possible configurations; everything from Java heap size to parquet block size. Snowflake is the opposite: you can't even specify partitions! There is only clustering. For a business, running snowflake is easy because engineers don't have to babysit it, and we like it because now we're free to work on more interesting problems. Everybody wins. Unless those problems are DB optimization. Then snowflake can actually get in your way.

mst 5y ago

Databricks and snowflake should pay an independent third party to re-run these. In-house benchmarks by either company don't count with results this different.

cmhill 5y ago
Databricks didn't run the Snowflake comparison in-house. From their article it says: "These results were corroborated by research from Barcelona Supercomputing Center, which frequently runs TPC-DS on popular data warehouses. Their latest research benchmarked Databricks and Snowflake, and found that Databricks was 2.7x faster and 12x better in terms of price performance."
- dekhn 5y ago
  I don't trust a supercomputer center to do a good job running a TPC benchmark (I do trust them to run LINPACK benchmarks).