11 ms·

Snowflake’s response to Databricks’ TPC-DS post

80 points by uvdn7 5y ago

maslam 5y ago
Databricks broke the record by 2x) and is 10x more cost effective, in an audited benchmark. Snowflake should participate in the official, audited benchmark. Customers win when businesses are open and transparent…
- jiggawatts 5y ago
  Audited how? If you look at the Snowflake response the numbers being posted by Databricks look outright faked or otherwise false.
  - maslam 5y ago
    Hey jiggawatts - TPC is the official way to audit benchmarks in the database industry. They’ve been around for a bit, but let me know if you want more info, I’m happy to share more about them.
    - lmeyerov 5y ago
      It sounds fundamentally busted if a competitor can submit benchmarks for someone else. TPC is great in general, but I didn't realize it had such a gaping flaw. TPC submissions take real time/$/energy/expertise, so I don't know anyone who has ever done it casually. Ex: It was a multi-company effort for the RAPIDS community to get enough API coverage & edge case optimization for an end-to-end GPU submission on the big data one (SQL, ...), and even there the TPC folks made them resubmit if I remember right. Also, note how the parent's response did not actually answer 'audited how'. Pushing the work to the questioner is on the shortlist of techniques studied by misinformation researchers. I'm a fan of both companies, so disappointing to see from a company rep.
      - rxin 5y ago
        Check my reply, Leo.
        lmeyerov 5y ago
        The audit question is on Databricks marketing unaudited Snowflake TPC numbers. I do think Snowflake is big enough to run TPC, but how you guys choose to market is on you. But: I think it's cool both companies got it to $200-300. Way better than years ago. Next stop: GPUs :)
        rxin 5y ago
        Ah ok. Wasn't clear. I think some repro scripts will be available soon.
        deleted 5y ago
        [deleted]
    - redis_mlc 5y ago
      > TPC is the official way to audit benchmarks in the database industry. TPC is a benchmark suite for a certain problem class. It says nothing about how the databases are configured or managed. In case you're thick, the above is a polite way of calling you a liar. This is why Oracle and other database vendors don't allow publishing of benchmarks. It's to protect them from incompetent or lazy authors primarily. Source: DBA.
  - Spivak 5y ago
    The results are so crazy different that either Snowflake or Databricks are wrong or outright lying.
    - jiggawatts 5y ago
      This is my point also, and I'm being downvoted for it. If two people are in disagreement about the same facts, then one of them is either misinformed or lying. It's that simple. If the only recourse seems to be to sink to the level of mud-slinging, with no clear ability to point to the audit trail and say "this is where it all went wrong", then it calls into question the value of that auditing process. I'm personally unimpressed with the TPC process in general. I remember one "benchmark" that showed the performance of a 2RU server breaking some record, and it was a minor footnote that it was using a disk array with 7,500 drives in it -- dedicated to that one server for the duration of the test. That's an absurd setup that will never exist at any customer, ever. I ran that same software myself on literally the exact same server, and it couldn't even begin to approach the posted TPC numbers on typical storage. It was at least two orders of magnitude slower. The rub was that its inefficient usage of storage was the main problem, and the vendor was pulling a smoke & mirrors trick to hide this deficiency of their product. The TPC numbers were an outright fraud in this case, at least in my mind. So to me, TPC looks like a staged show where the auditors are more like the referees in a WWE wrestling competition.
      - ttmahdy 5y ago
        The TPC audit process tends to be thorough and strict. Possibly you missed a configuration that was included in the Full Disclosure Report or Supporting Files? The Databricks official, audited benchmark was executed against Databricks SQL which is a PaaS service that doesn't allow special tuning btw.
        jiggawatts 5y ago
        I didn’t miss it. That doesn’t make it any less misleading.
        AtlasLion 5y ago
  - rxin 5y ago
    There's an official TPC process to audit and review the benchmark process. This debate can be easiest settled by everybody participating in the official benchmark, like we (Databricks) did. The official review process is significantly more complicated than just offering a static dataset that's been highly optimized for answering the exact set of queries. It includes data loading, data maintenance (insert and delete data), sequential query test, and concurrent query test. You can see the description of the official process in this 141 page document: http://tpc.org/tpc_documents_current_versions/pdf/tpc-ds_v3.2.0.pdf http://tpc.org/tpc_documents_current_versions/pdf/tpc-ds_v3.... Consider the following analogy: Professional athletes compete in the Olympics, and there are official judges and a lot of stringent rules and checks to ensure fairness. That's the real arena. That's what we (Databricks) have done with the official TPC-DS world record. For example, in data warehouse systems, data loading, ordering and updates can affect performance substantially, so it’s most useful to compare both systems on the official benchmark. But what’s really interesting to me is that even the Snowflake self-reported numbers ($267) are still more expensive than the Databricks’ numbers ($143 on spot, and $242 on demand). This is despite Databricks cost being calculated on our enterprise tier, while Snowflake used their cheapest tier without any enterprise features (e.g. disaster recovery). Edit: added link to audit process doc
    - jiggawatts 5y ago
      Please also refer to my comment below on the value of the TPC audit process: https://news.ycombinator.com/item?id=29208172 https://news.ycombinator.com/item?id=29208172
    - _dark_matter_ 5y ago
      Thanks for the additional context here. As someone who works for a company that pays for both databricks and snowflake, I will say that these results don't surprise me. Spark has always been infinitely configurable, in my experience. There are probably tens of thousands of possible configurations; everything from Java heap size to parquet block size. Snowflake is the opposite: you can't even specify partitions! There is only clustering. For a business, running snowflake is easy because engineers don't have to babysit it, and we like it because now we're free to work on more interesting problems. Everybody wins. Unless those problems are DB optimization. Then snowflake can actually get in your way.
- mst 5y ago
  Databricks and snowflake should pay an independent third party to re-run these. In-house benchmarks by either company don't count with results this different.
  - cmhill 5y ago
    Databricks didn't run the Snowflake comparison in-house. From their article it says: "These results were corroborated by research from Barcelona Supercomputing Center, which frequently runs TPC-DS on popular data warehouses. Their latest research benchmarked Databricks and Snowflake, and found that Databricks was 2.7x faster and 12x better in terms of price performance."
    - dekhn 5y ago
      I don't trust a supercomputer center to do a good job running a TPC benchmark (I do trust them to run LINPACK benchmarks).
PostThisTooFast 5y ago
Another douchey HN title. Obscurity isn’t cool. Nobody gives a shit.
aptxkid 5y ago
Personally I think it’s a great response and very well written. I didn’t jump on the congrats-Databricks wagon when the result first came out because of the weird front page comparison against snowflake. Both companies are doing great work. Focusing on building a better product for your customer is much more meaningful than making your competitor look bad.
- tyingq 5y ago
  It is well written, but there's some sleight of hand here and there too. Like using your lowest tier product to demonstrate price/performance against a competitor's highest tier. The Snowflake lowest tier doesn't have failover, for example...or compliance features.
  - uvdn7 5y ago
    Exactly. That’s why I think public benchmark war is just a waste of time. There will ALWAYS be some subtle differences between the two platforms that results will never be apple to apple.
  - buttaphingas 5y ago
    This is incorrect. Every edition of Snowflake is deployed across multiple availability zones with automatic failover in the case of failure or AZ outage. This is included in the price and requires no configuration by the customer. Cross-cloud/region failover requires the top edition and a few lines of SQL to configure (plus cloud egress costs for data replication). The higher editions of Snowflake include features like materialised views, dynamic data masking, BYOK, PCI & HIPAA compliance etc., non of which are required for the benchmark.
    - tyingq 5y ago
      I'm getting it from Snowflake's own page: https://www.snowflake.com/pricing/ https://www.snowflake.com/pricing/ Amongst other things, listed under the enterprise tier, and not lower tiers, is "Database failover and failback for business continuity". "The higher editions of Snowflake include features like materialised views, dynamic data masking, BYOK, PCI & HIPAA compliance etc., non of which are required for the benchmark." Yeah, but they are referencing a price/performance comparison to a Databricks tier that DOES have those things. That's the point. Update your own numbers with a lower tier, but don't update the competitor tier too?
ghostridr 5y ago
Hey 1990s, your TPC-DS results are in.
geoduck14 5y ago
I've been a customer/user of Snowflake. They make it simple to run SQL. There is a bunch of performance stuff that I don't need to worry about. I'm interested in using Databricks, but I haven't done it yet. I've heard good things about their product.
deleted 5y ago
[deleted]
hiyer 5y ago
Performance is only one part of the story. The major advantage Snowflake (and to some extent Presto/Trino) brings to the table is it's pretty much plug and play. Spark OTOH usually requires a lot of tweaking to work reliably for your workloads.
- EdwardDiego 5y ago
  Very much true. I saw a joke tweet recently something along the lines of - It's amazing how many data engineering scaling issues these days are being solved by just paying Snowflake more money. Spark does take a lot of tuning, but then I'm guessing Databricks offer that service as part of your licensing fee? (I'd hope so if they're selling a product based on FOSS code, there has to be a value add to justify it)
  - hiyer 5y ago
    > I'd hope so if they're selling a product based on FOSS code, there has to be a value add to justify it They have some proprietary features like DBIO [1]. They also have some cloud-specific features like storage autoscaling [2] that would not be available in OSS Spark. Even Delta Lake [3] used to be proprietary, but I suspect the rise of open-source frameworks like Iceberg led them to open-source it. Shameless plug - when working at a since-shutdown competitor to Databricks, I'd come up with storage autoscaling long before them [4], so it's not unlikely that they were "inspired" by us :-) . 1. https://docs.databricks.com/spark/latest/spark-sql/dbio-commit.html https://docs.databricks.com/spark/latest/spark-sql/dbio-comm... 2. https://databricks.com/blog/2017/12/01/transparent-autoscaling-of-instance-storage.html https://databricks.com/blog/2017/12/01/transparent-autoscali... 3. https://delta.io/ https://delta.io/ 4. https://www.qubole.com/blog/auto-scaling-in-qubole-with-aws-elastic-block-storage/ https://www.qubole.com/blog/auto-scaling-in-qubole-with-aws-...
    - glogla 5y ago
      The open source Delta is not a replacement for the real thing - they did not include features like optimizing small files (small file problem is well known in big data, and much more of a problem once streaming gets involved) and others. It is more of a demo of the real thing. Which does not stop them from repeating everywhere how open they are, of course. EDIT: the delta also still keeps partitioning information in the hive metastore, while iceberg keeps it in storage, making it a far superior design. Adopting iceberg is harder due to third party tools like AWS Redshift not supporting it - you have to go 100 % of the way.
blobbers 5y ago
This is the sort of FUD testing that gets thrown back and forth between companies of all kinds. If you're in networking, it's throughput, latency or fairness. If you're in graphics its your shaders or polygons or hashes. If you're in CPUs its your clock speed. If its cameras, it's megapixels (but nobody talks about lens or real measures of clarity) If you're in silicon it's your die size (None of that has mattered for years, those numbers are like versions not the largest block on your die) If you're in finance, it's about your returns or your drawdowns or your sharpe ratios. I'm a little bit surprised how seriously databricks is taking this, but maybe it's because one of the cofounders laid this claim. Ultimately what you find is one company is not very good at setting up the other company's system, and the result is the benchmarks are less than ideal. So why not have a showdown? Both founders, streamed live, running their benchmarks on the data. NETFLIX SPECIAL!
- rxin 5y ago
  Exactly. Not sure about Netflix special, but there are experts that have dedicated their professional careers to creating fair benchmarks. Snowflake should just participate in the official TPC benchmark. Disclaimer: Databricks cofounder who authored the original blog post.
  - AtlasLion 5y ago
    The benchmark itself is kinda useless, so I don't see why they should. If you look at tpc-h for years, you had exasol as a top dog, but in the real world that meant nothing for them.
    - deleted 5y ago
      [deleted]
    - ttmahdy 5y ago
      Exactly, companies learnt from Exasol Out of the box performance is the name of the game Executing a benchmark as complex as TPC-DS without tuning by Databricks or Snowflake is a big accomplishment
  - 5y ago
throwaway984393 5y ago
"Posting benchmark results is bad because it quickly becomes a race to the wrong solution. But somebody showed us sucking on a benchmark, so here's our benchmark results showing we're better."
- uvdn7 5y ago
  I disagree. It makes sense for Snowflake to response to what-they-think-is an unreasonably bad result published by Databricks. And they focused more on Snowflake’s result and only compared dollar cost against Databricks. It’s consistent with their philosophy that public benchmark war is beside the point and mostly a distraction.
- Rastonbury 5y ago
  I'm not familiar with this realm to comment on veracity of claims but it could very well be "Posting benchmark results is bad because it quickly becomes a race to the wrong solution. Someone misrepresented our performance in a benchmark, here are the actual results."
- AtlasLion 5y ago
  Their cofounder was behind vectorwise, which kicked ass in benchmarks, but died as no one even heard of it. You can run the benchmark queries fast, that's great, but can you handle code migrated from vertica? Will you optimiser come up with a good plan for queries built on 15 layers of views? That's what companies in the real world have, not some synthetic benchmark that you can make sure you can run for marketing purposes.
- feqgmmr2 5y ago
  The thing is even that response doesn't show them to be better. As someone pointed out, they're comparing their cheapest offering with Databricks' most expensive one and saying they're 3% better in price-perf. What does someone read into that?
pxc 5y ago
Can someone ELI5 what Snowflake and Databricks are? I spent a few minutes on the Databricks website once and couldn't really penetrate the marketing jargon. There are also some technical terms I don't know at all, and when I've searched for them, the top results are all more Azure stuff. Like wtf is a datalake?
- jeffreygoesto 5y ago
  People who downvoted this, please take a minute and reflect that your world is not the whole world. There is a serious question in this comment and there are myriads of topics _you_ have no clue about.
  - dekhn 5y ago
    sure, but if I see the term 'data lake' I'm gonna Bing it, with the first result being https://aws.amazon.com/big-data/datalakes-and-analytics/what-is-a-data-lake/ https://aws.amazon.com/big-data/datalakes-and-analytics/what... which explains it nicely. ELI5 is for reddit, generally here we expect you can google it to get the ELI5 explanation before giving us your hot take in a comment
    - pxc 5y ago
      Yeah, that's exactly the kind of content I found unsuitable when I did a web search for the term. It spends a whole two sentences giving an explanation that tells me very little about how data lakes are anything more specific than a cloud-hosted database solution, and moves on to > Organizations that successfully generate business value from their data, will outperform their peers. at which point I'm like > ok, I'm reading a covert advertisement about Fancy Cloud Technology aimed at some kind of big-spending manager, which is unlikely to tell me meaningfully what this actually is and I'm out. I was looking for content that was in a more neutral, purely educational genre, and wondering what collection of non-cloud analogues it replaces/is composed of. Someone writing in the comments > I used it to transform several terabytes of JSON into nice relational data for analysts without too much effort is way, way more direct and helpful than mentioning that 'unlike data warehouses, data lakes support non-relational data'. Like great, it's a cloud thing that supports a variety of databases. But what is it? > before giving us your hot take in a comment I didn't give any take at all? I just really found all the sources that came up on the first page of search results to be almost in the wrong genre for me, and expected (correctly) that people on this site would be able to produce descriptions in 1-5 sentences that worked way better for me. Pretty much all of the answers I got here were really good, and I'm glad I asked.
choppaface 5y ago
The audience for these posts are enterprise managers who don’t actually understand their compute needs. For the more technically inclined, don’t let any corporate blog post / comms piece live in your head rent-free. If you’re a customer, make them show you value for their money. If you’re not, make them provide you tools / services for free. Just don’t help them fuel the pissing contest, you’ll end up a bag holder (swag holder?).
deleted 5y ago
[deleted]
michaelhartm 5y ago
* Databricks is unethical * Nobody should benchmark anymore, just focus on customers instead * But hey, we just did some benchmarks and we look better than what Databricks claims * Btw, please sign up and do some benchmarks on Snowflake, we actually ship TPC-DS dataset with Snowflake * Btw, we agree with Databricks, let's remove the DeWitt clause, vendors should be able to benchmark each other! * Consistency is more important than anything else!!!
- kingkongv2 5y ago
  If people have never heard of Databricks, now is the time because a 100 billion company just started a war against them. Great marketing win Databricks.
  - geoduck14 5y ago
    To be fair, I've been equating Databricks for a month or so. Databricks is coming after Snowflake. Snowflake doesn't care. Snowflake has a pretty solid moat with: EASY SQL, data sharing (they have a marketplace), simple scaling
    - bpaneural 5y ago
      You'll need to revisit this again. In the last two years Databricks has built a lead and a bigger moat. They're essentially nice chaps with a huge community backing them. And we all love their open source tools which essentially powers not only their big data platforms, but everyone else's too (AWS, GCP).
    - ageek123 5y ago
      Databricks introduced an open source data sharing feature earlier this year. I don't know Databricks well enough to comment on the other two.
  - glogla 5y ago
    Databricks is $28B valuation and 2800 employees, Snowflake is $109 valuation and 2500 employees. They are both billion dolar companies, we're hardly talking David and Goliath here.
bjornsing 5y ago
> At the end of the script, the overall elapsed time and the geometric mean for all the queries is computed directly by querying the history view of all TPC-DS statements that have executed on the warehouse. The geometric mean? Really? Feels a lot easier to think in terms of arithmetic mean, and perhaps percentiles.
- rxin 5y ago
  Geometric mean is commonly used in benchmarks when the workloads consists of queries that have large (often orders of magnitude) differences in runtime. Consider 4 queries. Two run for 1sec, and the other two 1000sec. If we look at arithmetic mean, then we are really only taking into account the large queries. But improving geometric mean would require improving all queries. Note that I'm on the opposite side (Databricks cofounder here), so when I say that Snowflake didn't make a mistake here, you should trust me :)
  - bjornsing 5y ago
    > But improving geometric mean would require improving all queries. No. Improving the geometric mean only requires reducing the product of their execution times. So if you can make the two 1 ms queries execute in 0.5 ms at the expense of the two 1000 ms queries taking 1800 ms each then that’s an improvement in terms of geometric mean. So… kind of QED. The geometric mean is not easy to reason about.
    - ttmahdy 5y ago
      Usually making a 1 ms query execute in 0.5 ms is a lot harder than making a 10 second query execute in 5 second. One of the benefits of geometric mean is that all queries have "equal" weight in the metric, this keeps vendors from focusing on the long running queries and ignoring the short running ones. It is one way to balance between long and short query performance. A similar concept is applied to TPC-DS for data load, single user run (Power), multi user run (Throughput) and data maintenance (Concurrent Delete and Inserts). Check clause 7.6.3.1 in the TPC-Ds spec in http://tpc.org/tpc_documents_current_versions/pdf/tpc-ds_v3.2.0.pdf http://tpc.org/tpc_documents_current_versions/pdf/tpc-ds_v3....
AtlasLion 5y ago
The main question I have for DB is, how good is their query optimiser/compiler? It's fun that you can run some predefined set of queries fast. More important is, how good you can run queries in the real world, with suboptimal data models, layers upon layers of badly written views, CTEs, UDFs... That is what matters in the end. Not some synthetic benchmark based on known queries you can optimise specifically for.
- maslam 5y ago
  @AtlasLion you are right real world performance matters. We test extensively with actual workloads, and the speed up holds there too. For example: lots of real world BI queries are repeated over smallish data sets of 10 to 50 GB. We test that size factor and pattern all the time.
socaldata 5y ago
Take all the problems you have had with data warehousing and throw them in a proprietary cloud. That is Snowflake. They are the best today. Databricks started with the cloud datalake, sitting natively on parquet and using cloud native tools, fully open. Recently they added SQL to help democratize the data in the data lake versus moving it back and forth into a proprietary data warehouse. The selling point in Databricks is why move the data around when you can just have it in one place IF performance is the same or better. This is what led to the latest benchmark which in the writing appears to be unbiased. In snowflakes response however, they condemn it but then submit their own fundings. Sound a lot lot trump telling everyone he had billions of people attend his inauguration, doesn’t it? Anyhow, I trust independent studies more than I do coming from vendors. It cannot be argued or debated unless it was unfairly done. I think we are all smart enough to be careful with studies of any kind, but I can see why Databricks was excited about the findings.
- uvdn7 5y ago
  Whose result can be trusted is beside the point - I actually believe both experiments were likely conducted in good faith but with incomplete context. But that’s beside the point. The point is there’s no good reason to start a benchmark war to begin with.
  - joeharris76 5y ago
    > While performing the benchmarks, we noticed that the Snowflake pre-baked TPC-DS dataset had been recreated two days after our benchmark results were announced. An important part of the official benchmark is to verify the creation of the dataset. So, instead of using Snowflake’s pre-baked dataset, we uploaded an official TPC-DS dataset and used identical schema as Snowflake uses on its pre-baked dataset (including the same clustering column sets), on identical cluster size (4XL). We then ran and timed the POWER test three times. The first cold run took 10,085 secs, and the fastest of the three runs took 7,276 seconds. *Just to recap, we loaded the official TPC-DS dataset into Snowflake, timed how long it takes to run the power test, and it took 1.9x longer (best of 3) than what Snowflake reported in their blog.* https://databricks.com/blog/2021/11/15/snowflake-claims-similar-price-performance-to-databricks-but-not-so-fast.html?utm_source=bambu&utm_medium=social&utm_campaign=advocacy&blaid=2292581 https://databricks.com/blog/2021/11/15/snowflake-claims-simi...
uvdn7 5y ago
I genuinely think DeWitt clause is good for the users (bad for researchers). Without it, especially in the context of cooperate competitions, the company with the most marketing power will win. Users can always compare different products themselves. I am likely wrong but please help me understand.
glogla 5y ago
What do you know, here's an article[1] from 2017 about Databricks making an unfortunate mistake that showed Spark Streaming (which they sell) as a better streaming platform to Flink (which they don't sell). I really hope this is not the case again. (yes, I understand my sarcasm is unneeded, I couldn't help myself) [1]: https://www.ververica.com/blog/curious-case-broken-benchmark-revisiting-apache-flink-vs-databricks-runtime https://www.ververica.com/blog/curious-case-broken-benchmark...
kthejoker2 5y ago
Snowflake conceding they have a 700% markup between Standard and Premium editons which has zero impact on query performance is ... well, it's something. I'd start squeezing my sales engineers about that, definitely not sustainable... Also proof that lakehouse and spot compute price performance economics are here to stay, that's good for customers. Otherwise, as a vendor blog post with nothing but self-reported performance, this is worthless. Disclaimer: I work at Databricks but I admire Snowflake's product for what it is - iron sharpens iron.
- drawturkey 5y ago
  How do you get 700% markup? The difference between Standard and Enterprise is 50%. Enterprise does have features which do make workloads run faster, but this benchmark didn't need them.
- buttaphingas 5y ago
  I've used Snowflake for the past few years, and it's worth pointing out that when it comes to overall cost, there's a lot you get with Snowflake for free. For example, they have HA across 3 AZs out of the box, included in the price and with no configuration required. If I'm reading what Databricks published correctly, it seems that they've only used 1 driver node for this benchmark, in other words it's a dev setup. If they want to compare apples-to-apples then they should configure, and price, a multi-AZ HA set-up. I'm not sure if this is still applicable to Photon, however - can anyone confirm?
  - sagarm 5y ago
    The _data_ should be replicated, but the compute infrastructure doesn't need to be. Many companies I suspect would be fine having to restart pipelines on driver failure (increasing tail latency, basically) if it yields a substantial cost reduction.
bpaneural 5y ago
So much to read. TLDR; Databricks still holds the world record and they beat us on price/performance
falaki 5y ago
Linking to the discussion on the follow up from Databricks: https://news.ycombinator.com/item?id=29232346 https://news.ycombinator.com/item?id=29232346
imslowbutnice 5y ago
I dont get still how much optimization was done for the Snowflake TPC-DS power run. This is what I am seeing so far and what i am foggy on - DB1.Databricks generated the TPC-DS datasets from TPC-DS kit before time started. Databricks starts time then generated all queries. Then Databricks loaded from CSV to Delta format (also some delta tables were partitioned delta tables by date) and also computed statistics. Then all of the queries are executed 1-99 for TPCDS 100TB SF1. Databricks generated the TPC-DS datasets from TPC-DS kit before time started. Databricks starts time then generated all queries. Then load from S3 to Snowflake tables by - (i'm not sure about these next parts) - creating external stages and then "copy into" statements I guess? Or maybe just using copy into from an s3 bucket, that part doesnt matter much. But its not clear did they also allow target tables to be partitioned/clustering keys at all? Then all of the queries are executed 1-99 for TPCDS 100TB Its just hard to say exactly what "They were not allowed to apply any optimizations that would require deep understanding of the dataset or queries (as done in the Snowflake pre-baked dataset, with additional clustering columns)" means exactly. Like what does that exactly mean. At a glance though, this looks very impressive for Databricks, but just want to be sure before I submit to an opinion.

80 points by uvdn7 5y ago

Databricks broke the record by 2x) and is 10x more cost effective, in an audited benchmark. Snowflake should participate in the official, audited benchmark. Customers win when businesses are open and transparent…

jiggawatts 5y ago
Audited how? If you look at the Snowflake response the numbers being posted by Databricks look outright faked or otherwise false.
- maslam 5y ago
  Hey jiggawatts - TPC is the official way to audit benchmarks in the database industry. They’ve been around for a bit, but let me know if you want more info, I’m happy to share more about them.
  - lmeyerov 5y ago
    It sounds fundamentally busted if a competitor can submit benchmarks for someone else. TPC is great in general, but I didn't realize it had such a gaping flaw. TPC submissions take real time/$/energy/expertise, so I don't know anyone who has ever done it casually. Ex: It was a multi-company effort for the RAPIDS community to get enough API coverage & edge case optimization for an end-to-end GPU submission on the big data one (SQL, ...), and even there the TPC folks made them resubmit if I remember right. Also, note how the parent's response did not actually answer 'audited how'. Pushing the work to the questioner is on the shortlist of techniques studied by misinformation researchers. I'm a fan of both companies, so disappointing to see from a company rep.
    - rxin 5y ago
      Check my reply, Leo.
      - lmeyerov 5y ago
        The audit question is on Databricks marketing unaudited Snowflake TPC numbers. I do think Snowflake is big enough to run TPC, but how you guys choose to market is on you. But: I think it's cool both companies got it to $200-300. Way better than years ago. Next stop: GPUs :)
        rxin 5y ago
        Ah ok. Wasn't clear. I think some repro scripts will be available soon.
      - deleted 5y ago
        [deleted]
  - redis_mlc 5y ago
    > TPC is the official way to audit benchmarks in the database industry. TPC is a benchmark suite for a certain problem class. It says nothing about how the databases are configured or managed. In case you're thick, the above is a polite way of calling you a liar. This is why Oracle and other database vendors don't allow publishing of benchmarks. It's to protect them from incompetent or lazy authors primarily. Source: DBA.
- Spivak 5y ago
  The results are so crazy different that either Snowflake or Databricks are wrong or outright lying.
  - jiggawatts 5y ago
    This is my point also, and I'm being downvoted for it. If two people are in disagreement about the same facts, then one of them is either misinformed or lying. It's that simple. If the only recourse seems to be to sink to the level of mud-slinging, with no clear ability to point to the audit trail and say "this is where it all went wrong", then it calls into question the value of that auditing process. I'm personally unimpressed with the TPC process in general. I remember one "benchmark" that showed the performance of a 2RU server breaking some record, and it was a minor footnote that it was using a disk array with 7,500 drives in it -- dedicated to that one server for the duration of the test. That's an absurd setup that will never exist at any customer, ever. I ran that same software myself on literally the exact same server, and it couldn't even begin to approach the posted TPC numbers on typical storage. It was at least two orders of magnitude slower. The rub was that its inefficient usage of storage was the main problem, and the vendor was pulling a smoke & mirrors trick to hide this deficiency of their product. The TPC numbers were an outright fraud in this case, at least in my mind. So to me, TPC looks like a staged show where the auditors are more like the referees in a WWE wrestling competition.
    - ttmahdy 5y ago
      The TPC audit process tends to be thorough and strict. Possibly you missed a configuration that was included in the Full Disclosure Report or Supporting Files? The Databricks official, audited benchmark was executed against Databricks SQL which is a PaaS service that doesn't allow special tuning btw.
      - jiggawatts 5y ago
        I didn’t miss it. That doesn’t make it any less misleading.
      - AtlasLion 5y ago
- rxin 5y ago
  There's an official TPC process to audit and review the benchmark process. This debate can be easiest settled by everybody participating in the official benchmark, like we (Databricks) did. The official review process is significantly more complicated than just offering a static dataset that's been highly optimized for answering the exact set of queries. It includes data loading, data maintenance (insert and delete data), sequential query test, and concurrent query test. You can see the description of the official process in this 141 page document: http://tpc.org/tpc_documents_current_versions/pdf/tpc-ds_v3.2.0.pdf http://tpc.org/tpc_documents_current_versions/pdf/tpc-ds_v3.... Consider the following analogy: Professional athletes compete in the Olympics, and there are official judges and a lot of stringent rules and checks to ensure fairness. That's the real arena. That's what we (Databricks) have done with the official TPC-DS world record. For example, in data warehouse systems, data loading, ordering and updates can affect performance substantially, so it’s most useful to compare both systems on the official benchmark. But what’s really interesting to me is that even the Snowflake self-reported numbers ($267) are still more expensive than the Databricks’ numbers ($143 on spot, and $242 on demand). This is despite Databricks cost being calculated on our enterprise tier, while Snowflake used their cheapest tier without any enterprise features (e.g. disaster recovery). Edit: added link to audit process doc
  - jiggawatts 5y ago
    Please also refer to my comment below on the value of the TPC audit process: https://news.ycombinator.com/item?id=29208172 https://news.ycombinator.com/item?id=29208172
  - _dark_matter_ 5y ago
    Thanks for the additional context here. As someone who works for a company that pays for both databricks and snowflake, I will say that these results don't surprise me. Spark has always been infinitely configurable, in my experience. There are probably tens of thousands of possible configurations; everything from Java heap size to parquet block size. Snowflake is the opposite: you can't even specify partitions! There is only clustering. For a business, running snowflake is easy because engineers don't have to babysit it, and we like it because now we're free to work on more interesting problems. Everybody wins. Unless those problems are DB optimization. Then snowflake can actually get in your way.
mst 5y ago
Databricks and snowflake should pay an independent third party to re-run these. In-house benchmarks by either company don't count with results this different.
- cmhill 5y ago
  Databricks didn't run the Snowflake comparison in-house. From their article it says: "These results were corroborated by research from Barcelona Supercomputing Center, which frequently runs TPC-DS on popular data warehouses. Their latest research benchmarked Databricks and Snowflake, and found that Databricks was 2.7x faster and 12x better in terms of price performance."
  - dekhn 5y ago
    I don't trust a supercomputer center to do a good job running a TPC benchmark (I do trust them to run LINPACK benchmarks).

PostThisTooFast 5y ago

Another douchey HN title. Obscurity isn’t cool. Nobody gives a shit.

aptxkid 5y ago

Personally I think it’s a great response and very well written. I didn’t jump on the congrats-Databricks wagon when the result first came out because of the weird front page comparison against snowflake. Both companies are doing great work. Focusing on building a better product for your customer is much more meaningful than making your competitor look bad.

tyingq 5y ago
It is well written, but there's some sleight of hand here and there too. Like using your lowest tier product to demonstrate price/performance against a competitor's highest tier. The Snowflake lowest tier doesn't have failover, for example...or compliance features.
- uvdn7 5y ago
  Exactly. That’s why I think public benchmark war is just a waste of time. There will ALWAYS be some subtle differences between the two platforms that results will never be apple to apple.
- buttaphingas 5y ago
  This is incorrect. Every edition of Snowflake is deployed across multiple availability zones with automatic failover in the case of failure or AZ outage. This is included in the price and requires no configuration by the customer. Cross-cloud/region failover requires the top edition and a few lines of SQL to configure (plus cloud egress costs for data replication). The higher editions of Snowflake include features like materialised views, dynamic data masking, BYOK, PCI & HIPAA compliance etc., non of which are required for the benchmark.
  - tyingq 5y ago
    I'm getting it from Snowflake's own page: https://www.snowflake.com/pricing/ https://www.snowflake.com/pricing/ Amongst other things, listed under the enterprise tier, and not lower tiers, is "Database failover and failback for business continuity". "The higher editions of Snowflake include features like materialised views, dynamic data masking, BYOK, PCI & HIPAA compliance etc., non of which are required for the benchmark." Yeah, but they are referencing a price/performance comparison to a Databricks tier that DOES have those things. That's the point. Update your own numbers with a lower tier, but don't update the competitor tier too?

ghostridr 5y ago

Hey 1990s, your TPC-DS results are in.

geoduck14 5y ago

I've been a customer/user of Snowflake. They make it simple to run SQL. There is a bunch of performance stuff that I don't need to worry about. I'm interested in using Databricks, but I haven't done it yet. I've heard good things about their product.

deleted 5y ago

[deleted]

hiyer 5y ago

Performance is only one part of the story. The major advantage Snowflake (and to some extent Presto/Trino) brings to the table is it's pretty much plug and play. Spark OTOH usually requires a lot of tweaking to work reliably for your workloads.

EdwardDiego 5y ago
Very much true. I saw a joke tweet recently something along the lines of - It's amazing how many data engineering scaling issues these days are being solved by just paying Snowflake more money. Spark does take a lot of tuning, but then I'm guessing Databricks offer that service as part of your licensing fee? (I'd hope so if they're selling a product based on FOSS code, there has to be a value add to justify it)
- hiyer 5y ago
  > I'd hope so if they're selling a product based on FOSS code, there has to be a value add to justify it They have some proprietary features like DBIO [1]. They also have some cloud-specific features like storage autoscaling [2] that would not be available in OSS Spark. Even Delta Lake [3] used to be proprietary, but I suspect the rise of open-source frameworks like Iceberg led them to open-source it. Shameless plug - when working at a since-shutdown competitor to Databricks, I'd come up with storage autoscaling long before them [4], so it's not unlikely that they were "inspired" by us :-) . 1. https://docs.databricks.com/spark/latest/spark-sql/dbio-commit.html https://docs.databricks.com/spark/latest/spark-sql/dbio-comm... 2. https://databricks.com/blog/2017/12/01/transparent-autoscaling-of-instance-storage.html https://databricks.com/blog/2017/12/01/transparent-autoscali... 3. https://delta.io/ https://delta.io/ 4. https://www.qubole.com/blog/auto-scaling-in-qubole-with-aws-elastic-block-storage/ https://www.qubole.com/blog/auto-scaling-in-qubole-with-aws-...
  - glogla 5y ago
    The open source Delta is not a replacement for the real thing - they did not include features like optimizing small files (small file problem is well known in big data, and much more of a problem once streaming gets involved) and others. It is more of a demo of the real thing. Which does not stop them from repeating everywhere how open they are, of course. EDIT: the delta also still keeps partitioning information in the hive metastore, while iceberg keeps it in storage, making it a far superior design. Adopting iceberg is harder due to third party tools like AWS Redshift not supporting it - you have to go 100 % of the way.

blobbers 5y ago

This is the sort of FUD testing that gets thrown back and forth between companies of all kinds. If you're in networking, it's throughput, latency or fairness. If you're in graphics its your shaders or polygons or hashes. If you're in CPUs its your clock speed. If its cameras, it's megapixels (but nobody talks about lens or real measures of clarity) If you're in silicon it's your die size (None of that has mattered for years, those numbers are like versions not the largest block on your die) If you're in finance, it's about your returns or your drawdowns or your sharpe ratios. I'm a little bit surprised how seriously databricks is taking this, but maybe it's because one of the cofounders laid this claim. Ultimately what you find is one company is not very good at setting up the other company's system, and the result is the benchmarks are less than ideal. So why not have a showdown? Both founders, streamed live, running their benchmarks on the data. NETFLIX SPECIAL!

rxin 5y ago
Exactly. Not sure about Netflix special, but there are experts that have dedicated their professional careers to creating fair benchmarks. Snowflake should just participate in the official TPC benchmark. Disclaimer: Databricks cofounder who authored the original blog post.
- AtlasLion 5y ago
  The benchmark itself is kinda useless, so I don't see why they should. If you look at tpc-h for years, you had exasol as a top dog, but in the real world that meant nothing for them.
  - deleted 5y ago
    [deleted]
  - ttmahdy 5y ago
    Exactly, companies learnt from Exasol Out of the box performance is the name of the game Executing a benchmark as complex as TPC-DS without tuning by Databricks or Snowflake is a big accomplishment
- 5y ago

throwaway984393 5y ago

"Posting benchmark results is bad because it quickly becomes a race to the wrong solution. But somebody showed us sucking on a benchmark, so here's our benchmark results showing we're better."

uvdn7 5y ago
I disagree. It makes sense for Snowflake to response to what-they-think-is an unreasonably bad result published by Databricks. And they focused more on Snowflake’s result and only compared dollar cost against Databricks. It’s consistent with their philosophy that public benchmark war is beside the point and mostly a distraction.
Rastonbury 5y ago
I'm not familiar with this realm to comment on veracity of claims but it could very well be "Posting benchmark results is bad because it quickly becomes a race to the wrong solution. Someone misrepresented our performance in a benchmark, here are the actual results."
AtlasLion 5y ago
Their cofounder was behind vectorwise, which kicked ass in benchmarks, but died as no one even heard of it. You can run the benchmark queries fast, that's great, but can you handle code migrated from vertica? Will you optimiser come up with a good plan for queries built on 15 layers of views? That's what companies in the real world have, not some synthetic benchmark that you can make sure you can run for marketing purposes.
feqgmmr2 5y ago
The thing is even that response doesn't show them to be better. As someone pointed out, they're comparing their cheapest offering with Databricks' most expensive one and saying they're 3% better in price-perf. What does someone read into that?

pxc 5y ago

Can someone ELI5 what Snowflake and Databricks are? I spent a few minutes on the Databricks website once and couldn't really penetrate the marketing jargon. There are also some technical terms I don't know at all, and when I've searched for them, the top results are all more Azure stuff. Like wtf is a datalake?

jeffreygoesto 5y ago
People who downvoted this, please take a minute and reflect that your world is not the whole world. There is a serious question in this comment and there are myriads of topics _you_ have no clue about.
- dekhn 5y ago
  sure, but if I see the term 'data lake' I'm gonna Bing it, with the first result being https://aws.amazon.com/big-data/datalakes-and-analytics/what-is-a-data-lake/ https://aws.amazon.com/big-data/datalakes-and-analytics/what... which explains it nicely. ELI5 is for reddit, generally here we expect you can google it to get the ELI5 explanation before giving us your hot take in a comment
  - pxc 5y ago
    Yeah, that's exactly the kind of content I found unsuitable when I did a web search for the term. It spends a whole two sentences giving an explanation that tells me very little about how data lakes are anything more specific than a cloud-hosted database solution, and moves on to > Organizations that successfully generate business value from their data, will outperform their peers. at which point I'm like > ok, I'm reading a covert advertisement about Fancy Cloud Technology aimed at some kind of big-spending manager, which is unlikely to tell me meaningfully what this actually is and I'm out. I was looking for content that was in a more neutral, purely educational genre, and wondering what collection of non-cloud analogues it replaces/is composed of. Someone writing in the comments > I used it to transform several terabytes of JSON into nice relational data for analysts without too much effort is way, way more direct and helpful than mentioning that 'unlike data warehouses, data lakes support non-relational data'. Like great, it's a cloud thing that supports a variety of databases. But what is it? > before giving us your hot take in a comment I didn't give any take at all? I just really found all the sources that came up on the first page of search results to be almost in the wrong genre for me, and expected (correctly) that people on this site would be able to produce descriptions in 1-5 sentences that worked way better for me. Pretty much all of the answers I got here were really good, and I'm glad I asked.

choppaface 5y ago

The audience for these posts are enterprise managers who don’t actually understand their compute needs. For the more technically inclined, don’t let any corporate blog post / comms piece live in your head rent-free. If you’re a customer, make them show you value for their money. If you’re not, make them provide you tools / services for free. Just don’t help them fuel the pissing contest, you’ll end up a bag holder (swag holder?).

deleted 5y ago

[deleted]

michaelhartm 5y ago

* Databricks is unethical * Nobody should benchmark anymore, just focus on customers instead * But hey, we just did some benchmarks and we look better than what Databricks claims * Btw, please sign up and do some benchmarks on Snowflake, we actually ship TPC-DS dataset with Snowflake * Btw, we agree with Databricks, let's remove the DeWitt clause, vendors should be able to benchmark each other! * Consistency is more important than anything else!!!

kingkongv2 5y ago
If people have never heard of Databricks, now is the time because a 100 billion company just started a war against them. Great marketing win Databricks.
- geoduck14 5y ago
  To be fair, I've been equating Databricks for a month or so. Databricks is coming after Snowflake. Snowflake doesn't care. Snowflake has a pretty solid moat with: EASY SQL, data sharing (they have a marketplace), simple scaling
  - bpaneural 5y ago
    You'll need to revisit this again. In the last two years Databricks has built a lead and a bigger moat. They're essentially nice chaps with a huge community backing them. And we all love their open source tools which essentially powers not only their big data platforms, but everyone else's too (AWS, GCP).
  - ageek123 5y ago
    Databricks introduced an open source data sharing feature earlier this year. I don't know Databricks well enough to comment on the other two.
- glogla 5y ago
  Databricks is $28B valuation and 2800 employees, Snowflake is $109 valuation and 2500 employees. They are both billion dolar companies, we're hardly talking David and Goliath here.

bjornsing 5y ago

> At the end of the script, the overall elapsed time and the geometric mean for all the queries is computed directly by querying the history view of all TPC-DS statements that have executed on the warehouse. The geometric mean? Really? Feels a lot easier to think in terms of arithmetic mean, and perhaps percentiles.

rxin 5y ago
Geometric mean is commonly used in benchmarks when the workloads consists of queries that have large (often orders of magnitude) differences in runtime. Consider 4 queries. Two run for 1sec, and the other two 1000sec. If we look at arithmetic mean, then we are really only taking into account the large queries. But improving geometric mean would require improving all queries. Note that I'm on the opposite side (Databricks cofounder here), so when I say that Snowflake didn't make a mistake here, you should trust me :)
- bjornsing 5y ago
  > But improving geometric mean would require improving all queries. No. Improving the geometric mean only requires reducing the product of their execution times. So if you can make the two 1 ms queries execute in 0.5 ms at the expense of the two 1000 ms queries taking 1800 ms each then that’s an improvement in terms of geometric mean. So… kind of QED. The geometric mean is not easy to reason about.
  - ttmahdy 5y ago
    Usually making a 1 ms query execute in 0.5 ms is a lot harder than making a 10 second query execute in 5 second. One of the benefits of geometric mean is that all queries have "equal" weight in the metric, this keeps vendors from focusing on the long running queries and ignoring the short running ones. It is one way to balance between long and short query performance. A similar concept is applied to TPC-DS for data load, single user run (Power), multi user run (Throughput) and data maintenance (Concurrent Delete and Inserts). Check clause 7.6.3.1 in the TPC-Ds spec in http://tpc.org/tpc_documents_current_versions/pdf/tpc-ds_v3.2.0.pdf http://tpc.org/tpc_documents_current_versions/pdf/tpc-ds_v3....

AtlasLion 5y ago

The main question I have for DB is, how good is their query optimiser/compiler? It's fun that you can run some predefined set of queries fast. More important is, how good you can run queries in the real world, with suboptimal data models, layers upon layers of badly written views, CTEs, UDFs... That is what matters in the end. Not some synthetic benchmark based on known queries you can optimise specifically for.

maslam 5y ago
@AtlasLion you are right real world performance matters. We test extensively with actual workloads, and the speed up holds there too. For example: lots of real world BI queries are repeated over smallish data sets of 10 to 50 GB. We test that size factor and pattern all the time.

socaldata 5y ago

Take all the problems you have had with data warehousing and throw them in a proprietary cloud. That is Snowflake. They are the best today. Databricks started with the cloud datalake, sitting natively on parquet and using cloud native tools, fully open. Recently they added SQL to help democratize the data in the data lake versus moving it back and forth into a proprietary data warehouse. The selling point in Databricks is why move the data around when you can just have it in one place IF performance is the same or better. This is what led to the latest benchmark which in the writing appears to be unbiased. In snowflakes response however, they condemn it but then submit their own fundings. Sound a lot lot trump telling everyone he had billions of people attend his inauguration, doesn’t it? Anyhow, I trust independent studies more than I do coming from vendors. It cannot be argued or debated unless it was unfairly done. I think we are all smart enough to be careful with studies of any kind, but I can see why Databricks was excited about the findings.

uvdn7 5y ago
Whose result can be trusted is beside the point - I actually believe both experiments were likely conducted in good faith but with incomplete context. But that’s beside the point. The point is there’s no good reason to start a benchmark war to begin with.
- joeharris76 5y ago
  > While performing the benchmarks, we noticed that the Snowflake pre-baked TPC-DS dataset had been recreated two days after our benchmark results were announced. An important part of the official benchmark is to verify the creation of the dataset. So, instead of using Snowflake’s pre-baked dataset, we uploaded an official TPC-DS dataset and used identical schema as Snowflake uses on its pre-baked dataset (including the same clustering column sets), on identical cluster size (4XL). We then ran and timed the POWER test three times. The first cold run took 10,085 secs, and the fastest of the three runs took 7,276 seconds. *Just to recap, we loaded the official TPC-DS dataset into Snowflake, timed how long it takes to run the power test, and it took 1.9x longer (best of 3) than what Snowflake reported in their blog.* https://databricks.com/blog/2021/11/15/snowflake-claims-similar-price-performance-to-databricks-but-not-so-fast.html?utm_source=bambu&utm_medium=social&utm_campaign=advocacy&blaid=2292581 https://databricks.com/blog/2021/11/15/snowflake-claims-simi...

uvdn7 5y ago

I genuinely think DeWitt clause is good for the users (bad for researchers). Without it, especially in the context of cooperate competitions, the company with the most marketing power will win. Users can always compare different products themselves. I am likely wrong but please help me understand.

glogla 5y ago

What do you know, here's an article[1] from 2017 about Databricks making an unfortunate mistake that showed Spark Streaming (which they sell) as a better streaming platform to Flink (which they don't sell). I really hope this is not the case again. (yes, I understand my sarcasm is unneeded, I couldn't help myself) [1]: https://www.ververica.com/blog/curious-case-broken-benchmark-revisiting-apache-flink-vs-databricks-runtime https://www.ververica.com/blog/curious-case-broken-benchmark...

kthejoker2 5y ago

Snowflake conceding they have a 700% markup between Standard and Premium editons which has zero impact on query performance is ... well, it's something. I'd start squeezing my sales engineers about that, definitely not sustainable... Also proof that lakehouse and spot compute price performance economics are here to stay, that's good for customers. Otherwise, as a vendor blog post with nothing but self-reported performance, this is worthless. Disclaimer: I work at Databricks but I admire Snowflake's product for what it is - iron sharpens iron.

drawturkey 5y ago
How do you get 700% markup? The difference between Standard and Enterprise is 50%. Enterprise does have features which do make workloads run faster, but this benchmark didn't need them.
buttaphingas 5y ago
I've used Snowflake for the past few years, and it's worth pointing out that when it comes to overall cost, there's a lot you get with Snowflake for free. For example, they have HA across 3 AZs out of the box, included in the price and with no configuration required. If I'm reading what Databricks published correctly, it seems that they've only used 1 driver node for this benchmark, in other words it's a dev setup. If they want to compare apples-to-apples then they should configure, and price, a multi-AZ HA set-up. I'm not sure if this is still applicable to Photon, however - can anyone confirm?
- sagarm 5y ago
  The _data_ should be replicated, but the compute infrastructure doesn't need to be. Many companies I suspect would be fine having to restart pipelines on driver failure (increasing tail latency, basically) if it yields a substantial cost reduction.

bpaneural 5y ago

So much to read. TLDR; Databricks still holds the world record and they beat us on price/performance

falaki 5y ago

Linking to the discussion on the follow up from Databricks: https://news.ycombinator.com/item?id=29232346 https://news.ycombinator.com/item?id=29232346

imslowbutnice 5y ago

I dont get still how much optimization was done for the Snowflake TPC-DS power run. This is what I am seeing so far and what i am foggy on - DB1.Databricks generated the TPC-DS datasets from TPC-DS kit before time started. Databricks starts time then generated all queries. Then Databricks loaded from CSV to Delta format (also some delta tables were partitioned delta tables by date) and also computed statistics. Then all of the queries are executed 1-99 for TPCDS 100TB SF1. Databricks generated the TPC-DS datasets from TPC-DS kit before time started. Databricks starts time then generated all queries. Then load from S3 to Snowflake tables by - (i'm not sure about these next parts) - creating external stages and then "copy into" statements I guess? Or maybe just using copy into from an s3 bucket, that part doesnt matter much. But its not clear did they also allow target tables to be partitioned/clustering keys at all? Then all of the queries are executed 1-99 for TPCDS 100TB Its just hard to say exactly what "They were not allowed to apply any optimizations that would require deep understanding of the dataset or queries (as done in the Snowflake pre-baked dataset, with additional clustering columns)" means exactly. Like what does that exactly mean. At a glance though, this looks very impressive for Databricks, but just want to be sure before I submit to an opinion.