3 ms·

There aren't a lot of 'official' published tpcds results. Until a little less than 2 months ago, there was exactly one published result; it was at the 10TB scal

There aren't a lot of 'official' published tpcds results. Until a little less than 2 months ago, there was exactly one published result; it was at the 10TB scale. I like your site and comparisons, but you only have 100 gig and 1 TB. If you had 10TB results I would have wanted to use them for comparison purposes. How come you haven't go to 10T? We had to figure out how to get something to compare to and understand how we were doing. We looked at the recent Giga Om Microsoft 30T tpch results [1]. They used sum of query times to compare results (sum of times in terms of fraction of an hour to run them all * cost per hour). I decided to do the same sum of query time, except with TPC-DS 10T. As TPC-DS official results provide the power run time, we can get query times from there. We wanted to push ourselves to test 10TB TPC-DS. It was much more data, much larger intermediate results. Some databases don't support grouping sets, and that means they can't run the official queries as you said. When we added support for them in MemSQL we made a choice to implement features that customers want; customers want to run TPC-DS without changing the queries, too for testing and other reasons. There are suggested rewrites for systems that don't have grouping sets. MemSQL didn't always have that feature. It's another way to distinguish system capability, I don't know why some of those other companies choose not to add support. I don't think using grouping sets or rewriting matters too much for our perf. We never cache results; for us the warmup time separates the time to compile and code gen. Some companies do cache final or intermediate results. I will update our blog post with some of this info, but it might not be completed until tomorrow. Since you were interested in the first run, aka warmup only time I will provide it here first. It's about a 5% difference. MemSQL Sum of query time (avg of runs not including the warm-up time, already in the blog post): 6,494.74 MemSQL Warmup time only (just the first run, so includes optimize and compile time): 6,860.68 I work at MemSQL. [1] https://gigaom.com/report/data-warehouse-cloud-benchmark/ https://gigaom.com/report/data-warehouse-cloud-benchmark/ [2] https://fivetran.com/blog/warehouse-benchmark https://fivetran.com/blog/warehouse-benchmark

georgewfraser 7y ago
There's a ton of gray area in benchmarking, you have to make a lot of little choices along the way, so I appreciate you sharing your reasoning. > How come you haven't go to 10T? We don't see a lot of 10 TB single datasets "in the wild". We think 100 GB - 1 TB is more representative of the average real-world data warehouse user. > for us the warmup time separates the time to compile and code gen You should include compile time---your compile time is really good! Redshift sometimes takes longer to compile the query than to run it, though I think they may have improved this in the last 6 months.

georgewfraser 7y ago

There's a ton of gray area in benchmarking, you have to make a lot of little choices along the way, so I appreciate you sharing your reasoning. > How come you haven't go to 10T? We don't see a lot of 10 TB single datasets "in the wild". We think 100 GB - 1 TB is more representative of the average real-world data warehouse user. > for us the warmup time separates the time to compile and code gen You should include compile time---your compile time is really good! Redshift sometimes takes longer to compile the query than to run it, though I think they may have improved this in the last 6 months.