3 ms·
Looks like Google BigQuery which is based on the Dremel paper.
by filereaper 10y ago
Looks like Google BigQuery which is based on the Dremel paper.
- jeffbarr 10y agoThis is not BigQuery.
- puzzle 10y agoIt's not, but it resembles BQ a lot in the central part of the query execution, where it transparently performs work on your behalf on thousands of other machines that you don't have to be aware of. Is that correct?
- openasocket 10y ago1. No, you still need a Redshift cluster, which performs the work. A better example would be something like Athena. 2. That's not really a big similarity. Having many machines coordinate for data processing is a really common thing. So if BQ and Redshift Spectrum are similar, than so is Athena, Presto, any mapreduce-based system, Spark, etc.
- puzzle 10y agoMy main point is that there is work behind the scenes happening that you don't have to worry about, on an undefined number of machines that you don't have to care about. Yes, there is still a cluster which does some kind of planning and post-processing now, plus there is obviously a lot of coordination, but it's not like Hadoop or classic Redshift, where you are constrained only to the hardware that you paid for and set up. It's the provisioning in the central phase that resembles BQ the most.
- openasocket 10y agoIn that case Athena is more what you're talking about. Redshift Spectrum is weirder. You still have to have a Redshift cluster you provision and pay for, but for some S3 processing it calls out to other, independent, servers. I guess I'd consider that a hybrid approach.
- filereaper 10y agoJeff, Could you highlight the differences?