I've been working with all these tools for a while now (and a bunch more along the way). It's not the orthodox cloud-thinking, but you're often best off proces

I've been working with all these tools for a while now (and a bunch more along the way). It's not the orthodox cloud-thinking, but you're often best off processing at the point of creation (or ingestion). Normalize the data as much as you can, (probably) compress[1], and send as close to the target as possible. If you grab the data, send elsewhere, transform <repeat>... it all gets slow and expensive pretty quickly. Also a headache to manage failures. This is especially true if you're using BigQuery. Stage your near-raw data into BigQuery and then use it's muscle as much as you can. A classic example here might be de-duplicating data. A painful prospect for many distributed systems, but pretty easy on the BigQuery side. This is all especially true for time-series data. With BigQuery time partitions you can keep the queries fast (and the costs reasonable). Also limits the range of technologies and languages you need to wrangle too. 1: Choosing your data format and compression approach can make a huge difference.