Distributed over the storage mediums you intend to pull from for the computation. It's literally one of the basic building blocks for modern data warehouses wh

Distributed over the storage mediums you intend to pull from for the computation. It's literally one of the basic building blocks for modern data warehouses which have solved this problem. By distributing to different nodes over some key in your dataset when you ingest the data or in Google's case some intelligent chunking method [0], at query time you wind up with hopefully evenly distributed chunks already, and by doing things this way you minimize the amount of data you need to actually move or copy at query time before the data warehouses essentially run a map reduce job (with some really cool query planning [1] [2]) with your query to get you your results. As to your second point, of course it sounds like a server? Serverless just means I don't have to maintain the hardware resources which is a nightmare at any real scale, AWS Athena and Spectrum are great examples of not having to scale hardware, though there are tradeoffs. The point I'm arguing against is that it's expensive (it's not), because most clouds allow free data transfer within their networks and because of modern data techniques that minimize the amount of data transfer that's needed so run times are limited to how bad you've written a SQL query or chosen to initially distribute your data. Combine that with instance reservations for more regular workloads and modern Big Data can be pretty cheap. On the chance you have to move 100TB through a raw Spark job, if you have an idea that you know what you'll need the data for you can take a page or two from systems that were built to solve these problems and organize your data in such a way that lends itself to that fact. Storing your 100TB data as one contiguous block and then having to chunk + transfer it at query time in their rudimentary search implementation like the OP suggests is probably about the worst position you could have put yourself in, and would have been a naive thing for someone to do. [0] https://research.google/pubs/pub51/ https://research.google/pubs/pub51/ [1] https://docs.aws.amazon.com/redshift/latest/dg/c-query-planning.html https://docs.aws.amazon.com/redshift/latest/dg/c-query-plann... [2] https://cloud.google.com/files/BigQueryTechnicalWP.pdf https://cloud.google.com/files/BigQueryTechnicalWP.pdf