3 ms·

If you stored the data in a distributed fashion initially then it's not a problem. Redshift, Bigquery etc implement it this way and then have various schemes f

by Grimm1 6y ago

If you stored the data in a distributed fashion initially then it's not a problem. Redshift, Bigquery etc implement it this way and then have various schemes for computation on top. Redshift bundles individual compute with storage whereas other implementations scale the compute independent of the distributed storage. But this has allowed very cheap scale for querying large datasets and in practice, I imagine you very rarely have to worry about implementing the data transport yourself beyond initial loading with tools like those available. Edit: Also most clouds moving data within their networks is free so really it's just talking time for moving data which indirectly influcences cost in terms of run time.

Dylan16807 6y ago
> If you stored the data in a distributed fashion initially then it's not a problem. Distributed over what? Putting all the data in one service and transferring it into another is still going to look an awful lot like a big centralized transfer. If you're permanently storing chunks of data next to chunks of the compute that will run on your data, that sounds an awful lot like a server.
- Grimm1 6y ago
  Distributed over the storage mediums you intend to pull from for the computation. It's literally one of the basic building blocks for modern data warehouses which have solved this problem. By distributing to different nodes over some key in your dataset when you ingest the data or in Google's case some intelligent chunking method [0], at query time you wind up with hopefully evenly distributed chunks already, and by doing things this way you minimize the amount of data you need to actually move or copy at query time before the data warehouses essentially run a map reduce job (with some really cool query planning [1] [2]) with your query to get you your results. As to your second point, of course it sounds like a server? Serverless just means I don't have to maintain the hardware resources which is a nightmare at any real scale, AWS Athena and Spectrum are great examples of not having to scale hardware, though there are tradeoffs. The point I'm arguing against is that it's expensive (it's not), because most clouds allow free data transfer within their networks and because of modern data techniques that minimize the amount of data transfer that's needed so run times are limited to how bad you've written a SQL query or chosen to initially distribute your data. Combine that with instance reservations for more regular workloads and modern Big Data can be pretty cheap. On the chance you have to move 100TB through a raw Spark job, if you have an idea that you know what you'll need the data for you can take a page or two from systems that were built to solve these problems and organize your data in such a way that lends itself to that fact. Storing your 100TB data as one contiguous block and then having to chunk + transfer it at query time in their rudimentary search implementation like the OP suggests is probably about the worst position you could have put yourself in, and would have been a naive thing for someone to do. [0] https://research.google/pubs/pub51/ https://research.google/pubs/pub51/ [1] https://docs.aws.amazon.com/redshift/latest/dg/c-query-planning.html https://docs.aws.amazon.com/redshift/latest/dg/c-query-plann... [2] https://cloud.google.com/files/BigQueryTechnicalWP.pdf https://cloud.google.com/files/BigQueryTechnicalWP.pdf

by Grimm1 6y ago

Dylan16807 6y ago

> If you stored the data in a distributed fashion initially then it's not a problem. Distributed over what? Putting all the data in one service and transferring it into another is still going to look an awful lot like a big centralized transfer. If you're permanently storing chunks of data next to chunks of the compute that will run on your data, that sounds an awful lot like a server.

Grimm1 6y ago
Distributed over the storage mediums you intend to pull from for the computation. It's literally one of the basic building blocks for modern data warehouses which have solved this problem. By distributing to different nodes over some key in your dataset when you ingest the data or in Google's case some intelligent chunking method [0], at query time you wind up with hopefully evenly distributed chunks already, and by doing things this way you minimize the amount of data you need to actually move or copy at query time before the data warehouses essentially run a map reduce job (with some really cool query planning [1] [2]) with your query to get you your results. As to your second point, of course it sounds like a server? Serverless just means I don't have to maintain the hardware resources which is a nightmare at any real scale, AWS Athena and Spectrum are great examples of not having to scale hardware, though there are tradeoffs. The point I'm arguing against is that it's expensive (it's not), because most clouds allow free data transfer within their networks and because of modern data techniques that minimize the amount of data transfer that's needed so run times are limited to how bad you've written a SQL query or chosen to initially distribute your data. Combine that with instance reservations for more regular workloads and modern Big Data can be pretty cheap. On the chance you have to move 100TB through a raw Spark job, if you have an idea that you know what you'll need the data for you can take a page or two from systems that were built to solve these problems and organize your data in such a way that lends itself to that fact. Storing your 100TB data as one contiguous block and then having to chunk + transfer it at query time in their rudimentary search implementation like the OP suggests is probably about the worst position you could have put yourself in, and would have been a naive thing for someone to do. [0] https://research.google/pubs/pub51/ https://research.google/pubs/pub51/ [1] https://docs.aws.amazon.com/redshift/latest/dg/c-query-planning.html https://docs.aws.amazon.com/redshift/latest/dg/c-query-plann... [2] https://cloud.google.com/files/BigQueryTechnicalWP.pdf https://cloud.google.com/files/BigQueryTechnicalWP.pdf