3 ms·

there is a subset of embarrassingly parallel problems that are are heavily data intensive. E.g. suppose you have 100 TB of data files and you want to run some

by shoo 6y ago

there is a subset of embarrassingly parallel problems that are are heavily data intensive. E.g. suppose you have 100 TB of data files and you want to run some kind of keyword search over the data. If the data can be broken into 1000x100GB chunks then you can do some map-reduce-ish thing where each 100GB chunk is searched independently, then the search results from each of the 1000 chunks is aggregated. 1000x speedup! serverless! however, if you want to execute this across some fleet of rented "serverless" servers, a key factor that will influence cost and running time is (1) where is the 100 TB of data right now, (2) how are you going to copy each 100 GB chunk of the data to each serverless server, (3) how much time and money will that copy cost. I.e. in examples like this where the time required to read the data and send the data over the network is much larger than the time required to compute the data once the data is in memory, is going to be more efficient to move the code & the compute to where the data already is rather than moving the data and the code to some other physical compute device behind a bunch of abstraction layers and network pipes.

gopalv 6y ago
> there is a subset of embarrassingly parallel problems that are are heavily data intensive. There's an even smaller subset which is one-shot data access. > in examples like this where the time required to read the data and send the data over the network is much larger than the time required to compute the data once the data is in memory The annoying thing about lambda and other functional alternatives is that data-access patterns tend to be repetitive in somewhat predictable ways & there is no way to take advantage of that fact easily. However, if you don't have that & say you were reading from S3 for every pass, then lambda does look attractive because the container lifetime management is outsourced - but if you do have even temporal stickiness of data, then it helps to do your own container management & direct queries closer to previous access, rather than to entirely cold instances. If there's a thing that hadoop missed out on building into itself, it was a distributed work queue with functions with slight side-effects (i.e memoization).
- adev_ 6y ago
  > If there's a thing that hadoop missed out on building into itself, it was a distributed work queue with functions with slight side-effects (i.e memoization). Is that not named spark ? :)
Grimm1 6y ago
If you stored the data in a distributed fashion initially then it's not a problem. Redshift, Bigquery etc implement it this way and then have various schemes for computation on top. Redshift bundles individual compute with storage whereas other implementations scale the compute independent of the distributed storage. But this has allowed very cheap scale for querying large datasets and in practice, I imagine you very rarely have to worry about implementing the data transport yourself beyond initial loading with tools like those available. Edit: Also most clouds moving data within their networks is free so really it's just talking time for moving data which indirectly influcences cost in terms of run time.
- Dylan16807 6y ago
  > If you stored the data in a distributed fashion initially then it's not a problem. Distributed over what? Putting all the data in one service and transferring it into another is still going to look an awful lot like a big centralized transfer. If you're permanently storing chunks of data next to chunks of the compute that will run on your data, that sounds an awful lot like a server.
  - Grimm1 6y ago
    Distributed over the storage mediums you intend to pull from for the computation. It's literally one of the basic building blocks for modern data warehouses which have solved this problem. By distributing to different nodes over some key in your dataset when you ingest the data or in Google's case some intelligent chunking method [0], at query time you wind up with hopefully evenly distributed chunks already, and by doing things this way you minimize the amount of data you need to actually move or copy at query time before the data warehouses essentially run a map reduce job (with some really cool query planning [1] [2]) with your query to get you your results. As to your second point, of course it sounds like a server? Serverless just means I don't have to maintain the hardware resources which is a nightmare at any real scale, AWS Athena and Spectrum are great examples of not having to scale hardware, though there are tradeoffs. The point I'm arguing against is that it's expensive (it's not), because most clouds allow free data transfer within their networks and because of modern data techniques that minimize the amount of data transfer that's needed so run times are limited to how bad you've written a SQL query or chosen to initially distribute your data. Combine that with instance reservations for more regular workloads and modern Big Data can be pretty cheap. On the chance you have to move 100TB through a raw Spark job, if you have an idea that you know what you'll need the data for you can take a page or two from systems that were built to solve these problems and organize your data in such a way that lends itself to that fact. Storing your 100TB data as one contiguous block and then having to chunk + transfer it at query time in their rudimentary search implementation like the OP suggests is probably about the worst position you could have put yourself in, and would have been a naive thing for someone to do. [0] https://research.google/pubs/pub51/ https://research.google/pubs/pub51/ [1] https://docs.aws.amazon.com/redshift/latest/dg/c-query-planning.html https://docs.aws.amazon.com/redshift/latest/dg/c-query-plann... [2] https://cloud.google.com/files/BigQueryTechnicalWP.pdf https://cloud.google.com/files/BigQueryTechnicalWP.pdf
mihaaly 6y ago
Not to mention when the processing of huge set of data is not for having a short answer but to process the data into an other form, the same size or still big. Even if the data is massively distributed from the start the communication and aggregation of distributed data between processing or client nodes might be a considerable bottleneck. Different aspects might desire different logic for distribution otherwise the performance suffers, but rearranging for the sake of processing might ruin the overall performance of the task.

by shoo 6y ago

gopalv 6y ago

> there is a subset of embarrassingly parallel problems that are are heavily data intensive. There's an even smaller subset which is one-shot data access. > in examples like this where the time required to read the data and send the data over the network is much larger than the time required to compute the data once the data is in memory The annoying thing about lambda and other functional alternatives is that data-access patterns tend to be repetitive in somewhat predictable ways & there is no way to take advantage of that fact easily. However, if you don't have that & say you were reading from S3 for every pass, then lambda does look attractive because the container lifetime management is outsourced - but if you do have even temporal stickiness of data, then it helps to do your own container management & direct queries closer to previous access, rather than to entirely cold instances. If there's a thing that hadoop missed out on building into itself, it was a distributed work queue with functions with slight side-effects (i.e memoization).

adev_ 6y ago
> If there's a thing that hadoop missed out on building into itself, it was a distributed work queue with functions with slight side-effects (i.e memoization). Is that not named spark ? :)

Grimm1 6y ago

If you stored the data in a distributed fashion initially then it's not a problem. Redshift, Bigquery etc implement it this way and then have various schemes for computation on top. Redshift bundles individual compute with storage whereas other implementations scale the compute independent of the distributed storage. But this has allowed very cheap scale for querying large datasets and in practice, I imagine you very rarely have to worry about implementing the data transport yourself beyond initial loading with tools like those available. Edit: Also most clouds moving data within their networks is free so really it's just talking time for moving data which indirectly influcences cost in terms of run time.

Dylan16807 6y ago
> If you stored the data in a distributed fashion initially then it's not a problem. Distributed over what? Putting all the data in one service and transferring it into another is still going to look an awful lot like a big centralized transfer. If you're permanently storing chunks of data next to chunks of the compute that will run on your data, that sounds an awful lot like a server.
- Grimm1 6y ago
  Distributed over the storage mediums you intend to pull from for the computation. It's literally one of the basic building blocks for modern data warehouses which have solved this problem. By distributing to different nodes over some key in your dataset when you ingest the data or in Google's case some intelligent chunking method [0], at query time you wind up with hopefully evenly distributed chunks already, and by doing things this way you minimize the amount of data you need to actually move or copy at query time before the data warehouses essentially run a map reduce job (with some really cool query planning [1] [2]) with your query to get you your results. As to your second point, of course it sounds like a server? Serverless just means I don't have to maintain the hardware resources which is a nightmare at any real scale, AWS Athena and Spectrum are great examples of not having to scale hardware, though there are tradeoffs. The point I'm arguing against is that it's expensive (it's not), because most clouds allow free data transfer within their networks and because of modern data techniques that minimize the amount of data transfer that's needed so run times are limited to how bad you've written a SQL query or chosen to initially distribute your data. Combine that with instance reservations for more regular workloads and modern Big Data can be pretty cheap. On the chance you have to move 100TB through a raw Spark job, if you have an idea that you know what you'll need the data for you can take a page or two from systems that were built to solve these problems and organize your data in such a way that lends itself to that fact. Storing your 100TB data as one contiguous block and then having to chunk + transfer it at query time in their rudimentary search implementation like the OP suggests is probably about the worst position you could have put yourself in, and would have been a naive thing for someone to do. [0] https://research.google/pubs/pub51/ https://research.google/pubs/pub51/ [1] https://docs.aws.amazon.com/redshift/latest/dg/c-query-planning.html https://docs.aws.amazon.com/redshift/latest/dg/c-query-plann... [2] https://cloud.google.com/files/BigQueryTechnicalWP.pdf https://cloud.google.com/files/BigQueryTechnicalWP.pdf

mihaaly 6y ago

Not to mention when the processing of huge set of data is not for having a short answer but to process the data into an other form, the same size or still big. Even if the data is massively distributed from the start the communication and aggregation of distributed data between processing or client nodes might be a considerable bottleneck. Different aspects might desire different logic for distribution otherwise the performance suffers, but rearranging for the sake of processing might ruin the overall performance of the task.