6 ms·

Little secret... with Rust you can just do big data processing on a single core => http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http

Little secret... with Rust you can just do big data processing on a single core => http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...

shmerl 11y ago
In many cases distributed computation is needed when there is BIG data, which won't fit on a laptop like in that example. I.e. distribution is needed not only because computation can't be done on one node (i.e. it would take too long), but because data can't possibly fit on any one node.
- deleted 11y ago
  [deleted]
- maxdemarzi 11y ago
  See his next post: http://www.frankmcsherry.org/graph/scalability/cost/2015/02/04/COST2.html http://www.frankmcsherry.org/graph/scalability/cost/2015/02/... How big are the jobs being done? I remember reading a statistic[1] that is a couple of years old that 50% of the big data jobs were less than 10 GB. 90% was less than 100 GB and only something like 1% was bigger than a TB. You can fit multiple terabytes on a single machine. I wonder if anyone has new statistics about this. [1]: http://www.msr-waypoint.com/pubs/204499/a20-appuswamy.pdf http://www.msr-waypoint.com/pubs/204499/a20-appuswamy.pdf
  - shmerl 11y ago
    I'm talking about really big data, which requires hundreds of nodes just to store it and etc.
    - jsmthrowaway 11y ago
      And the point being made (and with which I agree) is there's probably fewer than a few hundred of such deployments in the world, and their problems are very likely not your problems. I have good context on this from seeing the analytics warehouse for a very popular social network. Even with a multi-terabyte hot database, the analytics dataset fit on a single 2U server full of drives. I have not seen a deployment of Hadoop yet that makes sense. Usually I see things like a 2 or 3 TB log spread across 60 nodes and a job to count entries matching a regular expression (because Hadoop becomes the hammer hunting for nails), which can easily be done in a VM on a Mac in minutes. That was the thrust of the original post, and I agree wholeheartedly.
      - shmerl 11y ago
        > because Hadoop becomes the hammer hunting for nails Yes, I agree that it's not an uncommon situation. But regardless whether legitimately big deployments are common or not, I never really appreciated some negligence with performance in using Java for such things. Just because it scales, it doesn't mean that overhead should be ignored, because the cost of the overhead scales too. So going back to my original point, having such kind of systems made with Rust would be really much better.
      - rakoo 11y ago
        If the data already lives on 60 nodes, then transferring all this data to a vm + performing work may be longer (and clog the pipes) than just doing the work where the data already is and aggregating results.
  - bad_user 11y ago
    A previous project I worked on was persisting more than 10 TB of data per week, data that had to be analyzed, the requirement being to extract meaningful user profiles from it. We also went back to analyze older data with new queries / algorithms quite a lot and playing with a petabyte is fun. It was a startup, it was dissolved at some point, so you've never heard of it. I don't like the 90% argument. 90% of apps are just dumb frontends to a MySQL database, so you don't need Rust, or Spark. As you can see, that doesn't tell you anything meaningful about the other 10%.
nickpsecurity 11y ago
That's a pretty sad result to get from 128 cores. I've seen amateur Beowulf clusters get better results.
- tacone 11y ago
  Also GNU utils http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html http://aadrake.com/command-line-tools-can-be-235x-faster-tha... The above experiment (which has an interesting github repo) is somewhat over (and real world unusable), but still is eye opening. Hadoop and Spark bring so much complexity that looking for simpler solutions is something worth considering.
  - tacone 11y ago
    Excuse me, I guess that post had no related github repo. I was probably recalling this repo: https://github.com/erikfrey/bashreduce https://github.com/erikfrey/bashreduce
  - nickpsecurity 11y ago
    Oh my. That's a treat. Did someone forget to tell these people that the performance from more nodes is supposed to scale in the other direction? Lol.
  - rakoo 11y ago
    Let's also not forget GNU parallel (https://www.gnu.org/software/parallel/ https://www.gnu.org/software/parallel/)
    - dagw 11y ago
      GNU parallel is one of the most underrated projects in the whole GNU universe. Sure it doesn't work for everything, but for all cases where it does work, spinning up 50 ec2 spot instances and just pointing parallel at them is by far the quickest and easiest way to do distributed computing.
      - 11y ago
sampo 11y ago
Many parallel algorithms become slower if you make them use too many cores, as they spend most of the time in communication, and very little in actual computations. Maybe the parallel systems would have been faster on a smaller number of cores than 128?

shmerl 11y ago

In many cases distributed computation is needed when there is BIG data, which won't fit on a laptop like in that example. I.e. distribution is needed not only because computation can't be done on one node (i.e. it would take too long), but because data can't possibly fit on any one node.

deleted 11y ago
[deleted]
maxdemarzi 11y ago
See his next post: http://www.frankmcsherry.org/graph/scalability/cost/2015/02/04/COST2.html http://www.frankmcsherry.org/graph/scalability/cost/2015/02/... How big are the jobs being done? I remember reading a statistic[1] that is a couple of years old that 50% of the big data jobs were less than 10 GB. 90% was less than 100 GB and only something like 1% was bigger than a TB. You can fit multiple terabytes on a single machine. I wonder if anyone has new statistics about this. [1]: http://www.msr-waypoint.com/pubs/204499/a20-appuswamy.pdf http://www.msr-waypoint.com/pubs/204499/a20-appuswamy.pdf
- shmerl 11y ago
  I'm talking about really big data, which requires hundreds of nodes just to store it and etc.
  - jsmthrowaway 11y ago
    And the point being made (and with which I agree) is there's probably fewer than a few hundred of such deployments in the world, and their problems are very likely not your problems. I have good context on this from seeing the analytics warehouse for a very popular social network. Even with a multi-terabyte hot database, the analytics dataset fit on a single 2U server full of drives. I have not seen a deployment of Hadoop yet that makes sense. Usually I see things like a 2 or 3 TB log spread across 60 nodes and a job to count entries matching a regular expression (because Hadoop becomes the hammer hunting for nails), which can easily be done in a VM on a Mac in minutes. That was the thrust of the original post, and I agree wholeheartedly.
    - shmerl 11y ago
      > because Hadoop becomes the hammer hunting for nails Yes, I agree that it's not an uncommon situation. But regardless whether legitimately big deployments are common or not, I never really appreciated some negligence with performance in using Java for such things. Just because it scales, it doesn't mean that overhead should be ignored, because the cost of the overhead scales too. So going back to my original point, having such kind of systems made with Rust would be really much better.
    - rakoo 11y ago
      If the data already lives on 60 nodes, then transferring all this data to a vm + performing work may be longer (and clog the pipes) than just doing the work where the data already is and aggregating results.
- bad_user 11y ago
  A previous project I worked on was persisting more than 10 TB of data per week, data that had to be analyzed, the requirement being to extract meaningful user profiles from it. We also went back to analyze older data with new queries / algorithms quite a lot and playing with a petabyte is fun. It was a startup, it was dissolved at some point, so you've never heard of it. I don't like the 90% argument. 90% of apps are just dumb frontends to a MySQL database, so you don't need Rust, or Spark. As you can see, that doesn't tell you anything meaningful about the other 10%.

nickpsecurity 11y ago

That's a pretty sad result to get from 128 cores. I've seen amateur Beowulf clusters get better results.

tacone 11y ago
Also GNU utils http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html http://aadrake.com/command-line-tools-can-be-235x-faster-tha... The above experiment (which has an interesting github repo) is somewhat over (and real world unusable), but still is eye opening. Hadoop and Spark bring so much complexity that looking for simpler solutions is something worth considering.
- tacone 11y ago
  Excuse me, I guess that post had no related github repo. I was probably recalling this repo: https://github.com/erikfrey/bashreduce https://github.com/erikfrey/bashreduce
- nickpsecurity 11y ago
  Oh my. That's a treat. Did someone forget to tell these people that the performance from more nodes is supposed to scale in the other direction? Lol.
- rakoo 11y ago
  Let's also not forget GNU parallel (https://www.gnu.org/software/parallel/ https://www.gnu.org/software/parallel/)
  - dagw 11y ago
    GNU parallel is one of the most underrated projects in the whole GNU universe. Sure it doesn't work for everything, but for all cases where it does work, spinning up 50 ec2 spot instances and just pointing parallel at them is by far the quickest and easiest way to do distributed computing.
    - 11y ago

sampo 11y ago

Many parallel algorithms become slower if you make them use too many cores, as they spend most of the time in communication, and very little in actual computations. Maybe the parallel systems would have been faster on a smaller number of cores than 128?