4 ms·

See his next post: http://www.frankmcsherry.org/graph/scalability/cost/2015/02/04/COST2.html http://www.frankmcsherry.org/graph/scalability/cost/2015/02/... Ho

See his next post: http://www.frankmcsherry.org/graph/scalability/cost/2015/02/04/COST2.html http://www.frankmcsherry.org/graph/scalability/cost/2015/02/... How big are the jobs being done? I remember reading a statistic[1] that is a couple of years old that 50% of the big data jobs were less than 10 GB. 90% was less than 100 GB and only something like 1% was bigger than a TB. You can fit multiple terabytes on a single machine. I wonder if anyone has new statistics about this. [1]: http://www.msr-waypoint.com/pubs/204499/a20-appuswamy.pdf http://www.msr-waypoint.com/pubs/204499/a20-appuswamy.pdf

shmerl 11y ago
I'm talking about really big data, which requires hundreds of nodes just to store it and etc.
- jsmthrowaway 11y ago
  And the point being made (and with which I agree) is there's probably fewer than a few hundred of such deployments in the world, and their problems are very likely not your problems. I have good context on this from seeing the analytics warehouse for a very popular social network. Even with a multi-terabyte hot database, the analytics dataset fit on a single 2U server full of drives. I have not seen a deployment of Hadoop yet that makes sense. Usually I see things like a 2 or 3 TB log spread across 60 nodes and a job to count entries matching a regular expression (because Hadoop becomes the hammer hunting for nails), which can easily be done in a VM on a Mac in minutes. That was the thrust of the original post, and I agree wholeheartedly.
  - shmerl 11y ago
    > because Hadoop becomes the hammer hunting for nails Yes, I agree that it's not an uncommon situation. But regardless whether legitimately big deployments are common or not, I never really appreciated some negligence with performance in using Java for such things. Just because it scales, it doesn't mean that overhead should be ignored, because the cost of the overhead scales too. So going back to my original point, having such kind of systems made with Rust would be really much better.
  - rakoo 11y ago
    If the data already lives on 60 nodes, then transferring all this data to a vm + performing work may be longer (and clog the pipes) than just doing the work where the data already is and aggregating results.
bad_user 11y ago
A previous project I worked on was persisting more than 10 TB of data per week, data that had to be analyzed, the requirement being to extract meaningful user profiles from it. We also went back to analyze older data with new queries / algorithms quite a lot and playing with a petabyte is fun. It was a startup, it was dissolved at some point, so you've never heard of it. I don't like the 90% argument. 90% of apps are just dumb frontends to a MySQL database, so you don't need Rust, or Spark. As you can see, that doesn't tell you anything meaningful about the other 10%.

shmerl 11y ago

I'm talking about really big data, which requires hundreds of nodes just to store it and etc.

jsmthrowaway 11y ago
And the point being made (and with which I agree) is there's probably fewer than a few hundred of such deployments in the world, and their problems are very likely not your problems. I have good context on this from seeing the analytics warehouse for a very popular social network. Even with a multi-terabyte hot database, the analytics dataset fit on a single 2U server full of drives. I have not seen a deployment of Hadoop yet that makes sense. Usually I see things like a 2 or 3 TB log spread across 60 nodes and a job to count entries matching a regular expression (because Hadoop becomes the hammer hunting for nails), which can easily be done in a VM on a Mac in minutes. That was the thrust of the original post, and I agree wholeheartedly.
- shmerl 11y ago
  > because Hadoop becomes the hammer hunting for nails Yes, I agree that it's not an uncommon situation. But regardless whether legitimately big deployments are common or not, I never really appreciated some negligence with performance in using Java for such things. Just because it scales, it doesn't mean that overhead should be ignored, because the cost of the overhead scales too. So going back to my original point, having such kind of systems made with Rust would be really much better.
- rakoo 11y ago
  If the data already lives on 60 nodes, then transferring all this data to a vm + performing work may be longer (and clog the pipes) than just doing the work where the data already is and aggregating results.

bad_user 11y ago

A previous project I worked on was persisting more than 10 TB of data per week, data that had to be analyzed, the requirement being to extract meaningful user profiles from it. We also went back to analyze older data with new queries / algorithms quite a lot and playing with a petabyte is fun. It was a startup, it was dissolved at some point, so you've never heard of it. I don't like the 90% argument. 90% of apps are just dumb frontends to a MySQL database, so you don't need Rust, or Spark. As you can see, that doesn't tell you anything meaningful about the other 10%.