6 ms·
I was just thinking, that Rust is a great candidate for big data processing tools. So much more than Java (which is annoyingly used a lot there). Something like
by shmerl 11y ago
I was just thinking, that Rust is a great candidate for big data processing tools. So much more than Java (which is annoyingly used a lot there). Something like Spark and HDFS should be implemented in Rust.
- maxdemarzi 11y agoLittle secret... with Rust you can just do big data processing on a single core => http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...
- shmerl 11y agoIn many cases distributed computation is needed when there is BIG data, which won't fit on a laptop like in that example. I.e. distribution is needed not only because computation can't be done on one node (i.e. it would take too long), but because data can't possibly fit on any one node.
- deleted 11y ago[deleted]
- maxdemarzi 11y agoSee his next post: http://www.frankmcsherry.org/graph/scalability/cost/2015/02/04/COST2.html http://www.frankmcsherry.org/graph/scalability/cost/2015/02/... How big are the jobs being done? I remember reading a statistic[1] that is a couple of years old that 50% of the big data jobs were less than 10 GB. 90% was less than 100 GB and only something like 1% was bigger than a TB. You can fit multiple terabytes on a single machine. I wonder if anyone has new statistics about this. [1]: http://www.msr-waypoint.com/pubs/204499/a20-appuswamy.pdf http://www.msr-waypoint.com/pubs/204499/a20-appuswamy.pdf
- shmerl 11y agoI'm talking about really big data, which requires hundreds of nodes just to store it and etc.
- jsmthrowaway 11y agoAnd the point being made (and with which I agree) is there's probably fewer than a few hundred of such deployments in the world, and their problems are very likely not your problems. I have good context on this from seeing the analytics warehouse for a very popular social network. Even with a multi-terabyte hot database, the analytics dataset fit on a single 2U server full of drives. I have not seen a deployment of Hadoop yet that makes sense. Usually I see things like a 2 or 3 TB log spread across 60 nodes and a job to count entries matching a regular expression (because Hadoop becomes the hammer hunting for nails), which can easily be done in a VM on a Mac in minutes. That was the thrust of the original post, and I agree wholeheartedly.
- shmerl 11y ago> because Hadoop becomes the hammer hunting for nails Yes, I agree that it's not an uncommon situation. But regardless whether legitimately big deployments are common or not, I never really appreciated some negligence with performance in using Java for such things. Just because it scales, it doesn't mean that overhead should be ignored, because the cost of the overhead scales too. So going back to my original point, having such kind of systems made with Rust would be really much better.
- rakoo 11y agoIf the data already lives on 60 nodes, then transferring all this data to a vm + performing work may be longer (and clog the pipes) than just doing the work where the data already is and aggregating results.
- bad_user 11y agoA previous project I worked on was persisting more than 10 TB of data per week, data that had to be analyzed, the requirement being to extract meaningful user profiles from it. We also went back to analyze older data with new queries / algorithms quite a lot and playing with a petabyte is fun. It was a startup, it was dissolved at some point, so you've never heard of it. I don't like the 90% argument. 90% of apps are just dumb frontends to a MySQL database, so you don't need Rust, or Spark. As you can see, that doesn't tell you anything meaningful about the other 10%.
- nickpsecurity 11y agoThat's a pretty sad result to get from 128 cores. I've seen amateur Beowulf clusters get better results.
- tacone 11y agoAlso GNU utils http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html http://aadrake.com/command-line-tools-can-be-235x-faster-tha... The above experiment (which has an interesting github repo) is somewhat over (and real world unusable), but still is eye opening. Hadoop and Spark bring so much complexity that looking for simpler solutions is something worth considering.
- tacone 11y agoExcuse me, I guess that post had no related github repo. I was probably recalling this repo: https://github.com/erikfrey/bashreduce https://github.com/erikfrey/bashreduce
- nickpsecurity 11y agoOh my. That's a treat. Did someone forget to tell these people that the performance from more nodes is supposed to scale in the other direction? Lol.
- rakoo 11y agoLet's also not forget GNU parallel (https://www.gnu.org/software/parallel/ https://www.gnu.org/software/parallel/)
- dagw 11y agoGNU parallel is one of the most underrated projects in the whole GNU universe. Sure it doesn't work for everything, but for all cases where it does work, spinning up 50 ec2 spot instances and just pointing parallel at them is by far the quickest and easiest way to do distributed computing.
- 11y ago
- sampo 11y agoMany parallel algorithms become slower if you make them use too many cores, as they spend most of the time in communication, and very little in actual computations. Maybe the parallel systems would have been faster on a smaller number of cores than 128?
- frankmcsherry 11y agoYou can contribute to: https://github.com/frankmcsherry/timely-dataflow https://github.com/frankmcsherry/timely-dataflow https://github.com/frankmcsherry/differential-dataflow https://github.com/frankmcsherry/differential-dataflow Or, just tell your friends. :) Better yet, write some python / pandas / dataframes / whatever_the_cool_kids_need layer on top, and rule the next big data drama cycle.
- shmerl 11y agoThanks, those look very interesting! I'll go read about Naiad first :)
- pron 11y agoThe more cores you have and the more RAM, the bigger advantage GC has. The thing with having lots of RAM is that it's very hard to take advantage of it with on-stack data (which can, at most, use about 1-2% of the total RAM available -- do the math) and thread-local heaps. Once you use thread-local heaps/arenas, you need to shard your data. Any cross-shard access would mean locking, which doesn't scale very well. That's exactly where GCs shine: they let you have scalable, concurrent access to data with arbitrary lifespan. That's why Java is used for those kind of applications -- it performs and scales much better than Rust can hope to do on large machines. You are right, though, that if the processing is extremely "batchy" and all data dies at the same time, then it doesn't make a difference.
- Meai 11y agoRust has heap memory as well..?
- pron 11y agoOf course, but concurrent access to shared data in Rust (let alone when that data has an arbitrary lifetime) carries more overhead (and is much trickier) than in GCed environments. As a general rule, GCs decrease overhead for concurrent memory access.
- frankmcsherry 11y agoIn my experience, concurrent access to shared data is relatively rare in a well-designed data-parallel platform. Data are exchanged rather than shared, at which point ownership of memory transfers from one worker to another. Rust is great at understanding ownership, and at managing memory in this sort of situation.
- pron 11y agoAgain, that depends on how batchy things are. If all the datais available in advance -- you're right. If it's streaming -in, it's not so simple any more.