4 ms·
Better headline, "Apache Spark Is Very Slow". Only 1 megabyte per second per node? (3.8GB/(8 nodes*500 seconds)) Even with single core nodes, that's utterly fee
by paulsutter 8y ago
Better headline, "Apache Spark Is Very Slow". Only 1 megabyte per second per node? (3.8GB/(8 nodes*500 seconds)) Even with single core nodes, that's utterly feeble (pro tip, please state the kind of node you're using)
Seems fishy that the benchmark is small enough to fit into GPU RAM as well
- twtw 8y agoPro tip: justify why dividing data set size by time to solution and saying number is small is a good analysis. Let's try imagenet training. Intel's best time is 3h25m on 128 nodes. Imagenet is ~150 gb. (150 GB)/(3 hr * 3600 sec/hr * 128 nodes) = less than one megabyte per second per node! Caffe is slow! CPUs are slow! Or bs metrics are bad? https://dawn.cs.stanford.edu/benchmark/ImageNet/train.html https://dawn.cs.stanford.edu/benchmark/ImageNet/train.html
- jmalicki 8y agoIntel's run wasn't 3:25 per epoch, which is the difference. A simple ETL should be one linear scan over the data, with maybe some merges, nothing I saw here seemed more complex than that?
- paulsutter 8y agoThis is ETL processing, not training a neural network. If their (undisclosed) actual task is not parallelizable in Spark, the comparison is even sillier so let's be generous and assume it is.
- felipe_aramburu 8y agoThe GPU workload is used in feeding XGBOOST to perform classification. We are omitting this timing on purpose because the training time on purpose becuase SPARK took 1000's of times longer for this part of the workflow. You can see the workflow for yourself and see how the task is quite parallelizable.
- scarejunba 8y agoSo just the ETL portion is that slow? That's really odd. I think I see faster performance than that ETLing on Hadoop. Unless there's something complex going on here.
- felipe_aramburu 8y agoDid you see the steps being performed in the workflow? How would you perform these steps on hadoop and what kind of times would you expect for the different steps in the workflow?
- scarejunba 8y agoI went back to see and only just realized you had a “TLDR here’s the full details” bit. Missed that and saw only the 5x faster ETL which I figured was what you were crowing about. I’ll have to look at the rest later. I still don’t get why the ETL phase has to be so slow but maybe it’ll be obvious when I look, like you’re doing extensive transformations or something. But no one would just call that “just ETL”. Anyway, I’ll see for sure later.
- felipe_aramburu 8y agoCacaw! I am glad the crowing reached you :). You probably did not get far if you are still on the old numbers that was in the first paragraph. ETL can be very extensive. For example, we first built this when we had to take data from 15 different database systems that represented individuals and their pension contributions and join across these systems. It was a largish join about ~4 tables from each system so around a 60 table join. That was "JUST ETL". The job was preparing the data for training. ETL is often times a large part of people's workflow. Looking for needle situations in a haystack. That can be JUST ETL. If you believe there is a more apt word for extracting data from a system, performing unspeakable transformations on it, then making that information available to another process then please tell me. Being of Peruvian stock myself I take great license with my language and grammar.
- felipe_aramburu 8y agoYou will notice that the second size does NOT fit into GPU ram on this benchmark since we ran it on a GPU that has less memory available than the DATA it is running on. Spark is feeble, perhaps but it is one of the tools being used to feed GPU workloads. Here we have replaced a workload where spark was previously used and now it is faster.
- pete23 8y agoI’d like a Postgres bake off. Playing against Spark on a 24Gb dataset seems silly, too.
- felipe_aramburu 8y ago1. It really isn't the kind of problem we are trying to solve. 2. Last time we did this we were over 100x faster and so we said. Ok its slow as &#@%. We are more interested in trying to compare to tools that can provide workflows which are: A. I have a ton of data in S3, HDFS, cloud storage solutions, centralized NFS B. I want to use the GPU to perform learning, graph analytics, classification, etc. I want to get from A to B as fast as possible utilizing the same resources for A that I will need to complete steb B.
- felipe_aramburu 8y agoAs for size we will be releasing much larger workloads for GTC and are working hard on finishing up our work to make the workloads public and reproducible as well as releasing a new version of our engine where you can run much much larger workloads through distribution.
- mr_toad 8y ago> Apache Spark Is Very Slow I’m betting that the bottleneck is running ML training. Otherwise I can’t see why an ETL job on that amount of data would take several minutes.
- felipe_aramburu 8y agoAn etl job could take an infinite amount of time. I am not trying to be cute by saying this but the point is that some ETL jobs might be as simple as loading data from a file others could involve millions of transformations. There is no limit to what you can do in ETL. This is not just loading the data from a file and providing it to xgboost.
- felipe_aramburu 8y agoAnd as we state in the post. This does not include ML times since we wanted to show how much faster we were making ETL. The ML timing differences are even starker than ETL.
- nl 8y agoI've done some fairly intense Spark turning. We were able to get a query that took 16 hours to 4 seconds. This took a long time, and included changing how the data was stored etc. Anyway, fast Spark performance is possible but it is very sensitive to tuning (this was in ~2016, so maybe things have improved).
- scarejunba 8y agoInteresting. This is worth writing a blog post on.
- nl 8y agoI did a talk on it at Hadoop World or some HortonWorks conference one year. Don't know where the slides when though.
- scarejunba 8y agoDo you remember the title? I can try finding it on the Internet if you happen to be able to find the title (maybe from your submission email?). Thanks for doing the talk.
- nl 8y agoThis was the conference: https://dataworkssummit.com/sydney-2017/ https://dataworkssummit.com/sydney-2017/, but they don't seem to have uploaded any of the sessions from Sydney to the SlideShare. The one I did the previous year is nicely linked from http://hadoopsummit.org/melbourne/agenda/ http://hadoopsummit.org/melbourne/agenda/
- rubyfan 8y agoThe workload size is goofy to say the least.
- einpoklum 8y agoDistributed workloads are not yet supported, so they showcase what they can fit on a single node right now.