5 ms·
It's interesting, but not earth-shattering. The "10x fewer nodes" means nothing; how powerful are the new nodes? What's the network? Do you use SSDs? etc. etc.
by discardorama 12y ago
It's interesting, but not earth-shattering. The "10x fewer nodes" means nothing; how powerful are the new nodes? What's the network? Do you use SSDs? etc. etc.
They also tuned their code to this specific problem:
"Exploiting Cache Locality: In the sort benchmark, each record is 100 bytes, where the sort key is the first 10 bytes. As we were profiling our sort program, we noticed the cache miss rate was high, because each comparison required an object pointer lookup that was random..... Combining TimSort with our new layout to exploit cache locality, the CPU time for sorting was reduced by a factor of 5."
I would love to see MR and Spark compete on the exact same hardware configuration.
- saryant 12y agoThe article says exactly what they ran on. EC2 i2.8xlarge instances which have 32 cores, 800GB SSD and 244GB RAM.
- discardorama 12y agoI read that. But how does that compare with the nodes they're comparing against ("10x fewer nodes")?
- rxin 12y agoThe old entry had 10Gb/s <full-duplex> (40 nodes/rack 160Gbps rack to spine. 2.5:1 subscription), 64GB of RAM, and 12 x 3TB SATA. The network part is probably the most important one here, and both have comparable network.
- discardorama 12y agoSince each node was handling 500GB of data (roughly), I think the disk speed may have been a more critical factor since each node had 244GB of memory. Their nodes used SSDs; the older nodes used spinning rust. The seek times alone will be a killer.
- rxin 12y agoNot sure why you mentioned seek time. In large scale, distributed sorting, I/O is mostly sequential.
- tlipcon 12y agoIf only that were true -- the shuffle is typically seek-bound when the intermediate data doesn't fit into cache (plenty of papers show this pretty conclusively).
- rxin 12y agoHi Todd, Except in the case of MR 2100 nodes the entire dataset fit in memory :)
- lmeyerov 12y agoDoesn't that speak to his point? Either smaller memory and more nodes, or more memory and less nodes. Why not do apples to apples? (This feels like the benchmarketing going on in browsers, which, at this point, is largely meaningless.) Edit: on the other hand, this is an endorsement of the current wave of "per node performance stinks, let's avoid rewriting software for an extra year or two by throwing SSDs at it." Great for hardware vendors!
- rxin 12y agoNo it doesn't. The old record used 2100 nodes so the entire data actually fit in memory. There shouldn't be much seek happening even in the MR 2100 case. In Spark's case, the data actually doesn't fit in memory. Also this was primary network bound. The old record had 2100 nodes with 10Gbps network.
- lmeyerov 12y agoAh, so you're saying as this is network bound, you want to launch much fewer wimpy nodes in exchange for a few bigger ones (those w/ SSDs -- over 1PB worth!). If it was memory-bound, you wouldn't care how they're spread as long as it was still in-memory. Another way of looking at this is performance per watt or dollar. The r3.2 has 60GB, so comparing to that, Spark cost the same ~$1.4K while giving a 3X speedup. (Or on host that charges per minute, it'd be the same performance at 3X cheaper.) This is me not knowing the space: would MR (or more modern things like Tez) perform worse on this HW setup, or is this a reminder that hardware/config tuning matters?
- Twirrim 12y ago3TB SATA would indicate spinning rust too, so slower storage. It's far from an apples to apples comparison.
- deleted 12y ago[deleted]
- grier 12y agoAlso used the "Enhanced Networking" option on the instances which means single root I/O virtualization underneath.
- discardorama 12y agoNot just 800GB of SSD; 8x 800GB of SSD!
- trhway 12y agoyes, that's the key. The local IO on previous reported systems was HDDs, thus 300-600Mb/s at best with 6 drives per machine, while this guys are getting 3.3Gb/s. Getting full performance - 1.1Gb/s - of 10Gb network - were they lucky or AWS now is that good ? (couple years ago i was able to get only 400Mb/s node-to-node on cluster compute nodes there)
- nchammas 12y ago> I would love to see MR and Spark compete on the exact same hardware configuration. You may find this benchmark [1] interesting to read. It needs some updating (a lot has changed since February 2014), but it compares Shark (which uses Spark as its execution engine) to Hive (using Hadoop 1 MapReduce as its execution engine) and a number of other systems. The benchmark is run on EC2 and is detailed in such a way that it should be independently verifiable. Hive and Shark are run on identically sized clusters, though I don't know if the other details of the configuration were identical. [1] https://amplab.cs.berkeley.edu/benchmark/ https://amplab.cs.berkeley.edu/benchmark/