3 ms·
Can someone explain to me why it took 26 minutes for hadoop? How can 7 machines be that many times slower than a single machine?
by charcircuit 5y ago
Can someone explain to me why it took 26 minutes for hadoop? How can 7 machines be that many times slower than a single machine?
- pjmlp 5y agoEasy lesson from distributed systems, most of the time is spent in communication instead of actual work.
- charcircuit 5y ago26 minutes worth? That can't be right. Sending 250 MB (uncompressed) to 7 machines shouldn't take that long.
- usrbinbash 5y agoHow many sends are we talking about? Into how many messages is the data turned, how often does it get sent around? If I send 1MiB of data by packing it up into messages of 10 byte each, it will ikely be slower than sending 10MiB in a single message. Messages == Overhead. Envelopes, packing, unpacking, parsing, assembling, etc. all eat up cycles.
- charcircuit 5y agoThis is a distributed sum. You don't need extra messages being sent around. You send 7 messages giving each node 1/7th of all games. Then 1 message per node is sent with all the tallies to be merged together. And then maybe an extra message to give you the results if you weren't the one that merged it. 15 messages shouldn't take 26 minutes to send.
- anonymoushn 5y agoThis is what we've found. For a certain kind of simulation, we can process data on a single core at ~.5x line rate (I think this could be a lot faster if we took this problem seriously) so if we move to some arrangement where hosts have to download the data in order to process it, they can only use 2 cores each, and every 2 cores we want use requires another storage box to download the data from! Paying for hardware to increase "line rate" and sending zstd-compressed files over the wire might get us a factor of ~60 here, but even then we pretty rapidly run into issues where the data has to be placed on a bunch of distinct hosts in order to be useful, since 120 cores is not a lot of machines. In the future we expect to have workloads that do less computation per piece of data, which makes the need to move our compute to our data much more acute.
- rstuart4133 5y ago> How can 7 machines be that many times slower than a single machine? A mere 7 machines is almost certainly going to be slower than 1. There was a recent post here recently titled "Latency Numbers Every Programmer Should Know": https://news.ycombinator.com/item?id=30546995 https://news.ycombinator.com/item?id=30546995 In this case the dataset was small enough to fix in my laptop's DRAM without straining it. If we assume the 7 machines are in the data centre, that means the two numbers to compare are the main memory reference (100ns) versus the Data Centre round trip time (500us). That's a factor of 5,000. If your intuition told you those 7 machines are going to be faster, then you really should invest the time to internalise those numbers. The article is 100% correct - every programmer should know them.
- charcircuit 5y agoI think you are missing my point. It's a difference between 12 seconds on a laptop and 26 minutes on 7 c1.medium instances. Yes there is overhead in shipping the data around, but there isn't 26 minutes worth of overhead. If a c1.medium can process at the same speed as his laptop it should take less than 3 seconds worst case. And yes 7 machines should be faster at this scale.
- rstuart4133 5y ago> If a c1.medium can process at the same speed as his laptop As it happens, I got to perform that experiment. Sort of. I was moving stuff on physical Dell hardware to a virtualised environment, at the behest of MBA's. I was a bit concerned about it as we pushed the existing hardware hard - it had overnight stuff it had to get finished by morning. It had a 30% buffer. It wasn't even close. The "virtual environment" was 15 times slower. It wasn't the CPU. It was slower but not by much and that could have been remedied by firing up more VM's on different physical machines. It was disc storage. They insisted their big beefy SAN they spent 100's of K on would be faster than a locally connected SSD. But a SAN operates over a network, and SSD's over 150mm gigabit links. The I/O was mostly random. If they had of consulted "Latency Numbers Every Programmer Should Know" the could have predicted the outcome - just as I had. BTW, I also benchmarked my the laptop. Despite what you apparently think it isn't much slower at doing a single task than the big Dell server, and I wouldn't expect it to be. The one difference is the Dell server comes with a factor of 10 more RAM, SSD, HDD and networking, and it seems to be able to drive all those devices at full speed in parallel. But that would not of helped in this instance as the task is too small. I know from experience for this sort of task my laptop will easily outperform a c1.medium.