4 ms·

> How can 7 machines be that many times slower than a single machine? A mere 7 machines is almost certainly going to be slower than 1. There was a recent post

> How can 7 machines be that many times slower than a single machine? A mere 7 machines is almost certainly going to be slower than 1. There was a recent post here recently titled "Latency Numbers Every Programmer Should Know": https://news.ycombinator.com/item?id=30546995 https://news.ycombinator.com/item?id=30546995 In this case the dataset was small enough to fix in my laptop's DRAM without straining it. If we assume the 7 machines are in the data centre, that means the two numbers to compare are the main memory reference (100ns) versus the Data Centre round trip time (500us). That's a factor of 5,000. If your intuition told you those 7 machines are going to be faster, then you really should invest the time to internalise those numbers. The article is 100% correct - every programmer should know them.

charcircuit 5y ago
I think you are missing my point. It's a difference between 12 seconds on a laptop and 26 minutes on 7 c1.medium instances. Yes there is overhead in shipping the data around, but there isn't 26 minutes worth of overhead. If a c1.medium can process at the same speed as his laptop it should take less than 3 seconds worst case. And yes 7 machines should be faster at this scale.
- rstuart4133 5y ago
  > If a c1.medium can process at the same speed as his laptop As it happens, I got to perform that experiment. Sort of. I was moving stuff on physical Dell hardware to a virtualised environment, at the behest of MBA's. I was a bit concerned about it as we pushed the existing hardware hard - it had overnight stuff it had to get finished by morning. It had a 30% buffer. It wasn't even close. The "virtual environment" was 15 times slower. It wasn't the CPU. It was slower but not by much and that could have been remedied by firing up more VM's on different physical machines. It was disc storage. They insisted their big beefy SAN they spent 100's of K on would be faster than a locally connected SSD. But a SAN operates over a network, and SSD's over 150mm gigabit links. The I/O was mostly random. If they had of consulted "Latency Numbers Every Programmer Should Know" the could have predicted the outcome - just as I had. BTW, I also benchmarked my the laptop. Despite what you apparently think it isn't much slower at doing a single task than the big Dell server, and I wouldn't expect it to be. The one difference is the Dell server comes with a factor of 10 more RAM, SSD, HDD and networking, and it seems to be able to drive all those devices at full speed in parallel. But that would not of helped in this instance as the task is too small. I know from experience for this sort of task my laptop will easily outperform a c1.medium.
  - randomswede 5y ago
    This is one of those "it depends" things. As usual, it's good to build an intuition for what you expect. But it is also worthwhile benchmarking it now and again. At a previous job, we were pushing against going to "networked temp disk" instead of "local temp disk", on the assumption that local storage would be faster than remote storage. But, actual benchmarking showed that the "networked temp disk" was about two to three times faster. Mostly because almost all IOPS on each machine went to servicing network disk requests, so trying to squeeze in on one machine's IOPS caused IO stall times that trying to squeeze into N machines' IO queue didn't see. It's also a "are you mostly doing read consecutive blocks" or "are you doing essentially random, scattered reads" (for read workloads, write workloads are a bit different, as it is approaching hard top speculatively write data that has not yet passed through a write(2) call).

charcircuit 5y ago

I think you are missing my point. It's a difference between 12 seconds on a laptop and 26 minutes on 7 c1.medium instances. Yes there is overhead in shipping the data around, but there isn't 26 minutes worth of overhead. If a c1.medium can process at the same speed as his laptop it should take less than 3 seconds worst case. And yes 7 machines should be faster at this scale.

rstuart4133 5y ago
> If a c1.medium can process at the same speed as his laptop As it happens, I got to perform that experiment. Sort of. I was moving stuff on physical Dell hardware to a virtualised environment, at the behest of MBA's. I was a bit concerned about it as we pushed the existing hardware hard - it had overnight stuff it had to get finished by morning. It had a 30% buffer. It wasn't even close. The "virtual environment" was 15 times slower. It wasn't the CPU. It was slower but not by much and that could have been remedied by firing up more VM's on different physical machines. It was disc storage. They insisted their big beefy SAN they spent 100's of K on would be faster than a locally connected SSD. But a SAN operates over a network, and SSD's over 150mm gigabit links. The I/O was mostly random. If they had of consulted "Latency Numbers Every Programmer Should Know" the could have predicted the outcome - just as I had. BTW, I also benchmarked my the laptop. Despite what you apparently think it isn't much slower at doing a single task than the big Dell server, and I wouldn't expect it to be. The one difference is the Dell server comes with a factor of 10 more RAM, SSD, HDD and networking, and it seems to be able to drive all those devices at full speed in parallel. But that would not of helped in this instance as the task is too small. I know from experience for this sort of task my laptop will easily outperform a c1.medium.
- randomswede 5y ago
  This is one of those "it depends" things. As usual, it's good to build an intuition for what you expect. But it is also worthwhile benchmarking it now and again. At a previous job, we were pushing against going to "networked temp disk" instead of "local temp disk", on the assumption that local storage would be faster than remote storage. But, actual benchmarking showed that the "networked temp disk" was about two to three times faster. Mostly because almost all IOPS on each machine went to servicing network disk requests, so trying to squeeze in on one machine's IOPS caused IO stall times that trying to squeeze into N machines' IO queue didn't see. It's also a "are you mostly doing read consecutive blocks" or "are you doing essentially random, scattered reads" (for read workloads, write workloads are a bit different, as it is approaching hard top speculatively write data that has not yet passed through a write(2) call).