5 ms·

Is this guy for real? He is limited by not having an "expensive macbook with 4 real cores" ?? He gave up on a test because it "paged out the 15GB of data". Of

by yazr 9y ago

Is this guy for real? He is limited by not having an "expensive macbook with 4 real cores" ?? He gave up on a test because it "paged out the 15GB of data". Of course at these scales a cluster is x5 slower. Can someone please explain - am i missing something here ?

tlb 9y ago
It'd obviously be a nice property of systems if they were efficient at any scale. But currently we have two kinds of systems, designed for small and large scale.
- philipkglass 9y ago
  If a system is designed for large scale problems, it seems like it really should be validated against problems that are impractical to run on a single commodity server. The problem with the results the blog author pokes fun at isn't (necessarily) that they trade high overhead for high scalability. The problem is more that these papers are mostly dealing with such small problems that simple one-laptop processing offers competitive or even superior performance. There's no useful evidence provided either way when a proposed "big data" approach is tested on inappropriately small inputs.
  - shub 9y ago
    Good luck finding anything public about graph processing on a dataset too large to fit on a single machine. I can launch an AWS instance with 128 cores and 4 TB RAM--how many triples is too many for that monster? Tens of billions? Hundreds of billions?
    - fnord123 9y ago
      The common crawl has 3.28 billion pages. That should result in plenty of edges for 2017. (Shouldn't it?)
      - shub 9y ago
        That's true. Thanks for reminding me! Next time I need to torture a graph store I'll try loading common crawl into it.
    - yazr 9y ago
      Yes. At 4$/hr these instances are truly mind blowing. BUT With only 2TB of local storage, your 4TB dataset (or output set) has to pass over the net for every execution. At 10Gbs - this alone can take 20 minutes to 1hr. If you want to write it to the local SSD - multiply that by x3 or x10 ?! .... This 4TB of memory is still bounded by 4 x 30MB of L3 cache. Which means your single thread implementation will be slowed down by x3 or x5 due to memory latency. Your multi core implementation will probably suffer even more. Distributed system are VERY hard, but dealing with them is inevitable for certain workloads.
      - shub 9y ago
        I was referring to x1e.32xlarge..2x1920GB ephemeral, 25 Gbps Ethernet + 14 Gbps EBS Ethernet. And at least for the application I'm considering, the working set would fit in RAM and you can run the algorithm online. Score! That's what I mean. You have to get really big to have Big Data. Fine, you can't do PageRank on a petabyte of web crawls using one machine. But the datasets that people use for benchmarking, at least the benchmarks that are made public, you definitely can. You can go far larger than a typical benchmark dataset and still do it on one machine.
JoachimSchipper 9y ago
He compares - with snark - the papers' authors' 160-core cluster (etc.), as reported by those authors, with a re-implementation running on his laptop. That is, the papers get 160 cores to his 1-2, and he still wins.
lambda 9y ago
> He is limited by not having an "expensive macbook with 4 real cores" ?? He gave up on a test because it "paged out the 15GB of data". Note that he already demonstrated that for many of the tests, his older MacBook was faster than these 160 core clusters. There were a couple where the older MacBook was not faster than a 160 core cluster, but he suspects that a newer one might be. The one that paged out happened to be one where he artificially broke it up into 160 pieces to simulate what it would be like if he had 160 similar cores, but that caused enough issues with memory locality that swapping dominated the results. This artificial partitioning was to give a sense for what results on a 160 core cluster "should" be, but the swapping distorts the results enough that for that problem it wouldn't even be a very good rough estimate. > Is this guy for real? This isn't supposed to be a publication-quality paper. This is a blog post, using the resources he has immediately at hand, to show why if you follow cutting edge distributed database research you might be led down a wrong, and very costly, path, which you could avoid with a much simpler implementation on much cheaper hardware. Note that he references an earlier paper that he did publish on this topic; this blog post is mostly just doing a similar check on some more recent published work, to see if much has improved in the field: https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf https://www.usenix.org/system/files/conference/hotos15/hotos...
shoo 9y ago
> Of course at these scales a cluster is x5 slower. So this complaint should be levelled at the papers and the academic communities- why is it acceptable to only provide empirical evidence for these distributed algorithms at data scales where they cannot out perform simpler implementations? Why not require empirical performance measurement on truly large scale examples for these kind of algorithms?
- wmf 9y ago
  Of course academics don't have access to truly big data, so this amounts to saying that they can't publish anything. That's fine with me but I don't think the academic CS establishment will agree.
  - deleted 9y ago
    [deleted]
  - mcguire 9y ago
    "the academic CS establishment" Strictly speaking, I think that term attributes more organization to the academic CS establishment than my experience supports. The point here is that, if they're trying to establish that Smart Technique 2 is better than Smart Technique 1, they need to first establish that Stupid Technique Q doesn't work at all. Which they haven't. If they don't have access to "truly big data" to do so, well, most of the people who want to use the Smart Techniques probably don't either.

by yazr 9y ago

tlb 9y ago

It'd obviously be a nice property of systems if they were efficient at any scale. But currently we have two kinds of systems, designed for small and large scale.

philipkglass 9y ago
If a system is designed for large scale problems, it seems like it really should be validated against problems that are impractical to run on a single commodity server. The problem with the results the blog author pokes fun at isn't (necessarily) that they trade high overhead for high scalability. The problem is more that these papers are mostly dealing with such small problems that simple one-laptop processing offers competitive or even superior performance. There's no useful evidence provided either way when a proposed "big data" approach is tested on inappropriately small inputs.
- shub 9y ago
  Good luck finding anything public about graph processing on a dataset too large to fit on a single machine. I can launch an AWS instance with 128 cores and 4 TB RAM--how many triples is too many for that monster? Tens of billions? Hundreds of billions?
  - fnord123 9y ago
    The common crawl has 3.28 billion pages. That should result in plenty of edges for 2017. (Shouldn't it?)
    - shub 9y ago
      That's true. Thanks for reminding me! Next time I need to torture a graph store I'll try loading common crawl into it.
  - yazr 9y ago
    Yes. At 4$/hr these instances are truly mind blowing. BUT With only 2TB of local storage, your 4TB dataset (or output set) has to pass over the net for every execution. At 10Gbs - this alone can take 20 minutes to 1hr. If you want to write it to the local SSD - multiply that by x3 or x10 ?! .... This 4TB of memory is still bounded by 4 x 30MB of L3 cache. Which means your single thread implementation will be slowed down by x3 or x5 due to memory latency. Your multi core implementation will probably suffer even more. Distributed system are VERY hard, but dealing with them is inevitable for certain workloads.
    - shub 9y ago
      I was referring to x1e.32xlarge..2x1920GB ephemeral, 25 Gbps Ethernet + 14 Gbps EBS Ethernet. And at least for the application I'm considering, the working set would fit in RAM and you can run the algorithm online. Score! That's what I mean. You have to get really big to have Big Data. Fine, you can't do PageRank on a petabyte of web crawls using one machine. But the datasets that people use for benchmarking, at least the benchmarks that are made public, you definitely can. You can go far larger than a typical benchmark dataset and still do it on one machine.

JoachimSchipper 9y ago

He compares - with snark - the papers' authors' 160-core cluster (etc.), as reported by those authors, with a re-implementation running on his laptop. That is, the papers get 160 cores to his 1-2, and he still wins.

lambda 9y ago

> He is limited by not having an "expensive macbook with 4 real cores" ?? He gave up on a test because it "paged out the 15GB of data". Note that he already demonstrated that for many of the tests, his older MacBook was faster than these 160 core clusters. There were a couple where the older MacBook was not faster than a 160 core cluster, but he suspects that a newer one might be. The one that paged out happened to be one where he artificially broke it up into 160 pieces to simulate what it would be like if he had 160 similar cores, but that caused enough issues with memory locality that swapping dominated the results. This artificial partitioning was to give a sense for what results on a 160 core cluster "should" be, but the swapping distorts the results enough that for that problem it wouldn't even be a very good rough estimate. > Is this guy for real? This isn't supposed to be a publication-quality paper. This is a blog post, using the resources he has immediately at hand, to show why if you follow cutting edge distributed database research you might be led down a wrong, and very costly, path, which you could avoid with a much simpler implementation on much cheaper hardware. Note that he references an earlier paper that he did publish on this topic; this blog post is mostly just doing a similar check on some more recent published work, to see if much has improved in the field: https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf https://www.usenix.org/system/files/conference/hotos15/hotos...

shoo 9y ago

> Of course at these scales a cluster is x5 slower. So this complaint should be levelled at the papers and the academic communities- why is it acceptable to only provide empirical evidence for these distributed algorithms at data scales where they cannot out perform simpler implementations? Why not require empirical performance measurement on truly large scale examples for these kind of algorithms?

wmf 9y ago
Of course academics don't have access to truly big data, so this amounts to saying that they can't publish anything. That's fine with me but I don't think the academic CS establishment will agree.
- deleted 9y ago
  [deleted]
- mcguire 9y ago
  "the academic CS establishment" Strictly speaking, I think that term attributes more organization to the academic CS establishment than my experience supports. The point here is that, if they're trying to establish that Smart Technique 2 is better than Smart Technique 1, they need to first establish that Stupid Technique Q doesn't work at all. Which they haven't. If they don't have access to "truly big data" to do so, well, most of the people who want to use the Smart Techniques probably don't either.