6 ms·

Bigger data; same laptop

80 points by mrry 12y ago

eklavya 12y ago
All that investment and involvement, those big systems have got to be good for somethings! Please can anybody knowledgable share some insights? mrry?
- jkot 12y ago
  It is one thing to run short benchmark on laptop. Other thing to sustain load 24/7 with demand spikes and hw failures.
  - mrry 12y ago
    This is absolutely true! However, I'm not aware of anybody doing this for the graph processing systems being discussed in the post. Nor am I sure what that would mean. Admittedly if something takes 100 times longer to run on multiple machines than it does on a single laptop (as in the previous COST post [1]) then you might care about fault tolerance, but this is a contrived problem if you could achieve the same results faster on yor laptop. [1] http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...
    - jkot 12y ago
      I was answering parent questions. There are usually good reasons for clustering and big iron.
  - angersock 12y ago
    In the spirit of the article: If you can write it so that it runs in 1/100th the time, do you even have to worry about it running for 24/7?
  - vardump 12y ago
    So true. If you can't accept the risk of random bit flip in memory, laptops are only good for using a real computer remotely. Memory errors do happen all too often on laptops. Have had it happening in the middle of a AES CBC encrypted data file... How hard would it be to get ECC RAM on laptops for better reliability?
  - felixgallo 12y ago
    can't help but feel that maybe you missed the point. The point is that an inexpensive single computer, programmed properly, can be seen to blow away gigantic, extremely expensive "big data"-programmed clusters. if you wanted to run this same algorithm on a 'real' single computer with ecc and high-performance cores, nothing would stop you, and it would get even faster, and still be radically less expensive in every dimension than the "big data" clusters, for this problem set.
    - vardump 12y ago
      Didn't miss the point at all. Lamenting lack of reliable laptops. Well, at least with reliable memory. You can buy a laptop with ECC RAM in the display adapter, but not as main memory! Sigh. Luckily networking is getting so good that remote desktops feel almost like local most of the time.
- bane 12y ago
  If you think of the network as part of the memory hierarchy, transferring data to nodes in a distributed system is usually slower than even spinning disk storage. About the only thing slower are USB, floppy drives and tape. So basically big systems only make sense if the computation time on a single system exceeds the transfer time off of that system to the remote nodes + the time for that data to work it's way up the local node memory hierarchy. This can happen when the data is big and the computation has unavoidable serialism in each part of the computation. Otherwise you'r likely to be able to coax a single system to just do it faster and with far less complexity than working on a compute cluster. (I'm ignoring big systems with fast shared memory, like classic super-computers, where there effectively is zero transfer time since each node uses the same memory pool)
  - eklavya 12y ago
    Makes sense, thanks :)
  - e12e 12y ago
    > If you think of the network as part of the memory hierarchy, transferring data to nodes in a distributed system is usually slower than even spinning disk storage. According to Google quick search, the bandwidth of DDR3 is ~12.8 GB/s, or ~104 gbps. As I understand it second-hand (read: cheap) infiniband givies ~40 gpbs, and newer, high-end gives ~100gbps[1]. Now, if you're using regular gigabit ethernet (or even 10gps with cheap-ish switches) -- you'd probably see around 5-8 gbps -- or if we're being very generous, on the order of 1/10th of cpu-ram bandwidth. Sata2 is rated at ~6 gbps for comparison. In other words, 10gpbs ethernet/infiniband should be on-par with local SSDs, but "proper" infiniband should push the node-node ram-ram performance up compared to that. It appears fiberchannel is also moving towards the ~100 gps mark[2], but isn't there quite yet. And you'd have to have something on the other end -- with high speed networking it is quite natural to assume that both source and destination is RAM. It's actually pretty crazy with ~40gps infiniband -- I'm considering getting some second-hand gear just to play with at home. Seems both cheaper and more reliable to work with than try to get 10gps ethernet to work (on a home/hobby budget!). Eg: http://www.ebay.com/itm/Flextronics-F-X430066-8-Port-SDR-InfiniBand-External-Switch-/181643992730 http://www.ebay.com/itm/Flextronics-F-X430066-8-Port-SDR-Inf... Or: http://www.ebay.com/itm/Mellanox-MHGA28-XTC-InfiniHost-III-Ex-20Gb-Dual-Port-InfiniBand-Adapter-card-/151577338944 http://www.ebay.com/itm/Mellanox-MHGA28-XTC-InfiniHost-III-E... All that said, I think the article makes a lot of great points -- and it does seem a bit funny that people seemingly are thinking of 1 gbps as "fast" when so much has happened on the cpu/ram side since that was remotely true. [1] http://www.hpcwire.com/2012/06/18/mellanox_cracks_100_gbps_with_new_infiniband_adapters/ http://www.hpcwire.com/2012/06/18/mellanox_cracks_100_gbps_w... [2] http://www.pcworld.com/article/2096980/fibre-channel-will-come-with-32gigabit-128gigabit-speeds-in-2016.html http://www.pcworld.com/article/2096980/fibre-channel-will-co...
- harperlee 12y ago
  Please understand that some of us are not native speakers and try really hard to understand your comments. Particularily in this place, making the effort seems to be obliged, as most of the community produce high-quality comments. What you want to convey with "mmry" is very opaque. I am still not sure of what you mean with it (I believe is "Am I right", but the reference is lost with me, and I don't think I'm the only one).
  - ShonM 12y ago
    In this case it's OPs username.
    - harperlee 12y ago
      Well now I feel a little bit embarrased, I was harsh without need. Also, thanks for pointing that out, it's better to know than to be treated with silence!
      - eklavya 12y ago
        Don't be, it was easy to miss :) I should have added an @
- mrry 12y ago
  Our point in the COST paper was neither that distributed computation systems are bad per se, nor that a laptop is a fine substitute for many of the real-world deployments of distributed systems. What these posts rail against is a tendency (post-MapReduce/Hadoop) to motivate and evaluate distributed systems solely in terms of their batch performance. This is a particularly common problem in graph processing research, where 2008-era general-purpose systems (i.e. Hadoop) have terrible performance, and so provide a straw-man baseline that a reasonably competent grad student should have had no difficulty in outperforming. "Does my new system outperform a single thread?" seems like a proper hypothesis to test for a distributed system that claims only a performance motivation. Instead our grad student finds that there is a long list of peer-reviewed systems to compare against, and doing that should be enough, right? It would be fine if those systems had been tested against the first hypothesis, but in most cases they have not, so the new system can be much worse than a single thread (for a small set of benchmarks) and this will go undetected. The COST work is a remedy for this issue. To return to your original question, I do think that distributed systems are good for some things. First, the OP concerns running graph algorithms on genuinely big data: 128 billion edges, and 3.5 billion vertices. A distributed system should be able to outperform Frank's laptop on that graph, and hopefully this post will encourage developers of new systems to aim higher and evaluate their systems on this scale of data. The second is that distributed systems offer far more than pure batch performance. For example, geo-distribution is critical to providing a good user experience in web properties; and serving interactive queries with low latency and high availability is a particular interest of mine. We touched on the first part in the Naiad paper [1], and I expect this will become especially important for inference in online learning systems [2]. Hopefully the COST work will force the authors of these systems to try harder in their implementation, and we will all benefit as a result. [1] http://dl.acm.org/citation.cfm?id=2522738 http://dl.acm.org/citation.cfm?id=2522738 [2] http://www.cidrdb.org/cidr2015/Papers/CIDR15_Paper19u.pdf http://www.cidrdb.org/cidr2015/Papers/CIDR15_Paper19u.pdf
  - bmh100 12y ago
    When you explain the purpose of COST in this way, it is insightful and the value becomes very apparent. Noting that so many systems have not been compared against a single thread is important, since many developers likely would assume that. Do you feel that your COST work helps provide scalability break points where a developer could choose one system over another when performance is the primary concern? I can see that being helpful in the prototype and early scaling stages, as well as for ad-hoc work.
bane 12y ago
I think we're starting to see the pendulum swing back towards caring about performance. For years we could just throw more hardware at a problem, and now performance curves are starting to flatten out. The bandaid has been to use more systems, but old-timers are starting to remember back to their youth, when systems were slow, that there used to be a phase in software development called "performance optimization" that's almost entirely disappeared in the modern discipline. The amount of personal computing power every person with a modern computer has at their fingertips is staggering and almost nobody has any idea how to use it. I'd like to think that's starting to change.
- gambiting 12y ago
  As someone who has recently obtained a Masters degree in Computing science and who has few friends doing PhDs in big data analysis - I don't see that trend. All the latest research techniques that I see being used currently concentrate at throwing more hardware at the problem if it allows the researches to express their ideas easier - I have a friend who writes algorithms for processing 300GB graphs in Java. And basically the idea is that it really doesn't matter if the algorithm takes 40 hours because 90% of the time is spent garbage collecting. This is still preferable to writing much faster program in C, and spending weeks trying to do memory optimizations and playing around with pointer logic. If there is a need, then Uni will gladly pay for any requested capacity on an Amazon cluster, it's part of the Russel group so they have nearly unlimited budget for research. I am sure that's not the case everywhere, but in my experience researchers gladly trade raw performance for ease of use.
  - srean 12y ago
    To see that performance matters, talk to the guys who are closer to the electricity bills that the big honking clusters demand. There is quite a bit of money to be saved by using hardware effectively. BTW if 90% time is spent in garbage collection, something is definitely wrong in the choice of tools or methods.
    - gambiting 12y ago
vardump 12y ago
First you should optimize performance locally. By choosing an algorithm and implementation that gives you the best results at data size n you need to deal with most. Lowest O complexity algorithm is not necessarily the best. Worst cases and especially pathological cases also matter. Then you need to deal with distribution between a large number of systems over network... This will likely force you to choose a different algorithm, but at least you'll have something the benchmark it against from local best case scenario. When it comes to graphs, I think good partitioning is very important in order to minimize average case latency. So it'd greatly help if you can somehow partition the data to minimize the number of nodes referring to a graph node on a remote system. When walking the graph, instead of immediately sending a query to the target system, it's probably better to batch the queries. Or not if latency counts more. Either way, latency will hurt badly. On a local system it's 100 ns. When the data is on a remote system, you're probably talking about 1000000 ns. To combat latency, maybe the best way is not to send the request in the first place? Maybe you can ensure nodes referring to a remote system are duplicated to some extent on multiple remote systems. It could help performance -- or hurt it more, like when you need to modify the nodes. Maybe you should have counters on nodes with connections to nodes on remote systems. If some remote system is referring to it more than local system, migrate the node there instead. It's of course not hard to see how this strategy could blow up. :-) Please don't take above ideas seriously if you're developing a graph processing system. I wrote those to illustrate the biggest basic problem of distributed systems, latency. Distributed systems are very hard.
deleted 12y ago
[deleted]
andor 12y ago
If you're wondering where the actual comparison is, open the link to "the previous post" in the first paragraph.
- frankmcsherry 12y ago
  That's a good point, thanks. I've added the timings from the original post to the tables in this post, which also makes clearer the intended point: the existing systems don't yet have measurements for data at this scale (except Facebook's modified Giraph, which I want to hear more about :))
TheLoneWolfling 12y ago
> it wears out my SSD No it doesn't. Reading doesn't cause (much) wear on an SSD - writing does. (To be technical - it does, but it only requires 1 write / 100,000 reads or so. - https://en.wikipedia.org/wiki/Flash_memory#Read_disturb https://en.wikipedia.org/wiki/Flash_memory#Read_disturb )
- frankmcsherry 12y ago
  This probably isn't crystal clear from this post alone, but for PageRank the 42GB working set has a good mix of read and write. Specifically, about 14GB of target rank vector gets +='d all over the place. If they got hammered randomly, then lots of paging and writing would happen.
  - TheLoneWolfling 12y ago
    Oh, ok. Nevermind then.

80 points by mrry 12y ago

All that investment and involvement, those big systems have got to be good for somethings! Please can anybody knowledgable share some insights? mrry?

jkot 12y ago
It is one thing to run short benchmark on laptop. Other thing to sustain load 24/7 with demand spikes and hw failures.
- mrry 12y ago
  This is absolutely true! However, I'm not aware of anybody doing this for the graph processing systems being discussed in the post. Nor am I sure what that would mean. Admittedly if something takes 100 times longer to run on multiple machines than it does on a single laptop (as in the previous COST post [1]) then you might care about fault tolerance, but this is a contrived problem if you could achieve the same results faster on yor laptop. [1] http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...
  - jkot 12y ago
    I was answering parent questions. There are usually good reasons for clustering and big iron.
- angersock 12y ago
  In the spirit of the article: If you can write it so that it runs in 1/100th the time, do you even have to worry about it running for 24/7?
- vardump 12y ago
  So true. If you can't accept the risk of random bit flip in memory, laptops are only good for using a real computer remotely. Memory errors do happen all too often on laptops. Have had it happening in the middle of a AES CBC encrypted data file... How hard would it be to get ECC RAM on laptops for better reliability?
- felixgallo 12y ago
  can't help but feel that maybe you missed the point. The point is that an inexpensive single computer, programmed properly, can be seen to blow away gigantic, extremely expensive "big data"-programmed clusters. if you wanted to run this same algorithm on a 'real' single computer with ecc and high-performance cores, nothing would stop you, and it would get even faster, and still be radically less expensive in every dimension than the "big data" clusters, for this problem set.
  - vardump 12y ago
    Didn't miss the point at all. Lamenting lack of reliable laptops. Well, at least with reliable memory. You can buy a laptop with ECC RAM in the display adapter, but not as main memory! Sigh. Luckily networking is getting so good that remote desktops feel almost like local most of the time.
bane 12y ago
If you think of the network as part of the memory hierarchy, transferring data to nodes in a distributed system is usually slower than even spinning disk storage. About the only thing slower are USB, floppy drives and tape. So basically big systems only make sense if the computation time on a single system exceeds the transfer time off of that system to the remote nodes + the time for that data to work it's way up the local node memory hierarchy. This can happen when the data is big and the computation has unavoidable serialism in each part of the computation. Otherwise you'r likely to be able to coax a single system to just do it faster and with far less complexity than working on a compute cluster. (I'm ignoring big systems with fast shared memory, like classic super-computers, where there effectively is zero transfer time since each node uses the same memory pool)
- eklavya 12y ago
  Makes sense, thanks :)
- e12e 12y ago
  > If you think of the network as part of the memory hierarchy, transferring data to nodes in a distributed system is usually slower than even spinning disk storage. According to Google quick search, the bandwidth of DDR3 is ~12.8 GB/s, or ~104 gbps. As I understand it second-hand (read: cheap) infiniband givies ~40 gpbs, and newer, high-end gives ~100gbps[1]. Now, if you're using regular gigabit ethernet (or even 10gps with cheap-ish switches) -- you'd probably see around 5-8 gbps -- or if we're being very generous, on the order of 1/10th of cpu-ram bandwidth. Sata2 is rated at ~6 gbps for comparison. In other words, 10gpbs ethernet/infiniband should be on-par with local SSDs, but "proper" infiniband should push the node-node ram-ram performance up compared to that. It appears fiberchannel is also moving towards the ~100 gps mark[2], but isn't there quite yet. And you'd have to have something on the other end -- with high speed networking it is quite natural to assume that both source and destination is RAM. It's actually pretty crazy with ~40gps infiniband -- I'm considering getting some second-hand gear just to play with at home. Seems both cheaper and more reliable to work with than try to get 10gps ethernet to work (on a home/hobby budget!). Eg: http://www.ebay.com/itm/Flextronics-F-X430066-8-Port-SDR-InfiniBand-External-Switch-/181643992730 http://www.ebay.com/itm/Flextronics-F-X430066-8-Port-SDR-Inf... Or: http://www.ebay.com/itm/Mellanox-MHGA28-XTC-InfiniHost-III-Ex-20Gb-Dual-Port-InfiniBand-Adapter-card-/151577338944 http://www.ebay.com/itm/Mellanox-MHGA28-XTC-InfiniHost-III-E... All that said, I think the article makes a lot of great points -- and it does seem a bit funny that people seemingly are thinking of 1 gbps as "fast" when so much has happened on the cpu/ram side since that was remotely true. [1] http://www.hpcwire.com/2012/06/18/mellanox_cracks_100_gbps_with_new_infiniband_adapters/ http://www.hpcwire.com/2012/06/18/mellanox_cracks_100_gbps_w... [2] http://www.pcworld.com/article/2096980/fibre-channel-will-come-with-32gigabit-128gigabit-speeds-in-2016.html http://www.pcworld.com/article/2096980/fibre-channel-will-co...
harperlee 12y ago
Please understand that some of us are not native speakers and try really hard to understand your comments. Particularily in this place, making the effort seems to be obliged, as most of the community produce high-quality comments. What you want to convey with "mmry" is very opaque. I am still not sure of what you mean with it (I believe is "Am I right", but the reference is lost with me, and I don't think I'm the only one).
- ShonM 12y ago
  In this case it's OPs username.
  - harperlee 12y ago
    Well now I feel a little bit embarrased, I was harsh without need. Also, thanks for pointing that out, it's better to know than to be treated with silence!
    - eklavya 12y ago
      Don't be, it was easy to miss :) I should have added an @
mrry 12y ago
Our point in the COST paper was neither that distributed computation systems are bad per se, nor that a laptop is a fine substitute for many of the real-world deployments of distributed systems. What these posts rail against is a tendency (post-MapReduce/Hadoop) to motivate and evaluate distributed systems solely in terms of their batch performance. This is a particularly common problem in graph processing research, where 2008-era general-purpose systems (i.e. Hadoop) have terrible performance, and so provide a straw-man baseline that a reasonably competent grad student should have had no difficulty in outperforming. "Does my new system outperform a single thread?" seems like a proper hypothesis to test for a distributed system that claims only a performance motivation. Instead our grad student finds that there is a long list of peer-reviewed systems to compare against, and doing that should be enough, right? It would be fine if those systems had been tested against the first hypothesis, but in most cases they have not, so the new system can be much worse than a single thread (for a small set of benchmarks) and this will go undetected. The COST work is a remedy for this issue. To return to your original question, I do think that distributed systems are good for some things. First, the OP concerns running graph algorithms on genuinely big data: 128 billion edges, and 3.5 billion vertices. A distributed system should be able to outperform Frank's laptop on that graph, and hopefully this post will encourage developers of new systems to aim higher and evaluate their systems on this scale of data. The second is that distributed systems offer far more than pure batch performance. For example, geo-distribution is critical to providing a good user experience in web properties; and serving interactive queries with low latency and high availability is a particular interest of mine. We touched on the first part in the Naiad paper [1], and I expect this will become especially important for inference in online learning systems [2]. Hopefully the COST work will force the authors of these systems to try harder in their implementation, and we will all benefit as a result. [1] http://dl.acm.org/citation.cfm?id=2522738 http://dl.acm.org/citation.cfm?id=2522738 [2] http://www.cidrdb.org/cidr2015/Papers/CIDR15_Paper19u.pdf http://www.cidrdb.org/cidr2015/Papers/CIDR15_Paper19u.pdf
- bmh100 12y ago
  When you explain the purpose of COST in this way, it is insightful and the value becomes very apparent. Noting that so many systems have not been compared against a single thread is important, since many developers likely would assume that. Do you feel that your COST work helps provide scalability break points where a developer could choose one system over another when performance is the primary concern? I can see that being helpful in the prototype and early scaling stages, as well as for ad-hoc work.

bane 12y ago

I think we're starting to see the pendulum swing back towards caring about performance. For years we could just throw more hardware at a problem, and now performance curves are starting to flatten out. The bandaid has been to use more systems, but old-timers are starting to remember back to their youth, when systems were slow, that there used to be a phase in software development called "performance optimization" that's almost entirely disappeared in the modern discipline. The amount of personal computing power every person with a modern computer has at their fingertips is staggering and almost nobody has any idea how to use it. I'd like to think that's starting to change.

gambiting 12y ago
As someone who has recently obtained a Masters degree in Computing science and who has few friends doing PhDs in big data analysis - I don't see that trend. All the latest research techniques that I see being used currently concentrate at throwing more hardware at the problem if it allows the researches to express their ideas easier - I have a friend who writes algorithms for processing 300GB graphs in Java. And basically the idea is that it really doesn't matter if the algorithm takes 40 hours because 90% of the time is spent garbage collecting. This is still preferable to writing much faster program in C, and spending weeks trying to do memory optimizations and playing around with pointer logic. If there is a need, then Uni will gladly pay for any requested capacity on an Amazon cluster, it's part of the Russel group so they have nearly unlimited budget for research. I am sure that's not the case everywhere, but in my experience researchers gladly trade raw performance for ease of use.
- srean 12y ago
  To see that performance matters, talk to the guys who are closer to the electricity bills that the big honking clusters demand. There is quite a bit of money to be saved by using hardware effectively. BTW if 90% time is spent in garbage collection, something is definitely wrong in the choice of tools or methods.
  - gambiting 12y ago

vardump 12y ago

First you should optimize performance locally. By choosing an algorithm and implementation that gives you the best results at data size n you need to deal with most. Lowest O complexity algorithm is not necessarily the best. Worst cases and especially pathological cases also matter. Then you need to deal with distribution between a large number of systems over network... This will likely force you to choose a different algorithm, but at least you'll have something the benchmark it against from local best case scenario. When it comes to graphs, I think good partitioning is very important in order to minimize average case latency. So it'd greatly help if you can somehow partition the data to minimize the number of nodes referring to a graph node on a remote system. When walking the graph, instead of immediately sending a query to the target system, it's probably better to batch the queries. Or not if latency counts more. Either way, latency will hurt badly. On a local system it's 100 ns. When the data is on a remote system, you're probably talking about 1000000 ns. To combat latency, maybe the best way is not to send the request in the first place? Maybe you can ensure nodes referring to a remote system are duplicated to some extent on multiple remote systems. It could help performance -- or hurt it more, like when you need to modify the nodes. Maybe you should have counters on nodes with connections to nodes on remote systems. If some remote system is referring to it more than local system, migrate the node there instead. It's of course not hard to see how this strategy could blow up. :-) Please don't take above ideas seriously if you're developing a graph processing system. I wrote those to illustrate the biggest basic problem of distributed systems, latency. Distributed systems are very hard.

deleted 12y ago

[deleted]

andor 12y ago

If you're wondering where the actual comparison is, open the link to "the previous post" in the first paragraph.

frankmcsherry 12y ago
That's a good point, thanks. I've added the timings from the original post to the tables in this post, which also makes clearer the intended point: the existing systems don't yet have measurements for data at this scale (except Facebook's modified Giraph, which I want to hear more about :))

TheLoneWolfling 12y ago

> it wears out my SSD No it doesn't. Reading doesn't cause (much) wear on an SSD - writing does. (To be technical - it does, but it only requires 1 write / 100,000 reads or so. - https://en.wikipedia.org/wiki/Flash_memory#Read_disturb https://en.wikipedia.org/wiki/Flash_memory#Read_disturb )

frankmcsherry 12y ago
This probably isn't crystal clear from this post alone, but for PageRank the 42GB working set has a good mix of read and write. Specifically, about 14GB of target rank vector gets +='d all over the place. If they got hammered randomly, then lots of paging and writing would happen.
- TheLoneWolfling 12y ago
  Oh, ok. Nevermind then.