5 ms·

Stranger in a Strange Land: “Big Data” programmer meets HPC community

108 points by drjohnson 12y ago

agibsonccc 12y ago
Despite the marketing stuff below, it's interesting to see the big players in each space partnering, the 2 communities have intersected for a while now: http://www.cloudera.com/content/cloudera/en/solutions/partner/cloudera-and-sgi.html http://www.cloudera.com/content/cloudera/en/solutions/partne... http://www.microway.com/hpc-tech-tips/hadoop-isnt-just-for-web-2-0-big-data-anymore-hadoop-for-hpc/ http://www.microway.com/hpc-tech-tips/hadoop-isnt-just-for-w... http://www.cloudera.com/content/cloudera/en/solutions/partner/Dell.html http://www.cloudera.com/content/cloudera/en/solutions/partne... http://inside-bigdata.com/2012/10/16/bringing-hadoop-to-hpc-panasas-partners-with-hortonworks/ http://inside-bigdata.com/2012/10/16/bringing-hadoop-to-hpc-... http://hortonworks.com/partner/sgi/ http://hortonworks.com/partner/sgi/
gfody 12y ago
"Programming against an abstraction of a reliable machine" is just plain old separation of concerns. This is good engineering practice. You don't worry about low level failures in high level code, and you shouldn't introduce more high level failure modes than are necessary. Addressing failures in the appropriate layer is perfectly apt. If ECC is insufficient then revise ECC. Declaring the approach to be a fundamentally flawed is to challenge one of the pillars of good engineering.
- dang 12y ago
  This comment is a middlebrow dismissal. It dismisses the entire article with a woolly generality like "good engineering practice". Not only that, but it dismisses the entire debate that the article is about, as if it were an obvious non-issue. This article is really substantive and deserves much better.
  - gfody 12y ago
    I beg your pardon, Separation of concerns is a fundamental principle of good engineering practice and that seems to be what the debate is about. I gave a couple examples in my comment, not just a vague dismissal (and it's not like SoC needs me to defend it). OP's article made a few unsubstantiated claims about ECC and silent corruption. These are presented as reasons to abandon the entire approach of "programming to an abstraction of a perfect machine" rather than just addressing the allegedly insufficient failsafes directly.
mindcrime 12y ago
I would argue that they aren't so much "separate communities" as two sides of the same coin. And to the extent that that isn't entirely true, I'd argue that it's becoming more true over time. Of course, I'm biased, as I went to school and did a degree specifically in "HPC" back when the NSF was pouring money into the field in the mid 2000's and pushing colleges to implement degree programs on the topic. But my background prior to that, and what I mostly did career wise even after getting that degree, is largely focused around the Java / Open Source / Hadoop / etc. world. So now I build data warehouses using Hadoop and friends, while being informed by the things I learned writing tons of MPI and OpenMP code back in the day. And now that YARN has positioned Hadoop and a general purpose cluster framework, not tied specifically to Map/Reduce (you can implement MPI on top of YARN now), it starts to look more and more like these world really are just different aspects of the same thing. Where there is a noticeable schism, to me, is the point made in TFA about how some jobs are more about IO and some are more about CPU cycles. IOW, sometimes a job is "big data" because the actual amount of data is literally very large, but other times it's "big data" (or "HPC") because of the nature of the calculation being performed, even though the actual data might not be that big. I think there will always be a place for the Beowulf type clusters and MPI-CH and OpenMP and their ilk, but I am really excited about the things that you can do in Hadoop and Friends world these days. With the new found flexibility in Hadoop with YARN, and things like Hama (for BSP) and Storm / S4 (for streaming data), SAMOA & Mahout (machine learning / analytics), it's a fun place to play.
- agibsonccc 12y ago
  You bring up a really good point. Being a bit younger, I hadn't seen much of HPC. I recently attended Big Data Innovation Summit and saw a joint talk between Intel and Dell. It didn't surprise me that a lot of the stuff hadoop did wasn't really newer per se, but it was great seeing the 2 converge and understanding the differences in the philosophies. YARN as a whole is going to be a great enabler for many kinds of platforms. It's a great time to be doing distributed systems. I might be biased in that I have a few cards on the table as well in that space, but I like getting up every morning so..meh.
- greatzebu 12y ago
  I'd say that the two communities here are just people working at opposite ends of a continuum of applications that run on large clusters. The HPC community is all the way at the end with tightly coupled applications, low data to computation ratios, and diverse communication patterns. The big data community is characterized by giant data to computation ratios, highly constrained and regular communication patterns, and loose coupling. The fundamental problems are similar (fault tolerance, load balancing, scheduling) but the best approaches depend on where you are on that continuum.
DerpDerpDerp 12y ago
The title was better before the edit, since "Stranger in a Strange Land" is a work of fiction, has been borrowed in many contexts, and the additional title material clarified that it was borrowing that title to talk about an experience in a particular field. I'm always sad when titles get edited to "match the post" in a way that actually removes clarity as to what the linked material is.
- ableal 12y ago
  > since "Stranger in a Strange Land" is a work of fiction A tad older: "And she bare him a son, and he called his name Gershom: for he said, I have been a stranger in a strange land." ( Exodus 2:22, e.g. http://www.bartleby.com/108/02/2.html#S2 http://www.bartleby.com/108/02/2.html#S2 ) A good portion of the titles in English are taken either from the KJV Bible or Shakespeare, not that there is much difference.
  - __Joker 12y ago
    " not that there is much difference.". Curios, can you explain, why there is not much difference ?
    - scott_karana 12y ago
      Each corpus is dauntingly large (arcane?), making knowledge of the source difficult to determine for casual onlookers. Both have exerted tremendous influences (both overt and subtle) on traditional Western culture.
    - ableal 12y ago
      They were contemporaneous, and both pretty much hailed as the best use of English around. To my (foreign) ear, they sound similar. Some writers even go so far as fictionalizing Shakespeare sticking his finger in the KJV - I think Anthony Burgess did that.
- dang 12y ago
  Ok, I restored the "original" title. For my part, I'm always sad when a meta comment that has nothing to do with the content of the post ends up getting voted overwhelmingly to the top. It's less your fault than that of the overwhelming upvoters. But it's an unfortunate glitch that reliably mars HN discussions, and I don't know of a good answer to it. By the way, the reason we reverted the title in the first place is that the editorialized subtitle immediately provoked a bikeshed debate about the "communities" and whether they're really distinct or not—when the meat of the post is about things like fault tolerance in distributed computing.
dekhn 12y ago
I used to work on supercomputers (Cray T3E to Seaborg era) and then joined google. The difference between big data and hpc is rapidly disappearing. Many aspects that people ascribe to MR have been removed or varied to enable more HPC-like computing.

108 points by drjohnson 12y ago

Despite the marketing stuff below, it's interesting to see the big players in each space partnering, the 2 communities have intersected for a while now: http://www.cloudera.com/content/cloudera/en/solutions/partner/cloudera-and-sgi.html http://www.cloudera.com/content/cloudera/en/solutions/partne... http://www.microway.com/hpc-tech-tips/hadoop-isnt-just-for-web-2-0-big-data-anymore-hadoop-for-hpc/ http://www.microway.com/hpc-tech-tips/hadoop-isnt-just-for-w... http://www.cloudera.com/content/cloudera/en/solutions/partner/Dell.html http://www.cloudera.com/content/cloudera/en/solutions/partne... http://inside-bigdata.com/2012/10/16/bringing-hadoop-to-hpc-panasas-partners-with-hortonworks/ http://inside-bigdata.com/2012/10/16/bringing-hadoop-to-hpc-... http://hortonworks.com/partner/sgi/ http://hortonworks.com/partner/sgi/

gfody 12y ago

"Programming against an abstraction of a reliable machine" is just plain old separation of concerns. This is good engineering practice. You don't worry about low level failures in high level code, and you shouldn't introduce more high level failure modes than are necessary. Addressing failures in the appropriate layer is perfectly apt. If ECC is insufficient then revise ECC. Declaring the approach to be a fundamentally flawed is to challenge one of the pillars of good engineering.

dang 12y ago
This comment is a middlebrow dismissal. It dismisses the entire article with a woolly generality like "good engineering practice". Not only that, but it dismisses the entire debate that the article is about, as if it were an obvious non-issue. This article is really substantive and deserves much better.
- gfody 12y ago
  I beg your pardon, Separation of concerns is a fundamental principle of good engineering practice and that seems to be what the debate is about. I gave a couple examples in my comment, not just a vague dismissal (and it's not like SoC needs me to defend it). OP's article made a few unsubstantiated claims about ECC and silent corruption. These are presented as reasons to abandon the entire approach of "programming to an abstraction of a perfect machine" rather than just addressing the allegedly insufficient failsafes directly.

mindcrime 12y ago

I would argue that they aren't so much "separate communities" as two sides of the same coin. And to the extent that that isn't entirely true, I'd argue that it's becoming more true over time. Of course, I'm biased, as I went to school and did a degree specifically in "HPC" back when the NSF was pouring money into the field in the mid 2000's and pushing colleges to implement degree programs on the topic. But my background prior to that, and what I mostly did career wise even after getting that degree, is largely focused around the Java / Open Source / Hadoop / etc. world. So now I build data warehouses using Hadoop and friends, while being informed by the things I learned writing tons of MPI and OpenMP code back in the day. And now that YARN has positioned Hadoop and a general purpose cluster framework, not tied specifically to Map/Reduce (you can implement MPI on top of YARN now), it starts to look more and more like these world really are just different aspects of the same thing. Where there is a noticeable schism, to me, is the point made in TFA about how some jobs are more about IO and some are more about CPU cycles. IOW, sometimes a job is "big data" because the actual amount of data is literally very large, but other times it's "big data" (or "HPC") because of the nature of the calculation being performed, even though the actual data might not be that big. I think there will always be a place for the Beowulf type clusters and MPI-CH and OpenMP and their ilk, but I am really excited about the things that you can do in Hadoop and Friends world these days. With the new found flexibility in Hadoop with YARN, and things like Hama (for BSP) and Storm / S4 (for streaming data), SAMOA & Mahout (machine learning / analytics), it's a fun place to play.

agibsonccc 12y ago
You bring up a really good point. Being a bit younger, I hadn't seen much of HPC. I recently attended Big Data Innovation Summit and saw a joint talk between Intel and Dell. It didn't surprise me that a lot of the stuff hadoop did wasn't really newer per se, but it was great seeing the 2 converge and understanding the differences in the philosophies. YARN as a whole is going to be a great enabler for many kinds of platforms. It's a great time to be doing distributed systems. I might be biased in that I have a few cards on the table as well in that space, but I like getting up every morning so..meh.
greatzebu 12y ago
I'd say that the two communities here are just people working at opposite ends of a continuum of applications that run on large clusters. The HPC community is all the way at the end with tightly coupled applications, low data to computation ratios, and diverse communication patterns. The big data community is characterized by giant data to computation ratios, highly constrained and regular communication patterns, and loose coupling. The fundamental problems are similar (fault tolerance, load balancing, scheduling) but the best approaches depend on where you are on that continuum.

DerpDerpDerp 12y ago

The title was better before the edit, since "Stranger in a Strange Land" is a work of fiction, has been borrowed in many contexts, and the additional title material clarified that it was borrowing that title to talk about an experience in a particular field. I'm always sad when titles get edited to "match the post" in a way that actually removes clarity as to what the linked material is.

ableal 12y ago
> since "Stranger in a Strange Land" is a work of fiction A tad older: "And she bare him a son, and he called his name Gershom: for he said, I have been a stranger in a strange land." ( Exodus 2:22, e.g. http://www.bartleby.com/108/02/2.html#S2 http://www.bartleby.com/108/02/2.html#S2 ) A good portion of the titles in English are taken either from the KJV Bible or Shakespeare, not that there is much difference.
- __Joker 12y ago
  " not that there is much difference.". Curios, can you explain, why there is not much difference ?
  - scott_karana 12y ago
    Each corpus is dauntingly large (arcane?), making knowledge of the source difficult to determine for casual onlookers. Both have exerted tremendous influences (both overt and subtle) on traditional Western culture.
  - ableal 12y ago
    They were contemporaneous, and both pretty much hailed as the best use of English around. To my (foreign) ear, they sound similar. Some writers even go so far as fictionalizing Shakespeare sticking his finger in the KJV - I think Anthony Burgess did that.
dang 12y ago
Ok, I restored the "original" title. For my part, I'm always sad when a meta comment that has nothing to do with the content of the post ends up getting voted overwhelmingly to the top. It's less your fault than that of the overwhelming upvoters. But it's an unfortunate glitch that reliably mars HN discussions, and I don't know of a good answer to it. By the way, the reason we reverted the title in the first place is that the editorialized subtitle immediately provoked a bikeshed debate about the "communities" and whether they're really distinct or not—when the meat of the post is about things like fault tolerance in distributed computing.

dekhn 12y ago

I used to work on supercomputers (Cray T3E to Seaborg era) and then joined google. The difference between big data and hpc is rapidly disappearing. Many aspects that people ascribe to MR have been removed or varied to enable more HPC-like computing.