9 ms·

"Big Data starts at 1.5 PB [because that's what fits in memory on Blue Waters]" -- Bill Gropp On today's top HPC installations, it currently takes about an hou

"Big Data starts at 1.5 PB [because that's what fits in memory on Blue Waters]" -- Bill Gropp On today's top HPC installations, it currently takes about an hour to read or write the contents of memory from/to global storage. If the workflow has broad dependencies (PDEs are especially bad, but lots of network analysis also fits the bill), it's much better to use more parallelism to run for a shorter period of time with the working set in memory. If you can't fit your working set in memory on the largest machine available, chances are you should either find time on a bigger machine or you can't afford to do the analysis. (The largest scientific allocations are in the 10s of millions of core hours per year, which would be burned in a few reads and writes on a million-core machine.) Also note that MapReduce is not very expressive. When IBM's Watson team was getting started, some people suggested using MapReduce/Hadoop, but the team quickly concluded it was way too slow/constraining and instead opted for an in-memory database with MPI for communication.

w_t_payne 13y ago
1.5 PB is pretty huge for most people outside of supercomputing. The biggest dataset I ever worked with fell some way short of that (It was approaching 1PB, I think, although we were not too sure how much data we had exactly, and it might have been slightly over, depending on how you measured). Even handling small(er) working sets could be a challenge in organisation, since we only had a small compute cluster (& associated infrastructure) to work with.
- dekhn 13y ago
  "Supercomputing" has never really been about big data. I used to work in supercomputing (NERSC) and have used most of the major machines in the field in the past. The supercomputer centers claim they're moving lots of data around, but it you look closely, it's almost entirely message passing during a calculation. And in most cases, the calculations they are doing are just wasted resources- simulating a protein over one long trajectory. Supercomputers, as compared to high throughput clusters, are just wasted money.
  - stingraycharles 13y ago
    It's not a waste of resources, it's just a different approach to solving a problem. Hadoop / "big data" clusters make the problem harder to solve (and probably even restrict the types of problems that can be solved) in exchange for cheap hardware. Supercomputers give engineers the ability to solve problems in a traditional manner, while moving the costs over to the hardware.
    - dekhn 13y ago
      Nobody solves problems on supercomputers "in the traditional manner". You have to completely rewrite your application around message passing.
  - deleted 13y ago
    [deleted]
  - boomzilla 13y ago
    On the other hand, Hadoop also serves a lot more than a computing platform for "big data". The OP probably never worked in a mid/big sized company where sharing data is big issue. HDFS can easily replace a bunch of high maintenance and under performing NFSes. Actually, I think someone should write a Samba clone for HDFS. The next huge thing in the Hadoop ecosystem is Hive. It's like having an unlimited Postgres server that hundreds of people can work on at the same time. Hive's got its quirks and developers need to be educated on how to write the Hive jobs properly, but it gives a lot more back by allowing joins between a lot of datasets that might not be able to fit in one single machine. There is a reason why Hadoop is so popular, especially in enterprises. I think that's why Cloudera got such a high valuation.
    - epistasis 13y ago
      There's a connector between Hadoop and the Gluster network file systems [1], giving a unified view of a large, distributed, POSIX compatible filesystem, which can be served up through Samba. We vastly prefer it to NFS, which we find to be far more flaky, slower, and when NFS fails it fails in really really painful ways. (Though looking again, I'm not sure if you meant NFS or NAS...) [1] http://gluster.org/community/documentation/index.php/Hadoop http://gluster.org/community/documentation/index.php/Hadoop
frozenport 13y ago
PDEs are kind compared to graphic codes. Molecular dynamics problems that are characterized by PDEs are some of the largest scaling codes - ever.

w_t_payne 13y ago

1.5 PB is pretty huge for most people outside of supercomputing. The biggest dataset I ever worked with fell some way short of that (It was approaching 1PB, I think, although we were not too sure how much data we had exactly, and it might have been slightly over, depending on how you measured). Even handling small(er) working sets could be a challenge in organisation, since we only had a small compute cluster (& associated infrastructure) to work with.

dekhn 13y ago
"Supercomputing" has never really been about big data. I used to work in supercomputing (NERSC) and have used most of the major machines in the field in the past. The supercomputer centers claim they're moving lots of data around, but it you look closely, it's almost entirely message passing during a calculation. And in most cases, the calculations they are doing are just wasted resources- simulating a protein over one long trajectory. Supercomputers, as compared to high throughput clusters, are just wasted money.
- stingraycharles 13y ago
  It's not a waste of resources, it's just a different approach to solving a problem. Hadoop / "big data" clusters make the problem harder to solve (and probably even restrict the types of problems that can be solved) in exchange for cheap hardware. Supercomputers give engineers the ability to solve problems in a traditional manner, while moving the costs over to the hardware.
  - dekhn 13y ago
    Nobody solves problems on supercomputers "in the traditional manner". You have to completely rewrite your application around message passing.
- deleted 13y ago
  [deleted]
- boomzilla 13y ago
  On the other hand, Hadoop also serves a lot more than a computing platform for "big data". The OP probably never worked in a mid/big sized company where sharing data is big issue. HDFS can easily replace a bunch of high maintenance and under performing NFSes. Actually, I think someone should write a Samba clone for HDFS. The next huge thing in the Hadoop ecosystem is Hive. It's like having an unlimited Postgres server that hundreds of people can work on at the same time. Hive's got its quirks and developers need to be educated on how to write the Hive jobs properly, but it gives a lot more back by allowing joins between a lot of datasets that might not be able to fit in one single machine. There is a reason why Hadoop is so popular, especially in enterprises. I think that's why Cloudera got such a high valuation.
  - epistasis 13y ago
    There's a connector between Hadoop and the Gluster network file systems [1], giving a unified view of a large, distributed, POSIX compatible filesystem, which can be served up through Samba. We vastly prefer it to NFS, which we find to be far more flaky, slower, and when NFS fails it fails in really really painful ways. (Though looking again, I'm not sure if you meant NFS or NAS...) [1] http://gluster.org/community/documentation/index.php/Hadoop http://gluster.org/community/documentation/index.php/Hadoop

frozenport 13y ago

PDEs are kind compared to graphic codes. Molecular dynamics problems that are characterized by PDEs are some of the largest scaling codes - ever.