5 ms·

1.5 PB is pretty huge for most people outside of supercomputing. The biggest dataset I ever worked with fell some way short of that (It was approaching 1PB, I t

1.5 PB is pretty huge for most people outside of supercomputing. The biggest dataset I ever worked with fell some way short of that (It was approaching 1PB, I think, although we were not too sure how much data we had exactly, and it might have been slightly over, depending on how you measured). Even handling small(er) working sets could be a challenge in organisation, since we only had a small compute cluster (& associated infrastructure) to work with.

dekhn 13y ago
"Supercomputing" has never really been about big data. I used to work in supercomputing (NERSC) and have used most of the major machines in the field in the past. The supercomputer centers claim they're moving lots of data around, but it you look closely, it's almost entirely message passing during a calculation. And in most cases, the calculations they are doing are just wasted resources- simulating a protein over one long trajectory. Supercomputers, as compared to high throughput clusters, are just wasted money.
- stingraycharles 13y ago
  It's not a waste of resources, it's just a different approach to solving a problem. Hadoop / "big data" clusters make the problem harder to solve (and probably even restrict the types of problems that can be solved) in exchange for cheap hardware. Supercomputers give engineers the ability to solve problems in a traditional manner, while moving the costs over to the hardware.
  - dekhn 13y ago
    Nobody solves problems on supercomputers "in the traditional manner". You have to completely rewrite your application around message passing.
- deleted 13y ago
  [deleted]
- boomzilla 13y ago
  On the other hand, Hadoop also serves a lot more than a computing platform for "big data". The OP probably never worked in a mid/big sized company where sharing data is big issue. HDFS can easily replace a bunch of high maintenance and under performing NFSes. Actually, I think someone should write a Samba clone for HDFS. The next huge thing in the Hadoop ecosystem is Hive. It's like having an unlimited Postgres server that hundreds of people can work on at the same time. Hive's got its quirks and developers need to be educated on how to write the Hive jobs properly, but it gives a lot more back by allowing joins between a lot of datasets that might not be able to fit in one single machine. There is a reason why Hadoop is so popular, especially in enterprises. I think that's why Cloudera got such a high valuation.
  - epistasis 13y ago
    There's a connector between Hadoop and the Gluster network file systems [1], giving a unified view of a large, distributed, POSIX compatible filesystem, which can be served up through Samba. We vastly prefer it to NFS, which we find to be far more flaky, slower, and when NFS fails it fails in really really painful ways. (Though looking again, I'm not sure if you meant NFS or NAS...) [1] http://gluster.org/community/documentation/index.php/Hadoop http://gluster.org/community/documentation/index.php/Hadoop
- davidmr 13y ago
  I wrote a different reply but deleted it. I'm curious about something: if, as you claim, supercomputers are wasted money, why are there so many of them? Have all the world's top supercomputing sites somehow colluded to convince all the world's largest governments that they're useful?
  - stingraycharles 13y ago
    I'm not disagreeing with you, but the argument of "people are using it and the government keeps paying for them" is hardly convincing.
    - davidmr 13y ago
      It's not just the government that buy these things. Oil and gas, heavy manufacturing, automotive, genomics, financial, etc. all have them. The poster I was replying to appeared to be under the gross misconception that all parallel problems were more effectively solved with high throughput clusters.
      - dekhn 13y ago
        Can you point to a genomics company that uses a supercomputer? Adn really claims that its unique capabilities are necessary> Remember, the human genome was assembled on two different architectures: a Dec Alpha cluster with large memory with fast network (for the day) and a bunch of cheap linux machines with slow network. It turns out you didn't actually need the former, although the people at Celera insisted you did.
        jerven 13y ago
        Genetics is normally very very data parallel. And often the ideal case for throughput computing. But there are rare times when doing modelling that a lot of chatter is required over interconnects. But even here people normally just get 4-6TB RAM machines instead of having lots of machines.

dekhn 13y ago

"Supercomputing" has never really been about big data. I used to work in supercomputing (NERSC) and have used most of the major machines in the field in the past. The supercomputer centers claim they're moving lots of data around, but it you look closely, it's almost entirely message passing during a calculation. And in most cases, the calculations they are doing are just wasted resources- simulating a protein over one long trajectory. Supercomputers, as compared to high throughput clusters, are just wasted money.

stingraycharles 13y ago
It's not a waste of resources, it's just a different approach to solving a problem. Hadoop / "big data" clusters make the problem harder to solve (and probably even restrict the types of problems that can be solved) in exchange for cheap hardware. Supercomputers give engineers the ability to solve problems in a traditional manner, while moving the costs over to the hardware.
- dekhn 13y ago
  Nobody solves problems on supercomputers "in the traditional manner". You have to completely rewrite your application around message passing.
deleted 13y ago
[deleted]
boomzilla 13y ago
On the other hand, Hadoop also serves a lot more than a computing platform for "big data". The OP probably never worked in a mid/big sized company where sharing data is big issue. HDFS can easily replace a bunch of high maintenance and under performing NFSes. Actually, I think someone should write a Samba clone for HDFS. The next huge thing in the Hadoop ecosystem is Hive. It's like having an unlimited Postgres server that hundreds of people can work on at the same time. Hive's got its quirks and developers need to be educated on how to write the Hive jobs properly, but it gives a lot more back by allowing joins between a lot of datasets that might not be able to fit in one single machine. There is a reason why Hadoop is so popular, especially in enterprises. I think that's why Cloudera got such a high valuation.
- epistasis 13y ago
  There's a connector between Hadoop and the Gluster network file systems [1], giving a unified view of a large, distributed, POSIX compatible filesystem, which can be served up through Samba. We vastly prefer it to NFS, which we find to be far more flaky, slower, and when NFS fails it fails in really really painful ways. (Though looking again, I'm not sure if you meant NFS or NAS...) [1] http://gluster.org/community/documentation/index.php/Hadoop http://gluster.org/community/documentation/index.php/Hadoop
davidmr 13y ago
I wrote a different reply but deleted it. I'm curious about something: if, as you claim, supercomputers are wasted money, why are there so many of them? Have all the world's top supercomputing sites somehow colluded to convince all the world's largest governments that they're useful?
- stingraycharles 13y ago
  I'm not disagreeing with you, but the argument of "people are using it and the government keeps paying for them" is hardly convincing.
  - davidmr 13y ago
    It's not just the government that buy these things. Oil and gas, heavy manufacturing, automotive, genomics, financial, etc. all have them. The poster I was replying to appeared to be under the gross misconception that all parallel problems were more effectively solved with high throughput clusters.
    - dekhn 13y ago
      Can you point to a genomics company that uses a supercomputer? Adn really claims that its unique capabilities are necessary> Remember, the human genome was assembled on two different architectures: a Dec Alpha cluster with large memory with fast network (for the day) and a bunch of cheap linux machines with slow network. It turns out you didn't actually need the former, although the people at Celera insisted you did.
      - jerven 13y ago
        Genetics is normally very very data parallel. And often the ideal case for throughput computing. But there are rare times when doing modelling that a lot of chatter is required over interconnects. But even here people normally just get 4-6TB RAM machines instead of having lots of machines.