5 ms·
Hadoop will not penetrate HPC. I'll say it again: Hadoop will not penetrate HPC. The reasons why vary quite a bit. But the core reasons are this: Data is ra
by batbomb 12y ago
Hadoop will not penetrate HPC.
I'll say it again:
Hadoop will not penetrate HPC.
The reasons why vary quite a bit. But the core reasons are this:
Data is rarely local.
Resources are shared.
Users are heterogenous, even if the cluster isn't.
Users don't typically have CS degrees or understand more than basic networking (Computational Chemists and biologists tend to be a but more advanced)
Users still heavily rely on Fortran.
Users rarely know much about SQL, so even Hive is often out of the question.
Clusters are rarely under the control of users. On top of that, Users are rarely able to launch VMs.
Lots of other reasons too.
- seanmcdirmid 12y agoAnd yet, CUDA is currently dominating in many HPC applications. There are reasons Hadoop/MapReduce is a poor fit for HPC, but your argument points above aren't very universal.
- KaiserPro 12y agoIn HPC they are. CUDA wins because it is wonderfully fast. There is a huge benifit for implementing it, even if it requires lots of work to get efficient MPI, because the rewards are literally massive. Hadoop uses REST ffs. You're never going to get low latency with REST. Its a nasty hack in the world of web, let alone HPC where there really isn't any excuse for using HTTP for something its really not designed to do. But the question is this: what do people actually use Hadoop for?
- growse 12y agoWhat do people use Hadoop for? Are you asking because you genuinely don't know? Hadoop lets me pick up PBs of (typically) record-format data and stick it on a bunch of cheap servers that are already standard items in my datacenter catalogue and are well-understood by my datacenter/ops teams. I can then open up any Java IDE, write a program now I can process that data. Up until this point, I've not really needed much specialist knowledge, perhaps with the exception of understanding the map/reduce paradigm which is a fairly basic programming construct anyway. There's lots of companies out there who have data problems that are bigger than a few TB, and they're bored of shovelling money at Oracle so they can have a fancy, expensive database when their data isn't actually relational to begin with.
- KaiserPro 12y agoThats the thing, So I get that it gives you network aware Fork() Its just the scheduling is a bit, well, lacking. In my datacenter, we run dirty cheap iron. The only thing special are the core switches(lots and lots of ten gig ports, to cope with the petabytes that we shove across it daily) we use standard linux tools, sharded posix file systems with a decent task dispatcher that understands dependencies.