4 ms·
As someone who has straddled the world of HPC and Big Data for years, this is a pretty solid summary of the issues. From my perspective, both sides could learn
by jandrewrogers 13y ago
As someone who has straddled the world of HPC and Big Data for years, this is a pretty solid summary of the issues. From my perspective, both sides could learn quite a bit from the other but there is little cross-fertilization.
Basically, Big Data is much more sophisticated at distributed systems computer science and HPC is much more sophisticated at massive parallelism algorithms computer science. HPC tends to throw money at hardware to solve distributed computing problems, and Big Data is still very limited when it comes to parallel algorithm design techniques because their sharding models are incredibly primitive (hash and range only? no thanks) partly because of their original use cases.
In my perfect world, a good platform would combine the areas of sophistication from both but people in one domain tend to lack the perspective and experience of the other. And in many cases, the only way you can learn the more advanced techniques is by working in the field; I can't tell you how much really sophisticated HPC computer science is not documented anywhere on the web but I used it every day when I was working on those systems.
- KaiserPro 13y agoThe one thing that HPC (and VFX) have over anyone else, is rational thought. A thing is used because it is the very fastest. we know its fast because we have the numbers to prove it. Infiniband is expensive, but it's much faster than ethernet for DMA. HTTP is far to inefficient for MPI. Posix file systems are brilliant. Object storage is far to slow and unreliable (see Http) ans usually miss important features like locking or concurrent access. when you're paying by the hour, that 5-30% VM/bytecode penalty is ruinously expensive.
- anonymousDan 13y agoWhat other sharing models are commonly used in HPC?