3 ms·
how has the the enterprise big data ecosystem (hadoop, spark, perhaps even more recently, cloud based stuff such as snowflake) influenced HPC?
by waynesonfire 5y ago
how has the the enterprise big data ecosystem (hadoop, spark, perhaps even more recently, cloud based stuff such as snowflake) influenced HPC?
- riskneutral 5y agoVery little, I think. They are very different. HPC has to do with solving certain rigorous scientific equations through the use of various numerical methods (differential equation solvers, linear algebra solvers, Monte Carlo simulation, etc). "Enterprise big data" is very far from having rigorous scientific equations that need solving.
- tonyarkles 5y agoThis was admittedly a while ago (2012 or so) but when I was in grad school, I would often challenge other people in my Distributed Systems lab to compare their Hadoop jobs against just running a batch job on a single system (without the Hadoop overhead etc, just sequential). More often than not, the threshold for "when does this actually make sense" was much larger than they'd expected. And then [1] came out and disappointed some people. By that time I was mostly out of the lab and just doing minor thesis edits, but when I brought it to our weekly reading group the discussion was... animated. [1] https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf https://www.usenix.org/system/files/conference/hotos15/hotos...