5 ms·
It looks like they are explicitly saying they want to make a machine that works for both types of HPC -- classic low-latency high-bandwidth internode communicat
by dekhn 11y ago
It looks like they are explicitly saying they want to make a machine that works for both types of HPC -- classic low-latency high-bandwidth internode communication (physics simulations) and modern Internet-driver high-bandwidth storage/node communications.
This is because the supercomputer community has long ignored the Internet-style of computation (MapReduce etc). But most of the new generation of scientists are adapting their codes to this new style, because dollar-for-dollar they can get more throughput than the classic style machines. Classic machines invest heavily in low-latency communication and typically require APIs like MPI to achieve it, while Internet HPC just uses well-designed TCP-based socket communications.
Building dual-design systems like this- especially when the community has little or no skill at building NG Internet HPC systems- is likely to produce a system that is good at few things.
Instead, build two systems. One is the largest (but not necessarily exaflop) you can afford and is a classic supercom[puter. Then, for the second, hire some datacenter designers from Google/Facebook and have them build a modern HPC cloud design.
The biologists will flock to the second one; they have long been underserved by the DOE supercomputing community.
- eitally 11y agoAnd it's not like there isn't precedent. I remember a couple years ago when Google showed off gene mapping at I/O (Urs' demo to visualize how easy it was to scale GCP From 1 to 10 to 1000 to 100000 cores), and now they've partnered with the Broad Institute to apply this more generally. http://www.broadinstitute.org/google http://www.broadinstitute.org/google
- dekhn 11y agoI was the person on stage with Urs who ran the demo (I'm a computational biologist who went to Google to help them build this kind of infrastructure, because DOE wasn't doing it). Read more about the demo here: https://cloud.google.com/compute/io https://cloud.google.com/compute/io I also helped create the original idea for the Broad collaboration. It was pretty obvious that standard HPC was a waste of money and not designed for high throughput biology, while Google published numerous papers (MapReduce, GFS, Bigtable) that demonstrated they were building infrastructure that was perfect for a wide range of computational biology programs.
- sevensor 11y ago> especially when the community has little or no skill at building NG Internet HPC systems I would argue that the community of people who actually have the skills to take advantage of the interconnects in a classic HPC system is vanishingly small, and in consequence we've overbuilt them on an epic scale. Allow me to vent. I had the good fortune to have a login on a "petascale" HPC system, and access to an allocation of hours. The /scratch filesystem would fail weekly, which killed everybody's jobs. If you had a big run going when /scratch failed, you lost everything. Scratch failed so much because the models that were being used often did wildly inappropriate amounts of file IO --- debugging print statements, detailed intermediate calculations, excessively verbose output --- that worked all right in development but when run in parallel brought the filesystem to its knees. Furthermore, the login nodes were almost unusably slow because of all the Python and Perl post-processing scripts running on them. This isn't even a matter of users being cheap with their hours --- post-processing would have been a tiny fraction of their allocations. Instead, it's that many of them gave no thought at all to how the post-processing might be structured and run through the batch scheduler, and saw no downside to abusing the login nodes for that purpose. In conclusion, I can attest to at least one HPC system that was badly mismatched to its users' needs and level of sophistication, despite allocations of hours being awarded only to a small number of researchers from across the country through a highly competitive process. Building these things serves national and institutional pride far more than any utilitarian interest.
- khm 11y agoAh, a Cray. Buck up; there are better supercomputers.
- sevensor 11y agoIt was, in fact. How did you infer that?
- ams6110 11y ago/scratch on a Cray is (last I used one) is a Lustre filesystem. It is the only file storage available to the compute nodes, which have no local storage of their own. No spinning disk, no SSD. So if the code is written such that it makes frequent small writes (e.g. it's peppered with print statements), the lustre nodes get hammered by all the compute nodes and become the bottleneck and they will eventually fall over.
- lorenzhs 11y agoOne of the largest challenges in building an exascale cluster is communication. Computing power increases at a higher rate than memory throughput does, and memory throughput increases faster than communication infrastructure advances. Many argue that an exascale computer can only be cost efficient if the communication capabilities scale highly sublinearly with the computation done in the subsystems [1]. In particular, you can't move the data, and new algorithms are needed that can deal with data that is arbitrarily distributed. This is quite challenging and unfortunately the theoretical computer science community seems to have decided that distributed memory algorithms have been covered since the 90s and are not worth their time. Yet they ignore the progress that has been made in other models of computation since, and many algorithmic improvements of the last decades are not applicable. It is high time to develop communication-efficient algorithms for the basic "toolbox". I guess what I'm trying to say is that you can't just throw MapReduce at an Exascale machine and expect it to perform well. Instead, you need an environment that is rich in primitives that have been implemented in a communication-efficient way. It's faster and cheaper to spend a little more effort on local communication if that allows for reduced communication volume (and/or the number of connections that need to be established!). The issue I have with the MapReduce approach is that it doesn't particularly care about data locality. Thus it is very hard to achieve communication volume sublinear in the input size, which is absolutely deadly in an exascale setting. I also understand the frustration with MPI, it is a very low-level API focused on data movement. It can be rather frustrating to use, but there do exist tools to make it more fun (Boost.MPI with C++11/14 is an excellent example). That said, with a well-engineered set of algorithmic tools, ideally you wouldn't need to use low-level MPI calls at all. However, MPI still remains a useful tool to implement these things. Exascale computing requires us to rethink a lot of things. [1] http://www.ipdps.org/ipdps2013/SBorkar_IPDPS_May_2013.pdf http://www.ipdps.org/ipdps2013/SBorkar_IPDPS_May_2013.pdf Shekhar Borkar (Intel), Keynote presentation at the 2013 IEEE International Parallel & Distributed Processing Symposium
- dekhn 11y agoYou are incorrect saying MapReduce isn't locality aware. Hadoop supports machine, rack, row, and cluster locality scheduling. Also, most modern Internet HPC systems dedicate a ton of design and equipment to having very high cross-sectional bandwidth, which enables the locality restrictions to be relaxed.
- jandrewrogers 11y agoIt is important to not conflate "massively parallel" (HPC) and "massively distributed" (Internet-scale), they have different architectural requirements and solve different classes of problem. People with competency in either of these areas tend to overestimate their understanding of the other but they are not solving the same computer science problems even though they look similar on the surface. Massively distributed systems do not get much benefit from low-latency interconnects. Massively parallel systems do, and in particular, it is a "throw hardware at the problem" kind of solution that helps cover for the fact that virtually no software designers can engineer efficient, non-trivial, massively parallel systems. MapReduce is a distributed model; outside of some trivial cases, it is a poor parallel model. And while the HPC community has a much better understanding of massive parallelism than the Internet-scale systems community, the HPC community largely doesn't grok massively distributed systems in the way that someone working on Google's infrastructure would. I benefitted from having spent several years designing software for both HPC and Internet-scale systems. They are not fungible, and both communities grok things that the other is oblivious to. Even within the HPC community though, the number of people skilled at the design of massively parallel software systems is quite small, much smaller than people that know massively distributed systems. You do not need two systems, you need one system and more people that have figured out how to design massively parallel software -- the real problem. It is difficult to overstate just how rare this skill is even within the HPC community.