17 ms·

Yes. In my computational history I have: used the largest non-classified DOE supercomputers, built my own modest closet clusters, developed an embarassingly pa

by dekhn 2y ago

Yes. In my computational history I have: used the largest non-classified DOE supercomputers, built my own modest closet clusters, developed an embarassingly parallel computing system using Google's idle prod cycles, and helped debug training of LLMs when I worked on the TPU team at Google. I work for big pharma now (and my phd is in biophysics) and I'm also comfortable with other HPC domains. I know the DOE/Nvidia history quite well as the Chief Scientist of NVIDIA visited LBL around 2005(6? 7?) and talked about their new hardware they were just starting to build and sell, with the goal of getting them into supercomputers. We asked if they had double precision performance yet (because that was a must for many supercomputer jobs), but at the time, nvidia DP was still lagging SP (I guess it still does?) and we also quibbled about their non-compliance with some esoteric details in IEEE 754. The best part of the whole talk was when he walked us through the idea of visualizing our operations by drawing the matrices as textures, because you can easily see the NaNs- they render as nvidia Green! I left DOE (Berkeley Lab) shortly after to work in industry because it was clear that ML wasn't going to be innovated in the government labs.

whoknw 2y ago
Thank you for that comment, as someone who has had exposure to the DOE as well (and LBL at that) I can only echo your sentiment. I'd even go further and state that is hard for me to believe that any innovation can happen in the calcified structures of governmental labs. Maybe the classified ones are different. The "DOE made NVIDIA" myth is a story I haven't seen pushed outside the DOE complex. It is true that the supercomputers the DOE pushes could be considered industry subsidies, by providing industry companies with a steady customer with a very high tolerance for unfinished products. That applies to NVIDIA, AMD, Intel, HPE/Cray and IBM more or less equally. I also want to stress what often gets overlooked: supercomputers are hell to operate and use. Aurora runs on Slingshot, a Cray interconnect. Those things look good on paper. Examples: Cray Aries (and "network quiesces") or Cray DataWarp. Who knows how Slingshot actually works in practice, for a hero run it only needs to hold things together for a few hours. As long as you get a high TOP500 ranking, a supercomputer is a success. There is no market for those things anymore and they are beholden to the same economics as everything else, hence codes that can't afford an army of PostDocs to work around the bugs and design decisions that are only necessary due to scale of those systems are better suited to plain old mid-range clusters. And I haven't even mentioned the eccentric userland of supercomputers. There are many reasons the DOE affords to run those behemoths. Some more trivial and petty than most people would like to believe. Like the author of the parent post, I have come to believe that the best bang for the buck on scientific output can be found elsewhere.
- close04 2y ago
  > I have come to believe that the best bang for the buck on scientific output can be found elsewhere. The best bang for the buck is never at the very top. The top is just for the biggest bang.

by dekhn 2y ago

whoknw 2y ago

Thank you for that comment, as someone who has had exposure to the DOE as well (and LBL at that) I can only echo your sentiment. I'd even go further and state that is hard for me to believe that any innovation can happen in the calcified structures of governmental labs. Maybe the classified ones are different. The "DOE made NVIDIA" myth is a story I haven't seen pushed outside the DOE complex. It is true that the supercomputers the DOE pushes could be considered industry subsidies, by providing industry companies with a steady customer with a very high tolerance for unfinished products. That applies to NVIDIA, AMD, Intel, HPE/Cray and IBM more or less equally. I also want to stress what often gets overlooked: supercomputers are hell to operate and use. Aurora runs on Slingshot, a Cray interconnect. Those things look good on paper. Examples: Cray Aries (and "network quiesces") or Cray DataWarp. Who knows how Slingshot actually works in practice, for a hero run it only needs to hold things together for a few hours. As long as you get a high TOP500 ranking, a supercomputer is a success. There is no market for those things anymore and they are beholden to the same economics as everything else, hence codes that can't afford an army of PostDocs to work around the bugs and design decisions that are only necessary due to scale of those systems are better suited to plain old mid-range clusters. And I haven't even mentioned the eccentric userland of supercomputers. There are many reasons the DOE affords to run those behemoths. Some more trivial and petty than most people would like to believe. Like the author of the parent post, I have come to believe that the best bang for the buck on scientific output can be found elsewhere.

close04 2y ago
> I have come to believe that the best bang for the buck on scientific output can be found elsewhere. The best bang for the buck is never at the very top. The top is just for the biggest bang.