11 ms·

I work in Livermore Computing at LLNL. We manage upwards of 30 different compute clusters (many listed here: https://hpc.llnl.gov/hardware/platforms https://hp

I work in Livermore Computing at LLNL. We manage upwards of 30 different compute clusters (many listed here: https://hpc.llnl.gov/hardware/platforms https://hpc.llnl.gov/hardware/platforms). You can read about the machine slated to hit the floor in 2022/2023 here: https://www.llnl.gov/news/llnl-and-hpe-partner-amd-el-capitan-projected-worlds-fastest-supercomputer https://www.llnl.gov/news/llnl-and-hpe-partner-amd-el-capita.... All the machines are highly utilized, and they have fast Infiniband/OmniPath networks that you simply cannot get in the cloud. For our workloads on "commodity" x86_64/no-GPU clusters, we pay 1/3 or less the cost of what you'd pay for equivalent cloud nodes, and for the really high end systems like Sierra, with NVIDIA GPUs and Power9's, we pay far less than that over the life of the machine. The way machines are procured here is different from what smaller shops might be used to. For example, the El Capitan machine mentioned above was procured via the CORAL-2 collaboration with 2 other national labs (ANL and ORNL). We write a 100+ page statement of work describing what the machine must do, and we release a set of benchmarks characterizing our workload. Vendors submit proposals for how they could meet our requirements, along with performance numbers and test results for the benchmarks. Then we pick the best proposal. We do something similar with LANL and SNL for the so-called commodity clusters (see https://hpc.llnl.gov/cts-2-rfi https://hpc.llnl.gov/cts-2-rfi for the latest one). As part of these processes, we learn a lot about what vendors are planning to offer 5 years out, so we're not picking off the shelf stuff -- we're getting large volumes of the latest hardware. In addition to the cost savings from running on-prem, it's our job to stay on the bleeding edge, and I'm not sure how we would do that without working with vendors through these procurements and running our own systems.

theyboosting 6y ago
didn't they EOL omnipath last year?
- tgamblin 6y ago
  Yes but we still have many OP systems on the floor (see the link above).
glacials 6y ago
I interned at LLNL on three separate occasions during school. Everything about what y'all do in high-performance computing is levels beyond anything I've gone on to see in the "real world". I cannot drop enough praise.
- tgamblin 6y ago
  thanks!
jedberg 6y ago
> it's our job to stay on the bleeding edge, and I'm not sure how we would do that without working with vendors through these procurements and running our own systems. I think this is key. Also the fact that you guys have a land grant so your datacenter costs are cheaper than most. :)
downerending 6y ago
In many similar contexts, buying hardware is different than buying services (e.g., AWS time), in terms of how you interact with funding agencies. Because of those realities, you might sometimes be unable to "go cloud" even if it's actually cheaper. Also, licensing can be a big problem. Some software packages understand licensing for a "real" cluster, but have no concept of how to license for the cloud. Finally, think about all of your interactions over the last ten years with companies behind the major cloud services. Which of them would you really trust, if it's your money/job/reputation on the line?
zozbot234 6y ago
> All the machines are highly utilized, and they have fast Infiniband/OmniPath networks that you simply cannot get in the cloud. It's weird that these networking technologies are not used more in "plain" datacentre settings, since networking latency and throughput has to be a significant challenge to scaling up non-trivial workloads and achieving true datacentre-scale computing. We hear a lot about how to "scale out", but that's only really feasible for relatively simple workloads where you just seek to do away with the whole issue of keeping different nodes in sync on a real-time basis, and accept the resulting compromises. In many cases, that's just not going to be enough.
- alexpotato 6y ago
  There are a lot of people from the National Lab super computer world who end up in High Frequency Trading for just the reason you describe. Specifically, how do you optimize a large cluster of computers to operate at the lowest possible latency. For the National Labs, those computers could be in the lab or with other labs around the world. For the HFT folks, the machines could be in an exchange or spread across multiple exchanges around the world. Source: I used to be head of Global Latency Monitoring for a HFT.
  - funfunfunction 6y ago
    Hey I’d love to pick your brain about what it takes to do this kind of work coming from a web/systems engineering background. Shoot me an email at samheutmaker at gmail if you’re open to it.
  - j88439h84 6y ago
    I'm curious why you moved to LLNL from HFT?
    - eyegor 6y ago
      Money is a safe guess. Research pay scales aren't even close to private sector, especially not finance.
ablekh 6y ago
Sorry about a slightly off-topic comment. Working on a deep tech startup, with a significant segment of potential customers being national labs, I'm very interested in learning various aspects [beyond the basics] about the process(es) that national labs use to procure application software (not the system one [OS distributions, monitoring etc.], but rather domain-specific one, e.g., scientific data management, modeling and simulations). I have looked at relevant sections on some of the labs' websites and my initial impression is quite fuzzy. If you are familiar with this subject, I would very much appreciate if you could clarify that. If you prefer, you can reach me at aleksandr <dot> blekh <at> Google's e-mail service. P.S. I do have connections at some national labs, but thought that you might shed some additional light on this.
- philipkglass 6y ago
  I used to work at Pacific Northwest National Laboratory. Their flagship computational chemistry application. NWChem, is developed mostly by their own employees. They also collaborate with researchers from other national labs and academia. Development started in the 1990s. When I was at the lab, it was free-as-in-beer but required onerous licensing agreements to get a copy of the source code. It has been open source since 2007. It has started using github to coordinate development in the last few years. http://www.nwchem-sw.org/index.php/Main_Page http://www.nwchem-sw.org/index.php/Main_Page https://github.com/nwchemgit/nwchem/ https://github.com/nwchemgit/nwchem/
  - ablekh 6y ago
    I appreciate your comment and links (BTW, I'm slightly familiar with NWChem and similar software :-). Having said that, I'm not sure how your comment addresses my question. You are talking about internally-developed software transfer from the lab to external users, whereas my use case is delivering externally-developed commercial software to national labs' users, a value transfer in the opposite direction. Care to comment on this aspect, if you're familiar with relevant procurement processes?
    - sseagull 6y ago
birdyrooster 6y ago
Does it run Fortran?
- tgamblin 6y ago
  Some, but LLNL codes are mostly C++, especially now that they need to make effective use of GPUs. More on codes: - https://computing.llnl.gov/sierra-supercomputer-dedication https://computing.llnl.gov/sierra-supercomputer-dedication Some repos of interest: - https://github.com/LLNL/axom https://github.com/LLNL/axom - https://github.com/LLNL/RAJA https://github.com/LLNL/RAJA
evanb 6y ago
I'm a former LLNL lattice QCD postdoc and current user of the LLNL machines (and others). I have to say, I was wildly spoiled. Every other computing center where I have an account cannot hold a candle to the helpfulness of the LLNL support, the ease of use (except for the 2-factor requirement, which I understand), and absolutely insane power.
awill88 6y ago
I grew up across the street from the the Lab on (Cheryl Dr.). Now early 30s, I’m a AWS architect and software engineer at a finance company in Bozeman, Montana, but I would kill to be an engineer for pursuit of knowledge ya know? I dream of working there some day.. and I miss home
- d-sc 6y ago
  Hey, I’m in Bozeman too. We should get lunch sometime.

theyboosting 6y ago

didn't they EOL omnipath last year?

tgamblin 6y ago
Yes but we still have many OP systems on the floor (see the link above).

glacials 6y ago

I interned at LLNL on three separate occasions during school. Everything about what y'all do in high-performance computing is levels beyond anything I've gone on to see in the "real world". I cannot drop enough praise.

tgamblin 6y ago
thanks!

jedberg 6y ago

> it's our job to stay on the bleeding edge, and I'm not sure how we would do that without working with vendors through these procurements and running our own systems. I think this is key. Also the fact that you guys have a land grant so your datacenter costs are cheaper than most. :)

downerending 6y ago

In many similar contexts, buying hardware is different than buying services (e.g., AWS time), in terms of how you interact with funding agencies. Because of those realities, you might sometimes be unable to "go cloud" even if it's actually cheaper. Also, licensing can be a big problem. Some software packages understand licensing for a "real" cluster, but have no concept of how to license for the cloud. Finally, think about all of your interactions over the last ten years with companies behind the major cloud services. Which of them would you really trust, if it's your money/job/reputation on the line?

zozbot234 6y ago

> All the machines are highly utilized, and they have fast Infiniband/OmniPath networks that you simply cannot get in the cloud. It's weird that these networking technologies are not used more in "plain" datacentre settings, since networking latency and throughput has to be a significant challenge to scaling up non-trivial workloads and achieving true datacentre-scale computing. We hear a lot about how to "scale out", but that's only really feasible for relatively simple workloads where you just seek to do away with the whole issue of keeping different nodes in sync on a real-time basis, and accept the resulting compromises. In many cases, that's just not going to be enough.

alexpotato 6y ago
There are a lot of people from the National Lab super computer world who end up in High Frequency Trading for just the reason you describe. Specifically, how do you optimize a large cluster of computers to operate at the lowest possible latency. For the National Labs, those computers could be in the lab or with other labs around the world. For the HFT folks, the machines could be in an exchange or spread across multiple exchanges around the world. Source: I used to be head of Global Latency Monitoring for a HFT.
- funfunfunction 6y ago
  Hey I’d love to pick your brain about what it takes to do this kind of work coming from a web/systems engineering background. Shoot me an email at samheutmaker at gmail if you’re open to it.
- j88439h84 6y ago
  I'm curious why you moved to LLNL from HFT?
  - eyegor 6y ago
    Money is a safe guess. Research pay scales aren't even close to private sector, especially not finance.

ablekh 6y ago

Sorry about a slightly off-topic comment. Working on a deep tech startup, with a significant segment of potential customers being national labs, I'm very interested in learning various aspects [beyond the basics] about the process(es) that national labs use to procure application software (not the system one [OS distributions, monitoring etc.], but rather domain-specific one, e.g., scientific data management, modeling and simulations). I have looked at relevant sections on some of the labs' websites and my initial impression is quite fuzzy. If you are familiar with this subject, I would very much appreciate if you could clarify that. If you prefer, you can reach me at aleksandr <dot> blekh <at> Google's e-mail service. P.S. I do have connections at some national labs, but thought that you might shed some additional light on this.

philipkglass 6y ago
I used to work at Pacific Northwest National Laboratory. Their flagship computational chemistry application. NWChem, is developed mostly by their own employees. They also collaborate with researchers from other national labs and academia. Development started in the 1990s. When I was at the lab, it was free-as-in-beer but required onerous licensing agreements to get a copy of the source code. It has been open source since 2007. It has started using github to coordinate development in the last few years. http://www.nwchem-sw.org/index.php/Main_Page http://www.nwchem-sw.org/index.php/Main_Page https://github.com/nwchemgit/nwchem/ https://github.com/nwchemgit/nwchem/
- ablekh 6y ago
  I appreciate your comment and links (BTW, I'm slightly familiar with NWChem and similar software :-). Having said that, I'm not sure how your comment addresses my question. You are talking about internally-developed software transfer from the lab to external users, whereas my use case is delivering externally-developed commercial software to national labs' users, a value transfer in the opposite direction. Care to comment on this aspect, if you're familiar with relevant procurement processes?
  - sseagull 6y ago

birdyrooster 6y ago

Does it run Fortran?

tgamblin 6y ago
Some, but LLNL codes are mostly C++, especially now that they need to make effective use of GPUs. More on codes: - https://computing.llnl.gov/sierra-supercomputer-dedication https://computing.llnl.gov/sierra-supercomputer-dedication Some repos of interest: - https://github.com/LLNL/axom https://github.com/LLNL/axom - https://github.com/LLNL/RAJA https://github.com/LLNL/RAJA

evanb 6y ago

I'm a former LLNL lattice QCD postdoc and current user of the LLNL machines (and others). I have to say, I was wildly spoiled. Every other computing center where I have an account cannot hold a candle to the helpfulness of the LLNL support, the ease of use (except for the 2-factor requirement, which I understand), and absolutely insane power.

awill88 6y ago

I grew up across the street from the the Lab on (Cheryl Dr.). Now early 30s, I’m a AWS architect and software engineer at a finance company in Bozeman, Montana, but I would kill to be an engineer for pursuit of knowledge ya know? I dream of working there some day.. and I miss home

d-sc 6y ago
Hey, I’m in Bozeman too. We should get lunch sometime.