3 ms·

USG has tons of compute. What we don't have is the tooling behind AWS or GCP (minus GovCloud which is mainly inaccessible due to infosec and budgetary insanity)

USG has tons of compute. What we don't have is the tooling behind AWS or GCP (minus GovCloud which is mainly inaccessible due to infosec and budgetary insanity). Our scientists look at what could be, look at their very interesting datasets, and then gnash their teeth for want of better tooling. People know this. Senior people from industry have screamed at senior government leadership that they are 20 years behind and falling faster. Nothing changes. Why? Every time someone gets out from under their grad school payoff contract, they run to Google or Facebook. Very few people are committing for the long term, and that's what's needed. Long-term commitment to steward a program. The people who remain often lack technical chops, have "certificates" in "cyber" something or other, but got through the Acquisitions school and now have authority to just pour money into blades, load Red Hat and mpi4py, and say "look, it's a supercomputer"! They don't need to invest in iron. They need to invest in people. But they think they can just keep hiring new grads and expect magic.

Retric 9y ago
AWS is a terrible match for super computing hardware. Sure, there are some embarrassingly parallel workloads that don't need fast interconnects, but mostly latency or memory becomes the issue.
- cobookman 9y ago
  Gcp has it's tpus though...
- batbomb 9y ago
  Dirty little secret: Much the computing on these machines isn't supercomputing and doesn't need fast interconnects. In fact, many (most?) workloads would be best done in the cloud. Except we already bought these machines. So there's tons of frameworks around meant to use these machines like they are AWS pizza boxes. If it's inefficient due to an architectural mismatch, that's fine. You're still using the machine that was paid for, and the only numbers that matter is utilization, and those metrics aren't even defined in terms of CPU utilization. If you reserve 10 nodes, and only run on one node, it's still reported that a 10 node job ran for 6 hours, not that a 10 node job job ran at 10% efficiency for 6 hours. As far as why the accounting is this way, it's assumed that users will attempt to maximize their usage since allocations are defined. But even then, you can blow through your allocation and generally ask for more. But that's not necessarily what happens. In summary: Because we bought these big computers, we have to use them, even if the workload isn't a good match.
  - Retric 9y ago
    Amdahl's law, if you spend more on interconnects you only speed up a subset of your workloads. But the reverse is also true so buying more nodes and less interconnects only speeds up a subset of your workloads. The optimum is going to seem wasteful on average. https://en.m.wikipedia.org/wiki/Amdahl%27s_law https://en.m.wikipedia.org/wiki/Amdahl%27s_law

Retric 9y ago

AWS is a terrible match for super computing hardware. Sure, there are some embarrassingly parallel workloads that don't need fast interconnects, but mostly latency or memory becomes the issue.

cobookman 9y ago
Gcp has it's tpus though...
batbomb 9y ago
Dirty little secret: Much the computing on these machines isn't supercomputing and doesn't need fast interconnects. In fact, many (most?) workloads would be best done in the cloud. Except we already bought these machines. So there's tons of frameworks around meant to use these machines like they are AWS pizza boxes. If it's inefficient due to an architectural mismatch, that's fine. You're still using the machine that was paid for, and the only numbers that matter is utilization, and those metrics aren't even defined in terms of CPU utilization. If you reserve 10 nodes, and only run on one node, it's still reported that a 10 node job ran for 6 hours, not that a 10 node job job ran at 10% efficiency for 6 hours. As far as why the accounting is this way, it's assumed that users will attempt to maximize their usage since allocations are defined. But even then, you can blow through your allocation and generally ask for more. But that's not necessarily what happens. In summary: Because we bought these big computers, we have to use them, even if the workload isn't a good match.
- Retric 9y ago
  Amdahl's law, if you spend more on interconnects you only speed up a subset of your workloads. But the reverse is also true so buying more nodes and less interconnects only speeds up a subset of your workloads. The optimum is going to seem wasteful on average. https://en.m.wikipedia.org/wiki/Amdahl%27s_law https://en.m.wikipedia.org/wiki/Amdahl%27s_law