3 ms·

You have never built a super computer, have you? You have to connect thousands of cables, have hundreds of nodes, with thousands of components, everything inte

You have never built a super computer, have you? You have to connect thousands of cables, have hundreds of nodes, with thousands of components, everything interconnected, and if you connect one wrong, the computer outputs incorrect results. You have to routinely update the software, and if a software upgrade introduces a 20% perf regression (which happens), then your 10 MWh cluster starts burning 2MWh for nothing. Or maybe your cooling system sucks, and after a minute of running at full capacity, you need to throttle your cluster to 0.1% of the peak to keep it cool enough that it runs "something". That's why all systems in the top500 i've been involved with (15 or so) run these benchmarks as an integration tests on every single cluster maintenance (node updates, servicing, OS updates, etc.). Submitting these results to the Top500 costs you nothing... if your cluster actually works. When you submit to the Top500, they ask for access so that they can re-run them themselves, which happens typically during / after the next maintenance to avoid impacting any users. If they haven't submitted, 100% sure their cluster does not deliver what they say it should deliver on paper. Maybe it delivers 1% of it, or 0.01% of it (seen both cases in real life). If they haven't fixed it, then maybe it can't be fixed. HPL, MLPerf, Spec, Stream, OSU.... these are not "benchmarks for bragging rights", these are tests that show that your system works.

chrisseaton 5y ago
> run these benchmarks as an integration tests on every single cluster maintenance Why run someone else's benchmark and not your own application to test performance? And what's the point of submitting to Top500? Why do you care how your system ranks? What's the business or technical purpose in that?
- cedilla 5y ago
  For the same reason that we don't test engines during a race. Only what's under test should change, with the rest being fixed. It would be much more difficult to adapt some of your own applications as a test. If the result are bad, how would you even know if it's the application's fault or a problem with the cluster? LINPACK on the other hand is well understood and there has been tons of work to make sure that it uses all the power your cluster can deliver.
- Sanguinaire 5y ago
  HPL is a stress test with a notionally useful output measurement - just about the most effective way of pushing CPU load to the limit, and it tells you what fraction of the theoretical maximum FLOPS you can actually achieve given the other system constraints like memory and network throughput.
- N1H1L 5y ago
  Is this some sort of a post-truth joke? If you are claiming something is the best, then without any comparable metrics across systems we are in the Twilight zone.
  - chrisseaton 5y ago
    But they've got no obligation to prove anything to random people on the internet. They aren't asking for peer review. They probably don't want to waste time and energy running a benchmark to compete in someone else's top N list. What's in it for them?
    - volta83 5y ago
      > But they've got no obligation to prove anything to random people on the internet. They are a publicly traded company. I'm a tesla investor and deserve to know. If they are right and their hardware is 10x better than the competitor, then I'm happy. If they are wrong and they could have get 10x better outcomes by spending 100x less money, I'll be very pissed off. > They probably don't want to waste time and energy running a benchmark to compete in someone else's top N list. It costs no time. It's the first thing anybody does when building any cluster. It's part of accepting the materials from your suppliers to check that they didn't sell you shit.
      - chrisseaton 5y ago
        > I'm a tesla investor and deserve to know. I'm not an expert in securities law, but I don't believe being an investor entitles you to demand arbitrary proprietary information you want from a company. Otherwise people would buy one Apple share and find out what specs the new iPhone will have.
- volta83 5y ago
  When a company claims that their new vaccine cures cancer, but is against applying for FDA approval because "no reasons given", you seem to be the kind of person that camps in front of their offices overnight to get the first 10 shots, all in the neck.
  - chrisseaton 5y ago
    But FDA approval is mandatory, and participation in top N lists is just marketing. And they aren't even asking you to use their machine!
    - volta83 5y ago
      Dude i don't know what you don't get it. Tesla is saying their hardware is 10x better than the competition. I don't believe it. They don't publish any numbers, and everybody else does. You believe them. Good for you. I don't. I think not doing that is extremely suspicious, because if their cluster can be turned on, they have the results. So the reality of this is 100% certain that their results just suck, and they don't publish them to save face. You believe their shit is so good, they didn't even test it while installing it. Good for you. You think there is a masterplan to keep the results secret. Good for you. I think their results are just horrible, and that's why they don't show them. With any serious company, this wouldn't even be a discussion, because they would just show the verified facts about what their cluster can do or shut up about it if its really so secret. You don't see the NSA or the DOD bragging around about being 100x better than "the competition". To me they look forced to sell that they are doing something about these clusters to... drive hype up and get the stocks back up, and since their numbers are actually horrible, this is what we get.
N1H1L 5y ago
Hi fellow supercomputer person! My experience is on Summit & am now working with Frontier setup.

chrisseaton 5y ago

> run these benchmarks as an integration tests on every single cluster maintenance Why run someone else's benchmark and not your own application to test performance? And what's the point of submitting to Top500? Why do you care how your system ranks? What's the business or technical purpose in that?

cedilla 5y ago
For the same reason that we don't test engines during a race. Only what's under test should change, with the rest being fixed. It would be much more difficult to adapt some of your own applications as a test. If the result are bad, how would you even know if it's the application's fault or a problem with the cluster? LINPACK on the other hand is well understood and there has been tons of work to make sure that it uses all the power your cluster can deliver.
Sanguinaire 5y ago
HPL is a stress test with a notionally useful output measurement - just about the most effective way of pushing CPU load to the limit, and it tells you what fraction of the theoretical maximum FLOPS you can actually achieve given the other system constraints like memory and network throughput.
N1H1L 5y ago
Is this some sort of a post-truth joke? If you are claiming something is the best, then without any comparable metrics across systems we are in the Twilight zone.
- chrisseaton 5y ago
  But they've got no obligation to prove anything to random people on the internet. They aren't asking for peer review. They probably don't want to waste time and energy running a benchmark to compete in someone else's top N list. What's in it for them?
  - volta83 5y ago
    > But they've got no obligation to prove anything to random people on the internet. They are a publicly traded company. I'm a tesla investor and deserve to know. If they are right and their hardware is 10x better than the competitor, then I'm happy. If they are wrong and they could have get 10x better outcomes by spending 100x less money, I'll be very pissed off. > They probably don't want to waste time and energy running a benchmark to compete in someone else's top N list. It costs no time. It's the first thing anybody does when building any cluster. It's part of accepting the materials from your suppliers to check that they didn't sell you shit.
    - chrisseaton 5y ago
      > I'm a tesla investor and deserve to know. I'm not an expert in securities law, but I don't believe being an investor entitles you to demand arbitrary proprietary information you want from a company. Otherwise people would buy one Apple share and find out what specs the new iPhone will have.
volta83 5y ago
When a company claims that their new vaccine cures cancer, but is against applying for FDA approval because "no reasons given", you seem to be the kind of person that camps in front of their offices overnight to get the first 10 shots, all in the neck.
- chrisseaton 5y ago
  But FDA approval is mandatory, and participation in top N lists is just marketing. And they aren't even asking you to use their machine!
  - volta83 5y ago
    Dude i don't know what you don't get it. Tesla is saying their hardware is 10x better than the competition. I don't believe it. They don't publish any numbers, and everybody else does. You believe them. Good for you. I don't. I think not doing that is extremely suspicious, because if their cluster can be turned on, they have the results. So the reality of this is 100% certain that their results just suck, and they don't publish them to save face. You believe their shit is so good, they didn't even test it while installing it. Good for you. You think there is a masterplan to keep the results secret. Good for you. I think their results are just horrible, and that's why they don't show them. With any serious company, this wouldn't even be a discussion, because they would just show the verified facts about what their cluster can do or shut up about it if its really so secret. You don't see the NSA or the DOD bragging around about being 100x better than "the competition". To me they look forced to sell that they are doing something about these clusters to... drive hype up and get the stocks back up, and since their numbers are actually horrible, this is what we get.

N1H1L 5y ago

Hi fellow supercomputer person! My experience is on Summit & am now working with Frontier setup.