8 ms·

This might be a super dumb question, but are super computers worth it? Meaning anything that requires custom hardware instead of just groups of coordinated com

• trishume 6y ago
  Note: I have never worked with an HPC system, take what I say with a grain of salt. I suspect the answer is in the networking, what this article spends most of its time talking about. HPC networking is so ridiculously much better than commodity networking. 1us latencies vs 100us+, 200Gbps bandwidth vs 10Gbps. I imagine certain kinds of especially simulations involve a ton of communication between nodes simulating nearby cells and everything is completely bottlenecked on that. It might even be that you can't even match a supercomputer's speed by just buying more AWS instances because the scaling factors on the communication mean more instances doesn't make your sim faster. Again this is mostly inference on my part from things I've read, no first-hand knowledge.
  • jiggawatts 6y ago
    The hilarity is that most business environments could benefit from the same networking "tricks" used by HPC, but don't bother to set any of it up. I've even seen a few places that have even purchased the prerequisite hardware but then never turned the features on. Things like hardware flow control, cut-through switching, jumbo frames, SR-IOV, RDMA, etc... are just a few checkboxes away and can give even small VMs crazy good bandwidth and latency.
    • wmf 6y ago
      When I was doing a private cloud last year these features were way more work than checkboxes, but maybe that's what we got for using OpenStack. I finished the planning for jumbo frames to the VM (this was complicated by double VXLAN), SR-IOV was blocked because we only had ConnectX-4 but SR-IOV over LAG requires ConnectX-5, and the storage team never got around to Ceph over RoCE.
      • jiggawatts 6y ago
        Most enterprises run VMware, Hyper-V, or XenServer. Almost always with Cisco Nexus 9K series switching. With that kit it's surprisingly easy, but not with the default Broadcom NICs, they generally don't have the required features. Even if they do, the advanced features often don't work properly. I've seen jaw-dropping performance when it was set up correctly, something like 99% of the theoretical wire rate of 20 Gbps. With dual NICs and Windows 2016 we got nearly 40 Gbps because it can use both NICs at once! Amazing file copy times. I tested it with the usual 4GB ISO file copy test and it was so fast there wasn't even a progress bar! Meanwhile, on most networks I see about 1.5-3.0 Gbps effective across dual 10 Gbps NICs. If you think about it, that's crazy bad. It means that in effect they're getting less than 10% of the available physical capacity for data traffic. The other 90% is being thrown away by "passive" secondary cables or by the lack of configuration on the switching layer.
        • exikyut 6y ago
          I discovered the other day that you can google `Xgb/Xgbps` and it'll give you a result in seconds. 4GB is 0.8 seconds :) And regarding that 1.5-3.0Gbps across 2x10Gbps... a) wat?!?! and b) I wonder if the number of horribly misconfigured networks exactly like that are out there giving 10Gig a bad name and hurting adoption.
      • timeu 6y ago
        Yeah, I can feel your pain. We deployed a fully virtualized HPC cluster on top of OpenStack and there was a fair amount of testing and playing with knobs (SRIOV, Jumbo frames, CPU pinning, etc) to get close to bare metal performance on a 100 GBit/s SDN fabric.
    • duxup 6y ago
      Managing flow control, misc switching changes, mtu is as much an art of trial and error as it is a science. Unless you are in a very controlled environment it is VERY easy to actually slow down a network while trying to make it more efficient.
    • eecc 6y ago
      Yup, and quite probably no one wants to turn them on for fear of bugs and sudden disruptions. Besides, IT is a cost right? /s The rock-bottom cheap support company outsourcing the infra maintenance is more careful about minimizing actual work and matching the exact wording in the support contract and not a cent more
    • gnufx 6y ago
      Indeed. Infiniband was even meant for "datacentre" use, and I have better experience of it in compute clusters than Ethernet apart from the performance.
      • duxup 6y ago
        Infiniband is pretty neat for a given use case but that cable length kinda killed it for many data center use cases. Still, if folks can deal with those short distances there's lots of inexpensive equipment out there.
        • gnufx 6y ago
          Point taken, but for some time Mellanox has touted at least cross-campus-type IB. I never persuaded anyone to evaluate it for use between two machine rooms, but there were obvious non-HPC cases for it within them. I'd be interested to know how the long distance stuff actually works, not being a beginner with sales talk.
          • duxup 6y ago
            That's a good question. I haven't used it but extending infinaband ... I'm guessing there's some sort of intermediary devices doing some sort of piplining or emulation, that's usually how that works out.
    • GhettoMaestro 6y ago
      > Things like hardware flow control, cut-through switching, jumbo frames, SR-IOV, RDMA, etc... are just a few checkboxes away and can give even small VMs crazy good bandwidth and latency. This comment's dismissive tone really leads me to believe you have never really dealt with the details of said technologies.
  • kijiki 6y ago
    You can get 400Gbps bandwidth in commodity networking: https://www.fs.com/products/96982.html https://www.fs.com/products/96982.html Not going to get that sort of latency without extreme custom chips though.
    • jiggawatts 6y ago
      Wow, that's not even that expensive! I generally don't get to play with hardware much these days, but my "dream" on-premises kit would be a hyperconverged cluster with AMD EPYC 2 CPUs, NVMe storage, and 400 Gbps networking. It would be interesting to see what kind of performance that would enable compared to the typical cloud platforms everyone is so excited about...
      • briffle 6y ago
        Don't forget to add about ~$1000 per port for a 400GB transceiver.
    • gnufx 6y ago
      Gbps isn't a unit of latency (or message rate).
    • exikyut 6y ago
      ($15K)
• monocasa 6y ago
  The custom hardware isn't in the processor chips, it's in the interconnect. In a lot of ways they are groups of coordinated commodity hardware. It's the coordination that's the secret sauce and what you're paying for. Not every problem works great in the tree interconnects you see in data centers, so these systems are big hypercubes. Also the latency is more like PCIe than TCP. In fact PCIe is closer to Infiniband than original PCI in a lot of ways.
  • gnufx 6y ago
    The node interconnect is crucial. However, things like V100s (and presumably the CORAL2 GPUS) are clearly built for high performance computation, if not necessarily "HPC" in this sense; arguably also the nodes used by Sierra/Summit, though IBM seem just to say "AI". K computer was all built for the task, and Fugaku may be similarly impressive. See also the Chinese systems, though it's arguable how successful those designs are. There's also a fair amount of more or less hush-hush work in Europe on HPC custom hardware, though it's not clear that's paying off. The majority of HPC systems probably use fat tree topologies (including Sierra and Summit as far as I remember). Something to hand which compares fat tree, hypercube, and the dragonfly topology discussed in the article is http://www.hpcadvisorycouncil.com/events/2015/swiss-workshop/pdf/day1/10_ETH.pdf http://www.hpcadvisorycouncil.com/events/2015/swiss-workshop...
  • deleted 6y ago
    [deleted]
  • dekhn 6y ago
    I recently did an analysis and it looks like for most supercomputers, only 15% of the total price is interconnect. I was surprised- I expected it to be more. Also, most supercomputers aren't hypercubes, they're using things like folded butterfly.
    • monocasa 6y ago
      I'm really curious what process you were able to use to separate the component costs on systems like these. The manufacturers go out of their way to obfuscate that information.
• som33 6y ago
  >This might be a super dumb question, but are super computers worth it? Yes, because that's where you get innovation in architecture. You have to understand the reason commodity hardware caught up supercomputers was because of getting lucky with materials. AKA the mateirals cpu's were made out of was easily shrinkable and frequency kept going up because of dennard scaling until roughly 2006. With the end of dennard scaling, research into new processor materials and computer architectures is the main issue. Single threaded performance is a huge priority because most problems we are interested in cannot be made parallel and hence do not benefit much from multicore. There are also things you are not hearing about and aren't privy too. Basically computers can run much faster if you use incredibly expensive or experimental materials. The major roadblock is really mass production. I'm certain there are black projects doing all sorts of custom hardware behind the scenes, most likely for military applications. But the expense to run those custom machines makes it non viable for anything like mass adoption.
  • LASR 6y ago
    It seems like you believe you know a lot on this subject.
  • Tuna-Fish 6y ago
    > Basically computers can run much faster if you use incredibly expensive or experimental materials. The major roadblock is really mass production. I'm certain there are black projects doing all sorts of custom hardware behind the scenes, most likely for military applications. No. The CPUs on supercomputers are basically the same as what you can purchase commercially. What distinguishes supercomputers from "the cloud" is better networking. > Single threaded performance is a huge priority because most problems we are interested in cannot be made parallel and hence do not benefit much from multicore. The few big problems very interesting to militaries are all embarassingly parallel. There are no super-secret CPUs that have better single-threaded performance than what is available commercially.
    • jiggawatts 6y ago
      That's not entirely true. Well, it is true that you can't get better chips with much higher clocks commercially, but there are specialist vendors making digital circuitry with crazy high clock speeds, and the primary market is the military. For example, the Rapid Single Flux Quantum (RSFQ) logic uses superconducting circuitry and experimental chips have hit 1 THz for simple ADC/DAC circuits, and I've heard of 100 GHz DSP chips in practical use. They're typically used for radio telescope amplifiers and digitisers, large static military radar installations, etc... People sometimes forget that the commodity market seeks a kind of Pareto Frontier optimisation where "cost effectiveness" and "low risk manufacturing" are key metrics. If you don't care about any of that and just want the maximum performance at any cost, there's some amazing technologies out there! https://en.wikipedia.org/wiki/Rapid_single_flux_quantum https://en.wikipedia.org/wiki/Rapid_single_flux_quantum
• mwilcox 6y ago
  Most of the cost of a supercomputer is from energy consumption
• dietr1ch 6y ago
  I'm far from an expert here, but I guess it really depends on how well you can scale performance out of a single machine. For "embarrassingly parallel" problems that's "trivial", but if you constantly need data that would end up on a different machine then it might start making sense to have specialized hardware that greatly reduced those costs. That being said, I'm also on your side in guessing that the problems that are money-wise worth running on a supercomputer are getting pretty limited as commodity hardware improves.
  • semi-extrinsic 6y ago
    HPC is almost entirely "commodity hardware" on the CPU and GPU side these days. Meaning there is no problem for someone with a bit of money and time to get one node (~ one motherboard with all its associated parts) equivalent to a top supercomputer. As others have said, it is the interconnect that makes HPC special. The ability to run a job across tens, hundreds, thousands of nodes, where all nodes need to exchange large amounts of data with all other nodes every millisecond, without network being a huge bottleneck that leaves your CPUs idling. If your job is never big enough to outgrow the biggest and baddest single server money can buy, then indeed you don't need HPC. But if you are running e.g. a full fluid dynamics simulation of an F1 racecar, a wind turbine, or a jet airplane, you sure do.
• wmf 6y ago
  Supercomputers are incrementally less custom over time. Slingshot is a good example of this, reusing many parts of the Ethernet protocol and even parts of a commodity Broadcom chip design. The rest of the Cray Shasta system is built with off-the-shelf CPUs and GPUs (although custom HPC GPUs are coming from AMD and Intel). Clearly the customers (mostly governments) think it's worth it. They also buy commodity clusters in addition to supercomputers so they're very aware of the differences. The biggest difference is the network and that's why Cray is developing that part themselves; some simulations are network-bound and using a commodity network it would either cost more than a supercomputer (due to lower efficiency) or it would never reach the desired performance at all (due to poor scaling).
  • gnufx 6y ago
    It's odd to class slingshot/dragonfly as less custom for HPC than even infiniband (and Mellanox is all-out on HPC-specific "offload"). Other interconnects designed for HPC include Bull's BXI, Fujitsu's Tofu, and European stuff, though they're even as common as Cray's, and I don't know how they compare.
    • wmf 6y ago
      I agree that Infiniband is off-the-shelf technology at this point, but previous Cray interconnects like SeaStar/Gemini/Aries were 100% custom and had no interop with standards.
      • gnufx 6y ago
        Right, if that's all custom means.
• the_svd_doctor 6y ago
  For many problems in HPC the bottleneck is the network. The compute node are relatively classical commodity CPU(/GPU). But without a (good, custom) network, you can't achieve much.
  • gnufx 6y ago
    For example, some of us have speculated that there are periods when around half of the time on the UK Archer system (with the previous generation of Cray interconnect) might be spent in one MPI collective. It's a pity there's no measurement of such things.
• TallGuyShort 6y ago
  I work in HPC and I think the technology and some of the things done with it are very cool, and it has secondary benefits, too. That said: look at where most of the Top 500 gets the funding for these machines and that might answer your question.
• fluffything 6y ago
  It's not a dumb question, that's just a problem that most people don't have. If you have hundreds of 80.000$ nodes with 8 V100 each consuming thousands of Watts per node, you start thinking about how much money is the 90% idle time of those nodes, waiting for data from the network, is costing you (probably millions of dollars per year). So yeah, a network that's 10x faster is worth every penny if it can turn the system from a 90% idle to <10% idle. That's why people always pay extra for NVLink, HDR Infiniband or HPC Ethernet interconnects, and also why Nvidia bought Mellanox last month... Essentially, when your machine has 1 ExaFLOP/s of compute performance, every second that it remains idle waiting for the network you... well lose 1 ExaFLOP. That's just a lot. Also, imagine millions of cores writing to the same harddrive simultaneously - hell try doing `ls` on a normal linux machine on a folder with 10 million empty files. The "storage" infrastructure to support instantaneous I/O on these systems is quite complex as well. Every second that the system is writing to disk you are wasting another ExaFLOP of compute.
  • bno1 6y ago
    But can't you make the same argument about bandwidth? Every second that the processing units are computing, terabits of bandwidth are unused.
    • magicalhippo 6y ago
      Except the goal is usually to compute, not to transfer data, so the interconnects are a necessary evil.
    • dooglius 6y ago
      No, computing is happening in parallel with data being sent/received
      • gnufx 6y ago
        What fraction of communication on a typical HPC system is overlapped with computation? My guess is rather little, despite the Mellanox sales pitch. I do wish people commonly made cluster-global profiling-type measurements to be able to answer that sort of question and do optimization.
• lazylizard 6y ago
  Its really easy to split a computer with 200gb of memory into 100 small ones with 2gb ram each. But what if you wanted to do something that would like 200tb of memory?
  • deleted 6y ago
    [deleted]
• dekhn 6y ago
  I worked in supercomputers for some time (molecular dynamics simulations, among other things). I am not convinced that the additional cost is worth it, with the exception of a few things. For example, if you're a nation-state and you need to make a submarine that is 5% quieter, or 5% faster, or whatever, for national defense purposes, having a CFD simulator that can work with sizes of problems or turnaround times that are significantly better can be a strategic advantage. There just aren't that many codes that can take full advantage of these monsters, and there are other, cheaper, faster ways to obtain roughly the same results. For example, in my ex-field of molecular dynamics, it seems like you can get away with minimal interconnect by running many simulations and pooling statistics across runs, which needs a bunch of IO bandwidth but doesn't need supercomputer-style interconnect. What's interesting is that supercomputer hardware is really great for high performance ML training, but the supercomputer folks came to that realization later, and only after companies like Uber ported the tensorflow distributed implementation to MPI (Horovod). And most of the ML people were poking around with slow interconnects. Now it's converging- the largest use of supercomputer-style systems in 10 years will be ML training, not physical simulations. Supercomputer spend has only limited impact on forward technological process.
  • C1sc0cat 6y ago
    But with modelling having much more grunt allows you to use much better models for things like hydrodynamics modelling / weather simulations. Which beats using a physical model and highspeed film projected onto a wall covered in graph paper and manually digitising data.
    • dekhn 6y ago
      if you can fit your model on a single GPU, do that, before parallelizing. Getting more grunt from tightly bound parallelism is one of the hardest problems in software engineering that there is. Weather prediction and climate modelling (two distinct fields in supercomputer) do qualify (you can't fit a high res model of earth in the ram of a single GPU).
  • gnufx 6y ago
    There are different types of molecular dynamics that non-ex people run, and different modes of the same code may be more or less latency sensitive (e.g. two "standard" Gromacs benchmark cases I'd have to look up). While I'm unconvinced of the overall merits of exascale, it's clearly not true that you can generally do all the large-scale, high-resolution, or whatever, calculations in an embarrassingly parallel way. I haven't kept up with the public applications for CORAL et al, but there are definitely enough for petascale and above. My bitter experience, particularly with "big data" people, is that they just won't be told by those with long relevant research computing experience in HPC and similar. It's hardly that HPC people don't know what systems can do, particularly if they live and breathe things like distributed linear algebra. I despaired, and largely gave up, when someone strode into a chair From Industry and assured us that the university didn't do Big Data, notwithstanding LHC, astronomy, sequencing, synchrotron work, etc., and was then going to build a Big Data Cluster from a few knackered PCs (and in a basement!) to better the HPC systems. Then I had MPI explained to me.