16 ms·
All 500 of the world's top 500 supercomputers are running Linux
- linarism 9y agoSerious question (not in IT), what else can they run?
- Manozco 9y agoNetBSD because "Of course it runs NetBSD" :)
- wmf 9y agoIIRC at one point Microsoft paid one of them to run Windows and years ago there was one running OS X. Decades ago during the Unix Wars there used to be proprietary HPC OSes.
- astrodust 9y agoWhen Cray and SGI were still in the game there was a lot more operating system diversity. Apple's Xserve cluster was an interesting experiment but it never really went anywhere.
- mschuster91 9y ago> Apple's Xserve cluster was an interesting experiment but it never really went anywhere. A pity Apple does not manufacture real servers any more. With the old cheesegrater-mac-pros, which were the last thing available after the Xserve death, you at least have the option to put in 4 disks and do a RAID setup, but no remote power/console management or a serial port...
- astrodust 9y agoThey've acknowledged the "trash can" design was a mis-step and they're working to correct it, so there's that. Competing in the server market was very difficult for Apple, their needs are often radically different from consumer or even "pro"-consumer. They realized the couldn't "win" at servers, so backing out was the best bet for them.
- mschuster91 9y ago> They realized the couldn't "win" at servers, so backing out was the best bet for them. The need for servers is there, or otherwise the "macOS Server" would not be in the App Store. It actually is really nice, it provides everything needed for a business: LDAP, file server, mail, calendar, webserver, TimeMachine centralized, print server, ... - but how am I supposed to recommend this to a customer when Apple does not have any kind of HA ensurance (dual power supply, remote management, auto power on after AC loss, proper SNMP support, Wake On LAN), a way to attach more than 4 disks (or a real RAID, not a software one like CoreStorage does, and they cut out RAID management of diskutil anyway) or actually hardware that I can mount in a rack without wasting immense amounts of (expensive) space? For the small-business scale, Microsoft actually has the SBS Server licenses, which are not tied to any specific vendor and are exactly the same tools used at big companies so a company can hire any competent consultant/admin for management. Hell, they could simply ask Dell, HP or whomever to sell a specific server as "macOS Server compatible", charge a bit of money for the OS and that's it - people would rush to buy it, and if only to run a CI/CD environment for mobile app building without having to run it on non-DC-grade hardware!
- astrodust 9y agoThere's still a market, but Apple's not interested in being a bit player in it. HP, Dell, and SuperMicro utterly own the server space. Taking those companies head-on with a single one-size-fits-all server is never going to work, and Apple wasn't prepared to build out a complete server line. The "Server" software they have is for small businesses and is pretty good for 10-20 users, but beyond that you'd probably use Google Apps, Microsoft Exchange or something more serious. It's a nice thing to have, but it can't compete at an enterprise level.
- rootbear 9y agoBoth Cray and SGI (now HPE) are both still very much in the game. Their systems all run Linux.
- astrodust 9y agoI mean with their respective UNIX-like operating systems. There was a time when you'd find server rooms with a mix of AIX, HPUX, IRIX, SunOS, VMS, A/UX, and Netware and that was considered normal. In an academic environment you'd have someone running Plan9 or BSD some other quirky thing as well just because. Then Windows and Linux came along and started to kill off this diversity one system at a time.
- reducesuffering 9y agoFreeBSD? Netflix runs it on their OpenConnect boxes, but that's about the extent of my knowledge.
- pera 9y agoA few years ago I remember some of them were running BSD derivatives.
- b3lvedere 9y agoDepends on what you want to do with it. Basically anything else that operates on their hardware infrastructure is ok. Though it may be very challenging to get a useful supercomputer on very old operating systems. I'm guessing one of the major factors to decide for Linux is this: "The licensing cost of a custom, self-supported Linux distribution is the same, whether you're using 20 nodes or 20-million nodes."
- IncRnd 9y agoThe Cray I used to program used an OS called Unicos, which was a variant of Unix.
- PeachPlum 9y agoBlue Gene runs Plan 9 and Inferno
- fencepost 9y agoThey could likely run any of the BSD variants. Some form of commercial RTOS might also be viable. Even Minix might be an option (hire away some of those Intel folks!) though I'm not sure of its hardware support. That said, why bother? Linux is customizable enough to strip out almost everything extraneous for nodes and has the largest pool of experienced developers, and it's unlikely that any of the other options would have any performance advantage - and less likely that they could keep any performance advantage once it was identified.
- caf 9y agoWhy on earth would you want a RTOS for HPC? RT is all about privileging latency over throughput, whereas in HPC throughput is everything.
- fencepost 9y agoIt's unlikely that you would, but once you make it through Linux, the BSDs (including macOS), Unix systems (legacy?), Windows and Minix there's just not that much else left that's modern. At least with a RTOS or something designed for embedded systems you have some hope that it's designed to be lightweight.
- dTal 9y agoPerhaps because a lot of tasks are not embarassingly parallel, which means moving data around, which means network interconnects, which means latency matters. Every millisecond delay in forwarding a packet is thousands of instructions stalled. I'm not saying this is actually a sensible tradeoff for any extant HPC architecture, but it could be. You certainly want the 'OS' that runs the pipeline on your CPU to be real time!
- jamiesonbecker 9y agoTrue, a lot of tasks are not embarrassingly parallel, as you point out, but most tasks that run on supercomputers are embarassingly parallel. That's why those supercomputers have value in that problem domain in the first place.
- ghaff 9y agoIn recent history, a fair bit of AIX on IBM Power. Further back, lots of other Unix variants and other vendor-specific OSs.
- martinpw 9y agoIf you go here: https://www.top500.org/statistics/overtime/ https://www.top500.org/statistics/overtime/ Then select "Operating System" and submit, you can see the OSs that used to be run on systems over the past 10 years.
- rootbear 9y agoNice chart! The speed with which Linux supplanted proprietary Unix systems is remarkable. It would be interesting to know what version of Linux is used most often on current supercomputers. The supercomputer here at Goddard runs SLES (Suse) and I believe other NASA supercomputers run that as well.
- b_emery 9y agoA win for open source. I was just thinking the other day that it's a shame that Linus is not a billionaire. Seems he's doing just fine however: "Finnish-American software engineer and hacker Linus Torvalds has as estimated net worth of $150 million and an estimated annual salary of $10 million. He earned his net worth as the principal force behind the development of the Linux kernel." Source: https://www.therichest.com/celebnetworth/celebrity-business/tech-millionaire/linus-torvalds-net-worth/ https://www.therichest.com/celebnetworth/celebrity-business/...
- bearbearbear 9y agoPlease keep in mind however that many of the world's "supercomputers" are simply "bigger dick" projects where some government or organization slapped a few thousand blades into a rack and connected them with fiber crossbars so they could say they "built a supercomptuer." And also that there's no standardized definition of what a "supercomputer" is. My iPhone is not a "supercomputer" even though it has thousands of times more computing power than a Cray 1, which is considered a "supercomputer."
- vram22 9y agoInteresting, that site says his background is actually Swedish(-speaking, at least). >Born Linus Benedict Torvalds on December 28, 1969 in Helsinki, Finland, Torvalds is the son of journalists Ana and Nils Torvalds who belong to the Swedish-speaking minority.
- rootbear 9y ago
- wmf 9y agoNote that BlueGene/Q is commonly reported to run "Linux" but 99% of the nodes are running CNK instead. https://computing.llnl.gov/tutorials/bgq/#Software https://computing.llnl.gov/tutorials/bgq/#Software
- bantunes 9y ago> This is an open source, Linux-like, light-weight, 64-bit operating system that runs on the compute nodes Seems to be a Linux kernel fork with performance optimizations. EDIT: I stand corrected https://en.wikipedia.org/wiki/CNK_operating_system https://en.wikipedia.org/wiki/CNK_operating_system
- BeefySwain 9y agoWith such a glaring misconception, it makes you question the rest of the information presented.
- icc97 9y agoI remember Linus mentioning [0] about the major benefits of tuning Linux for super computers that came to be really useful for when trying to get performance out of a mobile phones decades later. [0]: https://m.youtube.com/watch?v=MShbP3OpASA&t=39m https://m.youtube.com/watch?v=MShbP3OpASA&t=39m
- PeachPlum 9y agoRonald Minnich (if you don't know him see [1]) comments : [1] https://share-ng.sandia.gov/news/resources/news_releases/sandia-computer-scientists-successfully-boot-one-million-linux-kernels-as-virtual-machines/ https://share-ng.sandia.gov/news/resources/news_releases/san... ----------------- "The semiannual TOP500 Supercomputer List was released yesterday. It also shows that China now claims 202 systems within the TOP500, while the United States claims 143 systems." My fellow Americans: Did you ever want to directly experience the transition to being a Tier 2 country? This is what it's like. It starts with the cancellation of the collider in Texas in 1994 (see Tyson's comments on that one). It continues as steady decline through the next two decades, under both Republicans and Democrats, and now it is reaching its culmination in the Trump administration with the replacement of scientists by political hacks. It's not fun. It's quite sad to watch, that's for sure.
- pcunite 9y agoTrump is not the problem. It is all the non-elected dark figures behind it all.
- PeachPlum 9y agoI wanted to fully quote Ron because they're his feelings and I think that's the only fair thing to do. He's my friend (this is from Facebook (a public post so I don't feel it is violating any trust))
- hutzlibu 9y agoSure thing, blame everything on the Illuminati. But were there not actual people voting for him, a lot? (but well, the alternative? you americans degraded indeed ...)
- mr_toad 9y agoFrom what I can see anti-science, anti-technology attitude is endemic to modern English speaking western culture. Elected and un-elected officials are just a symptom of the problem. They'd all flip-flop if they thought the political and media climate was shifting.
- EthereumDublin 9y agoI think it's hilarious that so many run CentOS instead of RHEL. Redhat has managed to make their product worse than free with their licensing bullshit.
- caf 9y agoIt makes sense though, because a supercomputer installation is exactly the kind of place you'd expect to have its own experts that support and customise the OS. And there really aren't that many supercomputer installations, so any that RHEL does have are probably more for the prestige than the revenue.
- wideem 9y agoThe real question is are they all runix minix(Intel ME)
- vortico 9y agoMost of them run Xeon, so yes, but about 10% use either Opteron, SPARC64, Power, and Sunway.
- eggy 9y agoWhere does that put Minix if the supercomputers have the ME chip? Based on the findings this past month, Minix is on the most computers.
- deleted 9y ago[deleted]
- dxxvi 9y ago"For example, the new Linux 4.14 enables supercomputers to use Heterogeneous Memory Management (HMM). This enables GPUs and CPUs to access a process's shared address space. Exactly 102 of the Top500 are using now using GPU accelerator/co-processor technology. All of these will perform better, thanks to HMM": it sounds like some of the 500 supercomputers use Linux 4.14. I guess none of them uses Linux 4.14 yet.
- tammer 9y agoWith an anti-science regime in charge of the government, America will only continue to see its technological lead decline. Brutal to be confronted with such a cutting opinion in an otherwise congenial article and to have no thoughts of disagreement.
- erikpukinskis 9y agoAmerica is a bet that chaos has some advantages over good governance.
- jjtheblunt 9y agoOne of the most discerning and succinct comments i've ever read
- oldsklgdfth 9y agoIs good governance obtainable in a country of such scale and idological diversity?
- bassman9000 9y agoWith an anti-science regime in charge of the government, America will only continue to see its technological lead decline. This is idiotic. The new admin could well commission new supercomputers for espionage or other military purposes. The constant politicization of every single aspect of life is going too far.
- deathtrader666 9y agoHow does one go about learning more on this topic -- being a Linux admin for a supercomputer?
- jabl 9y agoI don't think there's a degree for supercomputer admin, it's mostly on the job training. And to be fair, a lot of it is similar to generic Linux sysadmin stuff. Especially so nowadays with the focus on devops/infrastructure-as-code/etc., or generally the idea of how to administer many nodes in a scalable fashion. If you want to play around, go to http://openhpc.community/ http://openhpc.community/ , fire up a few virtual machines, follow the installation instructions, and start playing around.
- ForFreedom 9y agoThese linux OS are customized, compiled and not out of the box, correct?
- kbart 9y agoMore correct title would be "All 500 of the world's top 500 supercomputers are running Linux kernel". Of course nobody uses off-the-shelve Ubuntu (or another distro) to power supercomputer. I'm not an expert, but I bet that it's heavily modified Linux kernel with custom userspace optimized for a given task in mind (scientific calculations, simulations etc.)
- pjvandehaar 9y agoAt https://www.top500.org/statistics/overtime/ https://www.top500.org/statistics/overtime/ click "Operating System" and "Performance Share" to see: - 300 PFlops: Linux - 260 PFlops: other - 150 PFlops: Cray Linux - 130 PFlops: CentOS - 1 PFlop: bullx supercomputer suite (supports RHEL/SUSE/etc)
- krylon 9y agoI vaguely remember that Microsoft offered version of Windows Server for HPC environments at one point. Are the reasons for not using Windows purely financial (in a cluster with thousands of nodes the licensing costs can become substantial) or are there technical reasons as well?
- microcolonel 9y agoI doubt licensing costs have much to do with it, it seems like flexibility would be the driving factor. Supercomputing centers, vendors, institutions which use supercomputers, etc. all frequently modify and contribute modifications to the Linux kernel and associated userland. You have to imagine they are making changes which Microsoft could not afford to care much about, nor turn around in competitive time.
- krylon 9y agoThat makes sense. Thank you!