10 ms·
Linux dominates supercomputers as never before
- CraigJPerry 12y agoWhat does Linux not dominate? I can think of only a few categories: desktops, ATM machines, real time, SCADA. What are the remaining big sectors Linux doesn't yet dominate?
- epochwolf 12y agoPoint of Sale machines, machine controllers, etc.
- marcosdumay 12y agoDesktop and laptop computers. Also CAD plataforms (that are a subset of the first, but an important subset).
- legulere 12y agoReal time computing, routers, sensors
- dublinben 12y agoMore routers run linux than you'd think. There's very little reason to reinvent basic networking and routing components and a kernel, when all the common SoCs are supported by a free OS. For the small price of stashing a source code bundle somewhere on their website (if even!) companies can get most of what makes their product tick off the shelf.
- evv 12y agoAFAIK, most routers use OpenBSD over Linux for security reasons.
- dublinben 12y agoI guess I'm only considering commodity, consumer routers. I'm sure you're right at higher levels.
- intorio 12y agoFor licensing issues, actually.
- bashinator 12y agoJuniper Networks' JunOS is a FreeBSD derivative.
- kjs3 12y agoWhat significant networking vendor runs OpenBSD? And I don't mean the low-end, low-support, OpenBSD on an embedded board from China stuff (so, no, Soekris isn't a significant vendor in the router space). I can't think of a single business or enterprise class one that does. Cisco doesn't use it (IOS is home grown, IOS XR is QNX, NX-OS is Linux), Juniper uses FreeBSD, Arista uses Linux, HP uses (I think) Comware (linux), Intel & Broadcom are pushing Cumulus (linux) on their reference networking platforms, Dell/Force10 is NetBSD based (and older Dell is VXWorks, I think), Pretty sure Brocade is Linux, but they've also used VXWorks. Don't know what Hauwei uses (that wasn't stolen from Cisco, or course). Not that this is authoritative, but I've talked to a couple of execs at companies who were using OSS platforms for their networking devices (Nokia, Juniper, Force10, Arista) and all of them basically said OpenBSD wasn't a consideration because of the open hostility the people leading the project had towards commercial enterprises. No one was interested in betting the company with the threat of a "DARPA funding" moment hanging over them, much less someone who was more interested in dictating terms and making "statements" than cooperation on business relationships. [edit for punctuation]
- noryungi 12y agoI believe the OpenBSD community of developpers is not too concerned with popularity or integration with vendor's hardware. They are more concerned with doing things right (or, some would say, doing things their own way). Nevertheless, there are some small companies selling things based on OpenBSD - even though, I'll admit, they are mostly companies started by OpenBSD developpers: http://www.openbsd.org/products.html http://www.openbsd.org/products.html You can also get some consulting & support: http://www.openbsd.org/support.html http://www.openbsd.org/support.html And, of course, there is always the strong possibility that some people are using OpenBSD while keeping it hush-hush and "under the radar" -- maybe even under the radar of their own management. Make of this what you will.
- ptfoobar 12y agoRT is a matter of time (no pun intended). In fact hard real-time is already done using Linux and RT_PREEMPT is believed to go in the main tree soon.
- nardi 12y agoHow has no one mentioned mobile? Smartphones/tablets? You know, the largest segment of general computation devices?
- gamegoblin 12y agoAndroid is based on the Linux kernel at least.
- azdle 12y ago> How has no one mentioned mobile? Because Linux does dominate mobile? Android is Linux and iOS is a *nux. (iOS is a BSD derivative.)
- JoshTheGeek 12y agoBSD isn't Linux. Android uses the Linux kernel, but not iOS. I'm not sure which dominates, though
- barbs 12y agoIn numbers, I'm pretty sure Android dominated (can't remember the exact numbers but Google should reveal them for you). In revenue (app-sales etc) iOS dominates.
- shmerl 12y ago> In revenue (app-sales etc) iOS dominates. Who made such research? I'm not sure if it's even possible to know exactly, unless all vendors who run their own stores publish sales statistics.
- TheSoftwareGuy 12y ago>unless all vendors who run their own stores publish sales statistics. They do, thats one of those things that companies that are traded publicly are required to do.
- robotresearcher 12y agoThe profitable part of the cellphone business.
- kiram9 12y agoA lot of broadcast TV is all windows/mac. Even for things like playout servers, automation, and real time graphics engines mostly run on windows or mac.
- pjc50 12y agoPretty much all end-user facing systems? (Android barely counts as Linux for reasons discussed in my other comment)
- odonnellryan 12y agoSmall and medium sized business servers?
- lucidguppy 12y agoWhat alternatives are there really?
- Alupis 12y agoI really amazes me, that Linux is so darn flexible that it runs on some of the world's largest machines, down to the smallest embedded devices. After years of being a Windows fanboy -- I look back now and am very glad I made the switch. With Linux -- it's never a case of "you can't do that".
- TallGuyShort 12y agoWhat I think is more amazing is that with the exception of the 2 Windows systems, EVERYTHING is either AIX UNIX or Linux, or a combination. UNIX started as a "microcomputer" OS, but it really has proven to be incredibly flexible. Personally, I think the ideas that have come out of the Solaris and BSD communities are invaluable and I always like to see them included in projects - I'm a little disappointed they have lost favor in the HPC community. They are similar enough to Linux and other UNIX-like OSs that targeting all of them is relatively practical in most cases, and users will find similar interfaces everywhere, but their ideas, communities and approaches are different enough that I think they would prevent it from being a monoculture.
- _delirium 12y agoOne of the things that imo has pushed some of the other Unixes out of HPC is the move to using cluster file systems. Any Unix supports a traditional NFS-mounted NAS solution for storage, as used to be typical in HPC clusters (and still is fairly common in smaller installations at universities). But only AIX and Linux really have solid cluster filesystems; there's nothing in the BSD world analogous to IBM GPFS, or to Lustre, and porting is nontrivial.
- TallGuyShort 12y agoI think you're right. DragonflyBSD probably isn't widely used enough to be trusted by high-budget HPC projects anyway, but I was under the impression their HAMMER filesystem was a cluster file system. Looking at the website now I can't seem to find support for that, though...
- onalark 12y agoIt's worth pointing out that this is what most people think of as the traditional "Linux Operating System" but sans the Linux kernel. Also, there are a few components on these supercomputers that you won't find on a typical workstation or cluster machine. First, the kernel itself is frequently very small, lightweight, and much closer to what you would find on an embedded system than on a traditional desktop computer. That's because once the program on a supercomputer is loaded, the kernel's job has been mostly to get out of the way. This isn't to say that you don't find the Linux kernel on these supercomputers, it just isn't as common as you would think from reading this piece. Fear not, many pieces of these operating systems are still open source. Here's IBM's fusedOS prototype:https://github.com/ibm-research/fusedos https://github.com/ibm-research/fusedos. Second, many of these computers primarily only run code in C, C++, Fortran, and Python. These tend to be the only major languages in play on the HPC machines, with acceleration frameworks such as OpenMP, OpenCL, and CUDA playing major roles. Finally, everything is glued together with MPI, a high-level (at least it was in the 90s) abstraction for scientific programming that maps down to very high-performance networks designed to help scientific codes "scale", that is, run effectively when millions of cores are simultaneously engaged. These are beautiful machines producing important science, and the GNU/Linux operating system plays an incredibly important role in both their implementation and culture.
- deleted 12y ago[deleted]
- gh02t 12y agoThis is not exactly true, most of the ones I know are running straight up Linux. Titan, for example, runs basically SLES on the login nodes and Compute Node Linux on the compute nodes. CNL is a pared down version of Linux, but it definitely is the "real" Linux kernel with the functionality you'd expect to be there. Most are just slightly spruced up commodity server hardware running Linux. I'm not sure if this is what you're suggesting, but they don't run C/Fortran/whatever on bare metal. They're run by the OS on the compute node just like a normal OS process, except that tasks are dispatched to compute nodes by a central cluster manager. Processes running in a gang communicate via MPI to share data, though coprocessors are also pretty popular as well so you see a lot of communication between the host processor and a coprocessor too. Titan and Tianhe both actually have most of their compute power in the coprocessors (Xeon Phi and Nvidia Tesla, respectively), but they're still arranged in a master-slave arrangement just like if you bought a Phi or Tesla and stuck it in a spare PCI-E slot. They use plain old PCI-E, too. The Cray XT/XE series (a popular model of which Titan is an example) is basically just really nice blades with integrated cooling and a network backbone in a custom cabinet and possibly coprocessors attached to each blade. You could just as easily run Windows XP and play Minesweeper on each blade if you really wanted to, except maybe for some driver issues. The most foreign thing is probably the network backbones, where fabric architectures like Infiniband are popular. They're also not limited to specific programming languages. In truth, you can run whatever you want if someone has paid the bill for your resource allocation. I watch people run MATLAB on large clusters all the time, which hurts me because it's so damned inefficient. That said, Fortran and C++ comprise the overwhelming majority of large and computationally taxing codes. Just because all that power is there doesn't mean that all of the users take proper advantage of it. One of the larger calculations run on Titan that I know of (Denovo, a nuclear reactor simulation code) didn't even use the GPUs, only the CPUs. Making codes that can take advantage of GPU processing ergo Titan and its predecessor Jaguar has been a major project at the DOE, with libraries like Trilinos being developed to make it easier on scientists, many of whom are only computer programmers as a secondary concern. The setup you're describing used to be how it was until up to maybe 10 years ago and there are still systems in the top 500 that work like that. Probably some new ones being built, too. But what I've described is what seems to be in fashion these days mostly and the machines I use are all like that. I've heard mumblings about FPGA coprocessors being the Next Big Thing, but we will see.
- nroose 12y agoUm... That graph goes up to 600. Kind of an epic fail when it is an article about the fastest computers in the world.
- mnw21cam 12y agoYes, and also a fairly basic research fail, referring to the ECMWF's old supercomputer at position 60 as the UK's weather predicting system. They have couple of new systems at positions 19 and 20. ECMWF is for Europe, not UK, even if it is based in Reading. The key is in the name.
- zurn 12y agoAnother way to put it is that specialized supercomputers no longer exist, and the top500 is now populated with super-branded racks of PCs running Linux with souped up network cards. For the young ones in the audience, the supercomputer term has historically been used to describe machines with largely custom hardware from the CPUs up - or at least from the CPUs onwards. Cray vector machines are the canonical example. These were shared-memory / single-system-image machines, software really saw it as one computer instead of a networked cluster.
- alephnil 12y agoI don't think this description is entirely fair. What is expensive with these computers is the fast interconnection buses between processors and modules, which is needed for many scientific problems that can't easily be divided into independent subproblems. Just having a large cluseter of computers connected with normal Ethernet would not cut it. OK, you say souped out network cards, and if you had said souped up switches as well, I could have given you some points, but this is the major cost of modern supercomputers. You can get a non souped-up cluster with the same number of processors for a one digit percentage of the price of the supercomputer.
- walshemj 12y agoSome hadoop clusters are looking at/using infiniband - cluster network design depends on what sort of problems your dealing with.
- weland 12y agoPretty sure everything in top500 is using infiniband by now. I was briefly involved with the supercomputing world about four years ago. "Should we use Infiniband?" was no longer considered a non-rhetorical question even then.
- alephnil 12y agoWithout Infiniband or similar proprietary buses, it is usually not considered a supercomputer at all, but merely a cluster.
- bch 12y agoWhat _are_ examples of HPC installations that *BSD is key? Free, Net, Dragon Fly, Open?
- piokuc 12y agoIn the article it was mentioned that ECMWF's supercomputer was the fastest one of those running not on Linux. It's worth mentioning that ECMWF is in the process of migrating to an XC30 Cray system which runs on Linux.
- jason_slack 12y agoI think that I read a long while back that there was a cluster of PlayStation 3's, all running Linux that was pretty fast too. Edit: Here is one of the articles: http://phys.org/news/2010-12-air-playstation-3s-supercomputer.html http://phys.org/news/2010-12-air-playstation-3s-supercompute...