10 ms·

Ask HN: Why are supercomputers all running Linux and not BSD?

132 points by dbosch 9y ago

loop0 9y ago
I would guess it is because linux has a wider hardware support than bsds. As you're building a supercomputer it makes sense to have the faster hardware, implying they are new technology.
vectorEQ 9y ago
This might shed some light on your question: https://en.wikipedia.org/wiki/Comparison_of_operating_system_kernels https://en.wikipedia.org/wiki/Comparison_of_operating_system...
8fingerlouie 9y ago
Some will say better hardware support. While Linux has better hardware support, i usually find this to be in the more exotic direction. I think it's simply down to Linux being where the money is. The big players (IBM, Dell, etc) are all actively promoting Linux, and trained personel is also somewhat easy to find. So Linux is "the beast you know". As for FreeBSD, it might be a technically better platform, but it is living in Linux' shadow. Personally i run FreeBSD for the excellent documentation, stability, features like ZFS, but nothing i run couldn't just as easily run on Linux.
- alvil 9y ago
  > While Linux has better hardware support Linux has wider, not better hardware support. If OpenBSD supports some hardware it simply works out of the box and is stable and rock solid.
  - scardine 9y ago
    From the famous poster hanging on Facebook's office: "Done is better than perfect".
    - xxxdarrenxxx 9y ago
      Yes the snowball effect, exponential. Let's take this example. Today I woke up and said to myself. I'm gonna make a supercomputer. I will research and research, and come to a point where I could choose Linux or BSD for example. I read about one thing, and another and it all already has real examples running Linux (the top 500). So I can re-invent the wheel for god knows how long, or use what works for the top 500, which seems to be Linux. Decision easily made. Now the top 501 super computers run Linux. Who is brave enough to not actively accept the wheel that is. I probably wouldn't, especially with the fact that I have * no insurance * that a non-Linux super computer can hold it's own in real world applicable problem solving. Theoretically one might get there, but the pragmatism is strong.
- jo909 9y ago
  And hardware support does not only mean that it "somehow" works, but that the hardware vendor and the operating system vendor have certified this combination and will support you with all your problems for many many years. Edit: support as in commercial support with SLA etc
- ska 9y ago
  While Linux has better hardware support, i usually find this to be in the more exotic direction. HPC, and supercomputing in specific, tends to the exotic (particularly with busses)
- aeorgnoieang 9y ago
  > features like ZFS Linux has ZFS.
  - wolrah 9y ago
    > Linux has ZFS. Kind of. You have the choice of the FUSE version that's legally free and clear but has obvious FUSE related performance limitations, or the kernel version which has great performance but is questionable at best from a legal standpoint because the CDDL is not compatible with the GPL. Canonical has decided they're willing to take the risk by bundling it in Ubuntu and so far it hasn't backfired on them, but there's good reason to believe that Oracle's lawyers may have something to say about it if they ever feel that ZFS-on-Linux is threatening any of their products.
VSpike 9y ago
I wonder if this is another validation of the "worse is better" philosophy as described recently in an HN article http://minnie.tuhs.org/pipermail/tuhs/2017-May/009935.html http://minnie.tuhs.org/pipermail/tuhs/2017-May/009935.html and also discussed at https://www.jwz.org/doc/worse-is-better.html https://www.jwz.org/doc/worse-is-better.html
- erikj 9y ago
  JWZ seems to dislike Hacker News for some reason, judging by the redirect.
  - VSpike 9y ago
    I forgot about that. Try https://web.archive.org/web/20171114181219/https://www.jwz.org/doc/worse-is-better.html https://web.archive.org/web/20171114181219/https://www.jwz.o...
smilesnd 9y ago
A comment I found while looking at linux OS that are run on super computers. "Originally, the top 500 list was populated entirely by proprietary Unix systems from vendors like Cray research, SGI, etc. In June 1998, the first Linux system entered the top 500 list. By June 2003, Linux systems passed the 25% mark, accounting for 139 of the top 500. By November of 2003, Linux systems comprised over 56% of the top 500. By November 2006, Linux made up more than 75% of the top 500. You get the idea. Over the years, there were a few attempts by microsoft to get into supercomputing, and there were BSD and Mac systems." Since time is sold on these supercomputers they probably want to run all the same/similar OS so they can compete selling time on them. Also if one person has success everyone else will copy them. https://linux.slashdot.org/story/17/11/14/2223227/all-500-of-the-worlds-top-500-supercomputers-are-running-linux https://linux.slashdot.org/story/17/11/14/2223227/all-500-of... Slashdot has a ton of comments discussing bsd vs linux on this subject matter, but I didn't see anything to helpful. My only thought is large companies like netflix use bsd more for CDN because from what I been told bsd has the best I/O handling. Why they don't use it for the rest of there infrastructure? Maybe linux is better at crunching numbers and bsd is better for network and security? No idea thats my best guess.
- KaiserPro 9y ago
  I think netflix use BSD because they wanted to use BSD. Sure some flavours of BSD have ZFS built in, but thats a pretty rare corner case. Linux has two things that are extremely useful compared to BSD: 1) commercial backing (should one choose it) 2) first class support for inifinband, top end ethernet (should they use it) and storage controllers
  - smilesnd 9y ago
    Netflix uses bsd because it has great IO handling. https://www.quora.com/Why-did-Netflix-choose-FreeBSD-over-Linux https://www.quora.com/Why-did-Netflix-choose-FreeBSD-over-Li... But all the supercomputers use their own custom linux. So no commerical backing. Also these computers are not your standard data center. They cut networking and storage to a minimal because those are bottlenecks. These things are just massive ram/cpu/gpu boxes connected properly through pci. Edit: I was looking at Sunway hardware specs the number one supercomputer they use a PCI-E 3.0 connection for all there nodes. Communication between the nodes is 12GB/second with a latency of 1 us. Their total ram is 1.31 PB
ktpsns 9y ago
Scientific high energy physicist here with regional HPC center on the same floor. My observation is that administrators tend to enterprise distributions such as Scientific linux, Suse linux enterprise server (SLES), together with commercial MPI implementations such as IBM MPI and Intel MPI. On the other hand, people are used to Linux, in my environment literally everybody has Ubuntu on their notebook and workstation. They know how to run their python analysis scripts there and the only thing they have to change when going to the cluster is the adoption of an environment managament system (such as http://modules.sourceforge.net/ http://modules.sourceforge.net/). (However, I have to admit I never got in touch with BSD and don't know the differences in user space)
- nerdponx 9y ago
  Modules looks really interesting. Makes me wonder why Continuum is out there trying to reinvent the wheel with Anaconda. Glad to have something I can use at work to replace Conda environments. Now all it needs is Powershell/CMD support so I don't have to use it inside Cygwin...
  - Certhas 9y ago
    These solve completely different problems. On my HPC I load the conda module to run Python. They are really different tools for different jobs.
  - godelski 9y ago
    Modules are really about environments (including software management). Anaconda doesn't handle this. For example, Conda its version of HDF5 and points to its environment path. Let's say you want to be using a different version of HDF5. An easy way to do this is just use a module so that you load this. You are creating an easy way for the user to set up their environment, where they really don't have to know anything about it. It also helps with versioning. It is not uncommon to see various versions of gcc and intel compilers. In essence the user should be able to load their environment with a few module loads. Here's some more info, if you're interested [1]http://www.admin-magazine.com/HPC/Articles/Managing-the-Build-Environment-with-Environment-Modules http://www.admin-magazine.com/HPC/Articles/Managing-the-Buil... [2]http://www.admin-magazine.com/HPC/Articles/Managing-Cluster-Software-Packages http://www.admin-magazine.com/HPC/Articles/Managing-Cluster-... [3]https://uisapp2.iu.edu/confluence-prd/pages/viewpage.action?pageId=115540061 https://uisapp2.iu.edu/confluence-prd/pages/viewpage.action?...
jpgvm 9y ago
HPC may look like COTS gear but it's not. BSD doesn't have drivers for Infiniband and other HPC interconnects. Nor does it have client drivers (let alone server implementation) for Lustre which is the distributed filesystem used by most super computers. I imagine MPI support on BSD is also likely non-existent. Then there is the matter of accelerator support, i.e NVidia GPUs and Intel Xeon Phi. It's not to say that some vendor couldn't reasonably build a BSD based supercomputer, it's just highly unlikely given how much stuff is missing.
- nl 9y ago
  This is the correct answer (especially Infiniband - and Aries on Crays) Also NUMA is very important on supercomputers, and it works well on Linux. The other thing worth noting is the much better support IBM has for Linux on PowerPC (2 in the top 10). I think Sunway (most powerful in the world) is a Linux shop too.
  - a012 9y ago
    Is it egg and chicken matter? Vendors don't write driver for BSD and BSDs lack of users because lacks of drivers. Honestly I hope I can run an OpenBSD and install whatever driver for my plugged in devices, both for my personal and production servers.
    - convolvatron 9y ago
      its very much chicken and egg. Cray used Linux because all the customers were using linux. There was never a technical meeting discussing their relative merits. The Tera MTA project was actually BSD based, because it was from an age where the BSD project had clear technical superiority (and they were probably worried about complying with the GPL) As others have mentioned there was a Mellanox stack in Free circa 2005 that I worked with. It was used at Isilon (BSD based) in production. There really isn't a technical discussion here at all, when an overwhelmingly large part of your userbase uses X, it would be pretty stupid to only support Y, and probably not defensible to support X and Y
snvzz 9y ago
Because BSDs don't scale in that direction. Dragonfly's design shows promise, but it's not anywhere near ready for supercomputers yet.
- frankharv 9y ago
  Yes I agree most with your comments. The reason is NUMA. FreeBSD performance on NUMA is poor. It was only implemented in 2015. It still needs tuning. Dragonfly seems to be hard at work on the problems. http://lists.dragonflybsd.org/pipermail/users/2017-February/313242.html http://lists.dragonflybsd.org/pipermail/users/2017-February/...
mkj 9y ago
Intel compilers aren't available for bsd. The improved optimisation versus gcc is worthwhile.
- reacharavindh 9y ago
  This! I’m a HPC sysadmin, and I used FreeBSD for all infrastructure services - DNS, DHCP, PF, ZFS based backup server et al And strictly CentOS with tightly controlled installations of intel MKL libraries and its ecosystem.
- kev009 9y ago
  They actually are, as is VTune and some other commercial stuff from intel and the open source libraries like ISA-L and IPP and frameworks like DPDK, SPDK and NV-DIMM stuff.. all work on FreeBSD. Last I heard from my rep, intel was discontinuing icc altogether because it didn't make a lot of sense to not put the optimizations in the compilers most people use.. gcc, llvm, vcpp.
  - mkj 9y ago
    Ah released in 2015, cool. I'd assumed Intel kept their compilers as a competitive advantage even if they weren't profitable by themselves. Could certainly see it happening though.
RantyDave 9y ago
Because the people who use supercomputers just want to crunch numbers - the operating system is a distraction at best, and Linux is the path of least resistance.
jacquesm 9y ago
Because of one man: Donald Becker. At the beginning of the commodity super computer era Donald did an absolutely amazing job squeezing out every last bit of performance from commodity networking hardware for 'Beowulf' style clusters. This gave Linux a head start and the self-reinforcing effects of such a head start did the rest, it made answering the question 'for which OS should we start writing drivers?' for specialty HPC hardware a no-brainer. https://en.wikipedia.org/wiki/Donald_Becker https://en.wikipedia.org/wiki/Donald_Becker
- pgtan 9y ago
  ~ $ dmesg | grep Beck 3c59x: Donald Becker and others. www.scyld.com/network/vortex.html
  - kelnage 9y ago
    Sadly that link is long dead, but the Wayback Machine comes to the rescue: https://web.archive.org/web/20000619092736/www.scyld.com/network/vortex.html https://web.archive.org/web/20000619092736/www.scyld.com/net...
SEJeff 9y ago
Because BSD's SMP support has traditionally been pretty terrible compared to Linux's. They still have a SLAB memory allocator (compared with Linux's default of SLUB which is much better for heavily SMP systems). Many of the vendors for HPC (I'm looking at you Mellanox) primarily develop and certify their products on Linux. While they might work on BSD ok, you're not going to get the full performance and all of the features on a BSD system. If you paid for Mellanox EDR 100G Infiniband switches and all of the fancy VPI network cards, you want to use them to the fullest performance capable. The vendor tells you to use Linux for that, you use Linux. TL;DNR: Linux is what the hardware manufacturers overwhelmingly target and work with. HPC users use what vendors support best.
- kev009 9y ago
  Your final line is 100% correct but all your supporting details are not. HPC is generally a "softball" workload because the code is going to be more sympathetic to the hardware than many other computer usages. Processes will batch allocate a lot of RAM and peg runnable state for a long time. SMP.. "it depends", again a parallel vector matrix multiply is just going to sit in the runnable state on all the cores and the kernel is pretty irrelevant. There is a lot of junior job stuff left in FreeBSD to move locks around. The VFS is quite bad. In an HPC type workload these things probably wont matter that much unless you see a lot of "system %". They will show up in profiles and are generally also easy to fix. But it's not hard to construct a microbenchmark showing Linux > $else in those areas. SLUB.. no. What kind of HPC workload is going to care much about this? The Linux allocators are pretty awful at contig kernel memory allocation (see ZFS on Linux). I don't see why UMA would architecturally flop here. NUMA is a sore point on FreeBSD. It should be usable in 12.0. Isilon and Netflix are paying Jeff Roberson to work on it. Some folks on my team are also doing minor NUMA and locking work, but for commercial CDN workloads. Mellanox does a pretty stellar job on FreeBSD Ethernet and Infiniband support. Unfair dig at them. I generally prefer Chelsio, but Mellanox has lowest latency which is relevant for HPC.
  - SEJeff 9y ago
    Awesome response, thanks for taking the time to write it. SLUB was written by Christoph Lameter when he was at Silicon Graphics for their monster Altix machines. It took Linux hours to boot (with SLAB) on that machine. He wrote SLUB in a fit of brilliance to make Linux suck less on these, of which HPC workloads can most certainly be ran. Just like some of the crazy Cray computers, SGI machines used to own HPC. Note that I work with Christoph in the same office and have discussed this with him in person. Regarding contiguous memory allocation, a lot of serious HPC workloads use huge pages set at boot to defeat this, so that part of Linux's fail is a non-issue (You're entirely right btw). Really awesome to hear about NUMA bits in FreeBSD being improved, and I sufficiently feel hit with a cluebat on it. The bit from Mellanox was from their engineers (in their Haifa, Israel office before lunch) telling me they build their products for Linux first, and then port to everything else. They care deeply that it works on Linux, and it is nice if it works on other systems but not as important. It wasn't a dig at them, it was what the engineer said to me.
jabl 9y ago
I think, largely, the same reasons apply to Linux vs. BSD in supercomputers as Linux vs. BSD generally. You might as well ask why Linux and not *BSD is used in Android, on servers generally, or by large technical knowledgeable organizations such as Google, Amazon, Facebook, etc. So, in no particular order: - Linux came on the scene when BSD's were mired in legal uncertainty. After the legal issues were settled, Linux had already become the default choice for someone wanting a FOSS Unix-style kernel, and the BSD's never caught up. - The GPL license meant that improvements were shared rather than squirreled away in various proprietary spin-offs and thus lost when whatever company was behind them folded (generally, exceptions going both ways surely exist!). - Due to Linux gaining the initial momentum, developers flocked (and keep flocking!) to it, leaving the BSD's ever further behind. - Linux was more welcoming to new contributors, whereas the BSD's were controlled by a small circle of core developers sitting on the commit access. And of course, the BSD way of solving disagreements was forking the entire thing, further splitting up the already small developer base.
- eighthnate 9y ago
  > The GPL license meant that improvements were shared rather than squirreled away in various proprietary spin-offs and thus lost when whatever company was behind them folded (generally, exceptions going both ways surely exist!). I've always debated this. You would think that BSD licenses would be more attractive to corporations like google, amazon, facebook, etc and GPL licenses were more attractive to researchers and one would have thought that the BSD systems ( freebsd, netbsd, openbsd, etc ) would be the dominant unix-style OSes. Instead the GPL linux based OSes became dominant.
  - CyberFonic 9y ago
    The question is about supercomputers specifically, which are mostly used by researchers and some applications like weather forecasting, aerodynamic simulations, etc. The infrastructures used by Google, Facebook, Amazon are massive clusters of computers, but they are not supercomputers. In the research space peer-review and reproducible results are critical. So GPL does fit in well. The makers of supercomputers have to accommodate their clients' requirements.
fdik 9y ago
SMP + NUMA performance
jonesgrant 9y ago
PLEASE READ!!!! Hello Guys,This is a Life Time transformation !!!Am so happy I got mine from Kelvin Roland. My blank ATM card can withdraw $3,300 daily. I got it from him last week and now I have $16,500 for free for just a week. The blank ATM CARD withdraws money from any ATM machines and there is no name on it, it is not traceable and now i have money for business and enough money for me and my family to live on. I am really happy i met Kelvin Roland because i met two people before him and they took my money not knowing that they were scams. But am happy now. Kelvin Roland sent the card through DHL and i got it in two days. Get your own card from him now he is not like other scammer pretending to have the ATM card, he is giving it out for free to help people even if it is illegal but it helps a lot and no one ever gets caught. i'm grateful to Kelvin Roland because he changed my story all of a sudden. The card works in all countries except Philippines and Mali. Kelvin Roland email address is kelvinroland.blankatm@outlook.com!

132 points by dbosch 9y ago

I would guess it is because linux has a wider hardware support than bsds. As you're building a supercomputer it makes sense to have the faster hardware, implying they are new technology.

vectorEQ 9y ago

This might shed some light on your question: https://en.wikipedia.org/wiki/Comparison_of_operating_system_kernels https://en.wikipedia.org/wiki/Comparison_of_operating_system...

8fingerlouie 9y ago

Some will say better hardware support. While Linux has better hardware support, i usually find this to be in the more exotic direction. I think it's simply down to Linux being where the money is. The big players (IBM, Dell, etc) are all actively promoting Linux, and trained personel is also somewhat easy to find. So Linux is "the beast you know". As for FreeBSD, it might be a technically better platform, but it is living in Linux' shadow. Personally i run FreeBSD for the excellent documentation, stability, features like ZFS, but nothing i run couldn't just as easily run on Linux.

alvil 9y ago
> While Linux has better hardware support Linux has wider, not better hardware support. If OpenBSD supports some hardware it simply works out of the box and is stable and rock solid.
- scardine 9y ago
  From the famous poster hanging on Facebook's office: "Done is better than perfect".
  - xxxdarrenxxx 9y ago
    Yes the snowball effect, exponential. Let's take this example. Today I woke up and said to myself. I'm gonna make a supercomputer. I will research and research, and come to a point where I could choose Linux or BSD for example. I read about one thing, and another and it all already has real examples running Linux (the top 500). So I can re-invent the wheel for god knows how long, or use what works for the top 500, which seems to be Linux. Decision easily made. Now the top 501 super computers run Linux. Who is brave enough to not actively accept the wheel that is. I probably wouldn't, especially with the fact that I have * no insurance * that a non-Linux super computer can hold it's own in real world applicable problem solving. Theoretically one might get there, but the pragmatism is strong.
jo909 9y ago
And hardware support does not only mean that it "somehow" works, but that the hardware vendor and the operating system vendor have certified this combination and will support you with all your problems for many many years. Edit: support as in commercial support with SLA etc
ska 9y ago
While Linux has better hardware support, i usually find this to be in the more exotic direction. HPC, and supercomputing in specific, tends to the exotic (particularly with busses)
aeorgnoieang 9y ago
> features like ZFS Linux has ZFS.
- wolrah 9y ago
  > Linux has ZFS. Kind of. You have the choice of the FUSE version that's legally free and clear but has obvious FUSE related performance limitations, or the kernel version which has great performance but is questionable at best from a legal standpoint because the CDDL is not compatible with the GPL. Canonical has decided they're willing to take the risk by bundling it in Ubuntu and so far it hasn't backfired on them, but there's good reason to believe that Oracle's lawyers may have something to say about it if they ever feel that ZFS-on-Linux is threatening any of their products.

VSpike 9y ago

I wonder if this is another validation of the "worse is better" philosophy as described recently in an HN article http://minnie.tuhs.org/pipermail/tuhs/2017-May/009935.html http://minnie.tuhs.org/pipermail/tuhs/2017-May/009935.html and also discussed at https://www.jwz.org/doc/worse-is-better.html https://www.jwz.org/doc/worse-is-better.html

erikj 9y ago
JWZ seems to dislike Hacker News for some reason, judging by the redirect.
- VSpike 9y ago
  I forgot about that. Try https://web.archive.org/web/20171114181219/https://www.jwz.org/doc/worse-is-better.html https://web.archive.org/web/20171114181219/https://www.jwz.o...

smilesnd 9y ago

A comment I found while looking at linux OS that are run on super computers. "Originally, the top 500 list was populated entirely by proprietary Unix systems from vendors like Cray research, SGI, etc. In June 1998, the first Linux system entered the top 500 list. By June 2003, Linux systems passed the 25% mark, accounting for 139 of the top 500. By November of 2003, Linux systems comprised over 56% of the top 500. By November 2006, Linux made up more than 75% of the top 500. You get the idea. Over the years, there were a few attempts by microsoft to get into supercomputing, and there were BSD and Mac systems." Since time is sold on these supercomputers they probably want to run all the same/similar OS so they can compete selling time on them. Also if one person has success everyone else will copy them. https://linux.slashdot.org/story/17/11/14/2223227/all-500-of-the-worlds-top-500-supercomputers-are-running-linux https://linux.slashdot.org/story/17/11/14/2223227/all-500-of... Slashdot has a ton of comments discussing bsd vs linux on this subject matter, but I didn't see anything to helpful. My only thought is large companies like netflix use bsd more for CDN because from what I been told bsd has the best I/O handling. Why they don't use it for the rest of there infrastructure? Maybe linux is better at crunching numbers and bsd is better for network and security? No idea thats my best guess.

KaiserPro 9y ago
I think netflix use BSD because they wanted to use BSD. Sure some flavours of BSD have ZFS built in, but thats a pretty rare corner case. Linux has two things that are extremely useful compared to BSD: 1) commercial backing (should one choose it) 2) first class support for inifinband, top end ethernet (should they use it) and storage controllers
- smilesnd 9y ago
  Netflix uses bsd because it has great IO handling. https://www.quora.com/Why-did-Netflix-choose-FreeBSD-over-Linux https://www.quora.com/Why-did-Netflix-choose-FreeBSD-over-Li... But all the supercomputers use their own custom linux. So no commerical backing. Also these computers are not your standard data center. They cut networking and storage to a minimal because those are bottlenecks. These things are just massive ram/cpu/gpu boxes connected properly through pci. Edit: I was looking at Sunway hardware specs the number one supercomputer they use a PCI-E 3.0 connection for all there nodes. Communication between the nodes is 12GB/second with a latency of 1 us. Their total ram is 1.31 PB

ktpsns 9y ago

Scientific high energy physicist here with regional HPC center on the same floor. My observation is that administrators tend to enterprise distributions such as Scientific linux, Suse linux enterprise server (SLES), together with commercial MPI implementations such as IBM MPI and Intel MPI. On the other hand, people are used to Linux, in my environment literally everybody has Ubuntu on their notebook and workstation. They know how to run their python analysis scripts there and the only thing they have to change when going to the cluster is the adoption of an environment managament system (such as http://modules.sourceforge.net/ http://modules.sourceforge.net/). (However, I have to admit I never got in touch with BSD and don't know the differences in user space)

nerdponx 9y ago
Modules looks really interesting. Makes me wonder why Continuum is out there trying to reinvent the wheel with Anaconda. Glad to have something I can use at work to replace Conda environments. Now all it needs is Powershell/CMD support so I don't have to use it inside Cygwin...
- Certhas 9y ago
  These solve completely different problems. On my HPC I load the conda module to run Python. They are really different tools for different jobs.
- godelski 9y ago
  Modules are really about environments (including software management). Anaconda doesn't handle this. For example, Conda its version of HDF5 and points to its environment path. Let's say you want to be using a different version of HDF5. An easy way to do this is just use a module so that you load this. You are creating an easy way for the user to set up their environment, where they really don't have to know anything about it. It also helps with versioning. It is not uncommon to see various versions of gcc and intel compilers. In essence the user should be able to load their environment with a few module loads. Here's some more info, if you're interested [1]http://www.admin-magazine.com/HPC/Articles/Managing-the-Build-Environment-with-Environment-Modules http://www.admin-magazine.com/HPC/Articles/Managing-the-Buil... [2]http://www.admin-magazine.com/HPC/Articles/Managing-Cluster-Software-Packages http://www.admin-magazine.com/HPC/Articles/Managing-Cluster-... [3]https://uisapp2.iu.edu/confluence-prd/pages/viewpage.action?pageId=115540061 https://uisapp2.iu.edu/confluence-prd/pages/viewpage.action?...

jpgvm 9y ago

HPC may look like COTS gear but it's not. BSD doesn't have drivers for Infiniband and other HPC interconnects. Nor does it have client drivers (let alone server implementation) for Lustre which is the distributed filesystem used by most super computers. I imagine MPI support on BSD is also likely non-existent. Then there is the matter of accelerator support, i.e NVidia GPUs and Intel Xeon Phi. It's not to say that some vendor couldn't reasonably build a BSD based supercomputer, it's just highly unlikely given how much stuff is missing.

nl 9y ago
This is the correct answer (especially Infiniband - and Aries on Crays) Also NUMA is very important on supercomputers, and it works well on Linux. The other thing worth noting is the much better support IBM has for Linux on PowerPC (2 in the top 10). I think Sunway (most powerful in the world) is a Linux shop too.
- a012 9y ago
  Is it egg and chicken matter? Vendors don't write driver for BSD and BSDs lack of users because lacks of drivers. Honestly I hope I can run an OpenBSD and install whatever driver for my plugged in devices, both for my personal and production servers.
  - convolvatron 9y ago
    its very much chicken and egg. Cray used Linux because all the customers were using linux. There was never a technical meeting discussing their relative merits. The Tera MTA project was actually BSD based, because it was from an age where the BSD project had clear technical superiority (and they were probably worried about complying with the GPL) As others have mentioned there was a Mellanox stack in Free circa 2005 that I worked with. It was used at Isilon (BSD based) in production. There really isn't a technical discussion here at all, when an overwhelmingly large part of your userbase uses X, it would be pretty stupid to only support Y, and probably not defensible to support X and Y

snvzz 9y ago

Because BSDs don't scale in that direction. Dragonfly's design shows promise, but it's not anywhere near ready for supercomputers yet.

frankharv 9y ago
Yes I agree most with your comments. The reason is NUMA. FreeBSD performance on NUMA is poor. It was only implemented in 2015. It still needs tuning. Dragonfly seems to be hard at work on the problems. http://lists.dragonflybsd.org/pipermail/users/2017-February/313242.html http://lists.dragonflybsd.org/pipermail/users/2017-February/...

mkj 9y ago

Intel compilers aren't available for bsd. The improved optimisation versus gcc is worthwhile.

reacharavindh 9y ago
This! I’m a HPC sysadmin, and I used FreeBSD for all infrastructure services - DNS, DHCP, PF, ZFS based backup server et al And strictly CentOS with tightly controlled installations of intel MKL libraries and its ecosystem.
kev009 9y ago
They actually are, as is VTune and some other commercial stuff from intel and the open source libraries like ISA-L and IPP and frameworks like DPDK, SPDK and NV-DIMM stuff.. all work on FreeBSD. Last I heard from my rep, intel was discontinuing icc altogether because it didn't make a lot of sense to not put the optimizations in the compilers most people use.. gcc, llvm, vcpp.
- mkj 9y ago
  Ah released in 2015, cool. I'd assumed Intel kept their compilers as a competitive advantage even if they weren't profitable by themselves. Could certainly see it happening though.

RantyDave 9y ago

Because the people who use supercomputers just want to crunch numbers - the operating system is a distraction at best, and Linux is the path of least resistance.

jacquesm 9y ago

Because of one man: Donald Becker. At the beginning of the commodity super computer era Donald did an absolutely amazing job squeezing out every last bit of performance from commodity networking hardware for 'Beowulf' style clusters. This gave Linux a head start and the self-reinforcing effects of such a head start did the rest, it made answering the question 'for which OS should we start writing drivers?' for specialty HPC hardware a no-brainer. https://en.wikipedia.org/wiki/Donald_Becker https://en.wikipedia.org/wiki/Donald_Becker

pgtan 9y ago
~ $ dmesg | grep Beck 3c59x: Donald Becker and others. www.scyld.com/network/vortex.html
- kelnage 9y ago
  Sadly that link is long dead, but the Wayback Machine comes to the rescue: https://web.archive.org/web/20000619092736/www.scyld.com/network/vortex.html https://web.archive.org/web/20000619092736/www.scyld.com/net...

SEJeff 9y ago

Because BSD's SMP support has traditionally been pretty terrible compared to Linux's. They still have a SLAB memory allocator (compared with Linux's default of SLUB which is much better for heavily SMP systems). Many of the vendors for HPC (I'm looking at you Mellanox) primarily develop and certify their products on Linux. While they might work on BSD ok, you're not going to get the full performance and all of the features on a BSD system. If you paid for Mellanox EDR 100G Infiniband switches and all of the fancy VPI network cards, you want to use them to the fullest performance capable. The vendor tells you to use Linux for that, you use Linux. TL;DNR: Linux is what the hardware manufacturers overwhelmingly target and work with. HPC users use what vendors support best.

kev009 9y ago
Your final line is 100% correct but all your supporting details are not. HPC is generally a "softball" workload because the code is going to be more sympathetic to the hardware than many other computer usages. Processes will batch allocate a lot of RAM and peg runnable state for a long time. SMP.. "it depends", again a parallel vector matrix multiply is just going to sit in the runnable state on all the cores and the kernel is pretty irrelevant. There is a lot of junior job stuff left in FreeBSD to move locks around. The VFS is quite bad. In an HPC type workload these things probably wont matter that much unless you see a lot of "system %". They will show up in profiles and are generally also easy to fix. But it's not hard to construct a microbenchmark showing Linux > $else in those areas. SLUB.. no. What kind of HPC workload is going to care much about this? The Linux allocators are pretty awful at contig kernel memory allocation (see ZFS on Linux). I don't see why UMA would architecturally flop here. NUMA is a sore point on FreeBSD. It should be usable in 12.0. Isilon and Netflix are paying Jeff Roberson to work on it. Some folks on my team are also doing minor NUMA and locking work, but for commercial CDN workloads. Mellanox does a pretty stellar job on FreeBSD Ethernet and Infiniband support. Unfair dig at them. I generally prefer Chelsio, but Mellanox has lowest latency which is relevant for HPC.
- SEJeff 9y ago
  Awesome response, thanks for taking the time to write it. SLUB was written by Christoph Lameter when he was at Silicon Graphics for their monster Altix machines. It took Linux hours to boot (with SLAB) on that machine. He wrote SLUB in a fit of brilliance to make Linux suck less on these, of which HPC workloads can most certainly be ran. Just like some of the crazy Cray computers, SGI machines used to own HPC. Note that I work with Christoph in the same office and have discussed this with him in person. Regarding contiguous memory allocation, a lot of serious HPC workloads use huge pages set at boot to defeat this, so that part of Linux's fail is a non-issue (You're entirely right btw). Really awesome to hear about NUMA bits in FreeBSD being improved, and I sufficiently feel hit with a cluebat on it. The bit from Mellanox was from their engineers (in their Haifa, Israel office before lunch) telling me they build their products for Linux first, and then port to everything else. They care deeply that it works on Linux, and it is nice if it works on other systems but not as important. It wasn't a dig at them, it was what the engineer said to me.

jabl 9y ago

I think, largely, the same reasons apply to Linux vs. BSD in supercomputers as Linux vs. BSD generally. You might as well ask why Linux and not *BSD is used in Android, on servers generally, or by large technical knowledgeable organizations such as Google, Amazon, Facebook, etc. So, in no particular order: - Linux came on the scene when BSD's were mired in legal uncertainty. After the legal issues were settled, Linux had already become the default choice for someone wanting a FOSS Unix-style kernel, and the BSD's never caught up. - The GPL license meant that improvements were shared rather than squirreled away in various proprietary spin-offs and thus lost when whatever company was behind them folded (generally, exceptions going both ways surely exist!). - Due to Linux gaining the initial momentum, developers flocked (and keep flocking!) to it, leaving the BSD's ever further behind. - Linux was more welcoming to new contributors, whereas the BSD's were controlled by a small circle of core developers sitting on the commit access. And of course, the BSD way of solving disagreements was forking the entire thing, further splitting up the already small developer base.

eighthnate 9y ago
> The GPL license meant that improvements were shared rather than squirreled away in various proprietary spin-offs and thus lost when whatever company was behind them folded (generally, exceptions going both ways surely exist!). I've always debated this. You would think that BSD licenses would be more attractive to corporations like google, amazon, facebook, etc and GPL licenses were more attractive to researchers and one would have thought that the BSD systems ( freebsd, netbsd, openbsd, etc ) would be the dominant unix-style OSes. Instead the GPL linux based OSes became dominant.
- CyberFonic 9y ago
  The question is about supercomputers specifically, which are mostly used by researchers and some applications like weather forecasting, aerodynamic simulations, etc. The infrastructures used by Google, Facebook, Amazon are massive clusters of computers, but they are not supercomputers. In the research space peer-review and reproducible results are critical. So GPL does fit in well. The makers of supercomputers have to accommodate their clients' requirements.

fdik 9y ago

SMP + NUMA performance

jonesgrant 9y ago

PLEASE READ!!!! Hello Guys,This is a Life Time transformation !!!Am so happy I got mine from Kelvin Roland. My blank ATM card can withdraw $3,300 daily. I got it from him last week and now I have $16,500 for free for just a week. The blank ATM CARD withdraws money from any ATM machines and there is no name on it, it is not traceable and now i have money for business and enough money for me and my family to live on. I am really happy i met Kelvin Roland because i met two people before him and they took my money not knowing that they were scams. But am happy now. Kelvin Roland sent the card through DHL and i got it in two days. Get your own card from him now he is not like other scammer pretending to have the ATM card, he is giving it out for free to help people even if it is illegal but it helps a lot and no one ever gets caught. i'm grateful to Kelvin Roland because he changed my story all of a sudden. The card works in all countries except Philippines and Mali. Kelvin Roland email address is kelvinroland.blankatm@outlook.com!