7 ms·

Systems software research is still irrelevant

61 points by mbrubeck 12y ago

justincormack 12y ago
I think there is a revival of interest, as mass scale automation is changing the way we use servers, and security issues are finally changing the desktop/mobile. I found a lot of interest when I organized a conference [1] which is tomorrow (videos will be available), which si encouraging. [1] https://operatingsystems.io/ https://operatingsystems.io/
donavanm 12y ago
Hrm. I think there are some decent xounter arguments. I will agree that pure academic systems research is being superceded by industry development or academic commercialization. As a trend I think commoditization, parallelism, and management are where effort (and sucess) has gone, as opposed to novel research. Make it cheaper, wider, and possible to manage. Influential counterpoints: distributed systems management/configuration/orchestration, distributed datastores (CAP, CALVIN, Dynamo, etc), system languages (Go, Rust), distributed processing (tilera, niagra, larrabe, gpus), network fabrics instead of trees, &stream processing jnstead of batch (kinesis, millwheel).
- Dewie 12y ago
  > , system languages (Go, Rust), How is Go not another example of how "programming languages have stagnated"[0] ? You could argue that it is really practical and productive, but that is a separate point to whether it or not it is "more of the same". Or maybe it is more of the same, just in the vein of languages that most people have forgotten about by now, so it is for all intents and purposes new. Certainly native compilation and some control over memory layout (I guess, with structs and such) and indirection have not been common features in recent memory (that is; those two points and automatic memory management, and overall simplicity compared to C/++). [0] From the article: > It included a chart lamenting the stagnation in [...] programming languages
notacoward 12y ago
A lot depends on how "systems software" is defined. Very few people have much use for a whole new kernel. Even little pieces of new kernel code are less useful than before, as various functions (e.g. storage) can be handled just as well in user space. So if "systems" == "kernel" then relevance really has been declining. On the other hand, if one considers "system" to include user-space storage and networking, distributed consensus and service discovery, package and system management, JIT and GC in language runtimes, etc. etc. etc. then we're actually in a bit of a renaissance. The fact is that kernel/user hasn't been a very interesting boundary for a long time. There are hundreds of systems-programming problems that can be solved on either side of that boundary, except that if you try to solve them in the kernel debugging will be harder and you'll get embroiled in the constant turf wars that are the hallmark of a declining specialty. The idea that only kernel hackers do systems programming needs to die.
- jdimov 12y ago
  Does this mean we probably need a new, more minimalistic concept for a modern kernel?
  - notacoward 12y ago
    We've been moving that way for quite some time, despite Linus's historic dismissal of microkernel ideas. Granular capabilities, FUSE, namespaces, cgroups, containers etc. all take us in that direction. Over the next several years, I think even the kernel grognards will start to see that the "center of gravity" for systems software has shifted to better languages and tools than are available in their sandbox.
  - steveklabnik 12y ago
    There's been plenty of designs which are minimalistic, but haven't caught on for a variety of other reasons. See exokernels, for example. They're basically the minimum possible kernel.
- noelwelsh 12y ago
  This is exactly right. The OS isn't a very interesting system any more. Unfortunately the author still identifies it as the system of interest. For example: > A quote that still holds true, "There has been much talk about component architectures but only one true success: Unix pipes. It should be possible to build interactive and distributed applications from piece parts." This is completely false. REST/JSON has been fantastically successful. There is a huge amount of relevant systems work going on, and distributed systems research in particular has never been more relevant. Major systems today (Cassandra, HDFS, Riak, etc.) are directly informed by research. It's just that no-one is really interested in ttys and pipes anymore, and that's what the author identifies as systems.
codeulike 12y ago
He seems to think progress would look like "a new wave of built-from-the-ground-up operating systems sweeping away all the old ones every decade or so". Say, if Plan 9 had become dominant in the 90s, would we by now be replacing it with something else? What would everyone moving to new operating systems every 10 years gain us? There has been mind boggling progress since the 90s, just not in the direction he expected.
api 12y ago
"There has been much talk about component architectures but only one true success: Unix pipes. It should be possible to build interactive and distributed applications from piece parts." Java has many success stories in this area, starting with the amazing reusability of POJOs (plain old Java objects). I can take Java code from 1996 and almost always compile it and use it now without major refactoring and without having to really look at the internals. I can do it with the API docs only. Beyond POJOs there are many abstraction techniques like beans that have been used to build huge software systems that have stood the test of time, albeit with a bit of clunkiness. But Unix pipes are a bit clunky too. I coded a lot with Java years ago. It's been out of vogue for quite some time -- though it's still very popular just not trendy. Lately I've been reflecting on some of the pain points of other languages and thinking "wow, Java really did have a lot of things right." I've also been thinking that the JVM is almost an OS, and that it sort of wants to be. There were some efforts in that direction, but none really went anywhere. It's not quite perfect, but I feel like it's the only thing we've had in the past 30 years that looks like a step toward a post-C/Unix operating system that could really work. I also feel like in the age of many-cores we could do away with problems like GC pauses completely. I could forsee a future many-core box where 2-4 (of 32-64) cores are just almost flat out dedicated to GC, JITing, tracing, optimization, etc. When you've got that many cores you could dedicate a few to the OS and not worry much about it. The benefits would be things like forgetting about manual memory management, etc. Also note that the JVM can run more than Java. Java would just be the "core language" of this hypothetical OS, taking the place of C on Unix.
- MrBuddyCasino 12y ago
  > I also feel like in the age of many-cores we could do away with problems like GC pauses completely. Unfortunately its not a resources problem, or else there wouldn't be an issue with GC pauses on fast hardware. There is a solution for that though, and Cliff Click of Azul Systems is willing to sell you the only pause-less JVM GC there is for the low low price of 7500$ per server (see http://www.azulsystems.com/products/zing/whatisit http://www.azulsystems.com/products/zing/whatisit).
  - api 12y ago
bjackman 12y ago
I don't think I entirely agree - as notacoward says I think we're approaching a renaissance in certain areas. - Rust and Go are becoming extremely popular extremely quickly. - I think/hope we may soon see some sort of battle between OS container tech (Docker/LXC/cgroups, FreeBSD Jails) vs unikernels (Mirage, OSv, Linux unikernels) on paravirtualised platforms (Xen) as the full-weight-VM-running-Debian model ages. edit: Oooh look, an exciting relevant systems software paper! http://anil.recoil.org/papers/2013-asplos-mirage.pdf http://anil.recoil.org/papers/2013-asplos-mirage.pdf
- pron 12y ago
  > Rust and Go are becoming extremely popular extremely quickly. Unfortunately, that's yet another SV bias. Rust has hardly any adoption, and Go has moderate adoption and is growing nicely, but nothing close to justifying "extremely".
  - erickt 12y ago
    To be fair, we in the rust community are telling people to wait for 1.0 unless they are okay with constantly updating their code as we clean up the language for our first release. It's fair though to speak about momentum. Comparing Go's contributor list (https://golang.org/CONTRIBUTORS https://golang.org/CONTRIBUTORS) with ours (https://github.com/rust-lang/rust/blob/master/AUTHORS.txt https://github.com/rust-lang/rust/blob/master/AUTHORS.txt) has us with about 100 more than Go. I'm not saying our community is better than Go's or anything, but I'm certainly impressed a new language has been able to gain so much interest so quickly. When I first started contributing a couple years ago, we had like 15 people in our irc channel. Now we got over 500.
    - Dewie 12y ago
      That makes a lot of sense to me; Rust is a much more ambitious and interesting - from a programming language point of view - project than Go. I think that it as a language would appeal much more to the prototypical "language geek" (or at least the impression that I have of such people, including myself). That's not to slam Rust for being "over-complex"/"ivory tower", or slam Go for being "simplistic". But Rust tries to solve such a hard problem - be expressive without any unnecessary overhead, retain memory safety - that it seems to naturally lead to a design and language that appeals to people that are interested in programming languages "for their own sake". Meanwhile Go tries to be sufficiently expressive and then get out of your way to solve whatever problem in whatever domain that you're trying to solve.
josephlord 12y ago
Xen largely came from the Systems group at Cambridge I think. Seems fairly relevant to the current state of computing to me. Not sure what is being worked on now that might be relevant in a decade and a half from now [edit to add: but there is probably something].
- amirmc 12y ago
  > Not sure what is being worked on now that might be relevant in a decade and a half from now. Mirage OS is one of those things - http://openmirage.org http://openmirage.org - though I expect (hope) it'll take much less than decade to be relevant. Some of the same team that worked on early Xen are working on Mirage.
  - pron 12y ago
    And there's OSv, http://osv.io/ http://osv.io/, from the people behind KVM.
  - josephlord 12y ago
    I've just realised that my last sentence which you quoted could be read as saying that what is currently being worked on won't be as relevant when it was really intended to mean that there is probably some things that are being done now that will be relevant in the future I just don't know what they are.
- zetazzed 12y ago
  Xen is an example, but, more generally, virtualization has been the big story in systems software for the period mentioned in this article (post-2000). VMware came out of systems research at Stanford and Xen out of Cambridge. The cloud world of AWS/Azure/GCE with tens of millions of VMs running for pennies per hour is a radical shift in how we think about systems. The new wave of Docker/containers promises to extend this idea even further up the stack. How can we call that "stagnation in software" since 2000?
  - wrs 12y ago
    Virtualization as an operating system design feature predates that entire chart. A lot of the hard problems in the Xen/VMWare research related to implementing virtualization on a processor architecture (x86) that was extremely popular, but didn't actually support virtualization. Which is just the kind of stagnation he's talking about, I think.
fasteo 12y ago
Not sure to agree; there are quite a few interesting projects in both OS and system programming language fields. To name a few: OS Mirage OS[1] and unikernels in general (ErlangOnXen, HalVM) OSv [2] Chromium OS [3] Programming languages Julia [4] Rust [5] D-Lang [6] Nimrod [7] [1] http://www.openmirage.org/ http://www.openmirage.org/ [2] http://osv.io/ http://osv.io/ [3] http://www.chromium.org/chromium-os http://www.chromium.org/chromium-os [4] http://julialang.org/ http://julialang.org/ [5] http://www.rust-lang.org/ http://www.rust-lang.org/ [6] http://dlang.org/ http://dlang.org/ [7] http://nimrod-lang.org/ http://nimrod-lang.org/
nhaehnle 12y ago
The interesting thing about the table shown in the article is that even while the author laments it, it actually illustrates a good development: the table has an "hourglass" structure. There is a stable part (fundamental technology such as the kernel, TCP/IP etc.) at the center which acts as an interface between the top part (hardware) and the bottom part (languages). All innovation has to be anchored to some core set of stable interfaces into which the new innovations "plug in". The nature of OSs is such that they are naturally part of these core sets. That may be frustrating for OS researchers, but they'll have to learn to live with it.
bcantrill 12y ago
Ugh, this old troll. Something that took me a while to figure out: when someone says that "nothing interesting is happening any longer in [discipline]", it is virtually always code for "I feel old." Yes, things change and different periods of innovation are (of course) different, but it is wrong to cast value judgement on one era over another. Speaking as an aging technologist, I can say that it can be hard to resist this -- and in particular, it's tempting to view one's late twenties and early thirties as a Golden Age, and everything later as derivative and uninteresting. If it needs to be said: for those who are making this claim, they are often projecting their own lives onto their discipline, and may be confusing their own personal Golden Age with a broader one that may or may not have existed. My domain -- operating systems kernel development -- has been particularly vulnerable to this (in part because it has become increasingly specialized) and this is something that I have heard for my entire career. Of course, there has been tons of innovation in the kernel over those two decades (and I've been lucky enough to be close to a bunch of it) -- and no one now would seriously go back to the systems from two decades ago. (If anyone disagrees, they should kindly put their money where their mouth is and run only systems software from 1994.) So yes, systems software is still relevant -- and so are a lot of things that might feel "done." And if you are an aging technologist like me (protected class, baby!) and you feel tempted to tell these youngsters that there is nothing new under the sun, please check yourself: yes, you should educate the rising generation about the problems that have been solved -- but never go so far as to say that there is no room left for new ideas. Aside from the fact that it's a demoralizing thing to say to a younger technologist, history will likely prove you wrong!
- linksnapzz 12y ago
  Love the work you've done Bryan, but your response above does nothing to engage the salient points made in the link && elsewhere. Yes, we have better ways of getting performance metrics than truss, circa Solaris 2.5…and thank you! But that isn't what this is about.
  - bcantrill 12y ago
    Sorry, what salient points am I not engaging? The author of the linked piece is decrying what they perceive as a lack of "OS research and innovation" -- and I'm saying that OS innovation remains alive and well, if perhaps more technical and less accessible than it once was. I can of course be more concrete and rattle these things off -- but I feel safe in leaving software systems advances since 2000 as an exercise to the reader...
mcguire 12y ago
Overall, I agree with Pike in 2000 and this article today. Systems software research has become stagnant and significantly derivative. The point isn't that a new OS would come along and sweep everything away, the point is that nobody seems to be developing new OSs at all. Sort-of. In 2000 or so, systems research had turned almost entirely into server performance measurement and tweaks. That trend seems to have changed, but things like Xen and VMware are not really new. IBM invented virtualization back in the middle ages; most of the more interesting things going on recently have involved using the idea. Likewise, moving stuff from Plan 9, etc., into Linux is cool (very, very cool), but it's not new, groundbreaking research. That being said, there are a lot of nitpicks one can make with this article and with Pike's original talk. The processor speed comparison in the table is not terribly valid, in my opinion. The more important number is front-side bus speed, which has not changed anywhere near as much. (Front-side bus speed dominates most of computing loads I see, and a great whopping chunk of the work in those areas where it doesn't dominate, video games and data-heavy computing, is in preventing the front-side bus from being the bottleneck.) Pike was wrong about the language environment in 2000 and the table is wrong today. Heck, he might have been wrong about 1990. Programming language research in general and its specific applicability to systems (Hi, ATS!) has hotted up a lot since the '80s and '90s. Pike's comment, "There has been much talk about component architectures but only one true success: Unix pipes. It should be possible to build interactive and distributed applications from piece parts," bothers me a lot. A great deal of component research has been a dead-end in general (See that pile of crap with CORBA and SOAP laying on top?) Further, I believe Pike never really understood distributed systems. I have seen Pike complain that Nagel's algorithm has no place in TCP/IP (although I haven't been able to find the reference in years), but if you want a more recent example, see Go---everybody likes CSP, right? Everyone except someone who has tried to implement a synchronous rendezvous on a network. Finally, Plan 9 didn't fail. But it also didn't have any chance to succeed, if by success you mean taking over the world. It was a great research project, but I'd believe making someone who didn't grow up at Bell Labs should be against the Geneva Convention. (Anyone know when Plan 9 started supporting color?) Plan 9 is my prima facie evidence that Not Invented Here syndrome was invented at Bell Labs.
- bcantrill 12y ago
  There's a lot that's wrong in here, but one thing that absolutely needs to be called out: ...things like Xen and VMware are not really new. IBM invented virtualization back in the middle ages Actually and in fact, what VMware did was new -- to the point that prior to this work, many reasonable people believed that x86 couldn't be virtualized without unacceptable performance penalty. Please read Keith and Ole's excellent ASPLOS paper[1] for an in-depth discussion of the novelty -- and the citation for this technology when it won the ACM Software Systems Award[2]. If you insist that VMware's binary translation work wasn't novel, please be specific about how these peer-reviewed bodies got it wrong. [1] https://www.vmware.com/pdf/asplos235_adams.pdf https://www.vmware.com/pdf/asplos235_adams.pdf [2] http://awards.acm.org/award_winners/bugnion_3886645.cfm http://awards.acm.org/award_winners/bugnion_3886645.cfm
Someone 12y ago
Meanwhile, on the HN front page, we have: https://news.ycombinator.com/item?id=8651499 https://news.ycombinator.com/item?id=8651499: Kill init by touching a bunch of files https://news.ycombinator.com/item?id=8650952 https://news.ycombinator.com/item?id=8650952: on Linux, 'less' can probably get you owned One can argue that the research needed to prevent those was done decades ago, but this article doesn't make that argument. We may have hit upon a 'good enough' solution that is so hard to displace that it may never happen, but I think systems software research is very much relevant. Someone will have to polish a more robust/secure OS (may already exist in unpolished form, but possibly even must still be written) until it is shiny enough to replace the C-based (or equivalent) mess we have now.

61 points by mbrubeck 12y ago

I think there is a revival of interest, as mass scale automation is changing the way we use servers, and security issues are finally changing the desktop/mobile. I found a lot of interest when I organized a conference [1] which is tomorrow (videos will be available), which si encouraging. [1] https://operatingsystems.io/ https://operatingsystems.io/

donavanm 12y ago

Hrm. I think there are some decent xounter arguments. I will agree that pure academic systems research is being superceded by industry development or academic commercialization. As a trend I think commoditization, parallelism, and management are where effort (and sucess) has gone, as opposed to novel research. Make it cheaper, wider, and possible to manage. Influential counterpoints: distributed systems management/configuration/orchestration, distributed datastores (CAP, CALVIN, Dynamo, etc), system languages (Go, Rust), distributed processing (tilera, niagra, larrabe, gpus), network fabrics instead of trees, &stream processing jnstead of batch (kinesis, millwheel).

Dewie 12y ago
> , system languages (Go, Rust), How is Go not another example of how "programming languages have stagnated"[0] ? You could argue that it is really practical and productive, but that is a separate point to whether it or not it is "more of the same". Or maybe it is more of the same, just in the vein of languages that most people have forgotten about by now, so it is for all intents and purposes new. Certainly native compilation and some control over memory layout (I guess, with structs and such) and indirection have not been common features in recent memory (that is; those two points and automatic memory management, and overall simplicity compared to C/++). [0] From the article: > It included a chart lamenting the stagnation in [...] programming languages

notacoward 12y ago

A lot depends on how "systems software" is defined. Very few people have much use for a whole new kernel. Even little pieces of new kernel code are less useful than before, as various functions (e.g. storage) can be handled just as well in user space. So if "systems" == "kernel" then relevance really has been declining. On the other hand, if one considers "system" to include user-space storage and networking, distributed consensus and service discovery, package and system management, JIT and GC in language runtimes, etc. etc. etc. then we're actually in a bit of a renaissance. The fact is that kernel/user hasn't been a very interesting boundary for a long time. There are hundreds of systems-programming problems that can be solved on either side of that boundary, except that if you try to solve them in the kernel debugging will be harder and you'll get embroiled in the constant turf wars that are the hallmark of a declining specialty. The idea that only kernel hackers do systems programming needs to die.

jdimov 12y ago
Does this mean we probably need a new, more minimalistic concept for a modern kernel?
- notacoward 12y ago
  We've been moving that way for quite some time, despite Linus's historic dismissal of microkernel ideas. Granular capabilities, FUSE, namespaces, cgroups, containers etc. all take us in that direction. Over the next several years, I think even the kernel grognards will start to see that the "center of gravity" for systems software has shifted to better languages and tools than are available in their sandbox.
- steveklabnik 12y ago
  There's been plenty of designs which are minimalistic, but haven't caught on for a variety of other reasons. See exokernels, for example. They're basically the minimum possible kernel.
noelwelsh 12y ago
This is exactly right. The OS isn't a very interesting system any more. Unfortunately the author still identifies it as the system of interest. For example: > A quote that still holds true, "There has been much talk about component architectures but only one true success: Unix pipes. It should be possible to build interactive and distributed applications from piece parts." This is completely false. REST/JSON has been fantastically successful. There is a huge amount of relevant systems work going on, and distributed systems research in particular has never been more relevant. Major systems today (Cassandra, HDFS, Riak, etc.) are directly informed by research. It's just that no-one is really interested in ttys and pipes anymore, and that's what the author identifies as systems.

codeulike 12y ago

He seems to think progress would look like "a new wave of built-from-the-ground-up operating systems sweeping away all the old ones every decade or so". Say, if Plan 9 had become dominant in the 90s, would we by now be replacing it with something else? What would everyone moving to new operating systems every 10 years gain us? There has been mind boggling progress since the 90s, just not in the direction he expected.

api 12y ago

"There has been much talk about component architectures but only one true success: Unix pipes. It should be possible to build interactive and distributed applications from piece parts." Java has many success stories in this area, starting with the amazing reusability of POJOs (plain old Java objects). I can take Java code from 1996 and almost always compile it and use it now without major refactoring and without having to really look at the internals. I can do it with the API docs only. Beyond POJOs there are many abstraction techniques like beans that have been used to build huge software systems that have stood the test of time, albeit with a bit of clunkiness. But Unix pipes are a bit clunky too. I coded a lot with Java years ago. It's been out of vogue for quite some time -- though it's still very popular just not trendy. Lately I've been reflecting on some of the pain points of other languages and thinking "wow, Java really did have a lot of things right." I've also been thinking that the JVM is almost an OS, and that it sort of wants to be. There were some efforts in that direction, but none really went anywhere. It's not quite perfect, but I feel like it's the only thing we've had in the past 30 years that looks like a step toward a post-C/Unix operating system that could really work. I also feel like in the age of many-cores we could do away with problems like GC pauses completely. I could forsee a future many-core box where 2-4 (of 32-64) cores are just almost flat out dedicated to GC, JITing, tracing, optimization, etc. When you've got that many cores you could dedicate a few to the OS and not worry much about it. The benefits would be things like forgetting about manual memory management, etc. Also note that the JVM can run more than Java. Java would just be the "core language" of this hypothetical OS, taking the place of C on Unix.

MrBuddyCasino 12y ago
> I also feel like in the age of many-cores we could do away with problems like GC pauses completely. Unfortunately its not a resources problem, or else there wouldn't be an issue with GC pauses on fast hardware. There is a solution for that though, and Cliff Click of Azul Systems is willing to sell you the only pause-less JVM GC there is for the low low price of 7500$ per server (see http://www.azulsystems.com/products/zing/whatisit http://www.azulsystems.com/products/zing/whatisit).
- api 12y ago

bjackman 12y ago

I don't think I entirely agree - as notacoward says I think we're approaching a renaissance in certain areas. - Rust and Go are becoming extremely popular extremely quickly. - I think/hope we may soon see some sort of battle between OS container tech (Docker/LXC/cgroups, FreeBSD Jails) vs unikernels (Mirage, OSv, Linux unikernels) on paravirtualised platforms (Xen) as the full-weight-VM-running-Debian model ages. edit: Oooh look, an exciting relevant systems software paper! http://anil.recoil.org/papers/2013-asplos-mirage.pdf http://anil.recoil.org/papers/2013-asplos-mirage.pdf

pron 12y ago
> Rust and Go are becoming extremely popular extremely quickly. Unfortunately, that's yet another SV bias. Rust has hardly any adoption, and Go has moderate adoption and is growing nicely, but nothing close to justifying "extremely".
- erickt 12y ago
  To be fair, we in the rust community are telling people to wait for 1.0 unless they are okay with constantly updating their code as we clean up the language for our first release. It's fair though to speak about momentum. Comparing Go's contributor list (https://golang.org/CONTRIBUTORS https://golang.org/CONTRIBUTORS) with ours (https://github.com/rust-lang/rust/blob/master/AUTHORS.txt https://github.com/rust-lang/rust/blob/master/AUTHORS.txt) has us with about 100 more than Go. I'm not saying our community is better than Go's or anything, but I'm certainly impressed a new language has been able to gain so much interest so quickly. When I first started contributing a couple years ago, we had like 15 people in our irc channel. Now we got over 500.
  - Dewie 12y ago
    That makes a lot of sense to me; Rust is a much more ambitious and interesting - from a programming language point of view - project than Go. I think that it as a language would appeal much more to the prototypical "language geek" (or at least the impression that I have of such people, including myself). That's not to slam Rust for being "over-complex"/"ivory tower", or slam Go for being "simplistic". But Rust tries to solve such a hard problem - be expressive without any unnecessary overhead, retain memory safety - that it seems to naturally lead to a design and language that appeals to people that are interested in programming languages "for their own sake". Meanwhile Go tries to be sufficiently expressive and then get out of your way to solve whatever problem in whatever domain that you're trying to solve.

josephlord 12y ago

Xen largely came from the Systems group at Cambridge I think. Seems fairly relevant to the current state of computing to me. Not sure what is being worked on now that might be relevant in a decade and a half from now [edit to add: but there is probably something].

amirmc 12y ago
> Not sure what is being worked on now that might be relevant in a decade and a half from now. Mirage OS is one of those things - http://openmirage.org http://openmirage.org - though I expect (hope) it'll take much less than decade to be relevant. Some of the same team that worked on early Xen are working on Mirage.
- pron 12y ago
  And there's OSv, http://osv.io/ http://osv.io/, from the people behind KVM.
- josephlord 12y ago
  I've just realised that my last sentence which you quoted could be read as saying that what is currently being worked on won't be as relevant when it was really intended to mean that there is probably some things that are being done now that will be relevant in the future I just don't know what they are.
zetazzed 12y ago
Xen is an example, but, more generally, virtualization has been the big story in systems software for the period mentioned in this article (post-2000). VMware came out of systems research at Stanford and Xen out of Cambridge. The cloud world of AWS/Azure/GCE with tens of millions of VMs running for pennies per hour is a radical shift in how we think about systems. The new wave of Docker/containers promises to extend this idea even further up the stack. How can we call that "stagnation in software" since 2000?
- wrs 12y ago
  Virtualization as an operating system design feature predates that entire chart. A lot of the hard problems in the Xen/VMWare research related to implementing virtualization on a processor architecture (x86) that was extremely popular, but didn't actually support virtualization. Which is just the kind of stagnation he's talking about, I think.

fasteo 12y ago

Not sure to agree; there are quite a few interesting projects in both OS and system programming language fields. To name a few: OS Mirage OS[1] and unikernels in general (ErlangOnXen, HalVM) OSv [2] Chromium OS [3] Programming languages Julia [4] Rust [5] D-Lang [6] Nimrod [7] [1] http://www.openmirage.org/ http://www.openmirage.org/ [2] http://osv.io/ http://osv.io/ [3] http://www.chromium.org/chromium-os http://www.chromium.org/chromium-os [4] http://julialang.org/ http://julialang.org/ [5] http://www.rust-lang.org/ http://www.rust-lang.org/ [6] http://dlang.org/ http://dlang.org/ [7] http://nimrod-lang.org/ http://nimrod-lang.org/

nhaehnle 12y ago

The interesting thing about the table shown in the article is that even while the author laments it, it actually illustrates a good development: the table has an "hourglass" structure. There is a stable part (fundamental technology such as the kernel, TCP/IP etc.) at the center which acts as an interface between the top part (hardware) and the bottom part (languages). All innovation has to be anchored to some core set of stable interfaces into which the new innovations "plug in". The nature of OSs is such that they are naturally part of these core sets. That may be frustrating for OS researchers, but they'll have to learn to live with it.

bcantrill 12y ago

Ugh, this old troll. Something that took me a while to figure out: when someone says that "nothing interesting is happening any longer in [discipline]", it is virtually always code for "I feel old." Yes, things change and different periods of innovation are (of course) different, but it is wrong to cast value judgement on one era over another. Speaking as an aging technologist, I can say that it can be hard to resist this -- and in particular, it's tempting to view one's late twenties and early thirties as a Golden Age, and everything later as derivative and uninteresting. If it needs to be said: for those who are making this claim, they are often projecting their own lives onto their discipline, and may be confusing their own personal Golden Age with a broader one that may or may not have existed. My domain -- operating systems kernel development -- has been particularly vulnerable to this (in part because it has become increasingly specialized) and this is something that I have heard for my entire career. Of course, there has been tons of innovation in the kernel over those two decades (and I've been lucky enough to be close to a bunch of it) -- and no one now would seriously go back to the systems from two decades ago. (If anyone disagrees, they should kindly put their money where their mouth is and run only systems software from 1994.) So yes, systems software is still relevant -- and so are a lot of things that might feel "done." And if you are an aging technologist like me (protected class, baby!) and you feel tempted to tell these youngsters that there is nothing new under the sun, please check yourself: yes, you should educate the rising generation about the problems that have been solved -- but never go so far as to say that there is no room left for new ideas. Aside from the fact that it's a demoralizing thing to say to a younger technologist, history will likely prove you wrong!

linksnapzz 12y ago
Love the work you've done Bryan, but your response above does nothing to engage the salient points made in the link && elsewhere. Yes, we have better ways of getting performance metrics than truss, circa Solaris 2.5…and thank you! But that isn't what this is about.
- bcantrill 12y ago
  Sorry, what salient points am I not engaging? The author of the linked piece is decrying what they perceive as a lack of "OS research and innovation" -- and I'm saying that OS innovation remains alive and well, if perhaps more technical and less accessible than it once was. I can of course be more concrete and rattle these things off -- but I feel safe in leaving software systems advances since 2000 as an exercise to the reader...

mcguire 12y ago

Overall, I agree with Pike in 2000 and this article today. Systems software research has become stagnant and significantly derivative. The point isn't that a new OS would come along and sweep everything away, the point is that nobody seems to be developing new OSs at all. Sort-of. In 2000 or so, systems research had turned almost entirely into server performance measurement and tweaks. That trend seems to have changed, but things like Xen and VMware are not really new. IBM invented virtualization back in the middle ages; most of the more interesting things going on recently have involved using the idea. Likewise, moving stuff from Plan 9, etc., into Linux is cool (very, very cool), but it's not new, groundbreaking research. That being said, there are a lot of nitpicks one can make with this article and with Pike's original talk. The processor speed comparison in the table is not terribly valid, in my opinion. The more important number is front-side bus speed, which has not changed anywhere near as much. (Front-side bus speed dominates most of computing loads I see, and a great whopping chunk of the work in those areas where it doesn't dominate, video games and data-heavy computing, is in preventing the front-side bus from being the bottleneck.) Pike was wrong about the language environment in 2000 and the table is wrong today. Heck, he might have been wrong about 1990. Programming language research in general and its specific applicability to systems (Hi, ATS!) has hotted up a lot since the '80s and '90s. Pike's comment, "There has been much talk about component architectures but only one true success: Unix pipes. It should be possible to build interactive and distributed applications from piece parts," bothers me a lot. A great deal of component research has been a dead-end in general (See that pile of crap with CORBA and SOAP laying on top?) Further, I believe Pike never really understood distributed systems. I have seen Pike complain that Nagel's algorithm has no place in TCP/IP (although I haven't been able to find the reference in years), but if you want a more recent example, see Go---everybody likes CSP, right? Everyone except someone who has tried to implement a synchronous rendezvous on a network. Finally, Plan 9 didn't fail. But it also didn't have any chance to succeed, if by success you mean taking over the world. It was a great research project, but I'd believe making someone who didn't grow up at Bell Labs should be against the Geneva Convention. (Anyone know when Plan 9 started supporting color?) Plan 9 is my prima facie evidence that Not Invented Here syndrome was invented at Bell Labs.

bcantrill 12y ago
There's a lot that's wrong in here, but one thing that absolutely needs to be called out: ...things like Xen and VMware are not really new. IBM invented virtualization back in the middle ages Actually and in fact, what VMware did was new -- to the point that prior to this work, many reasonable people believed that x86 couldn't be virtualized without unacceptable performance penalty. Please read Keith and Ole's excellent ASPLOS paper[1] for an in-depth discussion of the novelty -- and the citation for this technology when it won the ACM Software Systems Award[2]. If you insist that VMware's binary translation work wasn't novel, please be specific about how these peer-reviewed bodies got it wrong. [1] https://www.vmware.com/pdf/asplos235_adams.pdf https://www.vmware.com/pdf/asplos235_adams.pdf [2] http://awards.acm.org/award_winners/bugnion_3886645.cfm http://awards.acm.org/award_winners/bugnion_3886645.cfm

Someone 12y ago

Meanwhile, on the HN front page, we have: https://news.ycombinator.com/item?id=8651499 https://news.ycombinator.com/item?id=8651499: Kill init by touching a bunch of files https://news.ycombinator.com/item?id=8650952 https://news.ycombinator.com/item?id=8650952: on Linux, 'less' can probably get you owned One can argue that the research needed to prevent those was done decades ago, but this article doesn't make that argument. We may have hit upon a 'good enough' solution that is so hard to displace that it may never happen, but I think systems software research is very much relevant. Someone will have to polish a more robust/secure OS (may already exist in unpolished form, but possibly even must still be written) until it is shiny enough to replace the C-based (or equivalent) mess we have now.