7 ms·

Examining the Legendary Hurd Kernel (2008)

86 points by vezzy-fnord 11y ago

vander_elst 11y ago
It would be nice seeing Hurd on L4se now that its source is freely available
- lelf 11y ago
  https://www.gnu.org/software/hurd/history/port_to_another_microkernel.html https://www.gnu.org/software/hurd/history/port_to_another_mi...
  - vander_elst 11y ago
    I meant the mathematically proved fork of L4
    - coupdejarnac 11y ago
      Looks like your homemade mouse needs its buttons debounced.
  - vander_elst 11y ago
    I meant the mathematically proved fork of L4
  - vander_elst 11y ago
    I meant the mathematically proved fork of L4
  - vander_elst 11y ago
    I meant the mathematically proved fork of L4
  - vander_elst 11y ago
    I meant the mathematically proved fork of L4
InTheArena 11y ago
I sit on our company's architecture review board, and we do a pass on every product that the company does. There are three links that I regularly have grounds to give: The "Linux is obsolete" Debate - http://www.oreilly.com/openbook/opensources/book/appa.html http://www.oreilly.com/openbook/opensources/book/appa.html Taligents Guide to Designing Object Oriented Software: http://www.amazon.com/Taligents-Guide-Designing-Programs-Object-Oriented/dp/0201408880 http://www.amazon.com/Taligents-Guide-Designing-Programs-Obj... And "The Critique" http://walfield.org/papers/200707-walfield-critique-of-the-GNU-Hurd.pdf http://walfield.org/papers/200707-walfield-critique-of-the-G...
- Cymen 11y ago
  The Taligent book is online too: https://root.cern.ch/TaligentDocs/TaligentOnline/DocumentRoot/1.0/Docs/books/WM/WM_1.html https://root.cern.ch/TaligentDocs/TaligentOnline/DocumentRoo...
- nickpsecurity 11y ago
  And yet, the monolithic systems (esp Linux) proved to have all the problems predicted. Shapiro did a nice job showing how ridiculous Linus's claim is in Round 2 of the debate, which you left off: http://www.coyotos.org/docs/misc/linus-rebuttal.html http://www.coyotos.org/docs/misc/linus-rebuttal.html Can't help but repeat a main point: there are many high assurance (reliability or security) microkernel-based systems that have been fielded but not a single one has achieved this based on the monolithic model Linus loves. QED.
Animats 11y ago
It's sad. Neither Hurd nor Mach reached the performance level of QNX. L4 finally did, although L4 now uses shared memory too much. (While it simplifies the kernel to do message passing via shared memory, it makes it easier for one process to mess up another. Queues in shared memory have to be managed very carefully.) The real issues in microkernel performance are subtle. Message passing and CPU dispatching have to be well-integrated. Otherwise, when you pass a message from one process to another in a subroutine/service like way and get a response, you go to the end of the line for CPU time twice. A good test of a message passing system is to write a client/server which does some trivial computation on the server and then returns a reply. Benchmark it. Then repeat the benchmark with the system CPU-bound. If performance drops by orders of magnitude under load, the message passing / CPU dispatch relation has been botched. On the security front, the "critique" says "programs are assumed to represent the interests of the user and, as such, are run with the user’s authority." This, of course, comes from the UNIX/Linux model and still gives us trouble. We still have trouble running something like Flash or a browser on a desktop without giving it far too much authority. (Of course, if you do have a more granular authority model, apps ask for too much, as the mobile world has learned. Grumble of the week: Firefox on Android now demands access to my phone book. Not because the browser needs it, but because their ancillary products, Sync or Pocket, might.)
- betaby 11y ago
  I keep hearing those stories of "performance level of QNX" since at least 2000. Perhaps it's even true (while I didn't see that during 4.x QNX era, but I blame lack of experience back then). I've seen even various synthetic tests QNX vs L4 couple of years ago. Again QNX source code has been released about 5 years ago if I'm not mistaken. So what was that super performing sauce of QNX which other micro kernels lack. What are those concepts/ideas? Why they weren't re-implemented on L4/Hurd (you can't just borrow, QNS is NOT free software)?
  - nickpsecurity 11y ago
    I'd be interested in hearing Animats' opinion. My guess is that it stayed simpler than Mach but not too simple. Right tradeoffs in the kernel along with fitting into the tiniest of cache and [typical of RTOS's] smart choice of instructions for predictable timing. Here's a paper on an older version of it: http://cseweb.ucsd.edu/~voelker/cse221/papers/qnx-paper92.pdf http://cseweb.ucsd.edu/~voelker/cse221/papers/qnx-paper92.pd... In 1992. it could execute in 8KB of cache with the microkernel and interrupt handler. That by itself should buy it some performance. :)
krylon 11y ago
This makes me sad in three ways. 1. Hurd is still not here, yet. 2. With Duke Nukem Forever actually finished, and with Perl 6 looking like it might get finished this year, we kind of run out of good jokes to crack about the Hurd. What will be the next killer-app that will run on Hurd out of the box, once it's finished? 3. Unless I've been misinformed, there are multi-server microkernel systems out there that do work. So it's not like the Hurd was a totally misguided idea. And yet, it's still out there, like cold fusion, artificial intelligence and hypo-allergenic kittens, it's tantalizingly out of reach... Well, one day
- InTheArena 11y ago
  Meanwhile, Linux plays a dominate role in the world with a Kernel that is in absolutely everything. So yeah, Micro-kernels work. But monolithic and simple has been carrying the day for a decade plus. (and before anyone complains about NT, they lost their Microkernel card when the moved graphics into the kernel. They like XNU on the mac are Hybrid Kernels).
  - vezzy-fnord 11y ago
    Actually if we're going purely by adoption rates, the most common kernels are probably some variant of L4 (used in baseband processors among other places) and ITRON in miscellaneous electronics, both of which are microkernels and the latter also an RTOS.
- vezzy-fnord 11y ago
  As tired as it may sound, I think there just might be a convenient time for the Hurd to rise, albeit in an unconventional way. You already have a de facto GNU OS at the moment in the form of the Guix System Distribution (GuixSD), what with it letting you configure the entire system in Scheme, perform transactional upgrades, system state rollbacks, it has its own init daemon and service scripts in Scheme, etc. At the same time, we've been seeing some efforts in creating tiny builds of the Linux kernel [1] that go as far as to reduce syscalls, VMM algorithms, capabilities, character devices and so forth. There's also the recent Linux libOS effort to create a network stack in userspace as a shared library, though that seems about the extent of it. NetBSD then gives you a userland driver framework with rump kernels. So what one can do is build a really stripped down Linux kernel, write a Mach IPC compatibility layer, run the Hurd servers on top of it and plug in the resulting product with GuixSD. And you now have the complete GNU system. As far as you'll ever get, anyway. This has already been discussed before [2], but it's not something that has ever been given any priority. With the current climate, it might be worth a shot. As to why anyone would want to do this... well, for one you get a full OS that is entirely configurable in Scheme with all of its services running as userland file servers (translators) in a sort of Plan 9-ish way, all the while supporting the Linux API. That's still pretty ahead of GNU/Linux. The Hurd also has features like running multiple instances of itself in user mode (subhurds) that can serve a similar purpose to containers. It's obvious no one's going to port the Hurd servers away from GNU Mach, so this is the best shot anyone has. Any Hurd or Guix devs in here to comment on this? [1] https://tiny.wiki.kernel.org/ https://tiny.wiki.kernel.org/ [2] https://www.gnu.org/software/hurd/open_issues/linux_as_the_kernel.html https://www.gnu.org/software/hurd/open_issues/linux_as_the_k...
tomcam 11y ago
Firefox..fills up available RAM with page caches. Often, swapping these pages back in from disk can be slower than re-fetching from the Internet. WTF? I'm not a kernel guy and would like to know why this is. Very, very counterintuitive to me.
- icebraining 11y ago
  Non-SSD disks are really slow for random accesses, like you might need if the cached files are not contiguous, and often congested due to background processes competing for their use. Meanwhile, the server most likely has those files in memory, ready to push them.
  - MichaelGG 11y ago
    Disks are what, 10ms random seek? You'd need a good, nearby server to service in under that time. And that assumes you've got a connection already established. If not, then add another roundtrip. I'm not sure "often" is appropriate here.
    - TheLoneWolfling 11y ago
      That's best-case. (Well, not really. Best case is that your disk isn't doing anything, the data is in one contiguous region, the head is in the right position, and the data is about to pass under the head.) In actuality, it can be a lot worse. I've seen latencies of 500ms+ before, when the disk is in "seek hell". And right now my laptop has a median (!) latency of ~100ms. Mind you I'm listening to music and copying files in the background. But still.
    - justabystander 11y ago
      It's kind of a hyperbole, granted, but most of our assumptions about disk access being faster rely on only one piece of software accessing that disk. With modern multi-core processors, pre-fetching and caching both at the application and the kernel level that assumption is quickly proven to be faulty. 10ms random seek is fine when you're only accessing one thing. But when multiple things are loading and alternating between locations it becomes significantly slower. Windows startup can be a pretty good indicator of this. I've got one system with an SSD (containing the OS) and a spinning platter drive for data. When I start it up, I have to wait for more than a minute before clicking anything or I end up with an unresponsive application for 3+ minutes. It still induces a quiet, resigned rage that such a powerful system can be nigh-crippled on boot.
tomcam 11y ago
Firefox..fills up available RAM with page caches. Often, swapping these pages back in from disk can be slower than re-fetching from the Internet. WTF? I'm not a kernel guy and would like to know why this is. Very, very counterintuitive to me.
sergiolp 11y ago
Some years ago, I've spent a lot of time studying GNU Mach and Hurd (I've also made some small contributions). I think I can say that I now both pretty well. I even started a project to preserve OSF Mach + MkLinux source code (https://github.com/slp/mkunity https://github.com/slp/mkunity), a very cool project for its time (circa 1998). These days I prefer to do my kernel hacking on monolithic kernels, mainly NetBSD. I've stopped working on Mach, Hurd and other experimental microkernels (there're a bunch out there) because it was becoming increasingly frustrating. If you'd ask me to define the problem with microkernels with one word, that would be "complexity". And its a kind of complexity that impacts everything: - Debugging is hard: On monolithic kernels, you have a single image, with both code and state. Hunting a bug is just a matter of jumping into the internal debugger (or attaching an external one, or generating a dump, or...) and looking around. On Hurd, the state is spread among Mach and the servers, so you'll have to look at each one trying to follow the trail left by the bug. - Managing resources is hard: Mach knows everything about the machine, but nothing the user. The server knows everything about the user, but nothing about the machine. And keeping them in sync is too much expensive. Go figure. - Obtaining a reasonalbe performance is har... imposible: You want to read() a pair of bytes from disk? Good, prepare a message, call to Mach, yield a little while the server is scheduled, copy the message, unmarshall it, process the request, prepare another message to Mach to read from disk, call to Mach, yield waiting for rescheduling, obtain the data, prepare the answer, call to Mach, yield waiting for rescheduling, obtain your 2 bytes. Easy! In the end, Torvalds was right. The user doesn't want to work with the OS, he wants to work with his application. This means the OS should be as invisble as possible, and fulfill userland requests following the shortest path. Microkernels doesn't comply with this requirement, so from a user's perspective, they fail natural selection. That said, if you're into kernels, microkernels are different and fun! Don't miss the oportunity of doing some hacking with one of them. Just don't be a fool like me, and avoid become obsessed trying to achieve the imposible.
- vezzy-fnord 11y ago
  If you'd ask me to define the problem with microkernels with one word, that would be "complexity". The problem with Mach, you mean. All the examples you listed are specific to it.
octatoan 11y ago
> Since those days, HURD has acquired the same reputation in operating system circles that Duke Nukem Forever has among gamers. What reputation does Duke Nukem Forever have among gamers?
- detaro 11y ago
  Vaporware. Announced 1997. Restarted/largely rebuild multiple times, released finally in 2011.
chatman 11y ago
It’s possible to have a complete and usable system running nothing other than GNU code.

86 points by vezzy-fnord 11y ago

vander_elst 11y ago

It would be nice seeing Hurd on L4se now that its source is freely available

lelf 11y ago
https://www.gnu.org/software/hurd/history/port_to_another_microkernel.html https://www.gnu.org/software/hurd/history/port_to_another_mi...
- vander_elst 11y ago
  I meant the mathematically proved fork of L4
  - coupdejarnac 11y ago
    Looks like your homemade mouse needs its buttons debounced.
- vander_elst 11y ago
  I meant the mathematically proved fork of L4
- vander_elst 11y ago
  I meant the mathematically proved fork of L4
- vander_elst 11y ago
  I meant the mathematically proved fork of L4
- vander_elst 11y ago
  I meant the mathematically proved fork of L4

InTheArena 11y ago

I sit on our company's architecture review board, and we do a pass on every product that the company does. There are three links that I regularly have grounds to give: The "Linux is obsolete" Debate - http://www.oreilly.com/openbook/opensources/book/appa.html http://www.oreilly.com/openbook/opensources/book/appa.html Taligents Guide to Designing Object Oriented Software: http://www.amazon.com/Taligents-Guide-Designing-Programs-Object-Oriented/dp/0201408880 http://www.amazon.com/Taligents-Guide-Designing-Programs-Obj... And "The Critique" http://walfield.org/papers/200707-walfield-critique-of-the-GNU-Hurd.pdf http://walfield.org/papers/200707-walfield-critique-of-the-G...

Cymen 11y ago
The Taligent book is online too: https://root.cern.ch/TaligentDocs/TaligentOnline/DocumentRoot/1.0/Docs/books/WM/WM_1.html https://root.cern.ch/TaligentDocs/TaligentOnline/DocumentRoo...
nickpsecurity 11y ago
And yet, the monolithic systems (esp Linux) proved to have all the problems predicted. Shapiro did a nice job showing how ridiculous Linus's claim is in Round 2 of the debate, which you left off: http://www.coyotos.org/docs/misc/linus-rebuttal.html http://www.coyotos.org/docs/misc/linus-rebuttal.html Can't help but repeat a main point: there are many high assurance (reliability or security) microkernel-based systems that have been fielded but not a single one has achieved this based on the monolithic model Linus loves. QED.

Animats 11y ago

It's sad. Neither Hurd nor Mach reached the performance level of QNX. L4 finally did, although L4 now uses shared memory too much. (While it simplifies the kernel to do message passing via shared memory, it makes it easier for one process to mess up another. Queues in shared memory have to be managed very carefully.) The real issues in microkernel performance are subtle. Message passing and CPU dispatching have to be well-integrated. Otherwise, when you pass a message from one process to another in a subroutine/service like way and get a response, you go to the end of the line for CPU time twice. A good test of a message passing system is to write a client/server which does some trivial computation on the server and then returns a reply. Benchmark it. Then repeat the benchmark with the system CPU-bound. If performance drops by orders of magnitude under load, the message passing / CPU dispatch relation has been botched. On the security front, the "critique" says "programs are assumed to represent the interests of the user and, as such, are run with the user’s authority." This, of course, comes from the UNIX/Linux model and still gives us trouble. We still have trouble running something like Flash or a browser on a desktop without giving it far too much authority. (Of course, if you do have a more granular authority model, apps ask for too much, as the mobile world has learned. Grumble of the week: Firefox on Android now demands access to my phone book. Not because the browser needs it, but because their ancillary products, Sync or Pocket, might.)

betaby 11y ago
I keep hearing those stories of "performance level of QNX" since at least 2000. Perhaps it's even true (while I didn't see that during 4.x QNX era, but I blame lack of experience back then). I've seen even various synthetic tests QNX vs L4 couple of years ago. Again QNX source code has been released about 5 years ago if I'm not mistaken. So what was that super performing sauce of QNX which other micro kernels lack. What are those concepts/ideas? Why they weren't re-implemented on L4/Hurd (you can't just borrow, QNS is NOT free software)?
- nickpsecurity 11y ago
  I'd be interested in hearing Animats' opinion. My guess is that it stayed simpler than Mach but not too simple. Right tradeoffs in the kernel along with fitting into the tiniest of cache and [typical of RTOS's] smart choice of instructions for predictable timing. Here's a paper on an older version of it: http://cseweb.ucsd.edu/~voelker/cse221/papers/qnx-paper92.pdf http://cseweb.ucsd.edu/~voelker/cse221/papers/qnx-paper92.pd... In 1992. it could execute in 8KB of cache with the microkernel and interrupt handler. That by itself should buy it some performance. :)

krylon 11y ago

This makes me sad in three ways. 1. Hurd is still not here, yet. 2. With Duke Nukem Forever actually finished, and with Perl 6 looking like it might get finished this year, we kind of run out of good jokes to crack about the Hurd. What will be the next killer-app that will run on Hurd out of the box, once it's finished? 3. Unless I've been misinformed, there are multi-server microkernel systems out there that do work. So it's not like the Hurd was a totally misguided idea. And yet, it's still out there, like cold fusion, artificial intelligence and hypo-allergenic kittens, it's tantalizingly out of reach... Well, one day

InTheArena 11y ago
Meanwhile, Linux plays a dominate role in the world with a Kernel that is in absolutely everything. So yeah, Micro-kernels work. But monolithic and simple has been carrying the day for a decade plus. (and before anyone complains about NT, they lost their Microkernel card when the moved graphics into the kernel. They like XNU on the mac are Hybrid Kernels).
- vezzy-fnord 11y ago
  Actually if we're going purely by adoption rates, the most common kernels are probably some variant of L4 (used in baseband processors among other places) and ITRON in miscellaneous electronics, both of which are microkernels and the latter also an RTOS.
vezzy-fnord 11y ago
As tired as it may sound, I think there just might be a convenient time for the Hurd to rise, albeit in an unconventional way. You already have a de facto GNU OS at the moment in the form of the Guix System Distribution (GuixSD), what with it letting you configure the entire system in Scheme, perform transactional upgrades, system state rollbacks, it has its own init daemon and service scripts in Scheme, etc. At the same time, we've been seeing some efforts in creating tiny builds of the Linux kernel [1] that go as far as to reduce syscalls, VMM algorithms, capabilities, character devices and so forth. There's also the recent Linux libOS effort to create a network stack in userspace as a shared library, though that seems about the extent of it. NetBSD then gives you a userland driver framework with rump kernels. So what one can do is build a really stripped down Linux kernel, write a Mach IPC compatibility layer, run the Hurd servers on top of it and plug in the resulting product with GuixSD. And you now have the complete GNU system. As far as you'll ever get, anyway. This has already been discussed before [2], but it's not something that has ever been given any priority. With the current climate, it might be worth a shot. As to why anyone would want to do this... well, for one you get a full OS that is entirely configurable in Scheme with all of its services running as userland file servers (translators) in a sort of Plan 9-ish way, all the while supporting the Linux API. That's still pretty ahead of GNU/Linux. The Hurd also has features like running multiple instances of itself in user mode (subhurds) that can serve a similar purpose to containers. It's obvious no one's going to port the Hurd servers away from GNU Mach, so this is the best shot anyone has. Any Hurd or Guix devs in here to comment on this? [1] https://tiny.wiki.kernel.org/ https://tiny.wiki.kernel.org/ [2] https://www.gnu.org/software/hurd/open_issues/linux_as_the_kernel.html https://www.gnu.org/software/hurd/open_issues/linux_as_the_k...

tomcam 11y ago

Firefox..fills up available RAM with page caches. Often, swapping these pages back in from disk can be slower than re-fetching from the Internet. WTF? I'm not a kernel guy and would like to know why this is. Very, very counterintuitive to me.

icebraining 11y ago
Non-SSD disks are really slow for random accesses, like you might need if the cached files are not contiguous, and often congested due to background processes competing for their use. Meanwhile, the server most likely has those files in memory, ready to push them.
- MichaelGG 11y ago
  Disks are what, 10ms random seek? You'd need a good, nearby server to service in under that time. And that assumes you've got a connection already established. If not, then add another roundtrip. I'm not sure "often" is appropriate here.
  - TheLoneWolfling 11y ago
    That's best-case. (Well, not really. Best case is that your disk isn't doing anything, the data is in one contiguous region, the head is in the right position, and the data is about to pass under the head.) In actuality, it can be a lot worse. I've seen latencies of 500ms+ before, when the disk is in "seek hell". And right now my laptop has a median (!) latency of ~100ms. Mind you I'm listening to music and copying files in the background. But still.
  - justabystander 11y ago
    It's kind of a hyperbole, granted, but most of our assumptions about disk access being faster rely on only one piece of software accessing that disk. With modern multi-core processors, pre-fetching and caching both at the application and the kernel level that assumption is quickly proven to be faulty. 10ms random seek is fine when you're only accessing one thing. But when multiple things are loading and alternating between locations it becomes significantly slower. Windows startup can be a pretty good indicator of this. I've got one system with an SSD (containing the OS) and a spinning platter drive for data. When I start it up, I have to wait for more than a minute before clicking anything or I end up with an unresponsive application for 3+ minutes. It still induces a quiet, resigned rage that such a powerful system can be nigh-crippled on boot.

tomcam 11y ago

sergiolp 11y ago

Some years ago, I've spent a lot of time studying GNU Mach and Hurd (I've also made some small contributions). I think I can say that I now both pretty well. I even started a project to preserve OSF Mach + MkLinux source code (https://github.com/slp/mkunity https://github.com/slp/mkunity), a very cool project for its time (circa 1998). These days I prefer to do my kernel hacking on monolithic kernels, mainly NetBSD. I've stopped working on Mach, Hurd and other experimental microkernels (there're a bunch out there) because it was becoming increasingly frustrating. If you'd ask me to define the problem with microkernels with one word, that would be "complexity". And its a kind of complexity that impacts everything: - Debugging is hard: On monolithic kernels, you have a single image, with both code and state. Hunting a bug is just a matter of jumping into the internal debugger (or attaching an external one, or generating a dump, or...) and looking around. On Hurd, the state is spread among Mach and the servers, so you'll have to look at each one trying to follow the trail left by the bug. - Managing resources is hard: Mach knows everything about the machine, but nothing the user. The server knows everything about the user, but nothing about the machine. And keeping them in sync is too much expensive. Go figure. - Obtaining a reasonalbe performance is har... imposible: You want to read() a pair of bytes from disk? Good, prepare a message, call to Mach, yield a little while the server is scheduled, copy the message, unmarshall it, process the request, prepare another message to Mach to read from disk, call to Mach, yield waiting for rescheduling, obtain the data, prepare the answer, call to Mach, yield waiting for rescheduling, obtain your 2 bytes. Easy! In the end, Torvalds was right. The user doesn't want to work with the OS, he wants to work with his application. This means the OS should be as invisble as possible, and fulfill userland requests following the shortest path. Microkernels doesn't comply with this requirement, so from a user's perspective, they fail natural selection. That said, if you're into kernels, microkernels are different and fun! Don't miss the oportunity of doing some hacking with one of them. Just don't be a fool like me, and avoid become obsessed trying to achieve the imposible.

vezzy-fnord 11y ago
If you'd ask me to define the problem with microkernels with one word, that would be "complexity". The problem with Mach, you mean. All the examples you listed are specific to it.

octatoan 11y ago

> Since those days, HURD has acquired the same reputation in operating system circles that Duke Nukem Forever has among gamers. What reputation does Duke Nukem Forever have among gamers?

detaro 11y ago
Vaporware. Announced 1997. Restarted/largely rebuild multiple times, released finally in 2011.

chatman 11y ago

It’s possible to have a complete and usable system running nothing other than GNU code.