6 ms·

Except that the reworked 4.4BSD version likely wouldn't have at all been a microkernel. And remember that much of the reason for the Hurd's u-kernel architectur

Except that the reworked 4.4BSD version likely wouldn't have at all been a microkernel. And remember that much of the reason for the Hurd's u-kernel architecture was in line with FSF goals of user freedom. To get rid of the whole sysadmin/user ring and have the user be master of their own domain, even having private swapped versions of OS services if needed, on top of the flexibility of the translator concept which lets OS services speak any object protocol, including files. Past that, doing a microkernel at that time would be foolish and 4.4BSD integrated with GNU components was obvious choice. I wouldn't have expected such a defeatist attitude from you. Really - trying to push the boundaries of the mainstream was foolish and instead we should have just gotten a GNU-BSD? By the way, they did try adopting an MIT research kernel called TRIX, which had in-kernel RPC and was partially user mode, but that flunked due to the costliness of porting it from the 68000 at the time. After certain amount of time passed, there were a number of usable microkernels that could be split off and turned into working systems. Ah yes, sure. After you're done with a monolithic Unix that the entire industry will settle on, go do a Perl 6 with "we microkernel now" and expect everyone to follow. Ain't happening. Given all this, I think there's little justification for Hurd to keep trying failed strategy on the very microkernel that gave the rest a bad name. They're better off trying to rewrite parts from scratch on a L4-based kernel or QNX clone. Or joining MINIX 3. ;) Retroactively improving Mach is actually a more interesting undertaking than just cloning QNX or doing yet another L4 flavor (which they tried, but found serious issues with L4 and object-capability systems: https://www.gnu.org/software/hurd/faq/which_microkernel.html https://www.gnu.org/software/hurd/faq/which_microkernel.html). Also MINIX 3 doesn't really have the translator concept going for it, nor capability-based security through port rights.

chris_wot 11y ago
I have honestly started to look at your comment feed. Seriously, how the hell do you hold this much arcane (but amazing!) knowledge in your head?
- vezzy-fnord 11y ago
  Heh, I'm flattered. I recall we left a rather bitter taste in each other regarding the systemd question (speaking of which, you never got back on my paper). It's really nothing special. Copious amounts of time coupled with an interest in system software research that motivates me to read research papers, dive into project wikis and the like. If you'd like to discuss things further, just reply affirmatively and I'll send you an email.
  - chris_wot 11y ago
    I was quite depressed at the time, I can only apologise for my tone. I've started to read your piece, but I've honestly been interrupted constantly so haven't been able to give it my full attention, which I intend to do as soon as I can! My disagreement with you over systemd, incidentally, never had anything to do with my assessment of your abilities. It was pretty obvious we both have technical skills :-)
- nickpsecurity 11y ago
  Both he and I realized that there's lots of answers to modern problems in those old papers, as history repeats a lot in IT. The field has a problem of passing wisdom down to next generations or just generalizing solutions. The constraints and OS/hardware competitiveness also made lots of inherently interesting solutions to problems that you're unlikely to see re-appear outside of embedded or special-purpose. Fun stuff to read. If interested, Timeline of Operating Systems is a good start. Just pick summary of anything major (minus DOS's) and dig into references if it interests you. https://en.wikipedia.org/wiki/Timeline_of_operating_systems https://en.wikipedia.org/wiki/Timeline_of_operating_systems The capability systems were also forward-thinking with much modern work on OOP, isolation, etc slowly reinventing its methods. Easy extension plus strong security/reliability were their thing. https://homes.cs.washington.edu/~levy/capabook/index.html https://homes.cs.washington.edu/~levy/capabook/index.html Here's a paper vezzy-fnord shared with a lot of nice examples with features that defined future work and some still ahead of modern systems: http://brinch-hansen.net/papers/2001b.pdf http://brinch-hansen.net/papers/2001b.pdf Lots of old stuff that still beats modern, mainstream stuff on key metrics. Hopefully, you'll dig into some of it and enjoy the "arcane" stuff yourself. It's arcane, but sometimes awesome too. :)
  - chris_wot 11y ago
    Oh, when I say "arcane" it's not a criticism :-) and a little tongue in cheek, I Lo e stuff like this. Hell, I'm reviewing x86 segmented real mode at the moment, just because I can. I'm the last one to criticise someone for doing deep dived into technical material!
    - nickpsecurity 11y ago
      Oh, I knew you liked the arcane stuff: it's why I replied. :) Real mode I don't have as much on but segments are powerful in right hands. Even on Atom, they do in 2 cycles what paging isolation takes 8 to do. Called "descriptors" in capability book I referenced so definitely check that out or the dedicated chapter on descriptor architectures. The high-assurance GEMSOS kernel used segments when ported to Intel 286: http://aesec.com/eval/NCSC-FER-94-008.pdf http://aesec.com/eval/NCSC-FER-94-008.pdf Note: Mainly see design, layering, and assurance activities. The most modern use of them is CHERI processor and CheriBSD at Cambridge. https://www.cl.cam.ac.uk/research/security/ctsrd/cheri/ https://www.cl.cam.ac.uk/research/security/ctsrd/cheri/ https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-864.pdf https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-864.pdf Note: Paper references quite a few past uses of them in detail. A great, modern example in software is Code Pointer Integrity (see 2014 paper link): http://dslab.epfl.ch/proj/cpi/ http://dslab.epfl.ch/proj/cpi/ So, there you go for tonight. Have fun. :)
nickpsecurity 11y ago
"Except that the reworked 4.4BSD version likely wouldn't have at all been a microkernel. And remember that much of the reason for the Hurd's u-kernel architecture was in line with FSF goals of user freedom." It was part of it. Their main goal, per their site, was to replace UNIX('s) with a GNU OS. They failed that one totally far as I can tell and didn't do too well on others for over 10 years. I pick the 2007 papers as the cutoff point since knowledgeable people (eg you) can look at them and assess if they've eliminated those issues since. If not, the failure has lasted over 20 years. So, it's a failure and Linux was a success for the main goal. If the goal is just exploration, then I agree that trying to do something with Mach and UNIX would be an interesting challenge. Still not best idea given all the failures up to that point of people trying that exact thing and always due to inherent incompatibilities in their nature. This was true in 1996. But it could be interesting and challenging work from a research perspective that might lead to innovations in handling such things. I've never doubted it would be interesting to read on how they handle everything. "I wouldn't have expected such a defeatist attitude from you." Maybe it's me going on 4 hrs of sleep for days in a row due to work and whatnot. I decided to give it a fresh look. I spent past hour reading its original paper, skimmed the papers on its mechanisms, read the 2007 critique in your microkernel link (do read that one), their position paper on fixes, and so on. Brain still tired but here's a semi-fresh take on it. GNU was heavily invested into UNIX but not proprietary-related BS surrounding it. They wanted to maintain compatibility with their tools, POSIX, and especially a filesystem-centric model but with more flexibility, security, and reliability. Bushnell and supporters decided on a microkernel approach, built around Mach specifically, kept privileged the minimum apps needed, kept as much in the standard library as possible, and tried to rework that combination into something largely compatible with UNIX but different & distributed underneath. All systems up to that point had failed to even emulate UNIX on Mach without compatibility, performance, or security issues. However, they managed to mostly integrate it with better performance, decent compatibility, better (but not good) security, and unknown [to me] reliability. The proxying approach to service construction, used prior in capability & security kernel work IIRC, was also independently invented in Hurd with lots of improvements in usability and consistency. Met their goal for flexibility and debugging in non-kernel, OS modifications. Now I contrast it to the other work a bit. Work prior to it, esp KeyKOS, managed to decompose the system further with the desired properties, strong security, good reliability, flexibility, and high performance. QNX created an integrated, UNIX-like OS that was fast, real-time, flexible, multi-server, and lacked Mach's weaknesses. They made it distributed & SMP-aware in their post-2000 version. L4 and separation kernels stayed with custom components directly on microkernel and virtualizing away POSIX/Linux/BSD as user-mode process with lightweight spawning (eg Dresden TUD:OS). The latter partly due to perceived impossibility of integration without API-related problems like in old Mach+UNIX work. Further, many of these assumed and maintained POLA in much of their design while Hurd didn't. Two examples: too trusting of UNIX assumptions, necessitating full enumeration for defensive programming against many unknowns (Walfield & Brinkman 2007); its own assumptions like "you can't harm a process by giving it extra permission" (Bushnell 1996). So, doing the comparison, it's design has serious and often fundamental problems that others avoid which stem from the specific requirements of the authors mandating certain components and interfaces. Let summarize. It's interesting work given the challenges and what they achieved. It has too many problems that are designed in which alternatives designed out. Mostly due to trying to shoehorn monolithic, insecure UNIX/POSIX API into a multi-server, robust microkernel with tight integration. Hard to say if further work is worthwhile given inability by 2007 to fix fundamental issues caused by mismatches in their goals, Mach, and UNIX/POSIX API. (Have they been since?) Although some precedent existed, the project did seem to make a lasting contribution via their translators. These were effective and UNIX-like enough that they got uptake that led to FUSE and later Rump kernels. Both are being used in many application areas ranging from development to desktop/server use to cloud applications. So, that's the overall take. It was a failure in many respects, but not a total waste. The translator concept was a signficant win in FOSS with long-term benefits. The battles at mixing two incompatible systems might benefit other work extending UNIX's or even non-UNIX's using their RPC model. People wanting Hurd to succeed should read the critique and solution papers while knocking out as many of these issues as possible. By 2007 & maybe currently, numerous attributes of the system are inferior to other works. Anyone wanting to meet the goals of the project are better off building on other approaches, esp non-Mach, while considering selectively breaking POSIX model or even entirely virtualizing some aspects. That's my current take unless 2007-2015 brought in amazing, fundamental improvements. Thoughts on the semi-fresh analysis?

chris_wot 11y ago

I have honestly started to look at your comment feed. Seriously, how the hell do you hold this much arcane (but amazing!) knowledge in your head?

vezzy-fnord 11y ago
Heh, I'm flattered. I recall we left a rather bitter taste in each other regarding the systemd question (speaking of which, you never got back on my paper). It's really nothing special. Copious amounts of time coupled with an interest in system software research that motivates me to read research papers, dive into project wikis and the like. If you'd like to discuss things further, just reply affirmatively and I'll send you an email.
- chris_wot 11y ago
  I was quite depressed at the time, I can only apologise for my tone. I've started to read your piece, but I've honestly been interrupted constantly so haven't been able to give it my full attention, which I intend to do as soon as I can! My disagreement with you over systemd, incidentally, never had anything to do with my assessment of your abilities. It was pretty obvious we both have technical skills :-)
nickpsecurity 11y ago
Both he and I realized that there's lots of answers to modern problems in those old papers, as history repeats a lot in IT. The field has a problem of passing wisdom down to next generations or just generalizing solutions. The constraints and OS/hardware competitiveness also made lots of inherently interesting solutions to problems that you're unlikely to see re-appear outside of embedded or special-purpose. Fun stuff to read. If interested, Timeline of Operating Systems is a good start. Just pick summary of anything major (minus DOS's) and dig into references if it interests you. https://en.wikipedia.org/wiki/Timeline_of_operating_systems https://en.wikipedia.org/wiki/Timeline_of_operating_systems The capability systems were also forward-thinking with much modern work on OOP, isolation, etc slowly reinventing its methods. Easy extension plus strong security/reliability were their thing. https://homes.cs.washington.edu/~levy/capabook/index.html https://homes.cs.washington.edu/~levy/capabook/index.html Here's a paper vezzy-fnord shared with a lot of nice examples with features that defined future work and some still ahead of modern systems: http://brinch-hansen.net/papers/2001b.pdf http://brinch-hansen.net/papers/2001b.pdf Lots of old stuff that still beats modern, mainstream stuff on key metrics. Hopefully, you'll dig into some of it and enjoy the "arcane" stuff yourself. It's arcane, but sometimes awesome too. :)
- chris_wot 11y ago
  Oh, when I say "arcane" it's not a criticism :-) and a little tongue in cheek, I Lo e stuff like this. Hell, I'm reviewing x86 segmented real mode at the moment, just because I can. I'm the last one to criticise someone for doing deep dived into technical material!
  - nickpsecurity 11y ago
    Oh, I knew you liked the arcane stuff: it's why I replied. :) Real mode I don't have as much on but segments are powerful in right hands. Even on Atom, they do in 2 cycles what paging isolation takes 8 to do. Called "descriptors" in capability book I referenced so definitely check that out or the dedicated chapter on descriptor architectures. The high-assurance GEMSOS kernel used segments when ported to Intel 286: http://aesec.com/eval/NCSC-FER-94-008.pdf http://aesec.com/eval/NCSC-FER-94-008.pdf Note: Mainly see design, layering, and assurance activities. The most modern use of them is CHERI processor and CheriBSD at Cambridge. https://www.cl.cam.ac.uk/research/security/ctsrd/cheri/ https://www.cl.cam.ac.uk/research/security/ctsrd/cheri/ https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-864.pdf https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-864.pdf Note: Paper references quite a few past uses of them in detail. A great, modern example in software is Code Pointer Integrity (see 2014 paper link): http://dslab.epfl.ch/proj/cpi/ http://dslab.epfl.ch/proj/cpi/ So, there you go for tonight. Have fun. :)

nickpsecurity 11y ago

"Except that the reworked 4.4BSD version likely wouldn't have at all been a microkernel. And remember that much of the reason for the Hurd's u-kernel architecture was in line with FSF goals of user freedom." It was part of it. Their main goal, per their site, was to replace UNIX('s) with a GNU OS. They failed that one totally far as I can tell and didn't do too well on others for over 10 years. I pick the 2007 papers as the cutoff point since knowledgeable people (eg you) can look at them and assess if they've eliminated those issues since. If not, the failure has lasted over 20 years. So, it's a failure and Linux was a success for the main goal. If the goal is just exploration, then I agree that trying to do something with Mach and UNIX would be an interesting challenge. Still not best idea given all the failures up to that point of people trying that exact thing and always due to inherent incompatibilities in their nature. This was true in 1996. But it could be interesting and challenging work from a research perspective that might lead to innovations in handling such things. I've never doubted it would be interesting to read on how they handle everything. "I wouldn't have expected such a defeatist attitude from you." Maybe it's me going on 4 hrs of sleep for days in a row due to work and whatnot. I decided to give it a fresh look. I spent past hour reading its original paper, skimmed the papers on its mechanisms, read the 2007 critique in your microkernel link (do read that one), their position paper on fixes, and so on. Brain still tired but here's a semi-fresh take on it. GNU was heavily invested into UNIX but not proprietary-related BS surrounding it. They wanted to maintain compatibility with their tools, POSIX, and especially a filesystem-centric model but with more flexibility, security, and reliability. Bushnell and supporters decided on a microkernel approach, built around Mach specifically, kept privileged the minimum apps needed, kept as much in the standard library as possible, and tried to rework that combination into something largely compatible with UNIX but different & distributed underneath. All systems up to that point had failed to even emulate UNIX on Mach without compatibility, performance, or security issues. However, they managed to mostly integrate it with better performance, decent compatibility, better (but not good) security, and unknown [to me] reliability. The proxying approach to service construction, used prior in capability & security kernel work IIRC, was also independently invented in Hurd with lots of improvements in usability and consistency. Met their goal for flexibility and debugging in non-kernel, OS modifications. Now I contrast it to the other work a bit. Work prior to it, esp KeyKOS, managed to decompose the system further with the desired properties, strong security, good reliability, flexibility, and high performance. QNX created an integrated, UNIX-like OS that was fast, real-time, flexible, multi-server, and lacked Mach's weaknesses. They made it distributed & SMP-aware in their post-2000 version. L4 and separation kernels stayed with custom components directly on microkernel and virtualizing away POSIX/Linux/BSD as user-mode process with lightweight spawning (eg Dresden TUD:OS). The latter partly due to perceived impossibility of integration without API-related problems like in old Mach+UNIX work. Further, many of these assumed and maintained POLA in much of their design while Hurd didn't. Two examples: too trusting of UNIX assumptions, necessitating full enumeration for defensive programming against many unknowns (Walfield & Brinkman 2007); its own assumptions like "you can't harm a process by giving it extra permission" (Bushnell 1996). So, doing the comparison, it's design has serious and often fundamental problems that others avoid which stem from the specific requirements of the authors mandating certain components and interfaces. Let summarize. It's interesting work given the challenges and what they achieved. It has too many problems that are designed in which alternatives designed out. Mostly due to trying to shoehorn monolithic, insecure UNIX/POSIX API into a multi-server, robust microkernel with tight integration. Hard to say if further work is worthwhile given inability by 2007 to fix fundamental issues caused by mismatches in their goals, Mach, and UNIX/POSIX API. (Have they been since?) Although some precedent existed, the project did seem to make a lasting contribution via their translators. These were effective and UNIX-like enough that they got uptake that led to FUSE and later Rump kernels. Both are being used in many application areas ranging from development to desktop/server use to cloud applications. So, that's the overall take. It was a failure in many respects, but not a total waste. The translator concept was a signficant win in FOSS with long-term benefits. The battles at mixing two incompatible systems might benefit other work extending UNIX's or even non-UNIX's using their RPC model. People wanting Hurd to succeed should read the critique and solution papers while knocking out as many of these issues as possible. By 2007 & maybe currently, numerous attributes of the system are inferior to other works. Anyone wanting to meet the goals of the project are better off building on other approaches, esp non-Mach, while considering selectively breaking POSIX model or even entirely virtualizing some aspects. That's my current take unless 2007-2015 brought in amazing, fundamental improvements. Thoughts on the semi-fresh analysis?