8 ms·

Writing an OS in Rust: Returning from Exceptions

175 points by phil-opp 10y ago

kibwen 10y ago
The amount of effort that Phil puts into these posts is really fantastic. Not only are they a great example of how to leverage Rust's abstractions to provide some level of safety in an unsafe domain, but they're also (IMO) approachable enough to appeal to people who have never worked at such a low level before, growing the population of systems programmers in the process. Keep it up! :)
- phil-opp 10y ago
  Thank you so much!
adamnemecek 10y ago
I think that now that we have a language suitable for writing kernels, we also need to have a discussion about what do we really want from a new os. Over the years there have been quite a few OSs and some of them were more advanced than the currently popular OSs. E.g. AS/400 didn't make the distinction between memory and hdd which seems to me like a good idea. BeOS has a fully async API which resulted in a much better user experience and better CPU utilization. Tandem had erlang like processes. None of the popular OSs have this. I'm not sure I want another UNIX implementation. Furthermore, why are all the APIs so diverse? Why aren't there reactive operating systems (as in OS with reactive API)? All of these ideas can be explored in Rust but on some level I'm not sure what should be the feature set of the OS of the future. The current driver models aren't that great either.
- TickleSteve 10y ago
  Would be wiser to only change one variable at once... Prove that an existing RTOS style library can be written with a simple scheduler working well. That would go a long way to it gaining adoption in the embedded world, show that traditional solutions work correctly with the new language first.... then move on to an innovative OS.
  - adamnemecek 10y ago
    Sometimes you have to go top to bottom. I think that Alan Kay says something along the lines that you should start with the design of the API. Polya say something similar in how to prove it.
    - taeric 10y ago
      Polya says to start from the solution. The problem is you have to know a definitive solution for this to be applicable. In the case of this hypothetical OS idea.... Most of what people "would want" from an OS is a made up thing they can't really express. There is no working backwards from a generalization. You work backwards from concrete ideas. Realistically, the OS everyone wants is "one that runs all of their software." They would prefer one that "doesn't crash." However, starting over with no real software is a non-answer.
      - adamnemecek 10y ago
        > Realistically, the OS everyone wants is "one that runs all of their software." I want an OS that's easy to program to the point I don't actually have to rely on third party software too much. This might be a pipe dream but I'm not convinced it's not achievable on some level. > They would prefer one that "doesn't crash." However, starting over with no real software is a non-answer. I'm trying to open a discussion about what ideas have been tried in the past, seem good, but for one reason or another didn't catch on.
        taeric 10y ago
        How does any modern computer not fit this ideal? Linux, in particular, is very easy to start hacking at.
        adamnemecek 10y ago
        What percentage of the OS that you use every day was written by you?
- BinaryIdiot 10y ago
  > The current driver models aren't that great either. Driver developer is soooo far out of my frield but it's always fascinated me. I'm curious why hardware companies didn't go with something like an interface for products versus these highly complex drivers; wouldn't it be possible for each type of device a group comes together, decides on an interface, and now anytime you plug a device in it would work (and if a specialized driver could be better you could install that separately but the goal is making everything magically work, immediately). But since no one has done it I feel like my idea is either horrible or short sighted. But I do want to look into how it all works one day.
  - stusmall 10y ago
    You aren't shortsighted, they have. Look at USB class drivers.
  - deleted 10y ago
    [deleted]
  - sedachv 10y ago
    That is how the original IBM PC BIOS worked with basic hardware, more or less. Both Open Firmware (https://en.wikipedia.org/wiki/Open_Firmware https://en.wikipedia.org/wiki/Open_Firmware) and EFI (http://www.uefi.org/sites/default/files/resources/EBC_Driver_Presentation.pdf http://www.uefi.org/sites/default/files/resources/EBC_Driver...) include a bytecode interpreter for portable device drivers - the idea is that the hardware device stores the driver as bytecode, the Open Firmware controller loads that driver bytecode when the device is plugged in and uses the CPU to execute it, and provides a uniform API to the OS. From what I read EFI was developed at Intel to be a deliberately incompatible Open Firmware clone for the Itanium. There really is not that much savings or advantage to this approach so there are not a lot of Open Firmware and EFI drivers out there. You really do not want hardware manufacturers writing drivers either - they tend to be terrible and full of bugs. Having the manufacturer publish the hardware specifications so that drivers can be easily implemented by developers is a much better approach, sadly a lot of manufacturers cannot be bothered with this.
  - wott 10y ago
    During DOS days, standards or more often de-facto standards had appeared. So chip makers or board makers would make their chip or board compatible with another existing model, and you would just need one driver per kind of device to get most of the expected functions. I am thinking of : -- sound cards being compatible with AdLib and/or SoundBlaster and/or SoundBlaster Pro; -- network cards being compatible with NE2000; -- video cards being compatible with VESA standards. Then Windows took over, buses changed, the shitfest started and never stopped. And now the shitfest comes with NDAs too.
- paulsutter 10y ago
  Security.
  - xj9 10y ago
    Genode on seL4.
- bluejekyll 10y ago
  > Why aren't there reactive operating systems (as in OS with reactive API) This is something that's been interesting me lately. I've been working with the excellent futures and tokio libraries, and I've wondered what would it be like if that was the primitive for all OS interactions? Basically, imagine a message passing system like Mach, but where the fundamental messages are all futures based. There might be some interesting things possible without holding locks in the kernel.
  - phil-opp 10y ago
    I definitely plan to integrate the futures library in the kernel somehow. We just need some asynchronous things and threads first :).
    - bluejekyll 10y ago
      cool! Do you have an overarching design your working off of? Or still too low level?
      - phil-opp 10y ago
        I have some vague ideas for the next posts, but no overarching design. At our current level, there aren't many design choices anyway. It will get much more intesting when we create a threading/concurrency model in the future.
- legulere 10y ago
  IMO a sane syscall API that was built with sandboxing in mind. The unix API gives you access to basically all of the computer or at least all of the user's data. Nowadays we live with executables we can't trust. Random binaries don't need to be dangerous to execute, only the syscalls that they can make on windows and unixes make them dangerous. This also relieves you from the need to sign most programs by a central authority
- pjmlp 10y ago
  > I think that now that we have a language suitable for writing kernels We already had it in 1961 as ESPOL and NEWP. http://www.liquisearch.com/burroughs_large_systems/algol/espol_and_newp http://www.liquisearch.com/burroughs_large_systems/algol/esp... Followed by many others, before UNIX's adoption due to its source being available for free. > I'm not sure I want another UNIX implementation. Yeah, they just keep repeating what was already done and without much research besides adding mainframe features. As Rob Pike puts it: "We really are using a 1970s era operating system well past its sell-by date. We get a lot done, and we have fun, but let's face it, the fundamental design of Unix is older than many of the readers of Slashdot, while lots of different, great ideas about computing and networks have been developed in the last 30 years. Using Unix is the computing equivalent of listening only to music by David Cassidy." from https://interviews.slashdot.org/story/04/10/18/1153211/rob-pike-responds https://interviews.slashdot.org/story/04/10/18/1153211/rob-p... What we really need are OSes that take the ideas from human-computer interaction from Xerox PARC and Viewpoints Research Institute and take them further into mainstream, using safe languages in the process.
  - DblPlusUngood 10y ago
    > "...while lots of different, great ideas about computing and networks have been developed in the last 30 years." > What we really need are OSes that take the ideas from human-computer interaction from Xerox PARC and Viewpoints Research Institute and take them further into mainstream, using safe languages in the process. How are modern OSes prevented from using said ideas (which ideas exactly?) simply because they are a UNIX implementation?
    - pjmlp 10y ago
      Because most UNIX users see only this: http://pubs.opengroup.org/onlinepubs/9699919799/ http://pubs.opengroup.org/onlinepubs/9699919799/ Everything else is fluff, and could stay just like it was on UNIX System V. Hence why for macOS, iOS, tvOS, Android and ChromeOS, having an UNIX kernel is just a matter of convenience, given their application models and programming languages. Apple and Google could eventually migrate to something else as kernel while keeping their Objective-C, Swift, Java and JavaScript APIs. NeXT based NeXTStep on UNIX as a door into the then new UNIX worksations market, but it never had a CLI culture like the other UNIX vendors. GUI workflows like on Xerox PARC were always important to Steve.
- _pmf_ 10y ago
  Off the top of my head: - strongly typed syscalls. - easier way to circumvent the kernel / OS (unikernels are very active and tangible space, see HFT) while potentially staying relatively safe - safe zero copy IPC - projects like https://zinc.rs/ https://zinc.rs/ which fold Embedded Linux' Device Tree concept into a coherent declarative HW description that is consumable by the bootstrapper
- justaaron 10y ago
  I would like to see a very simple OS, built for single-usage, allowing a singular application full control of hardware resources (multi-"threaded", but then let's expose "kernel threads" and "processes" via a rather simple/brutal/direct scheduling API.) I basically want a simple RTOS and HAL/Driver body. I want timers and clocks. I want some block devices and perhaps file-system-primitives etc. I want TCP/IP and a USB stack. I don't want anything nearly as complex as a Unix, nor need it be Posix compliant etc. I would like it to be runnable from uboot, as it's ARM time now, and arm64 support should be a given. I would think there is a substantial market for a minimalistic ultra-simple braindead-easy alternative to the "embedded linux" (LOL) wave that seems to be proliferating on all "a" class arm processors... i need determinacy more than thoroughput, but a 64k ram microcontroller with 500mips isn't going to cut it either. there is a serious hole in the OS market for CPU's where *nix is overkill or unsuitable (and no, RTAI nor a realtime patched kernel count. the complexity of the entire codebase is still there) I've heard about MirageOS from the oCaml people, but I have not been able to find out any info about timing and concurrency etc... any info on edgy projects highly appreciated, please!
Animats 10y ago
Hm. Is that code using the user's stack to handle an exception or interrupt? That's unsafe. If there's not enough user stack space (something the user can force), the kernel will get a double fault, usually a kernel panic condition. Normally, OSs above the DOS level switch to a kernel stack on an exception or interrupt. There's hardware support to help with this; see "Task state segment" (16 and 32 bit x86 only, amd64 is different).
- monocasa 10y ago
  He doesn't have user mode yet AFAICT.
- amluto 10y ago
  No. In fact, that hardware support is mandatory. The SP0 field is used unconditionally on a cross-privilege exception. Sadly, AMD64 came up with a terrible design for SYSCALL, and an exception right after SYSCALL will not automatically switch stacks. The result is a big mess.
amluto 10y ago
One minor sort-of-error: > The iretq instruction is the one and only way to return from exceptions and is specifically designed for this purpose. Not quite true. STI; LRET works too, and it's faster for stupid reasons. Also, the AMD architects blew it badly here. That quote from the manual: > IRET “must be used to terminate the exception or interrupt handler associated with the exception”. Indicates that the architects didn't think about how multitasking works. Consider: 1. User process A goes to sleep using a system call (select, nanosleep, whatever) that uses the SYSCALL instruction. 2. The kernel does a context switch to process B. 3. B's time slice runs out. The kernel finds out about this due to an interrupt. The kernel switches back to process A. 4. The kernel returns to process A's user code using SYSRET. This is an entirely ordinary sequence of events. But think about it from the CPU's perspective: the CPU entered the kernel in step 3 via an interrupt and returned in step 4 using SYSRET, which is not the same thing as IRETQ. Oh no! It turns out that this actually causes a problem on AMD CPUs: SYSRET will screw up the hidden part of the SS descriptor, causing bizarre crashes. Go AMD. Intel, fortunately, implemented SYSRET a bit differently and it works fine. Linux has a specific workaround for this design failure -- search for SYSRET_SS_ATTRS in the kernel source. I don't know how other kernels deal with it. Of course, Intel made other absurd errors in their IA-32e design , but that's another story.
- phil-opp 10y ago
  > STI; LRET works too, and it's faster for stupid reasons. Interesting, didn't know that. > This is an entirely ordinary sequence of events. [...] It turns out that this actually causes a problem on AMD CPUs Sometimes I think that the hardware designers intentionally made kernel development complicated :D. Thanks for the heads-up!
  - amluto 10y ago
    > > STI; LRET works too, and it's faster for stupid reasons. > > Interesting, didn't know that. In case you're curious, here's an implementation for Linux: https://git.kernel.org/cgit/linux/kernel/git/luto/linux.git/log/?h=x86/lret-to-userspace https://git.kernel.org/cgit/linux/kernel/git/luto/linux.git/... There are a couple gotchas. RF and TF won't work right with the LRET hack. You need to make sure not to clear IF until the STI, as otherwise you lose the magic one-instruction no-interrupts window. And it's unclear in the spec if NMIs or MCEs honor that window, so, if you want to be robust and your kernel can recover from NMI or MCE, you should detect if this happens, rewind one instruction, and clear IF again before returning. Other than that, it appears to work perfectly. :)
haberman 10y ago
A very interesting article. One thing that stood out to me: > Unfortunately, Rust does not support [a save-all-registers calling convention]. It was proposed once, but did not get accepted for various reasons. The primary reason was that such calling conventions can be simulated by writing a naked wrapper function. Followed by: > However, auto-vectorization causes a problem for us: Most of the multimedia registers are caller-saved. [...] We don’t use any multimedia registers explicitly, but the Rust compiler might auto-vectorize our code (including the exception handlers). This seems like a pretty convincing argument in favor of supporting this calling convention explicitly: only Rust knows what registers it is actually using. The current approach devolves into preserving every register that Rust might possibly use. AVX-512 has 2kb of registers alone! That's a lot of junk to save to the stack on the off-chance that Rust decides to super-auto-vectorize something.
- pcwalton 10y ago
  > That's a lot of junk to save to the stack on the off-chance that Rust decides to super-auto-vectorize something. Note that LLVM loves to use the XMM registers to do memcpys. This is something that kernels definitely do. So it's definitely a tradeoff.
  - amluto 10y ago
    Using extended state (XMM, etc) in a kernel is asking for performance problems. Saving and restoring is quite slow, and manually saving just a couple of, say, XMM regs defeats the "init optimization".
  - haberman 10y ago
    Right but my point is that saving registers that are not actually used is needless waste. If a "save all registers" calling convention was natively supported by Rust, you would only pay the cost for registers that are actually used.
    - Manishearth 10y ago
      Not exactly, you would have to save all registers all the time. For caller-saved/scratch registers in interrupt handlers, not only do you have to avoid stomping over registers, anything you call has to do too. You have three options here: - Save all the registers in each interrupt - Only call "save all registers" functions in your function, enforce this somehow. Since things like pagefault handlers can get pretty involved, you probably don't want to do this. - Just compile your kernel with most extra registers disabled. There is the fourth option which involves whole program taint analysis or something to track the registers stomped on by all transitive calls from the exception handler. Requires special compiler support though.
mtnygard 10y ago
This series is wonderful. Somehow, Phil is able to introduce concepts from CPU state through compiler wonkiness to kernel design--all while showing how Rust helps handle them--but make them all accessible. One other thing the series does is show how much we dink around due to x86 backward compatibility: - GDT still required by CPU but not needed for programs. - Ridiculous structure of IDT pointers due to multiple generations of bit width extension. - Boot sequence. Compared to the low-level setup for an ARM chip, it's night and day. ARM is what Intel was in the days of the 8088: load your program at an address, CPU jumps to that address, end of story!

175 points by phil-opp 10y ago

The amount of effort that Phil puts into these posts is really fantastic. Not only are they a great example of how to leverage Rust's abstractions to provide some level of safety in an unsafe domain, but they're also (IMO) approachable enough to appeal to people who have never worked at such a low level before, growing the population of systems programmers in the process. Keep it up! :)

phil-opp 10y ago
Thank you so much!

adamnemecek 10y ago

I think that now that we have a language suitable for writing kernels, we also need to have a discussion about what do we really want from a new os. Over the years there have been quite a few OSs and some of them were more advanced than the currently popular OSs. E.g. AS/400 didn't make the distinction between memory and hdd which seems to me like a good idea. BeOS has a fully async API which resulted in a much better user experience and better CPU utilization. Tandem had erlang like processes. None of the popular OSs have this. I'm not sure I want another UNIX implementation. Furthermore, why are all the APIs so diverse? Why aren't there reactive operating systems (as in OS with reactive API)? All of these ideas can be explored in Rust but on some level I'm not sure what should be the feature set of the OS of the future. The current driver models aren't that great either.

TickleSteve 10y ago
Would be wiser to only change one variable at once... Prove that an existing RTOS style library can be written with a simple scheduler working well. That would go a long way to it gaining adoption in the embedded world, show that traditional solutions work correctly with the new language first.... then move on to an innovative OS.
- adamnemecek 10y ago
  Sometimes you have to go top to bottom. I think that Alan Kay says something along the lines that you should start with the design of the API. Polya say something similar in how to prove it.
  - taeric 10y ago
    Polya says to start from the solution. The problem is you have to know a definitive solution for this to be applicable. In the case of this hypothetical OS idea.... Most of what people "would want" from an OS is a made up thing they can't really express. There is no working backwards from a generalization. You work backwards from concrete ideas. Realistically, the OS everyone wants is "one that runs all of their software." They would prefer one that "doesn't crash." However, starting over with no real software is a non-answer.
    - adamnemecek 10y ago
      > Realistically, the OS everyone wants is "one that runs all of their software." I want an OS that's easy to program to the point I don't actually have to rely on third party software too much. This might be a pipe dream but I'm not convinced it's not achievable on some level. > They would prefer one that "doesn't crash." However, starting over with no real software is a non-answer. I'm trying to open a discussion about what ideas have been tried in the past, seem good, but for one reason or another didn't catch on.
      - taeric 10y ago
        How does any modern computer not fit this ideal? Linux, in particular, is very easy to start hacking at.
        adamnemecek 10y ago
        What percentage of the OS that you use every day was written by you?
BinaryIdiot 10y ago
> The current driver models aren't that great either. Driver developer is soooo far out of my frield but it's always fascinated me. I'm curious why hardware companies didn't go with something like an interface for products versus these highly complex drivers; wouldn't it be possible for each type of device a group comes together, decides on an interface, and now anytime you plug a device in it would work (and if a specialized driver could be better you could install that separately but the goal is making everything magically work, immediately). But since no one has done it I feel like my idea is either horrible or short sighted. But I do want to look into how it all works one day.
- stusmall 10y ago
  You aren't shortsighted, they have. Look at USB class drivers.
- deleted 10y ago
  [deleted]
- sedachv 10y ago
  That is how the original IBM PC BIOS worked with basic hardware, more or less. Both Open Firmware (https://en.wikipedia.org/wiki/Open_Firmware https://en.wikipedia.org/wiki/Open_Firmware) and EFI (http://www.uefi.org/sites/default/files/resources/EBC_Driver_Presentation.pdf http://www.uefi.org/sites/default/files/resources/EBC_Driver...) include a bytecode interpreter for portable device drivers - the idea is that the hardware device stores the driver as bytecode, the Open Firmware controller loads that driver bytecode when the device is plugged in and uses the CPU to execute it, and provides a uniform API to the OS. From what I read EFI was developed at Intel to be a deliberately incompatible Open Firmware clone for the Itanium. There really is not that much savings or advantage to this approach so there are not a lot of Open Firmware and EFI drivers out there. You really do not want hardware manufacturers writing drivers either - they tend to be terrible and full of bugs. Having the manufacturer publish the hardware specifications so that drivers can be easily implemented by developers is a much better approach, sadly a lot of manufacturers cannot be bothered with this.
- wott 10y ago
  During DOS days, standards or more often de-facto standards had appeared. So chip makers or board makers would make their chip or board compatible with another existing model, and you would just need one driver per kind of device to get most of the expected functions. I am thinking of : -- sound cards being compatible with AdLib and/or SoundBlaster and/or SoundBlaster Pro; -- network cards being compatible with NE2000; -- video cards being compatible with VESA standards. Then Windows took over, buses changed, the shitfest started and never stopped. And now the shitfest comes with NDAs too.
paulsutter 10y ago
Security.
- xj9 10y ago
  Genode on seL4.
bluejekyll 10y ago
> Why aren't there reactive operating systems (as in OS with reactive API) This is something that's been interesting me lately. I've been working with the excellent futures and tokio libraries, and I've wondered what would it be like if that was the primitive for all OS interactions? Basically, imagine a message passing system like Mach, but where the fundamental messages are all futures based. There might be some interesting things possible without holding locks in the kernel.
- phil-opp 10y ago
  I definitely plan to integrate the futures library in the kernel somehow. We just need some asynchronous things and threads first :).
  - bluejekyll 10y ago
    cool! Do you have an overarching design your working off of? Or still too low level?
    - phil-opp 10y ago
      I have some vague ideas for the next posts, but no overarching design. At our current level, there aren't many design choices anyway. It will get much more intesting when we create a threading/concurrency model in the future.
legulere 10y ago
IMO a sane syscall API that was built with sandboxing in mind. The unix API gives you access to basically all of the computer or at least all of the user's data. Nowadays we live with executables we can't trust. Random binaries don't need to be dangerous to execute, only the syscalls that they can make on windows and unixes make them dangerous. This also relieves you from the need to sign most programs by a central authority
pjmlp 10y ago
> I think that now that we have a language suitable for writing kernels We already had it in 1961 as ESPOL and NEWP. http://www.liquisearch.com/burroughs_large_systems/algol/espol_and_newp http://www.liquisearch.com/burroughs_large_systems/algol/esp... Followed by many others, before UNIX's adoption due to its source being available for free. > I'm not sure I want another UNIX implementation. Yeah, they just keep repeating what was already done and without much research besides adding mainframe features. As Rob Pike puts it: "We really are using a 1970s era operating system well past its sell-by date. We get a lot done, and we have fun, but let's face it, the fundamental design of Unix is older than many of the readers of Slashdot, while lots of different, great ideas about computing and networks have been developed in the last 30 years. Using Unix is the computing equivalent of listening only to music by David Cassidy." from https://interviews.slashdot.org/story/04/10/18/1153211/rob-pike-responds https://interviews.slashdot.org/story/04/10/18/1153211/rob-p... What we really need are OSes that take the ideas from human-computer interaction from Xerox PARC and Viewpoints Research Institute and take them further into mainstream, using safe languages in the process.
- DblPlusUngood 10y ago
  > "...while lots of different, great ideas about computing and networks have been developed in the last 30 years." > What we really need are OSes that take the ideas from human-computer interaction from Xerox PARC and Viewpoints Research Institute and take them further into mainstream, using safe languages in the process. How are modern OSes prevented from using said ideas (which ideas exactly?) simply because they are a UNIX implementation?
  - pjmlp 10y ago
    Because most UNIX users see only this: http://pubs.opengroup.org/onlinepubs/9699919799/ http://pubs.opengroup.org/onlinepubs/9699919799/ Everything else is fluff, and could stay just like it was on UNIX System V. Hence why for macOS, iOS, tvOS, Android and ChromeOS, having an UNIX kernel is just a matter of convenience, given their application models and programming languages. Apple and Google could eventually migrate to something else as kernel while keeping their Objective-C, Swift, Java and JavaScript APIs. NeXT based NeXTStep on UNIX as a door into the then new UNIX worksations market, but it never had a CLI culture like the other UNIX vendors. GUI workflows like on Xerox PARC were always important to Steve.
_pmf_ 10y ago
Off the top of my head: - strongly typed syscalls. - easier way to circumvent the kernel / OS (unikernels are very active and tangible space, see HFT) while potentially staying relatively safe - safe zero copy IPC - projects like https://zinc.rs/ https://zinc.rs/ which fold Embedded Linux' Device Tree concept into a coherent declarative HW description that is consumable by the bootstrapper
justaaron 10y ago
I would like to see a very simple OS, built for single-usage, allowing a singular application full control of hardware resources (multi-"threaded", but then let's expose "kernel threads" and "processes" via a rather simple/brutal/direct scheduling API.) I basically want a simple RTOS and HAL/Driver body. I want timers and clocks. I want some block devices and perhaps file-system-primitives etc. I want TCP/IP and a USB stack. I don't want anything nearly as complex as a Unix, nor need it be Posix compliant etc. I would like it to be runnable from uboot, as it's ARM time now, and arm64 support should be a given. I would think there is a substantial market for a minimalistic ultra-simple braindead-easy alternative to the "embedded linux" (LOL) wave that seems to be proliferating on all "a" class arm processors... i need determinacy more than thoroughput, but a 64k ram microcontroller with 500mips isn't going to cut it either. there is a serious hole in the OS market for CPU's where *nix is overkill or unsuitable (and no, RTAI nor a realtime patched kernel count. the complexity of the entire codebase is still there) I've heard about MirageOS from the oCaml people, but I have not been able to find out any info about timing and concurrency etc... any info on edgy projects highly appreciated, please!

Animats 10y ago

Hm. Is that code using the user's stack to handle an exception or interrupt? That's unsafe. If there's not enough user stack space (something the user can force), the kernel will get a double fault, usually a kernel panic condition. Normally, OSs above the DOS level switch to a kernel stack on an exception or interrupt. There's hardware support to help with this; see "Task state segment" (16 and 32 bit x86 only, amd64 is different).

monocasa 10y ago
He doesn't have user mode yet AFAICT.
amluto 10y ago
No. In fact, that hardware support is mandatory. The SP0 field is used unconditionally on a cross-privilege exception. Sadly, AMD64 came up with a terrible design for SYSCALL, and an exception right after SYSCALL will not automatically switch stacks. The result is a big mess.

amluto 10y ago

One minor sort-of-error: > The iretq instruction is the one and only way to return from exceptions and is specifically designed for this purpose. Not quite true. STI; LRET works too, and it's faster for stupid reasons. Also, the AMD architects blew it badly here. That quote from the manual: > IRET “must be used to terminate the exception or interrupt handler associated with the exception”. Indicates that the architects didn't think about how multitasking works. Consider: 1. User process A goes to sleep using a system call (select, nanosleep, whatever) that uses the SYSCALL instruction. 2. The kernel does a context switch to process B. 3. B's time slice runs out. The kernel finds out about this due to an interrupt. The kernel switches back to process A. 4. The kernel returns to process A's user code using SYSRET. This is an entirely ordinary sequence of events. But think about it from the CPU's perspective: the CPU entered the kernel in step 3 via an interrupt and returned in step 4 using SYSRET, which is not the same thing as IRETQ. Oh no! It turns out that this actually causes a problem on AMD CPUs: SYSRET will screw up the hidden part of the SS descriptor, causing bizarre crashes. Go AMD. Intel, fortunately, implemented SYSRET a bit differently and it works fine. Linux has a specific workaround for this design failure -- search for SYSRET_SS_ATTRS in the kernel source. I don't know how other kernels deal with it. Of course, Intel made other absurd errors in their IA-32e design , but that's another story.

phil-opp 10y ago
> STI; LRET works too, and it's faster for stupid reasons. Interesting, didn't know that. > This is an entirely ordinary sequence of events. [...] It turns out that this actually causes a problem on AMD CPUs Sometimes I think that the hardware designers intentionally made kernel development complicated :D. Thanks for the heads-up!
- amluto 10y ago
  > > STI; LRET works too, and it's faster for stupid reasons. > > Interesting, didn't know that. In case you're curious, here's an implementation for Linux: https://git.kernel.org/cgit/linux/kernel/git/luto/linux.git/log/?h=x86/lret-to-userspace https://git.kernel.org/cgit/linux/kernel/git/luto/linux.git/... There are a couple gotchas. RF and TF won't work right with the LRET hack. You need to make sure not to clear IF until the STI, as otherwise you lose the magic one-instruction no-interrupts window. And it's unclear in the spec if NMIs or MCEs honor that window, so, if you want to be robust and your kernel can recover from NMI or MCE, you should detect if this happens, rewind one instruction, and clear IF again before returning. Other than that, it appears to work perfectly. :)

haberman 10y ago

A very interesting article. One thing that stood out to me: > Unfortunately, Rust does not support [a save-all-registers calling convention]. It was proposed once, but did not get accepted for various reasons. The primary reason was that such calling conventions can be simulated by writing a naked wrapper function. Followed by: > However, auto-vectorization causes a problem for us: Most of the multimedia registers are caller-saved. [...] We don’t use any multimedia registers explicitly, but the Rust compiler might auto-vectorize our code (including the exception handlers). This seems like a pretty convincing argument in favor of supporting this calling convention explicitly: only Rust knows what registers it is actually using. The current approach devolves into preserving every register that Rust might possibly use. AVX-512 has 2kb of registers alone! That's a lot of junk to save to the stack on the off-chance that Rust decides to super-auto-vectorize something.

pcwalton 10y ago
> That's a lot of junk to save to the stack on the off-chance that Rust decides to super-auto-vectorize something. Note that LLVM loves to use the XMM registers to do memcpys. This is something that kernels definitely do. So it's definitely a tradeoff.
- amluto 10y ago
  Using extended state (XMM, etc) in a kernel is asking for performance problems. Saving and restoring is quite slow, and manually saving just a couple of, say, XMM regs defeats the "init optimization".
- haberman 10y ago
  Right but my point is that saving registers that are not actually used is needless waste. If a "save all registers" calling convention was natively supported by Rust, you would only pay the cost for registers that are actually used.
  - Manishearth 10y ago
    Not exactly, you would have to save all registers all the time. For caller-saved/scratch registers in interrupt handlers, not only do you have to avoid stomping over registers, anything you call has to do too. You have three options here: - Save all the registers in each interrupt - Only call "save all registers" functions in your function, enforce this somehow. Since things like pagefault handlers can get pretty involved, you probably don't want to do this. - Just compile your kernel with most extra registers disabled. There is the fourth option which involves whole program taint analysis or something to track the registers stomped on by all transitive calls from the exception handler. Requires special compiler support though.

mtnygard 10y ago

This series is wonderful. Somehow, Phil is able to introduce concepts from CPU state through compiler wonkiness to kernel design--all while showing how Rust helps handle them--but make them all accessible. One other thing the series does is show how much we dink around due to x86 backward compatibility: - GDT still required by CPU but not needed for programs. - Ridiculous structure of IDT pointers due to multiple generations of bit width extension. - Boot sequence. Compared to the low-level setup for an ARM chip, it's night and day. ARM is what Intel was in the days of the 8088: load your program at an address, CPU jumps to that address, end of story!