6 ms·

Is it actually 100% necessary to hand-craft assembly for bootstrapping the OS? I suppose it could be that it's simpler to write it out in assembly, since the C

by ipsi 10y ago

Is it actually 100% necessary to hand-craft assembly for bootstrapping the OS? I suppose it could be that it's simpler to write it out in assembly, since the C (/etc) code would look basically the same - just a lot of assigning cryptic variables to cryptic pointers? Or, based on a quick Google search (http://stackoverflow.com/questions/3022046/is-it-possible-to-access-32-bit-registers-in-c http://stackoverflow.com/questions/3022046/is-it-possible-to...), is it because you cannot write to these registers from C, thus needing to drop down to assembly?

all2well 10y ago
Yes, on x86 at least. There are special data structures that can only be initialized with special instructions, which you can't generate with normal C because they aren't useful for anything but initializing those special data structures.
yalue 10y ago
I personally would say yes, it is 100% necessary to write assembly for bootstrapping the OS (at least on x86). It's almost entirely due to your third point--x86 has dedicated instructions which only exist to write to certain control registers, and these, somewhere down the stack, need to be written in assembly. C, and pretty much any other multi-architecture programming language, just isn't going to include architecture-specific control register manipulations in its freestanding library.
- amluto 10y ago
  IMO a bigger issue is that most of these kernel entries have unusual calling conventions. Exception handlers need to preserve all registers, for example. On x86_64, it's more complicated due to GSBASE handling. (Linux had it subtly wrong literally forever. OpenBSD still had it wrong last time I checked, but other hardening measures seem to make it impossible to exploit. SYSCALL is even worse -- when the handler is invoked, there isn't even a stack.
  - bogomipz 10y ago
    I'm not familiar with the term GSBASE, could you explain?
    - bogomipz 10y ago
      I looked it up - segmentation in long mode. I'm curious how both Linux and BSD had/have gotten it "wrong" for so long? That just sounds unusual and likely not an oversight. Can you elaborate?
      - amluto 10y ago
        It's not segmentation. It's effectively just one extra register usable only for address offsets. Unfortunately, on x86, privilege transitions and exception handling is an incoherent mess, and, worse, privilege changes (via IRET) can fail. Most kernels messed up what happens when IRET fails for obscure reasons, resulting in corrupting the value of GSBASE and therefore allowing a malicious program to change the targets of certain kernel memory accesses. The result was "BadIRET", which I discovered, fixed on Linux, and reported via CERT (not a good experience) to other vendors. Someone else named it, and a few other people wrote exploits based on it.
kaushiks 10y ago
It generally doesn't matter how the machine code came to be. It is theoretically possible to write an operating system for (say) x86 in (say) Javascript. The reason people generally pick a "systems" language and drop down to assembly when necessary is due to two broad reasons: 1. As you'd pointed out, the language in question might not let you generate certain instructions. For instance how do you express LGDT in C? You can't. You'd instead write it in assembly (or inline assembly) and expose it to the C world as a function (or in a compiler that supports it, an intrinsic) that it can call. 2. A CPU's understanding of state is defined around the notion of registers and memory. A programming language however exposes a higher level abstraction. For instance, a function call in C is an abstraction that defines both transfer of control (which the CPU knows about) and persistence of local state (saving and restoring registers, stack pointer etc., which the CPU knows nothing about). This is called the ABI and is merely a previously agreed upon convention. Higher level languages (say Java) are able to provide richer abstractions than merely an ABI as their "agreed upon conventions" are defined at a much higher level than in terms of a CPU (registers and memory). Java, for instance even defines a virtual architecture in terms of which facilities offered in the language are defined. User code written in one of these higher level languages depends on these facilities being available to them at any time. This is what makes it unsuitable for writing certain sections of OS code. For instance, the CPU expects the OS to save and restore register state around an interrupt handler. If said handler was written in C (as a function) though, it'd expect its first six arguments in certain registers etc. - conventions, the CPU knows nothing about. You can think of the CPU as exposing its own, simple ABI which can only be implemented by writing certain sections of code in assembly (This is not entirely true though, as one might be able to mimic something like this in certain compilers, especially C compilers, that let the programmer write "naked" function bodies and custom prolog / epilog code). This is also why most parts of the OS, that don't directly interact with the CPU, can and are written in a higher level language. For e.g. The Singularity OS by MSR was written in C# and assembly.
- bogomipz 10y ago
  Isn't the LGDT just an address though? You provide a starting address and a length offset that's it. Why can't that be expressed in C? Does it live in a register not visible to the compiler? You mentioned: "For instance, a function call in C is an abstraction that defines both transfer of control (which the CPU knows about) and persistence of local state (saving and restoring registers, stack pointer etc., which the CPU knows nothing about). " Doesn't the CPU know about this "persistence of local state"? I mean it pushing and popping that state form the stack right? I'm curious what you mean there. What's a "naked function" in this context? I think of naked function as one that doesn't have a return statement. Cheers.
phil-opp 10y ago
As others have mentioned, you need assembly operations to access hardware specific control registers. However, most system languages have some kind of inline assembly, which can be used to encapsulate those operations into e.g. a C or Rust function. So you could eliminate a large portion of the bootstrap assembly and rewrite it in Rust. The problem is that we're in 32-bit mode at the beginning. So we would need two Rust libraries, one for the bootstrap 32-bit code and one for the 64-bit kernel code. We would need to link them together somehow or use GRUB modules. Doing it in assembly was somehow simpler and more fun :) So let's assume you wanted to create an OS with a minimal amount of assembly: The minimum requirement of all higher level languages is at least a valid stack pointer. On x86, you need to initialize the stack pointer explicitely through a `mov esp, 0x{stack_pointer}`. It might be possible to do it in a naked Rust function [1]. As soon as you have set up a stack, you can directly call into Rust or C code (see [2]). It's easier on ARM Cortex-M boards, since the stack pointer is loaded from address 0 on reset. So you can boot directly into Rust, without any assembly file (see [3], especially the `common.ld` linker script). [1]: https://github.com/rust-lang/rfcs/pull/1201 https://github.com/rust-lang/rfcs/pull/1201 [2]: http://wiki.osdev.org/Bare_Bones http://wiki.osdev.org/Bare_Bones [3]: https://github.com/japaric/cu https://github.com/japaric/cu

by ipsi 10y ago

all2well 10y ago

Yes, on x86 at least. There are special data structures that can only be initialized with special instructions, which you can't generate with normal C because they aren't useful for anything but initializing those special data structures.

yalue 10y ago

I personally would say yes, it is 100% necessary to write assembly for bootstrapping the OS (at least on x86). It's almost entirely due to your third point--x86 has dedicated instructions which only exist to write to certain control registers, and these, somewhere down the stack, need to be written in assembly. C, and pretty much any other multi-architecture programming language, just isn't going to include architecture-specific control register manipulations in its freestanding library.

amluto 10y ago
IMO a bigger issue is that most of these kernel entries have unusual calling conventions. Exception handlers need to preserve all registers, for example. On x86_64, it's more complicated due to GSBASE handling. (Linux had it subtly wrong literally forever. OpenBSD still had it wrong last time I checked, but other hardening measures seem to make it impossible to exploit. SYSCALL is even worse -- when the handler is invoked, there isn't even a stack.
- bogomipz 10y ago
  I'm not familiar with the term GSBASE, could you explain?
  - bogomipz 10y ago
    I looked it up - segmentation in long mode. I'm curious how both Linux and BSD had/have gotten it "wrong" for so long? That just sounds unusual and likely not an oversight. Can you elaborate?
    - amluto 10y ago
      It's not segmentation. It's effectively just one extra register usable only for address offsets. Unfortunately, on x86, privilege transitions and exception handling is an incoherent mess, and, worse, privilege changes (via IRET) can fail. Most kernels messed up what happens when IRET fails for obscure reasons, resulting in corrupting the value of GSBASE and therefore allowing a malicious program to change the targets of certain kernel memory accesses. The result was "BadIRET", which I discovered, fixed on Linux, and reported via CERT (not a good experience) to other vendors. Someone else named it, and a few other people wrote exploits based on it.

kaushiks 10y ago

It generally doesn't matter how the machine code came to be. It is theoretically possible to write an operating system for (say) x86 in (say) Javascript. The reason people generally pick a "systems" language and drop down to assembly when necessary is due to two broad reasons: 1. As you'd pointed out, the language in question might not let you generate certain instructions. For instance how do you express LGDT in C? You can't. You'd instead write it in assembly (or inline assembly) and expose it to the C world as a function (or in a compiler that supports it, an intrinsic) that it can call. 2. A CPU's understanding of state is defined around the notion of registers and memory. A programming language however exposes a higher level abstraction. For instance, a function call in C is an abstraction that defines both transfer of control (which the CPU knows about) and persistence of local state (saving and restoring registers, stack pointer etc., which the CPU knows nothing about). This is called the ABI and is merely a previously agreed upon convention. Higher level languages (say Java) are able to provide richer abstractions than merely an ABI as their "agreed upon conventions" are defined at a much higher level than in terms of a CPU (registers and memory). Java, for instance even defines a virtual architecture in terms of which facilities offered in the language are defined. User code written in one of these higher level languages depends on these facilities being available to them at any time. This is what makes it unsuitable for writing certain sections of OS code. For instance, the CPU expects the OS to save and restore register state around an interrupt handler. If said handler was written in C (as a function) though, it'd expect its first six arguments in certain registers etc. - conventions, the CPU knows nothing about. You can think of the CPU as exposing its own, simple ABI which can only be implemented by writing certain sections of code in assembly (This is not entirely true though, as one might be able to mimic something like this in certain compilers, especially C compilers, that let the programmer write "naked" function bodies and custom prolog / epilog code). This is also why most parts of the OS, that don't directly interact with the CPU, can and are written in a higher level language. For e.g. The Singularity OS by MSR was written in C# and assembly.

bogomipz 10y ago
Isn't the LGDT just an address though? You provide a starting address and a length offset that's it. Why can't that be expressed in C? Does it live in a register not visible to the compiler? You mentioned: "For instance, a function call in C is an abstraction that defines both transfer of control (which the CPU knows about) and persistence of local state (saving and restoring registers, stack pointer etc., which the CPU knows nothing about). " Doesn't the CPU know about this "persistence of local state"? I mean it pushing and popping that state form the stack right? I'm curious what you mean there. What's a "naked function" in this context? I think of naked function as one that doesn't have a return statement. Cheers.

phil-opp 10y ago

As others have mentioned, you need assembly operations to access hardware specific control registers. However, most system languages have some kind of inline assembly, which can be used to encapsulate those operations into e.g. a C or Rust function. So you could eliminate a large portion of the bootstrap assembly and rewrite it in Rust. The problem is that we're in 32-bit mode at the beginning. So we would need two Rust libraries, one for the bootstrap 32-bit code and one for the 64-bit kernel code. We would need to link them together somehow or use GRUB modules. Doing it in assembly was somehow simpler and more fun :) So let's assume you wanted to create an OS with a minimal amount of assembly: The minimum requirement of all higher level languages is at least a valid stack pointer. On x86, you need to initialize the stack pointer explicitely through a `mov esp, 0x{stack_pointer}`. It might be possible to do it in a naked Rust function [1]. As soon as you have set up a stack, you can directly call into Rust or C code (see [2]). It's easier on ARM Cortex-M boards, since the stack pointer is loaded from address 0 on reset. So you can boot directly into Rust, without any assembly file (see [3], especially the `common.ld` linker script). [1]: https://github.com/rust-lang/rfcs/pull/1201 https://github.com/rust-lang/rfcs/pull/1201 [2]: http://wiki.osdev.org/Bare_Bones http://wiki.osdev.org/Bare_Bones [3]: https://github.com/japaric/cu https://github.com/japaric/cu