5 ms·

For those looking to try their hand at building an OS, there's Linux From Scratch. https://www.linuxfromscratch.org/ https://www.linuxfromscratch.org/ I faile

by 0xEF 3y ago

For those looking to try their hand at building an OS, there's Linux From Scratch. https://www.linuxfromscratch.org/ https://www.linuxfromscratch.org/ I failed tremendously, but I had a lot of incorrect ideas about what makes an operating system go. The learning experience was worth it, though.

quintussss 3y ago
But isn't this just about compiling the system? Not really 'building', ie programming a operating system.
sph 3y ago
That teaches how to compile Linux kernel and software, not to create an OS out of thin air. The go-to guide for anyone that wants to build their own operating system has always been the OSDev community wiki and forum: https://wiki.osdev.org/Main_Page https://wiki.osdev.org/Main_Page
- PhilipRoman 3y ago
  To be fair there is a lot you can do in userland. Graphics protocols, package management, isolation, security etc. And even more can be done by incrementally patching the kernel where needed. It's just that Linux is kind of usable out of the box with just the kernel and /bin/sh. But for other OSes this isn't true, the kernel can expose a vastly different interface than the actual os which users interact with.
  - swozey 3y ago
    I've done every bit of the kernely/cgroupy/bpf/vfio/syscall stuff for a long time and I honestly don't think I could tell you pretty much anything about building an OS. I actually came into this thread thinking, "damn, I have no idea specifically WHAT I would do/not do if I made an OS, or where to even start" not that I could at all. But I mean feature-wise, etc. Like, it's obviously a ton of apis that you call to invoke things at the system level but I have no idea what to do at the system level. Maybe you just communicate with APIs over there for everything on the motherboard. I know Nvidia has NVAPI I've used that a bunch through their SDK, not more low level. And you start with probably a super baby kernel.. There are also different styles of OS, like windows with the registry which I have broken and fixed thousands of times, and then nix which is pretty much file based. I don't know what else would be better than either of these, I hate the registry thing but maybe it has some huge benefit I haven't figured out. When I look at C I see heiroglypics. Maybe me learning rust is helping with that. I do really need to learn it for security sake and (live)patching, etc. I write go and ruby so I don't touch memory much. I'm the same about filesystems. I know a bunch of them and hve done tons of performance testing for SANs etc but couldn't tell you a thing about the lower level, making one, stuff. edit: lol I guess a lot of this is installing drivers too
    - Workaccount2 3y ago
      I have always found it interesting how people who come from the top down (software -> hardware) get confused by the low level stuff, and myself who came from the bottom up (hardware -> software) gets so confused by non-low-level code. I.e. I have no idea how to write a program for windows, it's totally alien to me and I cannot make sense of the code.
      - swozey 3y ago
        Perfect! I've worked with a lot of C kernel security devs and I don't think I've ever seen one of them write something not Cish, like python/ruby/go/etc. So, what exactly happens here - you need to write a kernel in some super fast low level language like C. I THINK The kernel that you write is probably talking to, for instance an Intel/AMD CPU, Motherboard, GPU, etc API to access the hardware. Is this close/wrong? If I made a completely new kernel how do I boot it? Does the kernel have its own API that the BIOS calls? Does the BIOS have anything to do with the kernel but instead something starts Windows and that loads the kernel? I think the kernel mounts/becomes aware of all of the hardware (apis?) potentially by the motherboard telling it what's on it. Does the BIOS just start any C app you point it at and they're just C apps that start the kernel? Maybe it goes through the motherboard for most of it.. Maybe it accesses every piece of hardware directly without an API somehow? This may be stupid questions but hopefully they make sense.. And thanks if you or anyone has input! edit: There's a Boot processing of linux Wiki [1] thats pretty good > After being loaded into RAM, bootloader (also called first-stage bootloader or primary bootloader) will execute to load the second-stage bootloader[2] (also called secondary bootloader).[6] The second-stage bootloader will load the kernel image into memory, decompress and initialize it then pass control to this kernel image.[2] Second-stage bootloader also performs several operation on the system such as system hardware check, mounting the root device, loading the necessary kernel modules,...[2] Finally, the very first user-space process (init process) starts, and other high-level system initializations are performed (which involve with startup scripts).[2] > For each of these stages and components, there are different variations and approaches; for example, GRUB, coreboot or Das U-Boot can be used as bootloaders (historical examples are LILO, SYSLINUX or Loadlin), while the startup scripts can be either traditional init-style, or the system configuration can be performed through modern alternatives such as systemd or Upstart. > The intermediate stage loader (stage1.5, usually core.img) is loaded and executed by the stage1 loader. The second-stage loader (stage2, the /boot/grub/ files) is loaded by the stage1.5 and displays the GRUB startup menu that allows the user to choose an operating system or examine and edit startup parameters. After a menu entry is chosen and optional parameters are given, GRUB loads the linux kernel into memory and passes control to it. GRUB 2 is also capable of chain-loading of another bootloader. In UEFI systems, the stage1 and stage1.5 usually are the same UEFI application file (such as grubx64.efi for x64 UEFI systems). [1] - https://en.wikipedia.org/wiki/Booting_process_of_Linux#:~:text=After%20being%20loaded%20into%20RAM,control%20to%20this%20kernel%20image https://en.wikipedia.org/wiki/Booting_process_of_Linux#:~:te.... [2] - https://www.gnu.org/software/grub/manual/grub/html_node/Images.html https://www.gnu.org/software/grub/manual/grub/html_node/Imag... [3] - Potentially how systemd starts installing the kernel https://github.com/ivandavidov/systemd-boot/blob/master/project/src/kernel-install/kernel-install https://github.com/ivandavidov/systemd-boot/blob/master/proj... [4] - This looks like it's a TPM authentication shim that sits between the EFI and Kernel - https://github.com/ivandavidov/systemd-boot/blob/5521e37e77ac8bd1cff5d8f15d6533c815e7974b/project/src/sd-boot/shim.c#L125 https://github.com/ivandavidov/systemd-boot/blob/5521e37e77a... [5] - Ok this is super helpful, a project on writing a minimal kernel - https://www.codeproject.com/Articles/1225196/Create-Your-Own-Kernel-In-C-2 https://www.codeproject.com/Articles/1225196/Create-Your-Own... [6] - And wow the GNU bootloader spec is rough - https://www.gnu.org/software/grub/manual/multiboot/multiboot.pdf https://www.gnu.org/software/grub/manual/multiboot/multiboot...
- bmer 3y ago
  It seems to me that OSDev.org is unnecessarily married to the idea of "low level == C": https://wiki.osdev.org/Languages https://wiki.osdev.org/Languages Would be interesting if someone with knowledge (e.g. David Chisnall) could help clarify whether this seeming of mine is seemly.
  - throwaway44023 3y ago
    For a long time, that was what everyone used. But there always were tutorials about other languages - C++, Pascal, even C#. Now Rust is gaining traction.
  - kjs3 3y ago
    Since OSDev is community project, so I think that says way more about the community than any strong bias on the part of OSDev. That said, a lot of "low level" is concerned with shoving bits around efficiently, not higher level abstractions useful for writing web browsers and such. C (for all it's issues) has a pretty good impedance match for that use case. The reality is there have always been other languages[1], and not all of them codified the evil stream-of-bytes paradigm. We've been writing OSes in Ada since the 80s. Pascal variants were somewhat common back in the day (e.g. Charles River PERC, CDC Cyber NOS). Apple was on the Pascal train for a long time. Unisys Clearpath is still written (and sold) in their highly extended Algol variant (ESPOL)[2]. I've used OSes written in Modula-2 & Modula-3. There's a number of folks that are trying FreePascal as an OS language. Of course, there was a healthy LispOS population at one brief, shining point in time (sigh...Genera). Tektronix and a few others had Smalltalk machines. IBM still writes a lot of low-level stuff in PL/I dialects across it's proprietary product lines (i Series, z Series). Intel wrote it's first ones (e.g. iRMX, ISIS) in a PL/I dialect (PL/M). And CP/M is there too. Then you can get to the really niche OS writing languages: JOVIAL, anyone? [1] The source code for some of these legacy OSes is out there. Studying it might be instructive. [2] Clearpath is the current incarnation of MCP from the Burroughs Large System mainframes (e.g. B5000, B6500, etc.). Stack based and very interesting; now runs as a VM on top of Xeon servers. Not only can you still buy Clearpath, you can download a hobbyist version to try out locally: https://www.app5.unisys.com/library/gmmail/emails/documents/Learn_CP_OS_2200_MCP_OS/email.htm https://www.app5.unisys.com/library/gmmail/emails/documents/...
abhinavk 3y ago
This is building an OS as in the way people build a PC for themselves. TFA is more about creating an OS.
jwells89 3y ago
Have known about LFS for a long time but never messed with it. Would it be suitable for someone who’s not quite up to the task of writing an OS from scratch, but has higher level ideas (e.g. filesystem arrangement/structure) they want to play with?

by 0xEF 3y ago

quintussss 3y ago

But isn't this just about compiling the system? Not really 'building', ie programming a operating system.

sph 3y ago

That teaches how to compile Linux kernel and software, not to create an OS out of thin air. The go-to guide for anyone that wants to build their own operating system has always been the OSDev community wiki and forum: https://wiki.osdev.org/Main_Page https://wiki.osdev.org/Main_Page

PhilipRoman 3y ago
To be fair there is a lot you can do in userland. Graphics protocols, package management, isolation, security etc. And even more can be done by incrementally patching the kernel where needed. It's just that Linux is kind of usable out of the box with just the kernel and /bin/sh. But for other OSes this isn't true, the kernel can expose a vastly different interface than the actual os which users interact with.
- swozey 3y ago
  I've done every bit of the kernely/cgroupy/bpf/vfio/syscall stuff for a long time and I honestly don't think I could tell you pretty much anything about building an OS. I actually came into this thread thinking, "damn, I have no idea specifically WHAT I would do/not do if I made an OS, or where to even start" not that I could at all. But I mean feature-wise, etc. Like, it's obviously a ton of apis that you call to invoke things at the system level but I have no idea what to do at the system level. Maybe you just communicate with APIs over there for everything on the motherboard. I know Nvidia has NVAPI I've used that a bunch through their SDK, not more low level. And you start with probably a super baby kernel.. There are also different styles of OS, like windows with the registry which I have broken and fixed thousands of times, and then nix which is pretty much file based. I don't know what else would be better than either of these, I hate the registry thing but maybe it has some huge benefit I haven't figured out. When I look at C I see heiroglypics. Maybe me learning rust is helping with that. I do really need to learn it for security sake and (live)patching, etc. I write go and ruby so I don't touch memory much. I'm the same about filesystems. I know a bunch of them and hve done tons of performance testing for SANs etc but couldn't tell you a thing about the lower level, making one, stuff. edit: lol I guess a lot of this is installing drivers too
  - Workaccount2 3y ago
    I have always found it interesting how people who come from the top down (software -> hardware) get confused by the low level stuff, and myself who came from the bottom up (hardware -> software) gets so confused by non-low-level code. I.e. I have no idea how to write a program for windows, it's totally alien to me and I cannot make sense of the code.
    - swozey 3y ago
      Perfect! I've worked with a lot of C kernel security devs and I don't think I've ever seen one of them write something not Cish, like python/ruby/go/etc. So, what exactly happens here - you need to write a kernel in some super fast low level language like C. I THINK The kernel that you write is probably talking to, for instance an Intel/AMD CPU, Motherboard, GPU, etc API to access the hardware. Is this close/wrong? If I made a completely new kernel how do I boot it? Does the kernel have its own API that the BIOS calls? Does the BIOS have anything to do with the kernel but instead something starts Windows and that loads the kernel? I think the kernel mounts/becomes aware of all of the hardware (apis?) potentially by the motherboard telling it what's on it. Does the BIOS just start any C app you point it at and they're just C apps that start the kernel? Maybe it goes through the motherboard for most of it.. Maybe it accesses every piece of hardware directly without an API somehow? This may be stupid questions but hopefully they make sense.. And thanks if you or anyone has input! edit: There's a Boot processing of linux Wiki [1] thats pretty good > After being loaded into RAM, bootloader (also called first-stage bootloader or primary bootloader) will execute to load the second-stage bootloader[2] (also called secondary bootloader).[6] The second-stage bootloader will load the kernel image into memory, decompress and initialize it then pass control to this kernel image.[2] Second-stage bootloader also performs several operation on the system such as system hardware check, mounting the root device, loading the necessary kernel modules,...[2] Finally, the very first user-space process (init process) starts, and other high-level system initializations are performed (which involve with startup scripts).[2] > For each of these stages and components, there are different variations and approaches; for example, GRUB, coreboot or Das U-Boot can be used as bootloaders (historical examples are LILO, SYSLINUX or Loadlin), while the startup scripts can be either traditional init-style, or the system configuration can be performed through modern alternatives such as systemd or Upstart. > The intermediate stage loader (stage1.5, usually core.img) is loaded and executed by the stage1 loader. The second-stage loader (stage2, the /boot/grub/ files) is loaded by the stage1.5 and displays the GRUB startup menu that allows the user to choose an operating system or examine and edit startup parameters. After a menu entry is chosen and optional parameters are given, GRUB loads the linux kernel into memory and passes control to it. GRUB 2 is also capable of chain-loading of another bootloader. In UEFI systems, the stage1 and stage1.5 usually are the same UEFI application file (such as grubx64.efi for x64 UEFI systems). [1] - https://en.wikipedia.org/wiki/Booting_process_of_Linux#:~:text=After%20being%20loaded%20into%20RAM,control%20to%20this%20kernel%20image https://en.wikipedia.org/wiki/Booting_process_of_Linux#:~:te.... [2] - https://www.gnu.org/software/grub/manual/grub/html_node/Images.html https://www.gnu.org/software/grub/manual/grub/html_node/Imag... [3] - Potentially how systemd starts installing the kernel https://github.com/ivandavidov/systemd-boot/blob/master/project/src/kernel-install/kernel-install https://github.com/ivandavidov/systemd-boot/blob/master/proj... [4] - This looks like it's a TPM authentication shim that sits between the EFI and Kernel - https://github.com/ivandavidov/systemd-boot/blob/5521e37e77ac8bd1cff5d8f15d6533c815e7974b/project/src/sd-boot/shim.c#L125 https://github.com/ivandavidov/systemd-boot/blob/5521e37e77a... [5] - Ok this is super helpful, a project on writing a minimal kernel - https://www.codeproject.com/Articles/1225196/Create-Your-Own-Kernel-In-C-2 https://www.codeproject.com/Articles/1225196/Create-Your-Own... [6] - And wow the GNU bootloader spec is rough - https://www.gnu.org/software/grub/manual/multiboot/multiboot.pdf https://www.gnu.org/software/grub/manual/multiboot/multiboot...
bmer 3y ago
It seems to me that OSDev.org is unnecessarily married to the idea of "low level == C": https://wiki.osdev.org/Languages https://wiki.osdev.org/Languages Would be interesting if someone with knowledge (e.g. David Chisnall) could help clarify whether this seeming of mine is seemly.
- throwaway44023 3y ago
  For a long time, that was what everyone used. But there always were tutorials about other languages - C++, Pascal, even C#. Now Rust is gaining traction.
- kjs3 3y ago
  Since OSDev is community project, so I think that says way more about the community than any strong bias on the part of OSDev. That said, a lot of "low level" is concerned with shoving bits around efficiently, not higher level abstractions useful for writing web browsers and such. C (for all it's issues) has a pretty good impedance match for that use case. The reality is there have always been other languages[1], and not all of them codified the evil stream-of-bytes paradigm. We've been writing OSes in Ada since the 80s. Pascal variants were somewhat common back in the day (e.g. Charles River PERC, CDC Cyber NOS). Apple was on the Pascal train for a long time. Unisys Clearpath is still written (and sold) in their highly extended Algol variant (ESPOL)[2]. I've used OSes written in Modula-2 & Modula-3. There's a number of folks that are trying FreePascal as an OS language. Of course, there was a healthy LispOS population at one brief, shining point in time (sigh...Genera). Tektronix and a few others had Smalltalk machines. IBM still writes a lot of low-level stuff in PL/I dialects across it's proprietary product lines (i Series, z Series). Intel wrote it's first ones (e.g. iRMX, ISIS) in a PL/I dialect (PL/M). And CP/M is there too. Then you can get to the really niche OS writing languages: JOVIAL, anyone? [1] The source code for some of these legacy OSes is out there. Studying it might be instructive. [2] Clearpath is the current incarnation of MCP from the Burroughs Large System mainframes (e.g. B5000, B6500, etc.). Stack based and very interesting; now runs as a VM on top of Xeon servers. Not only can you still buy Clearpath, you can download a hobbyist version to try out locally: https://www.app5.unisys.com/library/gmmail/emails/documents/Learn_CP_OS_2200_MCP_OS/email.htm https://www.app5.unisys.com/library/gmmail/emails/documents/...

abhinavk 3y ago

This is building an OS as in the way people build a PC for themselves. TFA is more about creating an OS.

jwells89 3y ago

Have known about LFS for a long time but never messed with it. Would it be suitable for someone who’s not quite up to the task of writing an OS from scratch, but has higher level ideas (e.g. filesystem arrangement/structure) they want to play with?