6 ms·

How to create an OS from scratch

227 points by pykello 1y ago

codazoda 1y ago
The author dismisses this as out of date, but this is one of the most straightforward examples I’ve seen. At least reading 00 and 01.
noone_youknow 1y ago
While I agree this might be a fun resource and useful example code for various aspects of legacy x86 interfacing, I would urge anyone who hopes to actually get into OS development to ignore this (and in fact every other tutorial I’ve ever seen, including those hosted on the popular sites). For all the reasons stated in the link from the README [1] and agreed by the author, this project should not be followed if one wants to gain an understanding of the design and implementation of operating systems for modern systems. Following it will likely lead only to another abandoned “hello world plus shell” that runs only in emulation of decades old hardware. My advice is get the datasheets and programmers’ manuals (which are largely free) and use those to find ways to implement your own ideas. [1] https://github.com/cfenollosa/os-tutorial/issues/269 https://github.com/cfenollosa/os-tutorial/issues/269
- signa11 1y ago
  risc-v seems is a clean-sheet design and that should be a good starting point (imho). fwiw, xv-6, the pedagogical os has migrated to it.
- shetaye 1y ago
  People interested in a "read the manual and code it up on real hardware"-type guide should take a look at Stanford's CS140E[1] repo! Students write a bare metal OS for a Raspberry Pi Zero W (ARMv6) in a series of labs, and we open source each year's code. Disclaimer: I'm on the teaching team [1]https://github.com/dddrrreee/cs140e-25win https://github.com/dddrrreee/cs140e-25win
  - zeroq 1y ago
    not really my cup of tea, but a random feedback: take a look at nand2tetris as it's really "user friendly" and, if I'm not mistaken, was even made into a game on Steam.
    - kragen 1y ago
      It is user friendly, and it's astounding how much they managed to pack into a single semester. Highly recommended! However, it's arguably too idealized and predetermined. I think you could get all the way through building the computer in https://nandgame.com/ https://nandgame.com/ without really learning anything beyond basic logic design as puzzle solving, but computer organization, as they call it in EE classes, is more about designing the puzzles than solving them. Even there, most of what you learn is sort of wrong. I haven't worked through the software part, but it looks like it suffers from the same kind of problems. IIRC there's nothing about virtual memory, filesystems, race conditions, deadly embraces, interprocess communication, scheduling, or security. It's great but it's maybe kind of an appetizer.
      - shetaye 1y ago
        IMO this is kind of the tradeoff. In 140E we do touch on virtual memory (w/ coherency handling on our specific ARM core), FAT32 "from scratch", etc. but it comes at the expense of prerequisites. There is a lot of effort to "minify" the labs to their core lesson, but there is an inevitable amount of complexity that can't (or shouldn't) be erased.
  - firesteelrain 1y ago
    Do you have the course online? Looks like a bunch of files
    - shetaye 1y ago
      There aren't any 1hr+ lectures, just some readings (selected from the manuals in `docs/`) and a bit of exposition from the professor before diving into the lab. Lots of "as needed" assistance
- Gunax 1y ago
  I don't even know where to _begin_ writing an operating system. If i wanted to learn just so i have a concept of what an os does, what would you recommend? I'm not trying to write operating systems per se. I'm trying to become a better developer by understanding operating systems.
  - kragen 1y ago
    Tanenbaum's textbook is highly readable, comprehensive (surveys every major known solution to each major prpblem), and mostly correct. xv6 may be a smaller, more old-fashioned, and more practical approach. RISC-V makes the usually hairy and convoluted issues of paging and virtual memory seem simple. QEMU's GDB server, OpenOCD, JTAG, and SWD can greatly reduce the amount of time you waste wondering why things won't boot. Sigrok/Pulseview may greatly speed up your device driver debugging. But I haven't written an operating system beyond some simple cooperative task-switching code, so take this with a grain of salt.
    - remexre 1y ago
      Funny question since you bring up JTAG and RISC-V -- do you have a cheapish RISC-V device you'd recommend that actually exposes its JTAG? The Milk-V Duo S, Milk-V Jupiter, and Pine64 Oz64 all seem not to expose one; IIRC, the Jupiter even wires TDO as an input (on the other side of a logic level shifter)...
      - kragen 1y ago
        That doesn't seem off-topic at all to me! I don't know what to recommend there. I have no relevant experience, because all my RISC-V hardware leaves unimplemented the privileged ISA, which is the part that RISC-V makes so much simpler. The unprivileged ISA is okay, but it's nothing to write home about, unless you want to implement a CPU instead of an OS.
  - userbinator 1y ago
    Start with a simple program loader and file system, like DOS.
- zeroq 1y ago
  disclaimer: I really have no idea about OSes, but hey... Maybe it's a matter of marketing the product and managing expectations, but many of these projects are A ok being "legacy and obsolete" just for the sake of simplicity for introducing basic concepts. Let's take two random examples. (1) "let's create 3D graphics from scratch" It's quite easy to grab "graphics gems" and create a comprehensive tutorial on software renderer. Sure it won't be practical, and sure it will likely end on phong shading, but for those wanting to understand how 3d models are translated into pixels on screen it's way more approachable than studying papers on nanites. (2) "let's crate a browser from scratch" It's been widely discussed that creating a new browser today would be complete madness (yet there's Ladybird!), but shaving the scope, even if it wouldn't be able to run most modern websites would be a interesting journey for someone who'd interested in how things work. PS. Ages ago I've done a Flash webpage that was supposed to mimic a desktop computer for ad campaign for tv show. Webpage acted as a personal computer of main character and people could lurk into it between episodes to read his emails, check his browser history, etc. I took it as a learning opportunity to learn about OS architecture and spent ungodly amount of unpaid overtime to make it as close to win3.1 running on dos as possible. Was it really an OS? Of course not, but it was a learning opportunity to get a grasp of certain things and it was extremely rewarding to have an easter egg with a command.com you could launch and interact with the system. Would I ever try to build a real OS. Hell no, I'm not as smart as lcamtuf to invite couple friends for drinks and start Argante. :)
  - remexre 1y ago
    To pick on graphics, since I'm more familiar with that domain, the problem isn't that this tutorial is about software rasterization, it's that the tutorial is a raytracer that doesn't do shading, textures, shadows, or any geometry but spheres, and spends most of its word count talking about implementing the trig functions on fixed-point numbers instead of just using math.h functions on IEEE floats.
    - zeroq 1y ago
      great counterpoint :)
    - 1y ago
- sparkie 1y ago
  Obviously you'll need to read the manuals to get much done, but these kinds of tutorials are complimentary. The issue with x86_64 is that you need to understand some of the legacy "warts" to actually use 64-bits. The CPU does not start in "long mode" - you have to gradually enable certain features to get yourself into it. Getting into 32-bit protected mode is prerequisite knowledge to getting into long mode. I recall there was some effort by Intel to resolve some of this friction, breaking backward compatibility, but not sure where that's at. The reason most hobby OS projects die is more to do with drivers. While it's trivial to support VGA and serial ports, for a modern machine we need USB3+, SATA, PCI-E, GPU drivers, WIFI and so forth. The effort for all these drivers dwarfs getting a basic kernel up and running. The few novel operating systems that support more modern hardware tend to utilize Linux drivers by providing a compatible API over a hardware abstraction layer - which forces certain design constraints on the kernel, such as (at least partial) POSIX compatibility.
  - wmf 1y ago
    You can just copy and paste the initialization code. It only runs once and there's very little of value to learn from it (unless you're into retrocomputing).
  - noone_youknow 1y ago
    Even taking only x86_64 as an example, going from real to long modes is primarily of concern to those writing firmware these days - a modern operating system will take over from UEFI or a bootloader (itself usually a UEFI executable). The details of enabling A20, setting up segmentation and the GDT, loading sectors via BIOS etc are of course historically interesting (which is fine if that’s the goal!) but just aren’t that useful today. The primary issue with most tutorials that I’ve seen is they don’t, when completed, leave one in a position of understanding “what’s next” in developing a usable system. Sticking with x86_64, those following will of course have set up a basic GDT, and even a bare-bones TSS, but won’t have much understanding of why they’ve done this or what they’ll need to do to next to support syscall, say, or properly layout interrupt stacks for long mode. By focusing mainly on the minutiae of legacy initialisation (which nobody needs) and racing toward “bang for my buck” interactive features, the tutorials tend to leave those completing it with a patchy, outdated understanding of the basics and a simple baremetal program that is in no way architected as a good base upon which to continue toward building a usable OS kernel.
andsoitis 1y ago
Also see https://wiki.osdev.org/Getting_Started https://wiki.osdev.org/Getting_Started
owenpalmer 1y ago
from the readme: > College is hard so I don't remember most of it. Interesting how counter-productive high stress environments can be for ingraining knowledge.
markus_zhang 1y ago
On a related topic - has anyone tried to move the build process of a very old Linux version, say Linux 0.92/0.12 to modern toolchains on modern computers? I believe the original build process requires gcc 1.4.0 and other programs such as `as` which are not available on modern systems. The target is to be able to build Linux 0.12 using modern gcc and run it in QEMU, or preferable on a true 80386 machine. AFAIK, modern gcc still supports this architecture, so in concept this should be possible. There might be a LOT of changes in source code to be made, though. The idea behind is to obtain a very old Linux that can be built easily on modern systems, with modern C support, and gradually add my own stuffs using the same tools, so that I don't limit myself to very old toolchains. Edit: The reason to pick Linux 0.12/0.92 is because 1) It is kinda complete (without network I believe) but not too complex (I believe the loc is under 20K), and 2) It is not a toy OS and can run on real hardware (a 80386 machine), and 3) we have an excellent book for it: https://download.oldlinux.org/CLK-5.0-WithCover.pdf https://download.oldlinux.org/CLK-5.0-WithCover.pdf
- tscherno 1y ago
  "NCommander" on YouTube could probably be a good starting point to get a feel for what is required and to ask some questions. Features very interesting/insane archaeology builds like bootstrapping Linux from Scratch on WinXP SFU which ships with an incomplete toolchain from 1993 (?). 8)
- khrbtxyz 1y ago
  https://github.com/mariuz/linux-0.01 https://github.com/mariuz/linux-0.01 might give you some ideas.
- dxroshan 1y ago
  `as` the assembler is still available.
- siddbudd 1y ago
  in English: https://download.oldlinux.org/ECLK-5.0-WithCover.pdf https://download.oldlinux.org/ECLK-5.0-WithCover.pdf
EFreethought 1y ago
If you wish to create an OS from scratch, you must first invent the universe.
- allanbjorklund 1y ago
  Nice paraphrase of Sagan. :-)
notorandit 1y ago
x86 only, unfortunately. But some parts can be borrowed for other ISAs.
ljsprague 1y ago
I still don't understand what's "running" the boot sector.
- bandrami 1y ago
  That's why the word "boot" is there
  - a96 1y ago
    https://en.wikipedia.org/wiki/Bootstrapping https://en.wikipedia.org/wiki/Bootstrapping in case someone reading this doesn't know.
    - torben-friis 1y ago
      Uh, TIL. I knew the bootstrapping expression and origins, but as a second language speaker it had never occurred to me that “boot” as verb was related.
    - fuzztester 1y ago
      Partly related: The process of bootstrapping a compiler (not an operating system) is really interesting and the method used is ingenious. I had read about it some years back. Basically, at high level, it's a kind of chicken and egg situation: After writing a compiler for language A (in language A), where A is a new language, how can you compile that compiler to an executable, so that you can compile application programs written in A? Because there is not yet any runnable compiler for A. I might not have described the issue very well. And the concept of cross-compiling also comes into the picture, depending on the situation. I don't remember the details perfectly now. If somebody else who knows, describes it, I think it would be interesting for many people here. Some links: https://en.m.wikipedia.org/wiki/Bootstrapping_(compilers) https://en.m.wikipedia.org/wiki/Bootstrapping_(compilers) https://en.m.wikipedia.org/wiki/Cross_compiler https://en.m.wikipedia.org/wiki/Cross_compiler
      - smnplk 1y ago
notorandit 1y ago
And ... > Hey! This is an old, abandoned project, with both technical and design issues. Please have fun with this tutorial but do look for more modern and authoritative sources if you want to learn about OS design.
- sim7c00 1y ago
  and - i made similar comment with a bit more elaboration. it's important to realise this is how to make an OS from scratch in the 90s without any idea of what the OS is going to do or what hardware you want to run it on, etc. etc. - there are many things missing which will surely bite you, hard, if you want to roll an actual usable OS. it's a lot of fun to do these things, but it's good not to be convinced it's actually how modern OSes work, or modern hardware for that matter.
tombert 1y ago
I have noticed that every distributed app I build, as it gets more concurrent, I end up reinventing a lot of operating system work. I end up rebuilding scheduling, different caching techniques, heuristics to try and preemptively add things to cache, etc. I really should learn more about kernel design. I would probably still be a better distributed systems engineer if I lift more concepts from OS design.
- alexisread 1y ago
  ...and a little further on, the OS/app starts to look like a distributed database. I think it's a good case for including lattice types in the OS ie. From the ground up. Bear in mind that an OS has an API ie. A DSL/language for it, Micropython is a good example: https://www.neilconway.org/docs/socc2012_bloom_lattices.pdf https://www.neilconway.org/docs/socc2012_bloom_lattices.pdf
senko 1y ago
Creating an OS is fun. It's the drivers and hardware support in general that get you. Thankless grind without which you get nowhere.
sim7c00 1y ago
First off, this _is_ a nice tutorial for what it is. It goes a little further than a lot of similar ones and its easy to follow along. Wanna say that clearly before: I’ve always found it curious that most OS dev tutorials still focus on x86_32, even though nearly all CPUs today are 64-bit. It’s probably because older materials are easier to follow, but starting with x86_64 is more relevant and often cleaner. Yes, 64-bit requires paging from the start, which adds complexity early on. But you can skip the whole 32-bit protected mode setup and jump straight from real mode to long mode, or better yet, use UEFI. UEFI lets you boot directly into a 64-bit context, and it's much easier to work with than BIOS once you get the hang of it. You’ll learn more by writing your own boot code than copying legacy snippets. UEFI support is straightforward, and you can still support BIOS if needed (most old machines are x64 anyway? it's been around for ages now...). Since the ESP partition is FAT32, you can place it early on disk and still leave room for a legacy bootsector if you want dual support. You can even run different payloads depending on the boot method. EDK2 has a learning curve, but once you understand how to resolve protocols and call services, it’s a huge upgrade. Most of it can be written in plain C. I only use a small inline asm trampoline to set up the stack and jump to stage 2. Also, skip legacy stuff like PIC/PIT. They’re emulated now. Use LAPIC for interrupts and timers, and look into MSI/MSI-X for modern interrupt routing. One thing I often see missing in older tutorials is a partition table in the bootsector. Without it, AHCI in QEMU won’t detect your disk atleast on some versions, and this again shows how crumbly and differently implemented some things can be (the ahci nor sata specs require this, so it's a tricky one if it hits you :D...). It’s easy to add, and makes sense if you want multiple partitions. UEFI helps here too—you can build a disk image with real partitions (e.g., FAT32 for boot, ext2/4 for root) and implement them properly. If you don't take into account your system will be using partitions and filesystems within those partitions it's gonna be a lot of re-writing. Structure of the repo also matters. A clean layout makes it easier to grow your OS without it turning into a mess. This project seems to suggest / imply a nice structure from what I can tell, despite it's also ofcourse modelled around the tutorial itself. - thinking of architecture independence is also often forgotten. Especially when working in newer langauges like Rust that might be apealing ,but most C/C++ code can also be easily made portable if you put foresight into your OS running on different platforms. QEMU can emulate a lot of them for you to test things on. TL;DR: Most tutorials teach you how to build an OS for 1990s hardware. Fun, but not very useful today. If you want to learn how modern OSes work, start with modern protocols and hardware. Some are more complex, but many are easier to use and better documented which can actually speed up development and reduce the need to port higher level systems over to newer low level code once you decide you want it to run on something less than 20 years old. (Athlon64 was 2003!)
exDM69 1y ago
Having built a few bare metal and hobby OS projects over the years, I would not recommend the path taken in this tutorial series. Because if you want to write an OS, don't write a bootloader first. This article essentially describes a stage 1 bootloader on top of legacy BIOS firmware. It will teach you about historical x86 minutiae, which is nothing but a hindrance if you want to understand OS concepts. Instead you should try to get a bootable ELF (multiboot) or PE (UEFI) image written in a high level language (C, C++, Rust, Zig, etc) as soon as possible. You can boot it up in QEMU very easily (compared to a boot sector image) and get a real debugger (gdb), system monitor and all the other invaluable tooling up. This will greatly affect the velocity of your project and get to the interesting stuff faster. Because bare metal/OS projects may be hard to write but they are even harder to debug. You will need all the help you can get.
- hmry 1y ago
  I think you're totally right from a practical view. Trying to debug 16-bit x86 code is a nightmare, none of the debuggers properly support it. Leaving practicality aside and focusing on aesthetics... Normally, for hobby wheel reimplementation projects like this, I find doing it as close to the bare metal as possible, relying on the minimum amount of other people's code, a lot more fun. But AFAIK, these days legacy BIOS boot is just some emulated compatibility mode running under UEFI anyway. The bootloader already ran and configured the hardware for you, and then it un-configured a bunch of stuff so you can re-do it. It's role-playing as an 80s PC for you. I find that deeply unsatisfying. UEFI is the bare-metal API, for all intents and purposes. (Unless you want to go completely blobless, writing your own firmware.)
  - exDM69 1y ago
    > hobby wheel reimplementation projects like this, I find doing it as close to the bare metal as possible, relying on the minimum amount of other people's code, a lot more fun. As a chronic wheel reinventor, I can understand this. But if I felt like scratching this itch now, I would pick some other hardware than x86. The RP2040 could be a nice target, or maybe some ARM or RISC-V SoC. That said, I totally understand that there's something different to having a bare metal / hobby OS project running on your daily driver computer than some embedded gadget. I'm speaking from experience here, I've written bare metal projects in the style of this project (first project I did with MS-DOS debug.com, wrote to a floppy disk and rebooted my machine from the floppy) and the "modern" way with compilers, emulators, debuggers etc. The difference in productivity and learning the interesting stuff is huge.
KingLancelot 1y ago
[dead]
rkagerer 1y ago
The crucial external document that lesson #3 references (and is all about) is now a dead link. Here's an archive - page 14 is where they point you: https://web.archive.org/web/20241112015613if_/https://www.cs.bham.ac.uk/~exr/lectures/opsys/10_11/lectures/os-dev.pdf https://web.archive.org/web/20241112015613if_/https://www.cs... But here's a more succinct explanation of the reason for the memory offset: https://stackoverflow.com/a/51996005 https://stackoverflow.com/a/51996005
p0w3n3d 1y ago
I wonder - does the GUI really belong to the OS? In all the examples I can give, the GUI is an application that runs on top of kernel and hosts the graphics calls. However all the systems I know have the non-gui applications which can run before. So I'd say that GUI is not a part of the OS... Please tell if you agree or not
- mmphosis 1y ago
  I'd say it depends on the purpose of the OS. The original Mac had graphics routines in ROM. It would be cool to have instant on devices.
- shetaye 1y ago
  Arguably the OS includes the entire distribution, not just the kernel. MacOS ships the window server and the entire graphics stack, so the GUI is certainly part of MacOS, and so at least some OSes include a GUI.
deater 1y ago
as someone who has written my own OS from scratch (vmwOS) and teach a class on it, I have to agree with a lot of the other comments that x86-based OS projects do end up being exercises in 40-year old PC/x86 retrocomputing. A few years ago I would have recommended the path I took (writing an OS for the Raspberry Pi) but the Pis have gone off the rails recently. So writing a simple OS for a Pi-1B+ is relatively doable (simple enough, sort of OK documentation, biggest downside is needing USB for the keyboard). Things led to disaster once everyone wanted to use Pi4 (which was all we could manage to source during the CPU shortage of '23) as the documentation is poor, getting interrupts going became nearly impossible, and the virtual memory/cache/etc setup on the 64-bit cores (at least a few years ago) was not documented well at all.
- shetaye 1y ago
  If you are still interested in SMP on a 64-bit ARM, we have had some success with virtual memory/cache/peripherals on the Pi Zero 2 W
forbiddenvoid 1y ago
Every time one of these pops up on HN, it's always an abandoned, unfinished project. Why do these OS projects never get completed?
- tlb 1y ago
  Unlike Linux, MacOS, and Windows, which are all completely finished. But also: most of the work of an OS is supporting a variety of hardware. That's not very intellectually interesting work. Hardware is usually hacky, and since was only ever tested against the manufacturer's driver code, the only way to use it reliably is to slavishly follow their usage.

227 points by pykello 1y ago

The author dismisses this as out of date, but this is one of the most straightforward examples I’ve seen. At least reading 00 and 01.

noone_youknow 1y ago

While I agree this might be a fun resource and useful example code for various aspects of legacy x86 interfacing, I would urge anyone who hopes to actually get into OS development to ignore this (and in fact every other tutorial I’ve ever seen, including those hosted on the popular sites). For all the reasons stated in the link from the README [1] and agreed by the author, this project should not be followed if one wants to gain an understanding of the design and implementation of operating systems for modern systems. Following it will likely lead only to another abandoned “hello world plus shell” that runs only in emulation of decades old hardware. My advice is get the datasheets and programmers’ manuals (which are largely free) and use those to find ways to implement your own ideas. [1] https://github.com/cfenollosa/os-tutorial/issues/269 https://github.com/cfenollosa/os-tutorial/issues/269

signa11 1y ago
risc-v seems is a clean-sheet design and that should be a good starting point (imho). fwiw, xv-6, the pedagogical os has migrated to it.
shetaye 1y ago
People interested in a "read the manual and code it up on real hardware"-type guide should take a look at Stanford's CS140E[1] repo! Students write a bare metal OS for a Raspberry Pi Zero W (ARMv6) in a series of labs, and we open source each year's code. Disclaimer: I'm on the teaching team [1]https://github.com/dddrrreee/cs140e-25win https://github.com/dddrrreee/cs140e-25win
- zeroq 1y ago
  not really my cup of tea, but a random feedback: take a look at nand2tetris as it's really "user friendly" and, if I'm not mistaken, was even made into a game on Steam.
  - kragen 1y ago
    It is user friendly, and it's astounding how much they managed to pack into a single semester. Highly recommended! However, it's arguably too idealized and predetermined. I think you could get all the way through building the computer in https://nandgame.com/ https://nandgame.com/ without really learning anything beyond basic logic design as puzzle solving, but computer organization, as they call it in EE classes, is more about designing the puzzles than solving them. Even there, most of what you learn is sort of wrong. I haven't worked through the software part, but it looks like it suffers from the same kind of problems. IIRC there's nothing about virtual memory, filesystems, race conditions, deadly embraces, interprocess communication, scheduling, or security. It's great but it's maybe kind of an appetizer.
    - shetaye 1y ago
      IMO this is kind of the tradeoff. In 140E we do touch on virtual memory (w/ coherency handling on our specific ARM core), FAT32 "from scratch", etc. but it comes at the expense of prerequisites. There is a lot of effort to "minify" the labs to their core lesson, but there is an inevitable amount of complexity that can't (or shouldn't) be erased.
- firesteelrain 1y ago
  Do you have the course online? Looks like a bunch of files
  - shetaye 1y ago
    There aren't any 1hr+ lectures, just some readings (selected from the manuals in `docs/`) and a bit of exposition from the professor before diving into the lab. Lots of "as needed" assistance
Gunax 1y ago
I don't even know where to _begin_ writing an operating system. If i wanted to learn just so i have a concept of what an os does, what would you recommend? I'm not trying to write operating systems per se. I'm trying to become a better developer by understanding operating systems.
- kragen 1y ago
  Tanenbaum's textbook is highly readable, comprehensive (surveys every major known solution to each major prpblem), and mostly correct. xv6 may be a smaller, more old-fashioned, and more practical approach. RISC-V makes the usually hairy and convoluted issues of paging and virtual memory seem simple. QEMU's GDB server, OpenOCD, JTAG, and SWD can greatly reduce the amount of time you waste wondering why things won't boot. Sigrok/Pulseview may greatly speed up your device driver debugging. But I haven't written an operating system beyond some simple cooperative task-switching code, so take this with a grain of salt.
  - remexre 1y ago
    Funny question since you bring up JTAG and RISC-V -- do you have a cheapish RISC-V device you'd recommend that actually exposes its JTAG? The Milk-V Duo S, Milk-V Jupiter, and Pine64 Oz64 all seem not to expose one; IIRC, the Jupiter even wires TDO as an input (on the other side of a logic level shifter)...
    - kragen 1y ago
      That doesn't seem off-topic at all to me! I don't know what to recommend there. I have no relevant experience, because all my RISC-V hardware leaves unimplemented the privileged ISA, which is the part that RISC-V makes so much simpler. The unprivileged ISA is okay, but it's nothing to write home about, unless you want to implement a CPU instead of an OS.
- userbinator 1y ago
  Start with a simple program loader and file system, like DOS.
zeroq 1y ago
disclaimer: I really have no idea about OSes, but hey... Maybe it's a matter of marketing the product and managing expectations, but many of these projects are A ok being "legacy and obsolete" just for the sake of simplicity for introducing basic concepts. Let's take two random examples. (1) "let's create 3D graphics from scratch" It's quite easy to grab "graphics gems" and create a comprehensive tutorial on software renderer. Sure it won't be practical, and sure it will likely end on phong shading, but for those wanting to understand how 3d models are translated into pixels on screen it's way more approachable than studying papers on nanites. (2) "let's crate a browser from scratch" It's been widely discussed that creating a new browser today would be complete madness (yet there's Ladybird!), but shaving the scope, even if it wouldn't be able to run most modern websites would be a interesting journey for someone who'd interested in how things work. PS. Ages ago I've done a Flash webpage that was supposed to mimic a desktop computer for ad campaign for tv show. Webpage acted as a personal computer of main character and people could lurk into it between episodes to read his emails, check his browser history, etc. I took it as a learning opportunity to learn about OS architecture and spent ungodly amount of unpaid overtime to make it as close to win3.1 running on dos as possible. Was it really an OS? Of course not, but it was a learning opportunity to get a grasp of certain things and it was extremely rewarding to have an easter egg with a command.com you could launch and interact with the system. Would I ever try to build a real OS. Hell no, I'm not as smart as lcamtuf to invite couple friends for drinks and start Argante. :)
- remexre 1y ago
  To pick on graphics, since I'm more familiar with that domain, the problem isn't that this tutorial is about software rasterization, it's that the tutorial is a raytracer that doesn't do shading, textures, shadows, or any geometry but spheres, and spends most of its word count talking about implementing the trig functions on fixed-point numbers instead of just using math.h functions on IEEE floats.
  - zeroq 1y ago
    great counterpoint :)
  - 1y ago
sparkie 1y ago
Obviously you'll need to read the manuals to get much done, but these kinds of tutorials are complimentary. The issue with x86_64 is that you need to understand some of the legacy "warts" to actually use 64-bits. The CPU does not start in "long mode" - you have to gradually enable certain features to get yourself into it. Getting into 32-bit protected mode is prerequisite knowledge to getting into long mode. I recall there was some effort by Intel to resolve some of this friction, breaking backward compatibility, but not sure where that's at. The reason most hobby OS projects die is more to do with drivers. While it's trivial to support VGA and serial ports, for a modern machine we need USB3+, SATA, PCI-E, GPU drivers, WIFI and so forth. The effort for all these drivers dwarfs getting a basic kernel up and running. The few novel operating systems that support more modern hardware tend to utilize Linux drivers by providing a compatible API over a hardware abstraction layer - which forces certain design constraints on the kernel, such as (at least partial) POSIX compatibility.
- wmf 1y ago
  You can just copy and paste the initialization code. It only runs once and there's very little of value to learn from it (unless you're into retrocomputing).
- noone_youknow 1y ago
  Even taking only x86_64 as an example, going from real to long modes is primarily of concern to those writing firmware these days - a modern operating system will take over from UEFI or a bootloader (itself usually a UEFI executable). The details of enabling A20, setting up segmentation and the GDT, loading sectors via BIOS etc are of course historically interesting (which is fine if that’s the goal!) but just aren’t that useful today. The primary issue with most tutorials that I’ve seen is they don’t, when completed, leave one in a position of understanding “what’s next” in developing a usable system. Sticking with x86_64, those following will of course have set up a basic GDT, and even a bare-bones TSS, but won’t have much understanding of why they’ve done this or what they’ll need to do to next to support syscall, say, or properly layout interrupt stacks for long mode. By focusing mainly on the minutiae of legacy initialisation (which nobody needs) and racing toward “bang for my buck” interactive features, the tutorials tend to leave those completing it with a patchy, outdated understanding of the basics and a simple baremetal program that is in no way architected as a good base upon which to continue toward building a usable OS kernel.

andsoitis 1y ago

Also see https://wiki.osdev.org/Getting_Started https://wiki.osdev.org/Getting_Started

owenpalmer 1y ago

from the readme: > College is hard so I don't remember most of it. Interesting how counter-productive high stress environments can be for ingraining knowledge.

markus_zhang 1y ago

On a related topic - has anyone tried to move the build process of a very old Linux version, say Linux 0.92/0.12 to modern toolchains on modern computers? I believe the original build process requires gcc 1.4.0 and other programs such as `as` which are not available on modern systems. The target is to be able to build Linux 0.12 using modern gcc and run it in QEMU, or preferable on a true 80386 machine. AFAIK, modern gcc still supports this architecture, so in concept this should be possible. There might be a LOT of changes in source code to be made, though. The idea behind is to obtain a very old Linux that can be built easily on modern systems, with modern C support, and gradually add my own stuffs using the same tools, so that I don't limit myself to very old toolchains. Edit: The reason to pick Linux 0.12/0.92 is because 1) It is kinda complete (without network I believe) but not too complex (I believe the loc is under 20K), and 2) It is not a toy OS and can run on real hardware (a 80386 machine), and 3) we have an excellent book for it: https://download.oldlinux.org/CLK-5.0-WithCover.pdf https://download.oldlinux.org/CLK-5.0-WithCover.pdf

tscherno 1y ago
"NCommander" on YouTube could probably be a good starting point to get a feel for what is required and to ask some questions. Features very interesting/insane archaeology builds like bootstrapping Linux from Scratch on WinXP SFU which ships with an incomplete toolchain from 1993 (?). 8)
khrbtxyz 1y ago
https://github.com/mariuz/linux-0.01 https://github.com/mariuz/linux-0.01 might give you some ideas.
dxroshan 1y ago
`as` the assembler is still available.
siddbudd 1y ago
in English: https://download.oldlinux.org/ECLK-5.0-WithCover.pdf https://download.oldlinux.org/ECLK-5.0-WithCover.pdf

EFreethought 1y ago

If you wish to create an OS from scratch, you must first invent the universe.

allanbjorklund 1y ago
Nice paraphrase of Sagan. :-)

notorandit 1y ago

x86 only, unfortunately. But some parts can be borrowed for other ISAs.

ljsprague 1y ago

I still don't understand what's "running" the boot sector.

bandrami 1y ago
That's why the word "boot" is there
- a96 1y ago
  https://en.wikipedia.org/wiki/Bootstrapping https://en.wikipedia.org/wiki/Bootstrapping in case someone reading this doesn't know.
  - torben-friis 1y ago
    Uh, TIL. I knew the bootstrapping expression and origins, but as a second language speaker it had never occurred to me that “boot” as verb was related.
  - fuzztester 1y ago
    Partly related: The process of bootstrapping a compiler (not an operating system) is really interesting and the method used is ingenious. I had read about it some years back. Basically, at high level, it's a kind of chicken and egg situation: After writing a compiler for language A (in language A), where A is a new language, how can you compile that compiler to an executable, so that you can compile application programs written in A? Because there is not yet any runnable compiler for A. I might not have described the issue very well. And the concept of cross-compiling also comes into the picture, depending on the situation. I don't remember the details perfectly now. If somebody else who knows, describes it, I think it would be interesting for many people here. Some links: https://en.m.wikipedia.org/wiki/Bootstrapping_(compilers) https://en.m.wikipedia.org/wiki/Bootstrapping_(compilers) https://en.m.wikipedia.org/wiki/Cross_compiler https://en.m.wikipedia.org/wiki/Cross_compiler
    - smnplk 1y ago

notorandit 1y ago

And ... > Hey! This is an old, abandoned project, with both technical and design issues. Please have fun with this tutorial but do look for more modern and authoritative sources if you want to learn about OS design.

sim7c00 1y ago
and - i made similar comment with a bit more elaboration. it's important to realise this is how to make an OS from scratch in the 90s without any idea of what the OS is going to do or what hardware you want to run it on, etc. etc. - there are many things missing which will surely bite you, hard, if you want to roll an actual usable OS. it's a lot of fun to do these things, but it's good not to be convinced it's actually how modern OSes work, or modern hardware for that matter.

tombert 1y ago

I have noticed that every distributed app I build, as it gets more concurrent, I end up reinventing a lot of operating system work. I end up rebuilding scheduling, different caching techniques, heuristics to try and preemptively add things to cache, etc. I really should learn more about kernel design. I would probably still be a better distributed systems engineer if I lift more concepts from OS design.

alexisread 1y ago
...and a little further on, the OS/app starts to look like a distributed database. I think it's a good case for including lattice types in the OS ie. From the ground up. Bear in mind that an OS has an API ie. A DSL/language for it, Micropython is a good example: https://www.neilconway.org/docs/socc2012_bloom_lattices.pdf https://www.neilconway.org/docs/socc2012_bloom_lattices.pdf

senko 1y ago

Creating an OS is fun. It's the drivers and hardware support in general that get you. Thankless grind without which you get nowhere.

sim7c00 1y ago

First off, this _is_ a nice tutorial for what it is. It goes a little further than a lot of similar ones and its easy to follow along. Wanna say that clearly before: I’ve always found it curious that most OS dev tutorials still focus on x86_32, even though nearly all CPUs today are 64-bit. It’s probably because older materials are easier to follow, but starting with x86_64 is more relevant and often cleaner. Yes, 64-bit requires paging from the start, which adds complexity early on. But you can skip the whole 32-bit protected mode setup and jump straight from real mode to long mode, or better yet, use UEFI. UEFI lets you boot directly into a 64-bit context, and it's much easier to work with than BIOS once you get the hang of it. You’ll learn more by writing your own boot code than copying legacy snippets. UEFI support is straightforward, and you can still support BIOS if needed (most old machines are x64 anyway? it's been around for ages now...). Since the ESP partition is FAT32, you can place it early on disk and still leave room for a legacy bootsector if you want dual support. You can even run different payloads depending on the boot method. EDK2 has a learning curve, but once you understand how to resolve protocols and call services, it’s a huge upgrade. Most of it can be written in plain C. I only use a small inline asm trampoline to set up the stack and jump to stage 2. Also, skip legacy stuff like PIC/PIT. They’re emulated now. Use LAPIC for interrupts and timers, and look into MSI/MSI-X for modern interrupt routing. One thing I often see missing in older tutorials is a partition table in the bootsector. Without it, AHCI in QEMU won’t detect your disk atleast on some versions, and this again shows how crumbly and differently implemented some things can be (the ahci nor sata specs require this, so it's a tricky one if it hits you :D...). It’s easy to add, and makes sense if you want multiple partitions. UEFI helps here too—you can build a disk image with real partitions (e.g., FAT32 for boot, ext2/4 for root) and implement them properly. If you don't take into account your system will be using partitions and filesystems within those partitions it's gonna be a lot of re-writing. Structure of the repo also matters. A clean layout makes it easier to grow your OS without it turning into a mess. This project seems to suggest / imply a nice structure from what I can tell, despite it's also ofcourse modelled around the tutorial itself. - thinking of architecture independence is also often forgotten. Especially when working in newer langauges like Rust that might be apealing ,but most C/C++ code can also be easily made portable if you put foresight into your OS running on different platforms. QEMU can emulate a lot of them for you to test things on. TL;DR: Most tutorials teach you how to build an OS for 1990s hardware. Fun, but not very useful today. If you want to learn how modern OSes work, start with modern protocols and hardware. Some are more complex, but many are easier to use and better documented which can actually speed up development and reduce the need to port higher level systems over to newer low level code once you decide you want it to run on something less than 20 years old. (Athlon64 was 2003!)

exDM69 1y ago

Having built a few bare metal and hobby OS projects over the years, I would not recommend the path taken in this tutorial series. Because if you want to write an OS, don't write a bootloader first. This article essentially describes a stage 1 bootloader on top of legacy BIOS firmware. It will teach you about historical x86 minutiae, which is nothing but a hindrance if you want to understand OS concepts. Instead you should try to get a bootable ELF (multiboot) or PE (UEFI) image written in a high level language (C, C++, Rust, Zig, etc) as soon as possible. You can boot it up in QEMU very easily (compared to a boot sector image) and get a real debugger (gdb), system monitor and all the other invaluable tooling up. This will greatly affect the velocity of your project and get to the interesting stuff faster. Because bare metal/OS projects may be hard to write but they are even harder to debug. You will need all the help you can get.

hmry 1y ago
I think you're totally right from a practical view. Trying to debug 16-bit x86 code is a nightmare, none of the debuggers properly support it. Leaving practicality aside and focusing on aesthetics... Normally, for hobby wheel reimplementation projects like this, I find doing it as close to the bare metal as possible, relying on the minimum amount of other people's code, a lot more fun. But AFAIK, these days legacy BIOS boot is just some emulated compatibility mode running under UEFI anyway. The bootloader already ran and configured the hardware for you, and then it un-configured a bunch of stuff so you can re-do it. It's role-playing as an 80s PC for you. I find that deeply unsatisfying. UEFI is the bare-metal API, for all intents and purposes. (Unless you want to go completely blobless, writing your own firmware.)
- exDM69 1y ago
  > hobby wheel reimplementation projects like this, I find doing it as close to the bare metal as possible, relying on the minimum amount of other people's code, a lot more fun. As a chronic wheel reinventor, I can understand this. But if I felt like scratching this itch now, I would pick some other hardware than x86. The RP2040 could be a nice target, or maybe some ARM or RISC-V SoC. That said, I totally understand that there's something different to having a bare metal / hobby OS project running on your daily driver computer than some embedded gadget. I'm speaking from experience here, I've written bare metal projects in the style of this project (first project I did with MS-DOS debug.com, wrote to a floppy disk and rebooted my machine from the floppy) and the "modern" way with compilers, emulators, debuggers etc. The difference in productivity and learning the interesting stuff is huge.

KingLancelot 1y ago

[dead]

rkagerer 1y ago

The crucial external document that lesson #3 references (and is all about) is now a dead link. Here's an archive - page 14 is where they point you: https://web.archive.org/web/20241112015613if_/https://www.cs.bham.ac.uk/~exr/lectures/opsys/10_11/lectures/os-dev.pdf https://web.archive.org/web/20241112015613if_/https://www.cs... But here's a more succinct explanation of the reason for the memory offset: https://stackoverflow.com/a/51996005 https://stackoverflow.com/a/51996005

p0w3n3d 1y ago

I wonder - does the GUI really belong to the OS? In all the examples I can give, the GUI is an application that runs on top of kernel and hosts the graphics calls. However all the systems I know have the non-gui applications which can run before. So I'd say that GUI is not a part of the OS... Please tell if you agree or not

mmphosis 1y ago
I'd say it depends on the purpose of the OS. The original Mac had graphics routines in ROM. It would be cool to have instant on devices.
shetaye 1y ago
Arguably the OS includes the entire distribution, not just the kernel. MacOS ships the window server and the entire graphics stack, so the GUI is certainly part of MacOS, and so at least some OSes include a GUI.

deater 1y ago

as someone who has written my own OS from scratch (vmwOS) and teach a class on it, I have to agree with a lot of the other comments that x86-based OS projects do end up being exercises in 40-year old PC/x86 retrocomputing. A few years ago I would have recommended the path I took (writing an OS for the Raspberry Pi) but the Pis have gone off the rails recently. So writing a simple OS for a Pi-1B+ is relatively doable (simple enough, sort of OK documentation, biggest downside is needing USB for the keyboard). Things led to disaster once everyone wanted to use Pi4 (which was all we could manage to source during the CPU shortage of '23) as the documentation is poor, getting interrupts going became nearly impossible, and the virtual memory/cache/etc setup on the 64-bit cores (at least a few years ago) was not documented well at all.

shetaye 1y ago
If you are still interested in SMP on a 64-bit ARM, we have had some success with virtual memory/cache/peripherals on the Pi Zero 2 W

forbiddenvoid 1y ago

Every time one of these pops up on HN, it's always an abandoned, unfinished project. Why do these OS projects never get completed?

tlb 1y ago
Unlike Linux, MacOS, and Windows, which are all completely finished. But also: most of the work of an OS is supporting a variety of hardware. That's not very intellectually interesting work. Hardware is usually hacky, and since was only ever tested against the manufacturer's driver code, the only way to use it reliably is to slavishly follow their usage.