10 ms·

Operating Systems: From 0 to 1

709 points by tuhdo 10y ago

kriro 10y ago
Seems to be for a x86 operating system. I'd have preferred some other architecture because so much of OSdev for x86 that I remember was working around quirks of the architecture (A20 gate etc.). I guess it's a valuable lesson but I'd really enjoy a fork of the book for the hardware you find in a Beagle Bone or Pi3 or something. Maybe this could be crowdfunded if the x86 version is popular?
- tuhdo 10y ago
  Author here. Yes, it's a x86 operating system. However, rather than getting around A20, it focused on protected mode instead. The book not only teaches x86, but how to use the official resources from the hardware manufacturer to write the OS. In sum, a reader when reaching part 3 for writing the OS, he will need to use the official document, in this case, the "System Programming Guide" manual from Intel to write C code that complies with the documents. Once he learned how to do so, learning other platforms will be much easier given how complex x86 is.
  - wolfgke 10y ago
    > Author here. Yes, it's a x86 operating system. However, rather than getting around A20, it focused on protected mode instead. You still have to open the A20 gate in the bootloader if you want to access a memory adress that has bit 20 (counting from 0) be set to 1 (you probably want) - even if you switch to protected mode. The only exception is if you boot from UEFI instead of BIOS - in this case the A20 gate is already set. But the book uses BIOS as far as I see it.
    - tuhdo 10y ago
      In the Volume 3, it is said that: "In protected mode, the IA-32 architecture provides a normal physical address space of 4 GBytes (2 32 bytes). This is the address space that the processor can address on its address bus. This address space is flat (unsegmented), with addresses ranging continuously from 0 to FFFFFFFFH. This physical address space can be mapped to read- write memory, read-only memory, and memory mapped I/O. The memory mapping facilities described in this chapter can be used to divide this physical memory up into segments and/or pages." It correlates to my experience of developing in protected mode in QEMU. Once entered protected mode, I can access to any address above 0x10000 without being wrapped around. When I was writing my first kernel (https://github.com/tuhdo/os-study https://github.com/tuhdo/os-study) in real-mode, indeed A20 must be enabled.
      - Vendan 10y ago
        The A20 gate would affect both real and protected mode, so to be truely compliant, you should be opening it even for a protected mode OS. It was originally a purely hardware hack forcing an address line(A20, or the 21st address line) to stay low, thus "wrapping" addresses from 1mb-2mb to 0-1mb, so it would have affected real or protected mode identically. On modern hardware, there is a chance that A20 is (against the accepted spec) default open, or possible not implemented.
        wolfgke 10y ago
        > On modern hardware, there is a chance that A20 is (against the accepted spec) default open, or possible not implemented. To nitpick on this a little bit (consider it as a polite supplement): In Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 3 (3A, 3B, 3C & 3D): System Programming Guide in section 8.7.13.4 External Signal Compatibility one can read (emphasis by me): "A20M# pin — On an IA-32 processor, the A20M# pin is typically provided for compatibility with the Intel 286 processor. Asserting this pin causes bit 20 of the physical address to be masked (forced to zero) for all external bus memory accesses. Processors supporting Intel Hyper-Threading Technology provide one A20M# pin, which affects the operation of both logical processors within the physical processor. The functionality of A20M# is used primarily by older operating systems and not used by modern operating systems. On newer Intel 64 processors, A20M# may be absent.". TLDR: The accepted spec is that A20M# might not exist.
        Vendan 10y ago
        fair point, I hadn't bothered to look more, but it's still true that A20(when implemented) gates both real and protected modes, and therefore "ignoring it to focus on protected mode" is invalid
    - deleted 10y ago
      [deleted]
emiliobumachar 10y ago
It's a from-first-principles guide to building an OS. From the title, I had mistakenly assumed it was about the first OS ever.
- vortegne 10y ago
  Interesting. I was sure from the title that it's about building an OS, but it's fascinating how different people interpret things.
- devnonymous 10y ago
  Like almost everything in Computer science there would be no clear definition of first OS ever. There used to be monitor programs and schedulers before they iteratively evolved into OS programs. Just like, programming languages, servers, 'cloud'... There seldom is a first ever something in our field.
  - brett40324 10y ago
    Im curious what "monitor programs" means here in this context? The first OS was OS/360 at IBM in 1964.
    - skissane 10y ago
      OS/360 was not the first OS. Quite a number of operating systems came before it. Examples of earlier operating systems include SHARE Operating System (SOS), first version in 1959, for the IBM 709 mainframe; which in turn was itself was based on an earlier operating system, GM-NAA I/O, first version in 1956. A very notable pre-OS/360 operating system was MIT's CTSS operating system, the first timesharing OS, first version in 1961 for the IBM 7094. The first versions of OS/360, by contrast, notably lacked timesharing – that was initially provided by alternative IBM operating systems (TSS/360 was the official answer and CP/CMS, which later became VM/CMS, the principal unofficial one); OS/360 only gained timesharing support itself when TSO was released in 1971.
    - devnonymous 10y ago
      https://en.m.wikipedia.org/wiki/Resident_monitor https://en.m.wikipedia.org/wiki/Resident_monitor
laxentasken 10y ago
I have on my list to read something along the lines like this, but this seems incomplete. Does anyone have anything similar?
- ktta 10y ago
  Maybe not exactly what you're looking for, but here's a linux specific ebook similar to this[1] [1]:https://0xax.gitbooks.io/linux-insides/content/ https://0xax.gitbooks.io/linux-insides/content/
  - tuhdo 10y ago
    Well, the book is not Linux-specific but only leverages Linux as a development environment.
    - ktta 10y ago
      I feel like it focuses on a lot of linux specific details that a OS agnostic book would when using linux as a 'development environment' definitely wouldn't. There are many topics other OSes like Windows and Solaris do differently and talking about them even for a little bit would be beneficial, but I haven't seen any trace of it. Search for Windows, Solaris, shows up nothing, and search for unix shows a single page about unix signals.
- haffla 10y ago
  It not only seems, it says it is incomplete which is not unusual for something that is a work in progress
- nguyendat 10y ago
  The little book about OS development by Erik Helin, Adam Renberg http://littleosbook.github.io/#introduction http://littleosbook.github.io/#introduction
- sagischwarz 10y ago
  This is what I'm planning to read as soon as I've learned assembly: http://www.brokenthorn.com/Resources/OSDevIndex.html http://www.brokenthorn.com/Resources/OSDevIndex.html
js8 10y ago
Looks very nice so far. One minor nitpick - there should definitely be a chapter on inter-process communication, it's an important part of operating systems.
radisb 10y ago
From the book: "If a programmer writes software for a living, he should better be specialized in one or two problem domains outside of software if he does not want his job taken by domain experts who learn programming in their spare time." Seems a bizarre sentiment, but after reading this sentence, I feel like I really wanna donate some money to the guy. If he gives a way I will surely do.
- robjan 10y ago
  I think the entire section 1.1 can be removed as it seems a bit pompous. Other than that, it seems like a reasonable guide.
  - radisb 10y ago
    I can see your point, but the section created a nice "atmosphere" for me and it was not too long. I dont know how to express it clearly.
- shacharz 10y ago
  Can you explain more what inspired you in this sentence or are you still thinking it's bizarre? And if you do still think it's bizarre why would that convince you to donate?
  - karmajunkie 10y ago
    not OP, but one thing we as developers should recognize is that programming itself is likely to become merely a skill like operating a spreadsheet or typing. In much the same way, writing was once a skill available (or at least, practiced) by a very small subset of people, and the spread of literacy made the job of "scribe" largely irrelevant over time. Children today are learning how to program—perhaps not yet in the numbers we might have imagined, but a growing number. As they grow up, they are not going to pursue jobs as "programmers" or "developers", any more than I would pursue a job in a typing pool. They will merely incorporate this skill into whatever role they end up in. Those best suited to compete in that scenario are going to have deep experience in some area where programming is an applied skill, not an end unto itself.
koolba 10y ago
For those of you that have read, or at least skimmed, this, how does it compare to the Minix book? (which was a joy to read!)
jerianasmith 10y ago
The hallmark of a Good book is that it should leave you wanting more. That's what the book is all about.
- dbancajas 10y ago
  Have you read the book?
lorenzfx 10y ago
How much time would it approximately take to work through this?
seewhat 10y ago
On pg 38: Field Gate Programmable Array (FPGA) Surely: Field Programmable Gate Array (FPGA) ?
- deleted 10y ago
  [deleted]
- tuhdo 10y ago
  Thanks. I will fixed it.
lanbanger 10y ago
The grammar makes that book fairly painful to read :-|
- tuhdo 10y ago
  Hi, I would love to hear your feedback. It is the style or the correctness? I am really appreciate if you can show me the few incorrect usages of the grammar.
  - dbancajas 10y ago
    I can tell you're not native. Have the book proofread by someone.
    - tuhdo 10y ago
      Yes, I'm not native. For that reason I tried to use simple grammar and sentences, and did go through paragraph by paragraph through a grammar checker until the end of chapter 4. I will try to improve it gradually.
      - palerdot 10y ago
        It is better to proof read for correctness. I myself want to point out few grammatical stuffs and looking for the source in Github. You can simple create a repo or add the source text to the current repo, so that people can easily fix these issues for you by giving pull requests. You don't want people judging this excellent book based on few language quirks here and there.
        tuhdo 10y ago
        Thanks. I will upload the source soon. As it is written in Lyx, not sure everyone will find it comfortable though. You can always open an issue in the meantime though.
        10y ago
gigatexal 10y ago
Well this has got to be one of the more ambitious things I've seen on HN. I wonder if the guy behind SkyOS is still doing operating systems. http://www.skyos.org/ http://www.skyos.org/
- wolfgke 10y ago
  > I wonder if the guy behind SkyOS is still doing operating systems. http://www.skyos.org/ http://www.skyos.org/ No, he (Robert Szeleney, http://www.skyos.org/?q=node/411 http://www.skyos.org/?q=node/411) doesn't. He (together with Heiko Hufnagl) founded a studio for creating software and games for mobile devices: > https://en.wikipedia.org/wiki/Djinnworks https://en.wikipedia.org/wiki/Djinnworks > http://djinnworks.at/about-new/ http://djinnworks.at/about-new/
bogomipz 10y ago
This looks great! I would also like to recommend another free resource that might be a good complement(theory vs implementation) to this: "Operating Systems: Three Easy Pieces" available online at: http://pages.cs.wisc.edu/~remzi/OSTEP/ http://pages.cs.wisc.edu/~remzi/OSTEP/
- redpanda_ua 10y ago
  I haven't yet finished this book, but this is very good book imo. Can recommend.
itsmemattchung 10y ago
Every time an OS book (about weekly) is posted on HN, I immediately want to jump all over it but I gently remind myself to finish: CMU's Computer System's from a programmer's perspective and Elements of computing (nand2tetris).
- harrisi 10y ago
  It seems you've already decided that these are worth reading, but in case you want some motivation, I had CS:APP as my textbook last term and I thought it was quite nice. I also worked through TECS several years ago and it was also a great resource. CS:APP is quite dry but by the end of the course I was able to write a real memory manager in C, which was super fun! If you're ever looking for an internet book reading buddy, I also need motivation. I worked through TECS with a friend over a few weeks and found it very useful to have someone else seeing and working through the same problems. Good luck!
  - dbancajas 10y ago
    TECS? I would like to have online buddy to do this self-learning courses. Most of my projects fizzled due to lack of accountability.
    - harrisi 10y ago
      TECS is The Elements of Computing System. Feel free to send me an email!
ejanus 10y ago
I will read your book....and thanks for such a book
anhldbk 10y ago
Nice job man!
brw12 10y ago
tuhdo, can you say more about how the process works to convert your book from lyx text source to pdf? Since the current pdf seems to have particular formatting added to the source text (eg, terms not emphasized in remaining.txt are italicized in the pdf), it's a little tricky to tell. Is the raw lyx file where the formatting rules live? Does the lyx file store its text source in text files, but its formatting rules in binary?

709 points by tuhdo 10y ago

kriro 10y ago

Seems to be for a x86 operating system. I'd have preferred some other architecture because so much of OSdev for x86 that I remember was working around quirks of the architecture (A20 gate etc.). I guess it's a valuable lesson but I'd really enjoy a fork of the book for the hardware you find in a Beagle Bone or Pi3 or something. Maybe this could be crowdfunded if the x86 version is popular?

tuhdo 10y ago
Author here. Yes, it's a x86 operating system. However, rather than getting around A20, it focused on protected mode instead. The book not only teaches x86, but how to use the official resources from the hardware manufacturer to write the OS. In sum, a reader when reaching part 3 for writing the OS, he will need to use the official document, in this case, the "System Programming Guide" manual from Intel to write C code that complies with the documents. Once he learned how to do so, learning other platforms will be much easier given how complex x86 is.
- wolfgke 10y ago
  > Author here. Yes, it's a x86 operating system. However, rather than getting around A20, it focused on protected mode instead. You still have to open the A20 gate in the bootloader if you want to access a memory adress that has bit 20 (counting from 0) be set to 1 (you probably want) - even if you switch to protected mode. The only exception is if you boot from UEFI instead of BIOS - in this case the A20 gate is already set. But the book uses BIOS as far as I see it.
  - tuhdo 10y ago
    In the Volume 3, it is said that: "In protected mode, the IA-32 architecture provides a normal physical address space of 4 GBytes (2 32 bytes). This is the address space that the processor can address on its address bus. This address space is flat (unsegmented), with addresses ranging continuously from 0 to FFFFFFFFH. This physical address space can be mapped to read- write memory, read-only memory, and memory mapped I/O. The memory mapping facilities described in this chapter can be used to divide this physical memory up into segments and/or pages." It correlates to my experience of developing in protected mode in QEMU. Once entered protected mode, I can access to any address above 0x10000 without being wrapped around. When I was writing my first kernel (https://github.com/tuhdo/os-study https://github.com/tuhdo/os-study) in real-mode, indeed A20 must be enabled.
    - Vendan 10y ago
      The A20 gate would affect both real and protected mode, so to be truely compliant, you should be opening it even for a protected mode OS. It was originally a purely hardware hack forcing an address line(A20, or the 21st address line) to stay low, thus "wrapping" addresses from 1mb-2mb to 0-1mb, so it would have affected real or protected mode identically. On modern hardware, there is a chance that A20 is (against the accepted spec) default open, or possible not implemented.
      - wolfgke 10y ago
        > On modern hardware, there is a chance that A20 is (against the accepted spec) default open, or possible not implemented. To nitpick on this a little bit (consider it as a polite supplement): In Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 3 (3A, 3B, 3C & 3D): System Programming Guide in section 8.7.13.4 External Signal Compatibility one can read (emphasis by me): "A20M# pin — On an IA-32 processor, the A20M# pin is typically provided for compatibility with the Intel 286 processor. Asserting this pin causes bit 20 of the physical address to be masked (forced to zero) for all external bus memory accesses. Processors supporting Intel Hyper-Threading Technology provide one A20M# pin, which affects the operation of both logical processors within the physical processor. The functionality of A20M# is used primarily by older operating systems and not used by modern operating systems. On newer Intel 64 processors, A20M# may be absent.". TLDR: The accepted spec is that A20M# might not exist.
        Vendan 10y ago
        fair point, I hadn't bothered to look more, but it's still true that A20(when implemented) gates both real and protected modes, and therefore "ignoring it to focus on protected mode" is invalid
  - deleted 10y ago
    [deleted]

emiliobumachar 10y ago

It's a from-first-principles guide to building an OS. From the title, I had mistakenly assumed it was about the first OS ever.

vortegne 10y ago
Interesting. I was sure from the title that it's about building an OS, but it's fascinating how different people interpret things.
devnonymous 10y ago
Like almost everything in Computer science there would be no clear definition of first OS ever. There used to be monitor programs and schedulers before they iteratively evolved into OS programs. Just like, programming languages, servers, 'cloud'... There seldom is a first ever something in our field.
- brett40324 10y ago
  Im curious what "monitor programs" means here in this context? The first OS was OS/360 at IBM in 1964.
  - skissane 10y ago
    OS/360 was not the first OS. Quite a number of operating systems came before it. Examples of earlier operating systems include SHARE Operating System (SOS), first version in 1959, for the IBM 709 mainframe; which in turn was itself was based on an earlier operating system, GM-NAA I/O, first version in 1956. A very notable pre-OS/360 operating system was MIT's CTSS operating system, the first timesharing OS, first version in 1961 for the IBM 7094. The first versions of OS/360, by contrast, notably lacked timesharing – that was initially provided by alternative IBM operating systems (TSS/360 was the official answer and CP/CMS, which later became VM/CMS, the principal unofficial one); OS/360 only gained timesharing support itself when TSO was released in 1971.
  - devnonymous 10y ago
    https://en.m.wikipedia.org/wiki/Resident_monitor https://en.m.wikipedia.org/wiki/Resident_monitor

laxentasken 10y ago

I have on my list to read something along the lines like this, but this seems incomplete. Does anyone have anything similar?

ktta 10y ago
Maybe not exactly what you're looking for, but here's a linux specific ebook similar to this[1] [1]:https://0xax.gitbooks.io/linux-insides/content/ https://0xax.gitbooks.io/linux-insides/content/
- tuhdo 10y ago
  Well, the book is not Linux-specific but only leverages Linux as a development environment.
  - ktta 10y ago
    I feel like it focuses on a lot of linux specific details that a OS agnostic book would when using linux as a 'development environment' definitely wouldn't. There are many topics other OSes like Windows and Solaris do differently and talking about them even for a little bit would be beneficial, but I haven't seen any trace of it. Search for Windows, Solaris, shows up nothing, and search for unix shows a single page about unix signals.
haffla 10y ago
It not only seems, it says it is incomplete which is not unusual for something that is a work in progress
nguyendat 10y ago
The little book about OS development by Erik Helin, Adam Renberg http://littleosbook.github.io/#introduction http://littleosbook.github.io/#introduction
sagischwarz 10y ago
This is what I'm planning to read as soon as I've learned assembly: http://www.brokenthorn.com/Resources/OSDevIndex.html http://www.brokenthorn.com/Resources/OSDevIndex.html

js8 10y ago

Looks very nice so far. One minor nitpick - there should definitely be a chapter on inter-process communication, it's an important part of operating systems.

radisb 10y ago

From the book: "If a programmer writes software for a living, he should better be specialized in one or two problem domains outside of software if he does not want his job taken by domain experts who learn programming in their spare time." Seems a bizarre sentiment, but after reading this sentence, I feel like I really wanna donate some money to the guy. If he gives a way I will surely do.

robjan 10y ago
I think the entire section 1.1 can be removed as it seems a bit pompous. Other than that, it seems like a reasonable guide.
- radisb 10y ago
  I can see your point, but the section created a nice "atmosphere" for me and it was not too long. I dont know how to express it clearly.
shacharz 10y ago
Can you explain more what inspired you in this sentence or are you still thinking it's bizarre? And if you do still think it's bizarre why would that convince you to donate?
- karmajunkie 10y ago
  not OP, but one thing we as developers should recognize is that programming itself is likely to become merely a skill like operating a spreadsheet or typing. In much the same way, writing was once a skill available (or at least, practiced) by a very small subset of people, and the spread of literacy made the job of "scribe" largely irrelevant over time. Children today are learning how to program—perhaps not yet in the numbers we might have imagined, but a growing number. As they grow up, they are not going to pursue jobs as "programmers" or "developers", any more than I would pursue a job in a typing pool. They will merely incorporate this skill into whatever role they end up in. Those best suited to compete in that scenario are going to have deep experience in some area where programming is an applied skill, not an end unto itself.

koolba 10y ago

For those of you that have read, or at least skimmed, this, how does it compare to the Minix book? (which was a joy to read!)

jerianasmith 10y ago

The hallmark of a Good book is that it should leave you wanting more. That's what the book is all about.

dbancajas 10y ago
Have you read the book?

lorenzfx 10y ago

How much time would it approximately take to work through this?

seewhat 10y ago

On pg 38: Field Gate Programmable Array (FPGA) Surely: Field Programmable Gate Array (FPGA) ?

deleted 10y ago
[deleted]
tuhdo 10y ago
Thanks. I will fixed it.

lanbanger 10y ago

The grammar makes that book fairly painful to read :-|

tuhdo 10y ago
Hi, I would love to hear your feedback. It is the style or the correctness? I am really appreciate if you can show me the few incorrect usages of the grammar.
- dbancajas 10y ago
  I can tell you're not native. Have the book proofread by someone.
  - tuhdo 10y ago
    Yes, I'm not native. For that reason I tried to use simple grammar and sentences, and did go through paragraph by paragraph through a grammar checker until the end of chapter 4. I will try to improve it gradually.
    - palerdot 10y ago
      It is better to proof read for correctness. I myself want to point out few grammatical stuffs and looking for the source in Github. You can simple create a repo or add the source text to the current repo, so that people can easily fix these issues for you by giving pull requests. You don't want people judging this excellent book based on few language quirks here and there.
      - tuhdo 10y ago
        Thanks. I will upload the source soon. As it is written in Lyx, not sure everyone will find it comfortable though. You can always open an issue in the meantime though.
        10y ago

gigatexal 10y ago

Well this has got to be one of the more ambitious things I've seen on HN. I wonder if the guy behind SkyOS is still doing operating systems. http://www.skyos.org/ http://www.skyos.org/

wolfgke 10y ago
> I wonder if the guy behind SkyOS is still doing operating systems. http://www.skyos.org/ http://www.skyos.org/ No, he (Robert Szeleney, http://www.skyos.org/?q=node/411 http://www.skyos.org/?q=node/411) doesn't. He (together with Heiko Hufnagl) founded a studio for creating software and games for mobile devices: > https://en.wikipedia.org/wiki/Djinnworks https://en.wikipedia.org/wiki/Djinnworks > http://djinnworks.at/about-new/ http://djinnworks.at/about-new/

bogomipz 10y ago

This looks great! I would also like to recommend another free resource that might be a good complement(theory vs implementation) to this: "Operating Systems: Three Easy Pieces" available online at: http://pages.cs.wisc.edu/~remzi/OSTEP/ http://pages.cs.wisc.edu/~remzi/OSTEP/

redpanda_ua 10y ago
I haven't yet finished this book, but this is very good book imo. Can recommend.

itsmemattchung 10y ago

Every time an OS book (about weekly) is posted on HN, I immediately want to jump all over it but I gently remind myself to finish: CMU's Computer System's from a programmer's perspective and Elements of computing (nand2tetris).

harrisi 10y ago
It seems you've already decided that these are worth reading, but in case you want some motivation, I had CS:APP as my textbook last term and I thought it was quite nice. I also worked through TECS several years ago and it was also a great resource. CS:APP is quite dry but by the end of the course I was able to write a real memory manager in C, which was super fun! If you're ever looking for an internet book reading buddy, I also need motivation. I worked through TECS with a friend over a few weeks and found it very useful to have someone else seeing and working through the same problems. Good luck!
- dbancajas 10y ago
  TECS? I would like to have online buddy to do this self-learning courses. Most of my projects fizzled due to lack of accountability.
  - harrisi 10y ago
    TECS is The Elements of Computing System. Feel free to send me an email!

ejanus 10y ago

I will read your book....and thanks for such a book

anhldbk 10y ago

Nice job man!

brw12 10y ago

tuhdo, can you say more about how the process works to convert your book from lyx text source to pdf? Since the current pdf seems to have particular formatting added to the source text (eg, terms not emphasized in remaining.txt are italicized in the pdf), it's a little tricky to tell. Is the raw lyx file where the formatting rules live? Does the lyx file store its text source in text files, but its formatting rules in binary?