4 ms·

x86 Bare Metal Examples: Minimal operating systems, learn x86 system programming

279 points by peter_d_sherman 5y ago

markhahn 5y ago
Because even a Paramount Leader needs a little programming side-project on slow weekend.
deleted 5y ago
[deleted]
cirosantilli 5y ago
I don't understand why people like x86 bare metal programming so much!!! XD Just kidding. Related: - https://news.ycombinator.com/item?id=19428700 https://news.ycombinator.com/item?id=19428700 - https://news.ycombinator.com/item?id=18531393 https://news.ycombinator.com/item?id=18531393 - https://news.ycombinator.com/item?id=13773219 https://news.ycombinator.com/item?id=13773219
- userbinator 5y ago
  It might have to do with all the recent discussions about Windows 11, mandatory TPM, and "secure" boot restrictions; to show that, yes, people do try to write their own OSs, and that such things would be basically impossible if the megacorps had their way.
  - narrator 5y ago
    What's weird about Windows 11 is they are integrating Android apps. In some ways it's getting more open and, with the TPM requirement, in some ways it's getting more closed. Android apps run on Linux with AnBox, though I haven't tried it. It seems like a much more viable world for alternative operating systems when most applications are written for Android.
throwitaway1235 5y ago
Couldn't read. Author is too obsessed with going to war with China. Neocon trash.
rvz 5y ago
Great examples, well done!
gswdh 5y ago
Has anyone else seen the China section in the readme!?
knz_ 5y ago
The author appears to mentally unhinged: https://github.com/cirosantilli/china-dictatorship https://github.com/cirosantilli/china-dictatorship I get it, the PRC is probably one of the worst regimes on the planet, but writing a book about it on every README you publish in your toy projects is a classic sign of schizophrenia.
Fluid_Mechanics 5y ago
Appropriate avatar
emily-c 5y ago
These are good examples and this might be slightly off topic but I wish that there were more resources on how modern x86 platforms actually work instead of using these legacy interfaces that are either emulated in SMM with IO port trapping or relegated to requiring chipset hacks to keep them working (such as VGA device hacks in the PCI spec or well known chipset legacy IO ports decoding to the LPC bridge). While it is essentially impossible to truly understand every aspect of todays platforms I think that one can build a lot of intuition about how things work by studying how transaction routing is handled in the platform, e.g. how the uncore and chipset routes TLPs (whether it's raw PCIe or packets essentially wrapped by other interconnect protocols like IF/HyperTransport/DMI/QPI/etc) through the system with various base address registers and decoding windows. This gives way to understanding how modern MSI interrupts and DMA works as well. While I understand that truly going into things like PCI configuration, ACPI, UEFI, chipset interfaces on bus 0, modern booting, DMA and interrupt remapping, DRAM, register locking and security, security processors, power management, various types of runtime firmware, etc is a a many-years-long exercise I think that the core ideas don't need to be spread around all over the place for one to get familiarity with the ideas and patterns that underpin our PCs. I think that a general conceptual overview on which one can hang more detailed knowledge acquired by reading specs is needed with the amount of complexity that exists today. There are some really good resources that I've collected over the years but there is still something missing. Maybe I or someone else will get around to it some day.
- userbinator 5y ago
  "legacy" is the reason the PC still remains relatively open despite corporate threats, so don't dismiss it lightly. https://news.ycombinator.com/item?id=15730113 https://news.ycombinator.com/item?id=15730113
  - emily-c 5y ago
    I'm not sure why you have legacy in quotes, that is what it is. Modern platforms do not act like a PC AT and there is a ton of cruft to keep those interfaces working which is set up and facilitated by firmware (not so open, at least TianoCore exists regardless of your thoughts on UEFI). Modern x86 systems programming is so much more than these interfaces. I think that it would be beneficial for others to understand how things are actually working on modern platforms. Regarding your linked post: For corporate users legacy methods of booting /are/ a security threat. Firmware attacks are a real threat for enterprises. Having IBVs all hand roll things like the S3 boot script or communication with SMM (not using standardized com buffers) is not wise. Secure boot combined with modern security features like using DRTM to remove firmware from the trusted computing base are important and should not be discounted because it's not an 40 year old interface. The PC platform is historically extremely insecure and there is a ton of work still to be done to effectively provide the primitives needed to secure the platform against determined attackers. If you want to have a USB stack in SMM owning your USB controller to emulate port 60h with IO SMI traps that's fine but it's not a reason to hold the platform back.
sydthrowaway 5y ago
Now do multicore!!
- cirosantilli 5y ago
  Thankfully, I regained my sanity before going that far. The "a second CPU started" example at: https://github.com/cirosantilli/x86-bare-metal-examples/tree/528ab2becc4992218ca0b7e297f3e1f1674268de#smp https://github.com/cirosantilli/x86-bare-metal-examples/tree... though. I think some of the projects under https://github.com/cirosantilli/x86-bare-metal-examples/tree/528ab2becc4992218ca0b7e297f3e1f1674268de#small-educational-projects https://github.com/cirosantilli/x86-bare-metal-examples/tree... did it though.
  - xpuente 5y ago
    Take a look at https://pdos.csail.mit.edu/6.828/2019/xv6.html https://pdos.csail.mit.edu/6.828/2019/xv6.html
xony 5y ago
tech industry should be away from politics .. you made a wonderful software piece ..all sins forgiven :)
brutal_chaos_ 5y ago
"Read this before getting started." I enjoyed the jab at Pooh. I'm also looking forward to digging into this. It looks really neat and I have been interested in baremetal programming recently. Hopefully this coming weekend will be free enough to indulge!
andy_threos_io 5y ago
Nowadays, it's better to use x86_64 only. The 32 bit x86 has many drawbacks. The 64 bit is much cleaner and better in many many ways. -no segmentation, flat address space -better interrupt handling -better system call -more registers etc. Also we found that Qemu or other Virtualization technology may cover up some of the hardware issues. Eg. Beside, that every hardware device (keyboard, usb, display, disk etc.) will have some simplified behavior, if you emulate an ARM cpu on x86_64 host, than the cache coherency will totally messed up on real hardware.
- cirosantilli 5y ago
  Yes, absolutely correct about possible QEMU simplifications. One option would be to use gem5 which can simulate caches/memory more accurately (sometimes), but it is also way slower as a result, some indications at: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/38eb67ad2ec46b58dbc701073ce0e980fad7788b#gem5-buildroot-setup https://github.com/cirosantilli/linux-kernel-module-cheat/tr...

markhahn 5y ago

Because even a Paramount Leader needs a little programming side-project on slow weekend.

deleted 5y ago

[deleted]

cirosantilli 5y ago

I don't understand why people like x86 bare metal programming so much!!! XD Just kidding. Related: - https://news.ycombinator.com/item?id=19428700 https://news.ycombinator.com/item?id=19428700 - https://news.ycombinator.com/item?id=18531393 https://news.ycombinator.com/item?id=18531393 - https://news.ycombinator.com/item?id=13773219 https://news.ycombinator.com/item?id=13773219

userbinator 5y ago
It might have to do with all the recent discussions about Windows 11, mandatory TPM, and "secure" boot restrictions; to show that, yes, people do try to write their own OSs, and that such things would be basically impossible if the megacorps had their way.
- narrator 5y ago
  What's weird about Windows 11 is they are integrating Android apps. In some ways it's getting more open and, with the TPM requirement, in some ways it's getting more closed. Android apps run on Linux with AnBox, though I haven't tried it. It seems like a much more viable world for alternative operating systems when most applications are written for Android.

throwitaway1235 5y ago

Couldn't read. Author is too obsessed with going to war with China. Neocon trash.

rvz 5y ago

Great examples, well done!

gswdh 5y ago

Has anyone else seen the China section in the readme!?

knz_ 5y ago

The author appears to mentally unhinged: https://github.com/cirosantilli/china-dictatorship https://github.com/cirosantilli/china-dictatorship I get it, the PRC is probably one of the worst regimes on the planet, but writing a book about it on every README you publish in your toy projects is a classic sign of schizophrenia.

Fluid_Mechanics 5y ago

Appropriate avatar

emily-c 5y ago

These are good examples and this might be slightly off topic but I wish that there were more resources on how modern x86 platforms actually work instead of using these legacy interfaces that are either emulated in SMM with IO port trapping or relegated to requiring chipset hacks to keep them working (such as VGA device hacks in the PCI spec or well known chipset legacy IO ports decoding to the LPC bridge). While it is essentially impossible to truly understand every aspect of todays platforms I think that one can build a lot of intuition about how things work by studying how transaction routing is handled in the platform, e.g. how the uncore and chipset routes TLPs (whether it's raw PCIe or packets essentially wrapped by other interconnect protocols like IF/HyperTransport/DMI/QPI/etc) through the system with various base address registers and decoding windows. This gives way to understanding how modern MSI interrupts and DMA works as well. While I understand that truly going into things like PCI configuration, ACPI, UEFI, chipset interfaces on bus 0, modern booting, DMA and interrupt remapping, DRAM, register locking and security, security processors, power management, various types of runtime firmware, etc is a a many-years-long exercise I think that the core ideas don't need to be spread around all over the place for one to get familiarity with the ideas and patterns that underpin our PCs. I think that a general conceptual overview on which one can hang more detailed knowledge acquired by reading specs is needed with the amount of complexity that exists today. There are some really good resources that I've collected over the years but there is still something missing. Maybe I or someone else will get around to it some day.

userbinator 5y ago
"legacy" is the reason the PC still remains relatively open despite corporate threats, so don't dismiss it lightly. https://news.ycombinator.com/item?id=15730113 https://news.ycombinator.com/item?id=15730113
- emily-c 5y ago
  I'm not sure why you have legacy in quotes, that is what it is. Modern platforms do not act like a PC AT and there is a ton of cruft to keep those interfaces working which is set up and facilitated by firmware (not so open, at least TianoCore exists regardless of your thoughts on UEFI). Modern x86 systems programming is so much more than these interfaces. I think that it would be beneficial for others to understand how things are actually working on modern platforms. Regarding your linked post: For corporate users legacy methods of booting /are/ a security threat. Firmware attacks are a real threat for enterprises. Having IBVs all hand roll things like the S3 boot script or communication with SMM (not using standardized com buffers) is not wise. Secure boot combined with modern security features like using DRTM to remove firmware from the trusted computing base are important and should not be discounted because it's not an 40 year old interface. The PC platform is historically extremely insecure and there is a ton of work still to be done to effectively provide the primitives needed to secure the platform against determined attackers. If you want to have a USB stack in SMM owning your USB controller to emulate port 60h with IO SMI traps that's fine but it's not a reason to hold the platform back.

sydthrowaway 5y ago

Now do multicore!!

cirosantilli 5y ago
Thankfully, I regained my sanity before going that far. The "a second CPU started" example at: https://github.com/cirosantilli/x86-bare-metal-examples/tree/528ab2becc4992218ca0b7e297f3e1f1674268de#smp https://github.com/cirosantilli/x86-bare-metal-examples/tree... though. I think some of the projects under https://github.com/cirosantilli/x86-bare-metal-examples/tree/528ab2becc4992218ca0b7e297f3e1f1674268de#small-educational-projects https://github.com/cirosantilli/x86-bare-metal-examples/tree... did it though.
- xpuente 5y ago
  Take a look at https://pdos.csail.mit.edu/6.828/2019/xv6.html https://pdos.csail.mit.edu/6.828/2019/xv6.html

xony 5y ago

tech industry should be away from politics .. you made a wonderful software piece ..all sins forgiven :)

brutal_chaos_ 5y ago

"Read this before getting started." I enjoyed the jab at Pooh. I'm also looking forward to digging into this. It looks really neat and I have been interested in baremetal programming recently. Hopefully this coming weekend will be free enough to indulge!

andy_threos_io 5y ago

Nowadays, it's better to use x86_64 only. The 32 bit x86 has many drawbacks. The 64 bit is much cleaner and better in many many ways. -no segmentation, flat address space -better interrupt handling -better system call -more registers etc. Also we found that Qemu or other Virtualization technology may cover up some of the hardware issues. Eg. Beside, that every hardware device (keyboard, usb, display, disk etc.) will have some simplified behavior, if you emulate an ARM cpu on x86_64 host, than the cache coherency will totally messed up on real hardware.

cirosantilli 5y ago
Yes, absolutely correct about possible QEMU simplifications. One option would be to use gem5 which can simulate caches/memory more accurately (sometimes), but it is also way slower as a result, some indications at: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/38eb67ad2ec46b58dbc701073ce0e980fad7788b#gem5-buildroot-setup https://github.com/cirosantilli/linux-kernel-module-cheat/tr...