10 ms·

How to run a program without an operating system?

394 points by Cieplak 8y ago

beached_whale 8y ago
Saw a talk on Include OS that is a C++ library that allows one to do this http://www.includeos.org http://www.includeos.org
- cirosantilli 8y ago
  A similar effect can be achieved with Newlib, which is mentioned on the post, and exemplified at: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/54e15e04338c0fecc0be139a0da2d0d972c21419#baremetal-setup-getting-started https://github.com/cirosantilli/linux-kernel-module-cheat/tr... You might need to implement some syscalls though. Or similarly you can also use an open source embedded system like FreeRTOS or Zephyr which essentially implement the syscalls for you.
  - cmroanirgo 8y ago
    That was a spectacular SO answer btw. Thanks for writing it!
  - perbu 8y ago
    IncludeOS is modeled much like FreeRTOS or Zephyr. The difference is mainly that it addresses CPUs and not microcontrollers as the primary platform. With paging, you have better security, typically.
    - beached_whale 8y ago
      Along those lines, a bit, I have always wanted to play with targeting linux but making ones program the init process. All the resources but the benefit of Linux's ability to run on a lot of platforms. Not micro controller though, well arm, but that is pretty large.
      - cirosantilli 8y ago
        This is trivial with Buildroot btw, see e.g.: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/54e15e04338c0fecc0be139a0da2d0d972c21419#init https://github.com/cirosantilli/linux-kernel-module-cheat/tr... And a minimal hello world at: https://unix.stackexchange.com/questions/122717/how-to-create-a-custom-linux-distro-that-runs-just-one-program-and-nothing-else/238579#238579 https://unix.stackexchange.com/questions/122717/how-to-creat...
CoolGuySteve 8y ago
If you ever programmed a ROM for a Gameboy or something this should look familiar. The BIOS of those old systems just jumps right into a certain address and all you have to do is make sure your linker is setup properly to put crt0.s there. The Gameboy Advance in particular is pretty fun and simple, I recommend it.
- shshhdhs 8y ago
  Where should one go to get started with GBA dev work these days?
  - raptorfactor 8y ago
    http://www.gbadev.org/ http://www.gbadev.org/
  - deleted 8y ago
    [deleted]
  - pathompong 8y ago
    http://www.loirak.com/gameboy/gbatutor.php http://www.loirak.com/gameboy/gbatutor.php
  - GrantSolar 8y ago
    I've found this to be a good resource: https://www.coranac.com/tonc/text/ https://www.coranac.com/tonc/text/
- jchw 8y ago
  Something neat about the original Gameboy is how its firmware actually works. As a developer, the entire region from $0000-$3FFF is mapped to the cartridge, and the entrypoint is at $0100. When the CPU boots up, there is a special temporary mapping for $0000-$0100 to the firmware. The firmware largely just validates the ROM header and displays the Nintendo logo. But my favorite part is this: it executes all the way to $0100. There is no jump to $0100 - the last two bytes are an instruction that simply removes the firmware mapping. Now the IP lands on $0100 and everything is mapped as expected. This simple mechanism made it notoriously difficult to extract the Gameboy BIOS. I recall reading that one way it was eventually extracted was via fault injection: by messing with the clock very precisely, you could get the processor to 'skip' the instruction that unmapped the firmware, thus allowing you to dump it. Of course, another way you could read the firmware is by "simply" decapping the processor and taking shots of it with a powerful microscope, and then optically extracting the 'bits.' I think this was also done, but as a person with little understanding of the actual details of how a decapping is done it seems like an incredibly expensive, error prone operation.
everyone 8y ago
Prompted a fascinating answer. 'closed as too broad' -- modern Stack overflow in a nutshell.
- TeMPOraL 8y ago
  I wish there was a guide for "proper" SO questions; my impression is, the only ones are super-detailed questions about some super-particular APIs. Also interesting how this disconnect is still going on after so many years, mostly unaddressed - despite the many essays Jeff Atwood writes about true purpose of SO, people still want to use it differently.
  - AndrewDucker 8y ago
    There is a guide here : https://stackoverflow.com/help/asking https://stackoverflow.com/help/asking
  - CathyWest 8y ago
    > I wish there was a guide for "proper" SO questions; my impression is, the only ones are super-detailed questions about some super-particular APIs. And those questions can in most cases be answered by the docs anyway. The times I've used SO is when the problem can't be answered by the docs, my own experience or that of the team, but then I've received either no answer or a series of non-answers (that should have been comments instead, but a lot of people can't comment). I also see quite a bit of overt reputation farming there. Like vague questions that receive an improbably precise answer that immediately gets selected. Like "how do u process payments?" (the orthography is usually bad) immediately followed by very specific instructions from a very specific payment processor not mentioned in the question, probably taken straight from its documentation. And I rarely see "closed as too broad" on those, probably because of the quick turn-around.
- yesenadam 8y ago
  Yeah, I hate that. I've asked a few questions on those sites and came back later, excited to read what answers I got and...it had been closed immediately, before anyone could answer it. GRRRR! Maybe there could/should be a 'broad' SE site alongside every 'normal' one? The 'too broad' questions include most of the interesting ones to read. The ones that are 'intellectually interesting'.[0] It's a shame. I love how you can ask anything on HN without being shut down like that. It's so unfriendly, hurtful even. (At worst on here, no-one answers - no harm done.) Why they can't just let 'too broad' questions go for a day or two to see what responses come in, and delete the page if it sux, I don't know. The mods get points for closing things, I guess. Often they don't understand the question, it seems. [0] e.g. https://mathoverflow.net/questions/43690/whats-a-mathematician-to-do https://mathoverflow.net/questions/43690/whats-a-mathematici... Well, bizarrely the close notice on that says "This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet." Which seems absurd. (Although sure, the question doesn't help people, the answers do though.)
Arainach 8y ago
For another take on the subject, see "This OS is a bootloader" in PoC||GTFO 0x4: https://www.alchemistowl.org/pocorgtfo/ https://www.alchemistowl.org/pocorgtfo/ There's a sequel in 0x5 that explains how to add a basic form of multiprocessing support.
- cirosantilli 8y ago
  I have also covered baremetal multicore at: https://stackoverflow.com/questions/980999/what-does-multicore-assembly-language-look-like/33651438#33651438 https://stackoverflow.com/questions/980999/what-does-multico... The exemple is present in the same repo: https://github.com/cirosantilli/x86-bare-metal-examples/blob/3c35cabb8934d0dd92974f5826c8815572831bea/smp.S https://github.com/cirosantilli/x86-bare-metal-examples/blob...
deleted 8y ago
[deleted]
andrewstuart 8y ago
Stack Overflow is for very specific questions with very specific answers. You'll get better results for more general questions from a reddit forum.
- cirosantilli 8y ago
  What annoys me is that Reddit threads get locked after 6 months. Then people ask another one. And another one. I want to fix that with decent algorithms one day: https://github.com/cirosantilli/write-free-science-books-to-get-famous-website https://github.com/cirosantilli/write-free-science-books-to-...
- sureaboutthis 8y ago
  There are some fairly decent subs on reddit but, generally, I find it mostly populated by 15-year old kids and others who have never had a job in the industry so, no, I would not refer anyone to reddit for information.
  - softawre 8y ago
    How can you tell their age? I've always though if a forum like this had a little age indicator (or experience indicator, or something like 'this person is smart') it would be a lot more useful.
    - sureaboutthis 8y ago
      It's strange that I can tell by the way they write. I'm not always correct but many times I'd find someone writing as if they were an expert but you could just tell they were only regurgitating something they read. They couldn't make a proper connection between things they were saying. I'd trick them in a way to reveal their age and way too often it was 15 to 17 years old. Sometimes 18 or 19. You can also read them saying they wanted to take CS when they graduated from school so you can glean from that they are probably in high school. Or I just flat out ask them (or accuse them).
      - 8y ago
justincormack 8y ago
The Osdev wiki is a great resource on this type of thing https://wiki.osdev.org/Expanded_Main_Page https://wiki.osdev.org/Expanded_Main_Page
- cirosantilli 8y ago
  I only wish they would create a repo with all their examples like I did :-)
stevefan1999 8y ago
This is wrong, you'd still have a operating system: BIOS
- klez 8y ago
  Maybe it's just my limited knowledge on the subject, but it seems to me it's a bit too much to consider the BIOS an operating system. At that point, what isn't an operating system? You're still writing code against a microcode interpreter even once you remove the BIOS. Is that an OS too?
  - xxs 8y ago
    BIOS - is an abbreviation for "basic input/output system". System is in the name. BIOSes offer system calls that tend to be backward compatible, so it's an OS of sorts.
    - cyphar 8y ago
      UEFI is the successor to BIOS on modern machines, and it stands for "Universal Extensible Firmware Interface". Is it also an operating system (it doesn't have "system" in the name)? (I don't disagree with you -- UEFI provides BootServices which arguably are more numerous than the number of syscalls on most operating systems. Just that "system" being in the name isn't really the best argument. :P)
      - xxs 8y ago
        It was thought to be like that - back in the day when everything used interrupts for syscalls, hence 'system' was mentioned. (Apple II had fixed address calls, though - no interrupts) It's quite the same as DOS being 'disk operating system' - it was able to drive the floppy and provide a file system.
  - da_chicken 8y ago
    I would agree. The BIOS/UEFI POSTs the system, and then passes execution to another program (usually a bootloader). At that point, the BIOS is no longer performing the tasks that an OS does. It's not scheduling tasks for the processor or handling I/O at all. Instead, it switches to what's basically an ACPI interface with functions that can be called by the OS. It's like an old TSR program.
finchisko 8y ago
Very nice post. However I thought it would be about writing programs that could be executed from OS, but OS would be completely unaware that such a program is running. Not sure if that is even possible. I know it’s possible to not use libc or similar, but os is aware of such a program.
- cirosantilli 8y ago
  It is not possible from userland, except for extremely serious zero days, due to hardware protections which OSes use, see: https://stackoverflow.com/questions/18717016/what-are-ring-0-and-ring-3-in-the-context-of-operating-systems/44483439#44483439 https://stackoverflow.com/questions/18717016/what-are-ring-0...
- viraptor 8y ago
  Kind of. Check out IBM's isostack. It takes CPU cores which are not used by the main os (you explicitly disable them) and essentially runs a parallel operating system which only handles the network cards / network stack. The main os cooperates with it, but otherwise it's not really aware of it.
- pjc50 8y ago
  A bit of thought about what it means to "run" gives you the answer here: in order for the code to run, it has to be scheduled at some point. That means it has to be in the operating system's task list somewhere. Having said that, there are places you can put running code that get scheduled but don't appear in an obvious way: inside other processes, as device drivers, as code on peripherals (GPU etc), or on the notorious management engine. Variations of these methods are used by viruses and exploits: the core of any exploit is getting code to run inside some other program that did not originally contain it. The final place is to put it "outside" the operating system as a hypervisor, then run the "normal" operating system as a seamless virtual machine inside the hypervisor.
- 0xcde4c3db 8y ago
  I remember hearing that the PlayStation emulator bleem! did something like this for threading/timing purposes, essentially running its own microkernel and hooking interrupts to bypass the Windows scheduler. However, this supposedly worked by basically exploiting a bug in the way Windows 9x set up task switching and memory protection, and so didn't work on any NT-based Windows (or Wine, for that matter). Also, it obviously didn't try to hide the main program that managed the window and other I/O operations. Malware often has something akin to this as a goal, but as far as I know it usually works by hiding itself from the user rather than the kernel.
sureaboutthis 8y ago
People forget that an operating system is, itself, just a program.
- JdeBP 8y ago
  It is usually more than one program. * https://superuser.com/a/329479/38062 https://superuser.com/a/329479/38062
amelius 8y ago
Now try to do it on a smartphone.
- 0xTJ 8y ago
  I really wish it were possible to do this kind of thing on smartphones. Unfortunately, odds are that anything you try will quickly brick it, and best-case scenario you can restore it using something like the low-level Broadcom interface
jancsika 8y ago
> printf '\364%509s\125\252' > main.img It appears that bash `printf` wraps coreutils printf "with ARGUMENTs converted to proper type first," according to the manual. I guess that's how the "%509s" specifier is able to work without any args. So how does bash `printf` actually determine the proper arg types? Is coreutils printf implemented with hooks that let wrappers fetch the proper arg types without generating an output string? Or does bash `printf` implement its own separate parser for the format specifier?
- kevinoid 8y ago
  > It appears that bash `printf` wraps coreutils printf "with ARGUMENTs converted to proper type first," according to the manual. I guess that's how the "%509s" specifier is able to work without any args. Actually, coreutils printf follows POSIX/SUS which states that "Any extra b, c, or s conversion specifiers shall be evaluated as if a null string argument were supplied".[1] So that trick will work for many printf(1) implementations (including both bash and coreutils). Bash printf(1) doesn't have any hooks or wrappers, but it does parse the format string to determine how to convert arguments before calling libc vsnprintf(3). If you are interested in the details, check out how %d/%i is handled in bash[2] and coreutils[3]. 1. http://pubs.opengroup.org/onlinepubs/9699919799/utilities/printf.html http://pubs.opengroup.org/onlinepubs/9699919799/utilities/pr... (item 9 under Extended Description) 2. https://git.savannah.gnu.org/cgit/bash.git/tree/builtins/printf.def?h=bash-4.4#n598 https://git.savannah.gnu.org/cgit/bash.git/tree/builtins/pri... 3. https://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;f=src/printf.c;hb=v8.30#l374 https://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;...
  - jancsika 8y ago
    Thanks for the links, those are really helpful! Why didn't libc include a printf function that expands a format specifier with a pointer to a byte array (and/or possibly an array of c strings)? That way there would at least be a standard way of handling arbitrary input and erroring out with less risk of buffer overflows and stuff.
- sfoley 8y ago
  I don’t think bash’s `printf` “wraps” coreutils; pretty sure it just implements the entire call itself.
Communitivity 8y ago
For any "How do I...X?" question involving connecting two pieces of software together in a different way, the first thing I try to do is look at the API(s) that connects them. In the case of a program and the OS, it's the system calls, device drivers, and standard libraries. There have been efforts to provide the capability of running a program without an OS before, but any such effort is going to need to provide the system calls and standard libraries used by the program, and the infrastructure to support it (device drivers, management, etc.). That that point it becomes a mini-OS. An example is Erlang on Xen. Xen is more often run with a guest OS running inside it, and then the program runs within that guest os. The http://erlangonxen.org/ http://erlangonxen.org/ folks made Ling (https://github.com/cloudozer/ling https://github.com/cloudozer/ling), software that enables an Erlang BEAM VM to be run directly on Xen, and thereby run a single Erlang program on Ling.
babuskov 8y ago
Not directly related to the topic, but I wrote a simple puzzle game that emulates a computer which executes programs directly in the memory and you can modify memory to rewrite running programs. Here's a web-based demo if anyone wants to check it out: https://roguebit.bigosaur.com/ https://roguebit.bigosaur.com/
vectorEQ 8y ago
i always feel these questions are answered to best effort of a person just starting out in the field of os / bare metal programming. there is a whole plethora of information missing in this post crucial to x86/64 early development which will get new programmers of this type into trouble if they don't become aware of it. i'd be careful following such advices. for example this person isn't aware you really need to build a cross toolchain for your OS to compile to your target. on 64-bit there's some more things to take into account about compilers (red-zone?). even if you go all assembly your assembler will probably be an optimising one and shoot you in the back on more than one occasion! Thats just the tool-chain surface issues (deeper ones ignored... like file formats..? binary is fun, but good luck scaling that up to a complex project... bios can't load you up that high in memory and limited amounts...) Next you get to things omitted which on baremetal would cause problems, like no mention of power management and such topics. or even assumptions that your code is loaded at 0x0000:0x7c00 while it could be loaded at that address but via a different CS value, completely screwing you over if you don't flush it immediately on your mbr... Take an example of opengl tutorials. people tell you to write your first polygon because they just managed to do so. They forget to mention to throttle the loop, so you burn out your VGA card rendering 10000000000fps of 1 triangle.... good job! this tutorials are kind of like that. fun to put into qemu which is sort of 'safe' learning and get acquainted but really doesn't do what it says on the box. I'm not against these type of tutorials, i like that more people become interested in this, but really take some time and state the scope of the offered information properly. like this is a bit of clickbait / misinformation which is a shame
emilfihlman 8y ago
>Technically, a program that runs without an OS, is an OS. It is not. A program that runs without an OS just doesn't need the OS provided services but instead manages everything itself. A perfectly valid program might be: #include <avr/io.h> int main(void) { DDRB=255; while(1) { PORTB++; } }
- vectorEQ 8y ago
  an OS is so much more than an application, amen. just a simple google would help: the low-level software that supports a computer's basic functions, such as scheduling tasks and controlling peripherals. this poor author of the post. >.> and then to think critisism gets a downvote :'D.. must be some angry kids mob who found where on memory mbr is put from some youtube1337hack0r.. The accepted answer was correct: "You place your binary code to a place where processor looks for after rebooting " That was a good anwer to the question. the rest is just filler nonsense.
- blitmap 8y ago
  If it's managing resources itself, how is it not an OS? What is art?
  - toxik 8y ago
    Programs manage resources. Operating systems explicitly manage resources for the benefit of other programs.
    - rhacker 8y ago
      I think that is the key notion. https://en.wikipedia.org/wiki/Operating_system https://en.wikipedia.org/wiki/Operating_system Handling resources isn't the definition of an OS. Also asking "what is art?" is not the same as asking "what is an operating system?" Art covers just about all human activity, the definition is extremely wide. I wouldn't say an OS has a definition that wide.
      - 8y ago
nmstoker 8y ago
Would be interesting to hear discussion on why one might consider doing this. My take is that it's most likely just being asked for the theoretical details, and few people would practically benefit, but does anyone think this would be practically useful? Presumably you'd potentially cut overhead from anything you didn't explicitly want happening on the host, but as soon as I start to think about most of the uses typically have, the advantages quickly sink below the disadvantages of having to actually manage everything. It's reminiscent of that person who set out to build a toaster from scratch.
- Tade0 8y ago
  One use case I can think of is heavy real time audio processing. In such an arrangement every millisecond of latency counts.
- mFixman 8y ago
  Because it's fun?
- Isamu 8y ago
  >does anyone think this would be practically useful? Most embedded software projects I worked on did not have an operating system. You'd just have C or C++ program and some ASM to initialize everything. I would expect really small IoT devices would be the same. With your average single chip device I wouldn't bother trying to get an operating system on there. I have only just started to use a Raspberry Pi and it does have an operating system - the advantages there are you can pull in IP networking and run services and do much more complex things than on the bare metal.
- farazzz 8y ago
  It’s good for learning, and firmware/embedded systems engineering exists too
- cirosantilli 8y ago
  If you want to understand and improve the Linux / some other OSs boot process.
lostgame 8y ago
A lot of people here seem to be questioning the usefulness of this, why someone would want to do this, et cetera. Embedded systems are one of the most extremely relevant uses of this today. However, historically, I think my first case of programming without an OS would be ASM development for the Sega Genesis? Another addition is that I saw a comment ‘a program that runs an OS, is an OS.’ This is absolutely not true. An Operating System is a multi-faceted, complex piece of software, usually with the capacity to run other pieces of software within it, as well as usually offering high-level API’s to improve ease-of-use for developers to access common functionality, such as playing sound, or displaying images. A ROM for the NES, while a program, is certainly not an OS, and to be very clear, the NES does not have an OS, like I believe the PS4 and XB1 have. It contains a certain number of hardware features that can be accessed and manipulated by low-level software functions.

394 points by Cieplak 8y ago

Saw a talk on Include OS that is a C++ library that allows one to do this http://www.includeos.org http://www.includeos.org

cirosantilli 8y ago
A similar effect can be achieved with Newlib, which is mentioned on the post, and exemplified at: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/54e15e04338c0fecc0be139a0da2d0d972c21419#baremetal-setup-getting-started https://github.com/cirosantilli/linux-kernel-module-cheat/tr... You might need to implement some syscalls though. Or similarly you can also use an open source embedded system like FreeRTOS or Zephyr which essentially implement the syscalls for you.
- cmroanirgo 8y ago
  That was a spectacular SO answer btw. Thanks for writing it!
- perbu 8y ago
  IncludeOS is modeled much like FreeRTOS or Zephyr. The difference is mainly that it addresses CPUs and not microcontrollers as the primary platform. With paging, you have better security, typically.
  - beached_whale 8y ago
    Along those lines, a bit, I have always wanted to play with targeting linux but making ones program the init process. All the resources but the benefit of Linux's ability to run on a lot of platforms. Not micro controller though, well arm, but that is pretty large.
    - cirosantilli 8y ago
      This is trivial with Buildroot btw, see e.g.: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/54e15e04338c0fecc0be139a0da2d0d972c21419#init https://github.com/cirosantilli/linux-kernel-module-cheat/tr... And a minimal hello world at: https://unix.stackexchange.com/questions/122717/how-to-create-a-custom-linux-distro-that-runs-just-one-program-and-nothing-else/238579#238579 https://unix.stackexchange.com/questions/122717/how-to-creat...

CoolGuySteve 8y ago

If you ever programmed a ROM for a Gameboy or something this should look familiar. The BIOS of those old systems just jumps right into a certain address and all you have to do is make sure your linker is setup properly to put crt0.s there. The Gameboy Advance in particular is pretty fun and simple, I recommend it.

shshhdhs 8y ago
Where should one go to get started with GBA dev work these days?
- raptorfactor 8y ago
  http://www.gbadev.org/ http://www.gbadev.org/
- deleted 8y ago
  [deleted]
- pathompong 8y ago
  http://www.loirak.com/gameboy/gbatutor.php http://www.loirak.com/gameboy/gbatutor.php
- GrantSolar 8y ago
  I've found this to be a good resource: https://www.coranac.com/tonc/text/ https://www.coranac.com/tonc/text/
jchw 8y ago
Something neat about the original Gameboy is how its firmware actually works. As a developer, the entire region from $0000-$3FFF is mapped to the cartridge, and the entrypoint is at $0100. When the CPU boots up, there is a special temporary mapping for $0000-$0100 to the firmware. The firmware largely just validates the ROM header and displays the Nintendo logo. But my favorite part is this: it executes all the way to $0100. There is no jump to $0100 - the last two bytes are an instruction that simply removes the firmware mapping. Now the IP lands on $0100 and everything is mapped as expected. This simple mechanism made it notoriously difficult to extract the Gameboy BIOS. I recall reading that one way it was eventually extracted was via fault injection: by messing with the clock very precisely, you could get the processor to 'skip' the instruction that unmapped the firmware, thus allowing you to dump it. Of course, another way you could read the firmware is by "simply" decapping the processor and taking shots of it with a powerful microscope, and then optically extracting the 'bits.' I think this was also done, but as a person with little understanding of the actual details of how a decapping is done it seems like an incredibly expensive, error prone operation.

everyone 8y ago

Prompted a fascinating answer. 'closed as too broad' -- modern Stack overflow in a nutshell.

TeMPOraL 8y ago
I wish there was a guide for "proper" SO questions; my impression is, the only ones are super-detailed questions about some super-particular APIs. Also interesting how this disconnect is still going on after so many years, mostly unaddressed - despite the many essays Jeff Atwood writes about true purpose of SO, people still want to use it differently.
- AndrewDucker 8y ago
  There is a guide here : https://stackoverflow.com/help/asking https://stackoverflow.com/help/asking
- CathyWest 8y ago
  > I wish there was a guide for "proper" SO questions; my impression is, the only ones are super-detailed questions about some super-particular APIs. And those questions can in most cases be answered by the docs anyway. The times I've used SO is when the problem can't be answered by the docs, my own experience or that of the team, but then I've received either no answer or a series of non-answers (that should have been comments instead, but a lot of people can't comment). I also see quite a bit of overt reputation farming there. Like vague questions that receive an improbably precise answer that immediately gets selected. Like "how do u process payments?" (the orthography is usually bad) immediately followed by very specific instructions from a very specific payment processor not mentioned in the question, probably taken straight from its documentation. And I rarely see "closed as too broad" on those, probably because of the quick turn-around.
yesenadam 8y ago
Yeah, I hate that. I've asked a few questions on those sites and came back later, excited to read what answers I got and...it had been closed immediately, before anyone could answer it. GRRRR! Maybe there could/should be a 'broad' SE site alongside every 'normal' one? The 'too broad' questions include most of the interesting ones to read. The ones that are 'intellectually interesting'.[0] It's a shame. I love how you can ask anything on HN without being shut down like that. It's so unfriendly, hurtful even. (At worst on here, no-one answers - no harm done.) Why they can't just let 'too broad' questions go for a day or two to see what responses come in, and delete the page if it sux, I don't know. The mods get points for closing things, I guess. Often they don't understand the question, it seems. [0] e.g. https://mathoverflow.net/questions/43690/whats-a-mathematician-to-do https://mathoverflow.net/questions/43690/whats-a-mathematici... Well, bizarrely the close notice on that says "This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet." Which seems absurd. (Although sure, the question doesn't help people, the answers do though.)

Arainach 8y ago

For another take on the subject, see "This OS is a bootloader" in PoC||GTFO 0x4: https://www.alchemistowl.org/pocorgtfo/ https://www.alchemistowl.org/pocorgtfo/ There's a sequel in 0x5 that explains how to add a basic form of multiprocessing support.

cirosantilli 8y ago
I have also covered baremetal multicore at: https://stackoverflow.com/questions/980999/what-does-multicore-assembly-language-look-like/33651438#33651438 https://stackoverflow.com/questions/980999/what-does-multico... The exemple is present in the same repo: https://github.com/cirosantilli/x86-bare-metal-examples/blob/3c35cabb8934d0dd92974f5826c8815572831bea/smp.S https://github.com/cirosantilli/x86-bare-metal-examples/blob...

deleted 8y ago

[deleted]

andrewstuart 8y ago

Stack Overflow is for very specific questions with very specific answers. You'll get better results for more general questions from a reddit forum.

cirosantilli 8y ago
What annoys me is that Reddit threads get locked after 6 months. Then people ask another one. And another one. I want to fix that with decent algorithms one day: https://github.com/cirosantilli/write-free-science-books-to-get-famous-website https://github.com/cirosantilli/write-free-science-books-to-...
sureaboutthis 8y ago
There are some fairly decent subs on reddit but, generally, I find it mostly populated by 15-year old kids and others who have never had a job in the industry so, no, I would not refer anyone to reddit for information.
- softawre 8y ago
  How can you tell their age? I've always though if a forum like this had a little age indicator (or experience indicator, or something like 'this person is smart') it would be a lot more useful.
  - sureaboutthis 8y ago
    It's strange that I can tell by the way they write. I'm not always correct but many times I'd find someone writing as if they were an expert but you could just tell they were only regurgitating something they read. They couldn't make a proper connection between things they were saying. I'd trick them in a way to reveal their age and way too often it was 15 to 17 years old. Sometimes 18 or 19. You can also read them saying they wanted to take CS when they graduated from school so you can glean from that they are probably in high school. Or I just flat out ask them (or accuse them).
    - 8y ago

justincormack 8y ago

The Osdev wiki is a great resource on this type of thing https://wiki.osdev.org/Expanded_Main_Page https://wiki.osdev.org/Expanded_Main_Page

cirosantilli 8y ago
I only wish they would create a repo with all their examples like I did :-)

stevefan1999 8y ago

This is wrong, you'd still have a operating system: BIOS

klez 8y ago
Maybe it's just my limited knowledge on the subject, but it seems to me it's a bit too much to consider the BIOS an operating system. At that point, what isn't an operating system? You're still writing code against a microcode interpreter even once you remove the BIOS. Is that an OS too?
- xxs 8y ago
  BIOS - is an abbreviation for "basic input/output system". System is in the name. BIOSes offer system calls that tend to be backward compatible, so it's an OS of sorts.
  - cyphar 8y ago
    UEFI is the successor to BIOS on modern machines, and it stands for "Universal Extensible Firmware Interface". Is it also an operating system (it doesn't have "system" in the name)? (I don't disagree with you -- UEFI provides BootServices which arguably are more numerous than the number of syscalls on most operating systems. Just that "system" being in the name isn't really the best argument. :P)
    - xxs 8y ago
      It was thought to be like that - back in the day when everything used interrupts for syscalls, hence 'system' was mentioned. (Apple II had fixed address calls, though - no interrupts) It's quite the same as DOS being 'disk operating system' - it was able to drive the floppy and provide a file system.
- da_chicken 8y ago
  I would agree. The BIOS/UEFI POSTs the system, and then passes execution to another program (usually a bootloader). At that point, the BIOS is no longer performing the tasks that an OS does. It's not scheduling tasks for the processor or handling I/O at all. Instead, it switches to what's basically an ACPI interface with functions that can be called by the OS. It's like an old TSR program.

finchisko 8y ago

Very nice post. However I thought it would be about writing programs that could be executed from OS, but OS would be completely unaware that such a program is running. Not sure if that is even possible. I know it’s possible to not use libc or similar, but os is aware of such a program.

cirosantilli 8y ago
It is not possible from userland, except for extremely serious zero days, due to hardware protections which OSes use, see: https://stackoverflow.com/questions/18717016/what-are-ring-0-and-ring-3-in-the-context-of-operating-systems/44483439#44483439 https://stackoverflow.com/questions/18717016/what-are-ring-0...
viraptor 8y ago
Kind of. Check out IBM's isostack. It takes CPU cores which are not used by the main os (you explicitly disable them) and essentially runs a parallel operating system which only handles the network cards / network stack. The main os cooperates with it, but otherwise it's not really aware of it.
pjc50 8y ago
A bit of thought about what it means to "run" gives you the answer here: in order for the code to run, it has to be scheduled at some point. That means it has to be in the operating system's task list somewhere. Having said that, there are places you can put running code that get scheduled but don't appear in an obvious way: inside other processes, as device drivers, as code on peripherals (GPU etc), or on the notorious management engine. Variations of these methods are used by viruses and exploits: the core of any exploit is getting code to run inside some other program that did not originally contain it. The final place is to put it "outside" the operating system as a hypervisor, then run the "normal" operating system as a seamless virtual machine inside the hypervisor.
0xcde4c3db 8y ago
I remember hearing that the PlayStation emulator bleem! did something like this for threading/timing purposes, essentially running its own microkernel and hooking interrupts to bypass the Windows scheduler. However, this supposedly worked by basically exploiting a bug in the way Windows 9x set up task switching and memory protection, and so didn't work on any NT-based Windows (or Wine, for that matter). Also, it obviously didn't try to hide the main program that managed the window and other I/O operations. Malware often has something akin to this as a goal, but as far as I know it usually works by hiding itself from the user rather than the kernel.

sureaboutthis 8y ago

People forget that an operating system is, itself, just a program.

JdeBP 8y ago
It is usually more than one program. * https://superuser.com/a/329479/38062 https://superuser.com/a/329479/38062

amelius 8y ago

Now try to do it on a smartphone.

0xTJ 8y ago
I really wish it were possible to do this kind of thing on smartphones. Unfortunately, odds are that anything you try will quickly brick it, and best-case scenario you can restore it using something like the low-level Broadcom interface

jancsika 8y ago

> printf '\364%509s\125\252' > main.img It appears that bash `printf` wraps coreutils printf "with ARGUMENTs converted to proper type first," according to the manual. I guess that's how the "%509s" specifier is able to work without any args. So how does bash `printf` actually determine the proper arg types? Is coreutils printf implemented with hooks that let wrappers fetch the proper arg types without generating an output string? Or does bash `printf` implement its own separate parser for the format specifier?

kevinoid 8y ago
> It appears that bash `printf` wraps coreutils printf "with ARGUMENTs converted to proper type first," according to the manual. I guess that's how the "%509s" specifier is able to work without any args. Actually, coreutils printf follows POSIX/SUS which states that "Any extra b, c, or s conversion specifiers shall be evaluated as if a null string argument were supplied".[1] So that trick will work for many printf(1) implementations (including both bash and coreutils). Bash printf(1) doesn't have any hooks or wrappers, but it does parse the format string to determine how to convert arguments before calling libc vsnprintf(3). If you are interested in the details, check out how %d/%i is handled in bash[2] and coreutils[3]. 1. http://pubs.opengroup.org/onlinepubs/9699919799/utilities/printf.html http://pubs.opengroup.org/onlinepubs/9699919799/utilities/pr... (item 9 under Extended Description) 2. https://git.savannah.gnu.org/cgit/bash.git/tree/builtins/printf.def?h=bash-4.4#n598 https://git.savannah.gnu.org/cgit/bash.git/tree/builtins/pri... 3. https://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;f=src/printf.c;hb=v8.30#l374 https://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;...
- jancsika 8y ago
  Thanks for the links, those are really helpful! Why didn't libc include a printf function that expands a format specifier with a pointer to a byte array (and/or possibly an array of c strings)? That way there would at least be a standard way of handling arbitrary input and erroring out with less risk of buffer overflows and stuff.
sfoley 8y ago
I don’t think bash’s `printf` “wraps” coreutils; pretty sure it just implements the entire call itself.

Communitivity 8y ago

For any "How do I...X?" question involving connecting two pieces of software together in a different way, the first thing I try to do is look at the API(s) that connects them. In the case of a program and the OS, it's the system calls, device drivers, and standard libraries. There have been efforts to provide the capability of running a program without an OS before, but any such effort is going to need to provide the system calls and standard libraries used by the program, and the infrastructure to support it (device drivers, management, etc.). That that point it becomes a mini-OS. An example is Erlang on Xen. Xen is more often run with a guest OS running inside it, and then the program runs within that guest os. The http://erlangonxen.org/ http://erlangonxen.org/ folks made Ling (https://github.com/cloudozer/ling https://github.com/cloudozer/ling), software that enables an Erlang BEAM VM to be run directly on Xen, and thereby run a single Erlang program on Ling.

babuskov 8y ago

Not directly related to the topic, but I wrote a simple puzzle game that emulates a computer which executes programs directly in the memory and you can modify memory to rewrite running programs. Here's a web-based demo if anyone wants to check it out: https://roguebit.bigosaur.com/ https://roguebit.bigosaur.com/

vectorEQ 8y ago

i always feel these questions are answered to best effort of a person just starting out in the field of os / bare metal programming. there is a whole plethora of information missing in this post crucial to x86/64 early development which will get new programmers of this type into trouble if they don't become aware of it. i'd be careful following such advices. for example this person isn't aware you really need to build a cross toolchain for your OS to compile to your target. on 64-bit there's some more things to take into account about compilers (red-zone?). even if you go all assembly your assembler will probably be an optimising one and shoot you in the back on more than one occasion! Thats just the tool-chain surface issues (deeper ones ignored... like file formats..? binary is fun, but good luck scaling that up to a complex project... bios can't load you up that high in memory and limited amounts...) Next you get to things omitted which on baremetal would cause problems, like no mention of power management and such topics. or even assumptions that your code is loaded at 0x0000:0x7c00 while it could be loaded at that address but via a different CS value, completely screwing you over if you don't flush it immediately on your mbr... Take an example of opengl tutorials. people tell you to write your first polygon because they just managed to do so. They forget to mention to throttle the loop, so you burn out your VGA card rendering 10000000000fps of 1 triangle.... good job! this tutorials are kind of like that. fun to put into qemu which is sort of 'safe' learning and get acquainted but really doesn't do what it says on the box. I'm not against these type of tutorials, i like that more people become interested in this, but really take some time and state the scope of the offered information properly. like this is a bit of clickbait / misinformation which is a shame

emilfihlman 8y ago

>Technically, a program that runs without an OS, is an OS. It is not. A program that runs without an OS just doesn't need the OS provided services but instead manages everything itself. A perfectly valid program might be: #include <avr/io.h> int main(void) { DDRB=255; while(1) { PORTB++; } }

vectorEQ 8y ago
an OS is so much more than an application, amen. just a simple google would help: the low-level software that supports a computer's basic functions, such as scheduling tasks and controlling peripherals. this poor author of the post. >.> and then to think critisism gets a downvote :'D.. must be some angry kids mob who found where on memory mbr is put from some youtube1337hack0r.. The accepted answer was correct: "You place your binary code to a place where processor looks for after rebooting " That was a good anwer to the question. the rest is just filler nonsense.
blitmap 8y ago
If it's managing resources itself, how is it not an OS? What is art?
- toxik 8y ago
  Programs manage resources. Operating systems explicitly manage resources for the benefit of other programs.
  - rhacker 8y ago
    I think that is the key notion. https://en.wikipedia.org/wiki/Operating_system https://en.wikipedia.org/wiki/Operating_system Handling resources isn't the definition of an OS. Also asking "what is art?" is not the same as asking "what is an operating system?" Art covers just about all human activity, the definition is extremely wide. I wouldn't say an OS has a definition that wide.
    - 8y ago

nmstoker 8y ago

Would be interesting to hear discussion on why one might consider doing this. My take is that it's most likely just being asked for the theoretical details, and few people would practically benefit, but does anyone think this would be practically useful? Presumably you'd potentially cut overhead from anything you didn't explicitly want happening on the host, but as soon as I start to think about most of the uses typically have, the advantages quickly sink below the disadvantages of having to actually manage everything. It's reminiscent of that person who set out to build a toaster from scratch.

Tade0 8y ago
One use case I can think of is heavy real time audio processing. In such an arrangement every millisecond of latency counts.
mFixman 8y ago
Because it's fun?
Isamu 8y ago
>does anyone think this would be practically useful? Most embedded software projects I worked on did not have an operating system. You'd just have C or C++ program and some ASM to initialize everything. I would expect really small IoT devices would be the same. With your average single chip device I wouldn't bother trying to get an operating system on there. I have only just started to use a Raspberry Pi and it does have an operating system - the advantages there are you can pull in IP networking and run services and do much more complex things than on the bare metal.
farazzz 8y ago
It’s good for learning, and firmware/embedded systems engineering exists too
cirosantilli 8y ago
If you want to understand and improve the Linux / some other OSs boot process.

lostgame 8y ago

A lot of people here seem to be questioning the usefulness of this, why someone would want to do this, et cetera. Embedded systems are one of the most extremely relevant uses of this today. However, historically, I think my first case of programming without an OS would be ASM development for the Sega Genesis? Another addition is that I saw a comment ‘a program that runs an OS, is an OS.’ This is absolutely not true. An Operating System is a multi-faceted, complex piece of software, usually with the capacity to run other pieces of software within it, as well as usually offering high-level API’s to improve ease-of-use for developers to access common functionality, such as playing sound, or displaying images. A ROM for the NES, while a program, is certainly not an OS, and to be very clear, the NES does not have an OS, like I believe the PS4 and XB1 have. It contains a certain number of hardware features that can be accessed and manipulated by low-level software functions.