5 ms·

Is there is there a good deep dive explaining how the actual portable executable works? The original blog post is kind of light on the details.

grimgrin 4y ago
Not _exactly_ what you're asking for, because certainly you're looking for a write-up/blog, however, if you haven't peeked at ape.S you absolutely must: https://github.com/jart/cosmopolitan/blob/master/ape/ape.S https://github.com/jart/cosmopolitan/blob/master/ape/ape.S It is very commented, in a good way.
- jart 4y ago
  Don't forget the linker script: https://github.com/jart/cosmopolitan/blob/master/ape/ape.lds https://github.com/jart/cosmopolitan/blob/master/ape/ape.lds It has a longform explanation in the comment at the top.
IshKebab 4y ago
Sadly not. Everything I've read assumes you are already very familiar with the obsolete executable formats it uses (COM, 6th edition Unix shell scripts). I presume it uses tricks similar to those files that are both ZIPs and PDFs simultaneously or whatever - basically they've chosen formats where the parser starts by reading different bits of the file. But that's an educated guess because I don't think there's a clear explanation anywhere.
- jart 4y ago
  It can't be easily explained. There's no school for it. You have to learn by fiddling around. Try this "Tiny Portable ELF" tutorial I wrote a few months ago: https://justine.lol/sizetricks/#elf https://justine.lol/sizetricks/#elf It shows you how to build your own executable from scratch that runs on four UNIX systems. If you build that code and try to improve upon it, then that should give you a pretty rock solid intuition and background for appreciating APE, since it takes that to the next level, by supporting Mac and Windows too.
  - IshKebab 4y ago
    It can definitely be explained! For instant your tiny ELF tutorial is currently missing any explanation at all - it says what you're trying to do, and a dump of code to do it, and a demo of it working but nothing about why it works. But it's easy to add. From skimming the code I think the explanation should be something like this: All four operating systems support the ELF executable format, and we're using the same x86 hardware in all four cases. When you compile a function like `int fib(...` the exact same x86 instructions end up running on all OSes. So why doesn't a Linux binary run out of the box on OpenBSD? There are a few reasons. First, the ELF format itself requires us to specify the OS ABI (conventions for how to functions at a binary/register level). Fortunately we can get around this by just setting it to the FreeBSD ABI because (insert explanation here). There are a few other minor differences in how the OSes interpret ELF fields, like (insert explanation here) But they are broadly compatible and the following ELF header will allow us to start executing x86 instructions on all four OSes. At that point we can run our `fib()` code and it will work! But the second issue occurs when we actually want to print the result. The print a string on all four OSes you can use their `write()` syscalls to tell the kernel to write a string to stdout (file descriptor 1 on all the OSes). Unfortunately although they all have the same ABI for making syscalls (maybe this is standard on x86??) they gave the write syscall different ID numbers - on Linux it is 1, but on the BSDs it is 4. If we were to try to execute Linux's `write()` on BSD it would actually call `exit()` - not what we want! So we need some way of detecting the OS at runtime. There are probably many ways to do this but I have done it by detecting differences in how the OSes set up the registers and program arguments before calling `start()`. Specifically... (insert explanation here). Totally doable! I'm sure some of that is wrong, but hopefully you get the point. It definitely is harder to explain something when you are an expert on it, but it's still possible. Probably the best way to do it is to work with someone who is interested but doesn't understand it and then they'll ask questions that you assumed they new the answer to. Anyway I'm not saying you need do this - it does take some time. I just wanted to impart that it could be explained if you wanted to.
    - jart 4y ago
      Please take a look. https://justine.lol/sizetricks/#elf https://justine.lol/sizetricks/#elf

grimgrin 4y ago

Not _exactly_ what you're asking for, because certainly you're looking for a write-up/blog, however, if you haven't peeked at ape.S you absolutely must: https://github.com/jart/cosmopolitan/blob/master/ape/ape.S https://github.com/jart/cosmopolitan/blob/master/ape/ape.S It is very commented, in a good way.

jart 4y ago
Don't forget the linker script: https://github.com/jart/cosmopolitan/blob/master/ape/ape.lds https://github.com/jart/cosmopolitan/blob/master/ape/ape.lds It has a longform explanation in the comment at the top.

IshKebab 4y ago

Sadly not. Everything I've read assumes you are already very familiar with the obsolete executable formats it uses (COM, 6th edition Unix shell scripts). I presume it uses tricks similar to those files that are both ZIPs and PDFs simultaneously or whatever - basically they've chosen formats where the parser starts by reading different bits of the file. But that's an educated guess because I don't think there's a clear explanation anywhere.

jart 4y ago
It can't be easily explained. There's no school for it. You have to learn by fiddling around. Try this "Tiny Portable ELF" tutorial I wrote a few months ago: https://justine.lol/sizetricks/#elf https://justine.lol/sizetricks/#elf It shows you how to build your own executable from scratch that runs on four UNIX systems. If you build that code and try to improve upon it, then that should give you a pretty rock solid intuition and background for appreciating APE, since it takes that to the next level, by supporting Mac and Windows too.
- IshKebab 4y ago
  It can definitely be explained! For instant your tiny ELF tutorial is currently missing any explanation at all - it says what you're trying to do, and a dump of code to do it, and a demo of it working but nothing about why it works. But it's easy to add. From skimming the code I think the explanation should be something like this: All four operating systems support the ELF executable format, and we're using the same x86 hardware in all four cases. When you compile a function like `int fib(...` the exact same x86 instructions end up running on all OSes. So why doesn't a Linux binary run out of the box on OpenBSD? There are a few reasons. First, the ELF format itself requires us to specify the OS ABI (conventions for how to functions at a binary/register level). Fortunately we can get around this by just setting it to the FreeBSD ABI because (insert explanation here). There are a few other minor differences in how the OSes interpret ELF fields, like (insert explanation here) But they are broadly compatible and the following ELF header will allow us to start executing x86 instructions on all four OSes. At that point we can run our `fib()` code and it will work! But the second issue occurs when we actually want to print the result. The print a string on all four OSes you can use their `write()` syscalls to tell the kernel to write a string to stdout (file descriptor 1 on all the OSes). Unfortunately although they all have the same ABI for making syscalls (maybe this is standard on x86??) they gave the write syscall different ID numbers - on Linux it is 1, but on the BSDs it is 4. If we were to try to execute Linux's `write()` on BSD it would actually call `exit()` - not what we want! So we need some way of detecting the OS at runtime. There are probably many ways to do this but I have done it by detecting differences in how the OSes set up the registers and program arguments before calling `start()`. Specifically... (insert explanation here). Totally doable! I'm sure some of that is wrong, but hopefully you get the point. It definitely is harder to explain something when you are an expert on it, but it's still possible. Probably the best way to do it is to work with someone who is interested but doesn't understand it and then they'll ask questions that you assumed they new the answer to. Anyway I'm not saying you need do this - it does take some time. I just wanted to impart that it could be explained if you wanted to.
  - jart 4y ago
    Please take a look. https://justine.lol/sizetricks/#elf https://justine.lol/sizetricks/#elf