6 ms·

Manually creating an ELF executable

79 points by muoncf 14y ago

kunai 14y ago
So essentially, this is writing an application in machine code? This is so fascinating I'm starting to get palpitations. I've always wanted to know how to program in machine code. I've looked through the web countless times in search of something like this, and haven't ever found it, until now. Thanks for the great insight. This kind of stuff makes C/C++ look like stuff for total newbies. Getting this close to the hardware really is quite mind-boggling.
- 0x0 14y ago
  Here's another article which starts with a .c program and refines it through a series of improvements to end up with a very tiny elf executable: http://www.muppetlabs.com/~breadbox/software/tiny/teensy.html http://www.muppetlabs.com/~breadbox/software/tiny/teensy.htm... Manually picking out hex opcodes is maybe a bit over the top. A nice assembler like nasm should be "low level enough" while still providing a lot more convenience and flexibility.
  - kunai 14y ago
    Well of course, nobody writes any applications in hex; it's more of a proof of concept, but a very interesting proof of concept. Asm, at least for me, is still too low-level for me to do any sort of serious work. I make desktop/mobile apps, and don't really do any sort of kernel hacking, so currently, I'll stick with C++ and Java. It is a nice thing to know, however, I don't see myself learning it right now. It's inevitable that I'll have to learn it sometime in the future if I ever want to broaden my prospects, but it's nice to play around with, but isn't all too useful for me.
- robinh 14y ago
  Hi, I'm the author. In this particular example, I basically skipped the compiler and linker to create an executable directly, yes. It's certainly a lot of fun, but if you're really trying to become proficient at programming in assembly language, I would recommend reading Assembly Language Step-by-Step: Programming with Linux.
  - kunai 14y ago
    Thanks. I'll be sure to check it out when I get into programming asm.
- Xcelerate 14y ago
  It is pretty fun. If you have a TI-83 or TI-84 calculator you can do the same thing a little bit easier. In high school, I would print out 2 pages of all the Z80 opcodes and put them on my desk during class. The calculators let you directly input hex and then run your program with asm(), so I would write my program by hand on paper, and then next to each line translate it to machine code and put it into the calculator. A lot of TI routines are easily accessible with bcalls, so you don't have to reinvent the wheel with each program. Plus Z80 is a lot easier than x86. (I still remember the return code: C9) I had a lot of fun taking friend's calculators and disabling the LCD and power button in assembly :)
  - kunai 14y ago
    Neat. I still have my old TI-83 lying around in my cluttered desk somewhere, I'll be sure to try this when/if I find it.
- femto 14y ago
  > I've always wanted to know how to program in machine code. 1) Decide which processor you want to program. I'm going to use a Z80, since that's one I'm familiar with. 2) Get hold of a good book, making sure it includes a section that exhaustively describes each instruction and the machine code for each. Eg. "Programming the Z80" by Rodnay Zaks. The microprocessor's datasheet will generally do if your familiar with other processors. 3) Read the book, cover to cover. 4) Make three vertical columns on a sheet of paper. 5) Write your program, in assembly language, in the middle column. Use alphanumeric labels (label means variable name) for all addresses. 6) Decide at what address you want your program to start (the origin) 7) Write the numeric address in the left hand column, next to the first instruction. 8) Look up the machine code, for the instruction in the middle column, in the Zaks book. 9) Write the machine code in the right hand column. 10) Add the length of the instruction (# bytes) that you have just written down to the address, and write the answer on the next row of the left hand column. You might have to leave labels in place for instructions that refer to parts of the program that you have not yet assembled. 11) Repeat from step 8, until the entire program has been assembled. 12) Go back and fill in the numerical addresses for any addresses that are still labels. 13) Now you have to get the program in memory. We're going to assume that a programming language, such as BASIC is available. The alternative is a keypad with address/data entry functions and a method to get an initial value into the program counter. 14) Create a big array, in BASIC, with all the numbers from the right hand column. 15) Write a loop which POKES each numeric value into memory, starting from the origin address. 16) Transfer execution to the origin address, using the a USR(nnnn) instruction, where nnnn is the origin address. 17) Hopefully your program works, but if not, make changes and go back to step 4 until your program works. Enjoy! In time, the process gets quicker, as you eventually remember the machine code for the the most common instructions and don't have to refer to the book. If you're writing timing critical code, have a fourth column, in which you record the number of clock cycles for each instruction to execute, and sum those numbers to determine execution time. If you have a target execution time, you will need to add/remove instructions to achieve the target.
  - ganley 14y ago
    If I had a nickel for every hour I spent doing this as a teen... (Sinclair ZX-81, then TRS-80 Model 1.) Also, that Zaks book rocks. To this day one of the best programming references I've ever used. http://www.z80.info/zaks.html http://www.z80.info/zaks.html
- pjmlp 14y ago
  We did it all the time in the early eighties, with machine language monitors.
Turing_Machine 14y ago
We used to do it with toggle switches, back in the day (long before ELF binaries existed, though).
- groby_b 14y ago
  Well, only the boot loader. You could use paper tape after you had that keyed in :)
  - femto 14y ago
    My favourite boot-loader story was told to me by one of the old-hands on the Jindalee Radar, when it was still a research project. Jindalee had a manned receiver station and a remote transmitter station, about 100km away. They were connected via a radio link, and to start with, the PDP at the remote site had to be booted by keying in the boot-loader over the radio link. If you lost the radio link, you had to start again...and again... and again...
ChuckMcM 14y ago
One of the cool things about "personal computers", when they were not synonymous with "IBM PC" or x86 box, was that you had total control over every bit and byte that went into them. Sometimes you had to use switches to manually set memory contents, sometimes there was a hard coded 'boot loader' which would do that for you. The coolest demo I think I've seen in this space was an FPGA computer design where the first download into the FPGA made it a serial device that could write memory, and then you erased that and loaded a computer which would then read that memory and execute it. I believe that one of the reasons the Arduino has the appeal it does is because you can know everything there is to know about the program it is running. That is a lot of fun for a technologist. It would be fun to have this blog posting written from within a debugger. I always found it intriguing to 'debug a program into existence' as it were.
- kunai 14y ago
  It's kind of the same deal now with the Raspberry Pi, except for the fact that it uses RISC and not x86. It's a great tinkering platform. Now, if only somebody would make a ring-0 OS for ARM... Now THAT would be fun.
  - pjmlp 14y ago
    You mean like Active Oberon? http://www.ocp.inf.ethz.ch/wiki/ http://www.ocp.inf.ethz.ch/wiki/ Not sure how workable this version is, but you can check on the forums, ftp://ftp.lrz.de/transfer/BlueBottle-A2/ARMRelease.zip You just need to enjoy using Pascal syntax languages for system programming.
  - nitrogen 14y ago
    There is FreeRTOS (http://www.freertos.org/ http://www.freertos.org/).
deleted 14y ago
[deleted]
ObnoxiousJul 14y ago
I am voiceless: why would the f*ck would someone loose such that much time on a trivial, yet annoying task? That is the first lesson you have in assembly language: program in assembly language, then in C because it spares you the time (and the mistakes involved in the process) to do such a thing. Thousands of men year efforts and wisdom ruined in a post, showing how people have too much time on their hands, and so few imaginations (well he could have written something interesting on how to display p0rn pictures in ASCII art at least).
- daeken 14y ago
  Having spent my years climbing up and down the ladder of abstraction, I think your analysis is way off the mark. This is a great way to learn how ELF works (and binaries in general), learn how machine code is formed, etc. Is it an actually 'useful' task, that produces a useful end product? Yes: knowledge. Knowledge that isn't easy to acquire. You won't ever do this in the Real World (TM), but it's a fantastic way to get into things.
  - ObnoxiousJul 14y ago
    I studied physics, and I learnt actual programming and assembly language: I actually did this in the real world! What do you learn in Computer Science then?! Gibbering useless concepts that makes you non-sensical experts in the field of not delivering your software neither in time, nor in the frame of the specification? Or just
    - milkshakes 14y ago
      in the unlikely case that your question is sincere, this article on wikipedia touches lightly upon some of daeken's "real world" contributions: https://en.wikipedia.org/wiki/Cody_Brocious https://en.wikipedia.org/wiki/Cody_Brocious
    - adamnemecek 14y ago
      Why exactly does this article make you so enraged?
kragen 14y ago
If you liked this, you'll probably like even more "A Whirlwind Tutorial on Creating Really Teensy ELF Executables for Linux" (http://www.muppetlabs.com/~breadbox/software/tiny/teensy.html http://www.muppetlabs.com/~breadbox/software/tiny/teensy.htm...). Last time I checked, the teensy executable had stopped working around Linux 2.6.26 or 2.6.27, and I don't know why. Relatedly, I wrote a tiny quasi-Forth in itself that generates ELF executables directly at https://github.com/kragen/stoneknifeforth https://github.com/kragen/stoneknifeforth. Its executables also stopped running around Linux 2.6.27, and I still haven't bothered to figure out why.
- robinh 14y ago
  Agreed. I would heartily recommend reading the article you linked to, I certainly liked it a lot. The project you've worked on looks interesting, too! It's a little late for me to be reading too much code, but I'll make sure to take a good look at it tomorrow. Thanks for the link.
  - kragen 14y ago
    I hope you enjoy it!
- caf 14y ago
  If you'd reported that at the time as a regression, then whatever change had stopped them working would most likely have been reverted. They take binary backwards-compatibility pretty seriously.
  - kragen 14y ago
    StoneKnifeForth is a toy, and the executables it generates were probably actually invalid in some way I don't understand because I haven't read the ELF spec. Preventing the kernel from barfing on them would have an insignificant real benefit, I think, and possibly a significant cost. If we were talking about PulseAudio or something, I would agree, but no. The obvious first step to reporting the bug (after looking at diffs, which I did try) would be to instrument the ELF loading code with printk calls to explain why it's failing to exec a supposedly ELF executable, so that I could report what it was about the executable it was that made the kernel barf. And then I could fix the executable. I just haven't gotten around to recompiling my kernel with a modified ELF loader.
drivers99 14y ago
This is great. I've had low-level stuff on my mind lately. I was thinking about where to start, either writing my own binary to the boot sector of a disk, or booting up MS-DOS and going from there, or from linux or windows and starting with machine code from there. So this would be perfect. (I was thinking of maybe implementing FORTH. Interesting that kragen mentions a "tiny quasi-Forth" that he made.)
Joeboy 14y ago
At some point soonish I intend to have a go at loading dynamic libraries in bare metal on the Raspberry Pi. I'll be on the lookout for stuff like this for .so's. Suggestions welcome.

79 points by muoncf 14y ago

kunai 14y ago

So essentially, this is writing an application in machine code? This is so fascinating I'm starting to get palpitations. I've always wanted to know how to program in machine code. I've looked through the web countless times in search of something like this, and haven't ever found it, until now. Thanks for the great insight. This kind of stuff makes C/C++ look like stuff for total newbies. Getting this close to the hardware really is quite mind-boggling.

0x0 14y ago
Here's another article which starts with a .c program and refines it through a series of improvements to end up with a very tiny elf executable: http://www.muppetlabs.com/~breadbox/software/tiny/teensy.html http://www.muppetlabs.com/~breadbox/software/tiny/teensy.htm... Manually picking out hex opcodes is maybe a bit over the top. A nice assembler like nasm should be "low level enough" while still providing a lot more convenience and flexibility.
- kunai 14y ago
  Well of course, nobody writes any applications in hex; it's more of a proof of concept, but a very interesting proof of concept. Asm, at least for me, is still too low-level for me to do any sort of serious work. I make desktop/mobile apps, and don't really do any sort of kernel hacking, so currently, I'll stick with C++ and Java. It is a nice thing to know, however, I don't see myself learning it right now. It's inevitable that I'll have to learn it sometime in the future if I ever want to broaden my prospects, but it's nice to play around with, but isn't all too useful for me.
robinh 14y ago
Hi, I'm the author. In this particular example, I basically skipped the compiler and linker to create an executable directly, yes. It's certainly a lot of fun, but if you're really trying to become proficient at programming in assembly language, I would recommend reading Assembly Language Step-by-Step: Programming with Linux.
- kunai 14y ago
  Thanks. I'll be sure to check it out when I get into programming asm.
Xcelerate 14y ago
It is pretty fun. If you have a TI-83 or TI-84 calculator you can do the same thing a little bit easier. In high school, I would print out 2 pages of all the Z80 opcodes and put them on my desk during class. The calculators let you directly input hex and then run your program with asm(), so I would write my program by hand on paper, and then next to each line translate it to machine code and put it into the calculator. A lot of TI routines are easily accessible with bcalls, so you don't have to reinvent the wheel with each program. Plus Z80 is a lot easier than x86. (I still remember the return code: C9) I had a lot of fun taking friend's calculators and disabling the LCD and power button in assembly :)
- kunai 14y ago
  Neat. I still have my old TI-83 lying around in my cluttered desk somewhere, I'll be sure to try this when/if I find it.
femto 14y ago
> I've always wanted to know how to program in machine code. 1) Decide which processor you want to program. I'm going to use a Z80, since that's one I'm familiar with. 2) Get hold of a good book, making sure it includes a section that exhaustively describes each instruction and the machine code for each. Eg. "Programming the Z80" by Rodnay Zaks. The microprocessor's datasheet will generally do if your familiar with other processors. 3) Read the book, cover to cover. 4) Make three vertical columns on a sheet of paper. 5) Write your program, in assembly language, in the middle column. Use alphanumeric labels (label means variable name) for all addresses. 6) Decide at what address you want your program to start (the origin) 7) Write the numeric address in the left hand column, next to the first instruction. 8) Look up the machine code, for the instruction in the middle column, in the Zaks book. 9) Write the machine code in the right hand column. 10) Add the length of the instruction (# bytes) that you have just written down to the address, and write the answer on the next row of the left hand column. You might have to leave labels in place for instructions that refer to parts of the program that you have not yet assembled. 11) Repeat from step 8, until the entire program has been assembled. 12) Go back and fill in the numerical addresses for any addresses that are still labels. 13) Now you have to get the program in memory. We're going to assume that a programming language, such as BASIC is available. The alternative is a keypad with address/data entry functions and a method to get an initial value into the program counter. 14) Create a big array, in BASIC, with all the numbers from the right hand column. 15) Write a loop which POKES each numeric value into memory, starting from the origin address. 16) Transfer execution to the origin address, using the a USR(nnnn) instruction, where nnnn is the origin address. 17) Hopefully your program works, but if not, make changes and go back to step 4 until your program works. Enjoy! In time, the process gets quicker, as you eventually remember the machine code for the the most common instructions and don't have to refer to the book. If you're writing timing critical code, have a fourth column, in which you record the number of clock cycles for each instruction to execute, and sum those numbers to determine execution time. If you have a target execution time, you will need to add/remove instructions to achieve the target.
- ganley 14y ago
  If I had a nickel for every hour I spent doing this as a teen... (Sinclair ZX-81, then TRS-80 Model 1.) Also, that Zaks book rocks. To this day one of the best programming references I've ever used. http://www.z80.info/zaks.html http://www.z80.info/zaks.html
pjmlp 14y ago
We did it all the time in the early eighties, with machine language monitors.

Turing_Machine 14y ago

We used to do it with toggle switches, back in the day (long before ELF binaries existed, though).

groby_b 14y ago
Well, only the boot loader. You could use paper tape after you had that keyed in :)
- femto 14y ago
  My favourite boot-loader story was told to me by one of the old-hands on the Jindalee Radar, when it was still a research project. Jindalee had a manned receiver station and a remote transmitter station, about 100km away. They were connected via a radio link, and to start with, the PDP at the remote site had to be booted by keying in the boot-loader over the radio link. If you lost the radio link, you had to start again...and again... and again...

ChuckMcM 14y ago

One of the cool things about "personal computers", when they were not synonymous with "IBM PC" or x86 box, was that you had total control over every bit and byte that went into them. Sometimes you had to use switches to manually set memory contents, sometimes there was a hard coded 'boot loader' which would do that for you. The coolest demo I think I've seen in this space was an FPGA computer design where the first download into the FPGA made it a serial device that could write memory, and then you erased that and loaded a computer which would then read that memory and execute it. I believe that one of the reasons the Arduino has the appeal it does is because you can know everything there is to know about the program it is running. That is a lot of fun for a technologist. It would be fun to have this blog posting written from within a debugger. I always found it intriguing to 'debug a program into existence' as it were.

kunai 14y ago
It's kind of the same deal now with the Raspberry Pi, except for the fact that it uses RISC and not x86. It's a great tinkering platform. Now, if only somebody would make a ring-0 OS for ARM... Now THAT would be fun.
- pjmlp 14y ago
  You mean like Active Oberon? http://www.ocp.inf.ethz.ch/wiki/ http://www.ocp.inf.ethz.ch/wiki/ Not sure how workable this version is, but you can check on the forums, ftp://ftp.lrz.de/transfer/BlueBottle-A2/ARMRelease.zip You just need to enjoy using Pascal syntax languages for system programming.
- nitrogen 14y ago
  There is FreeRTOS (http://www.freertos.org/ http://www.freertos.org/).

deleted 14y ago

[deleted]

ObnoxiousJul 14y ago

I am voiceless: why would the f*ck would someone loose such that much time on a trivial, yet annoying task? That is the first lesson you have in assembly language: program in assembly language, then in C because it spares you the time (and the mistakes involved in the process) to do such a thing. Thousands of men year efforts and wisdom ruined in a post, showing how people have too much time on their hands, and so few imaginations (well he could have written something interesting on how to display p0rn pictures in ASCII art at least).

daeken 14y ago
Having spent my years climbing up and down the ladder of abstraction, I think your analysis is way off the mark. This is a great way to learn how ELF works (and binaries in general), learn how machine code is formed, etc. Is it an actually 'useful' task, that produces a useful end product? Yes: knowledge. Knowledge that isn't easy to acquire. You won't ever do this in the Real World (TM), but it's a fantastic way to get into things.
- ObnoxiousJul 14y ago
  I studied physics, and I learnt actual programming and assembly language: I actually did this in the real world! What do you learn in Computer Science then?! Gibbering useless concepts that makes you non-sensical experts in the field of not delivering your software neither in time, nor in the frame of the specification? Or just
  - milkshakes 14y ago
    in the unlikely case that your question is sincere, this article on wikipedia touches lightly upon some of daeken's "real world" contributions: https://en.wikipedia.org/wiki/Cody_Brocious https://en.wikipedia.org/wiki/Cody_Brocious
  - adamnemecek 14y ago
    Why exactly does this article make you so enraged?

kragen 14y ago

If you liked this, you'll probably like even more "A Whirlwind Tutorial on Creating Really Teensy ELF Executables for Linux" (http://www.muppetlabs.com/~breadbox/software/tiny/teensy.html http://www.muppetlabs.com/~breadbox/software/tiny/teensy.htm...). Last time I checked, the teensy executable had stopped working around Linux 2.6.26 or 2.6.27, and I don't know why. Relatedly, I wrote a tiny quasi-Forth in itself that generates ELF executables directly at https://github.com/kragen/stoneknifeforth https://github.com/kragen/stoneknifeforth. Its executables also stopped running around Linux 2.6.27, and I still haven't bothered to figure out why.

robinh 14y ago
Agreed. I would heartily recommend reading the article you linked to, I certainly liked it a lot. The project you've worked on looks interesting, too! It's a little late for me to be reading too much code, but I'll make sure to take a good look at it tomorrow. Thanks for the link.
- kragen 14y ago
  I hope you enjoy it!
caf 14y ago
If you'd reported that at the time as a regression, then whatever change had stopped them working would most likely have been reverted. They take binary backwards-compatibility pretty seriously.
- kragen 14y ago
  StoneKnifeForth is a toy, and the executables it generates were probably actually invalid in some way I don't understand because I haven't read the ELF spec. Preventing the kernel from barfing on them would have an insignificant real benefit, I think, and possibly a significant cost. If we were talking about PulseAudio or something, I would agree, but no. The obvious first step to reporting the bug (after looking at diffs, which I did try) would be to instrument the ELF loading code with printk calls to explain why it's failing to exec a supposedly ELF executable, so that I could report what it was about the executable it was that made the kernel barf. And then I could fix the executable. I just haven't gotten around to recompiling my kernel with a modified ELF loader.

drivers99 14y ago

This is great. I've had low-level stuff on my mind lately. I was thinking about where to start, either writing my own binary to the boot sector of a disk, or booting up MS-DOS and going from there, or from linux or windows and starting with machine code from there. So this would be perfect. (I was thinking of maybe implementing FORTH. Interesting that kragen mentions a "tiny quasi-Forth" that he made.)

Joeboy 14y ago

At some point soonish I intend to have a go at loading dynamic libraries in bare metal on the Raspberry Pi. I'll be on the lookout for stuff like this for .so's. Suggestions welcome.