9 ms·
How is a binary executable organized? Let's explore it (2014)
- EmilioPeJu 3y agoA book about this topic which I enjoyed is "Learning Linux Binary Analysis" by Ryan O'neil.
- EmilioPeJu 3y agoIf you are curious about how small a ELF binary file can be, you might like the following amusing article: https://www.muppetlabs.com/~breadbox/software/tiny/teensy.html https://www.muppetlabs.com/~breadbox/software/tiny/teensy.ht...
- jstrieb 3y agoAs I've said in other threads https://news.ycombinator.com/item?id=38847750#38862450 https://news.ycombinator.com/item?id=38847750#38862450, I highly recommend writing an ELF by hand at least once. It's a great exercise to understand the basic parts of an executable. It's also helpful if you want to go the opposite direction of this article - bottom-up instead of top-down. Lots of other great discussion in various threads on that other HN post.
- Retr0id 3y agoSimilarly, I'd recommend writing a simple ELF loader. There's a fair bit of implementation complexity in dynamic linking, but if you only support static ELFs then it's straight-forward.
- mkoubaa 3y agoI've seriously considered writing an ELF loader that uses a special symbol (like _resolve) where dynamic library resolving is done imperatively. The flexibility from libdl always feel underwhelming.
- Isamu 3y agoYes, likewise I wrote a reader that simply tried to parse every bit of a complex ELF binary to report its structure and quickly found myself in poorly documented territory. It’s an education if you want it.
- eichin 3y agoI assume there's a better modern source (assuming for some reason you don't want to reference libbfd &c, but really, you at least want to cross-check with it) but BITD there was an AT&T System V book - I think it was https://books.google.com/books?id=mrImAAAAMAAJ https://books.google.com/books?id=mrImAAAAMAAJ but only at about 70% confidence, it's part of a series and might have been one of the others - that was "arcane but true" for ELF on existing platforms (at the time, which was the mid 1990s, which is why I hope there's a better starting point now...)
- matheusmoreira 3y agoAny good resources on the matter? I'm gonna need to write a fully featured ELF loader for my language soon. I need to prepare.
- toast0 3y agoI had written a static ELF loader for reasons, but when I was no longer to compile a static version of the binary I wanted to load, I found it wasn't too hard to load the system's dynamic loader instead. That's kind of the best of both worlds --- I can run a dynamicly linked binary, and I didn't have to do the linking and relocations.
- akdas 3y agoWriting an ELF file by hand is something I did recently: https://github.com/avik-das/garlic/blob/master/recursive/elf-exploration/write-elf.rb https://github.com/avik-das/garlic/blob/master/recursive/elf... To explain the format to myself and others, I also created an interactive visualization for the bytes in the file. It helps me to click on a byte, see an explanation for it and see related bytes elsewhere in the file highlighted. https://scratchpad.avikdas.com/elf-explanation/elf-explanation.html https://scratchpad.avikdas.com/elf-explanation/elf-explanati...
- bruce343434 3y agoThat's an awesome interactive page! Did you write it by hand, or did you use some sort of generator/tool?
- sunk1st 3y agoI agree it’s very nice. I’d also like to know how it was done. Also if you click two bytes that are in the same caption group one after another then it bugs out.
- akdas 3y agoThanks for the feedback. I replied in a sibling comment about how I made it. For the bug, feel free to email me at avik at avikdas dot com if you'd like. The behavior I verified just now (for me) is that if you click one byte to highlight it, then clicking any other byte in the same group will remove the highlighting.
- akdas 3y agoI wrote it all by hand :) Lately, I've been using Svelte for interactive visualizations (see my post on using a tool called Astro with Svelte: https://avikdas.com/2023/12/30/interactive-demos-using-astro.html https://avikdas.com/2023/12/30/interactive-demos-using-astro... ). But this one is all hand-written JS!
- 3y ago
- mito88 3y agois ELF unique to x86/x64?
- inkyoto 3y agoIt is not, and it has never been. ELF is platform agnostic, and has been used in operating systems on nearly every existing CPU platform since mid 90's (with a few notable exceptions being OS X, AIX, the embedded world and Windows).
- ddingus 3y agoSeconded. Doing it is basically a hand assembly. One reads the documentation, selects the bytes needed using a processor data sheet, orders them into the various sections, populates the ELF fields and then it really does boil down to typing them all in. Pre-ELF times, on say an 8 bit Apple 2, the machine code monitor, allowed input of the program bytes directly. Those are then executed. Storing to disk is only a bit more involved, and there is another opportunity! Disk sector editors allow one to create a file... ...and so it goes!
- matheusmoreira 3y agoModifying existing ELFs can also be extremely educational and fun. It's a bit frustrating at first because it's more or less impossible to debug this stuff when it doesn't work but when things finally start working it's awesome. Turns out it's possible to patch ELFs in all sorts of interesting ways. With the auxiliary vector you can even have introspection at runtime: Linux gives us the address of the program header table and from there you can get to anywhere. Just gotta extend a LOAD segment to cover the whole binary. For example I wrote tools to embed lisp modules and code right into my lisp's interpreter executable. The embedded segment is loaded from the ELF automatically, the interpreter just finds it and runs the code. I'm so proud of this little feature I wrote an article about it. https://www.matheusmoreira.com/articles/self-contained-lone-lisp-applications https://www.matheusmoreira.com/articles/self-contained-lone-... Would be cool if mainstream languages adopted this method.
- deleted 3y ago[deleted]
- kazinator 3y agoThese tricks can be easily done simply and portably, without caring about the underlying executable format. One trick is that you can reserve some global array in the executable, which is prefixed by a byte sequence that doesn't occur anywhere. A small utility can find that byte sequence and write custom data after it to create a customized executable. I think that, also, many executable formats don't mind if something is appended to the executable. If the executable somehow knows its original size (you can write that size somewhere using the previous trick: no grotty executable format parsing required), it can open itself, seek to that offset and read the data. I think this might be how CLISP creates an .exe file on Windows; I think it takes the base clisp.exe and combines it with the lispinit.mem image into one file.
- matheusmoreira 3y ago> reserve some global array in the executable This is neat but has a limitation in that it cannot be expanded after the program is compiled and linked. Resizing the array would invalidate all pointers that follow as well as render incorrect any code that takes its size. This can be solved with a layer of indirection: just append the data to the executable and write its size and file offset in the array. That way the data block can be freely resized. That's the solution people told me to use and indeed the one that I usually see in existing repositories. The problem is you run into some additional complexity later which results in the loss of portability and thus the main reason to choose this method. > it can open itself That's the crux of the issue. How does an executable open itself? That's where portability goes out the window. I've seen source code that opens argv[0] which is under the control of the parent program and therefore unreliable. I've seen code which opens /proc/self/exe which is Linux specific. I've seen code that calls Win32 API functions to get the path to the executable. All this just so it can open and load into memory a file which the kernel has already loaded, just so it can read some additional data off of it. My solution sidesteps that question entirely. It just adds a LOAD segment for the embedded data which instructs the kernel to map it in automatically before the program even runs. There's no need to open, seek or read anything, it will already be there by the time execution begins. The auxiliary vector contains a pointer to the program's segments table so it can reach the data from there. Then it's just a matter of walking this table looking for the custom descriptor segment. It's all done in a structured way, using the standard magic number locations and ranges. There's no chance of a magic number being recognized by mistake. The only possible portability issue is the availability of the AT_PHDR, AT_PHENT and AT_PHNUM entries in the auxiliary vector. I'm not sure if they're standard. I know Linux has them and it's all I personally care about but if these entries do turn out to be standard then I can confidently say that my method is portable to any ELF-based operating system.
- cylinder714 3y agoChris Wellons' "A Magnetized Needle and a Steady Hand," a piece on building an ELF executable from scratch: https://nullprogram.com/blog/2016/11/17/ https://nullprogram.com/blog/2016/11/17/
- sva_ 3y ago> When the program starts running, you might think it starts at main. It doesn’t! It actually goes to _start. This does a bunch of Very Important Things that I don’t understand very well, including calling main. So I won’t explain them. The way I understand it, the symbol main is a C-specific thing. The symbol _start is a language-agnostic entry point for the binary that will in this case call main. A convention of i.e. calling the entry point _start with main's argc/argv would make the format a lot less flexible.
- a-priori 3y agoTechnically the name _start is not special either. The binary lists its entry point address in a header and that’s where the OS starts execution from. That symbol is just called _start by convention by C and other languages, which is what the linker uses to set the entry point when writing the ELF headers, but if you’re writing your own linker scripts you could call the entry point whatever you want.
- Keyframe 3y agoto extend on it, _start is where .text begins and address of that is set by linker
- a-priori 3y agoThe entry point can be anywhere in the .text section, and often won't be at the beginning of the section.
- Retr0id 3y agoTechnically it doesn't even need to be in the .text section, it could be anywhere in the address space. You'll get a segfault if it's not somewhere executable though (assuming you're on a system with an appropriately configured MMU)
- Keyframe 3y agoyes and then you'll have a bad time, but at the same time per convention _start is where .text begins. You can see where it starts with readelf --file-header <executable> and look at Entry point address field. You can change it, yes.
- zerotolerance 3y agoJulia's articles are always excellent. I've always had great results teaching people that compiled code doesn't keep secrets by demoing `strings`.
- actionfromafar 3y agoCan you elaborate?
- shzhdbi09gv8ioi 3y agoman strings
- dilyevsky 3y agoIf you put something like if mySecretPassword == "Qwerty123" { ... then "Qwerty123" will be easily seen by strings utility. Which is pretty obvious but I'm guessing some junior folks will be surprised.
- trealira 3y agoYou probably know this, but in C, if you wrote the comparison like this if (password[0] == 'Q' && password[1] == 'w' /* the rest of the letters... */ && password[8] == '3' // C strings are 0 terminated && password[9] == '\0') { ... } It will probably compile to something like this on x86-64, assuming a "password" is a pointer held in RDI (although I didn't look at a compiler's output): cmp byte [rdi], 0x51 ; 'Q' jne false ;; the rest of the letters... cmp byte [rdi + 9], 0x0 ; '\0' jne false The benefit is that the strings utility wouldn't see the string "Qwerty123" anywhere in the binary. Unicode and bounds checking code probably complicate this generated assembly in other languages. It would be nice if there were some way to write a macro or an inline function so that you could write a comparison like if inlineStringEqualP(mySecretPassword, "Qwerty123") { ... And then it would expand into something like the assembly I wrote above, so that way, the string literal "Qwerty123" isn't embedded in the final binary executable anywhere. I bet it's possible with C++ templates somehow, but it would be messy.
- actionfromafar 3y agoNot a criticism, not even a nit-pick, but a reflection "(binaries are kind of the definition of platform-specific, so this is all platform-specific) (this is true!) When "Actually Portable Executable" took the (geek) proved that the same binary could run on a bunch of platforms, that was a surreal moment I still haven't mentally recovered from. Here we spent decades trying to solve the cross-platform problem, in so many fractals of ways (Java, cross-platform libraries, etc etc) and the solution was right under our noses all this time.
- norir 3y agoI personally am not convinced that portable binaries are a net positive. I believe in the era of fast computers that source distribution and local compilation is superior to binary distribution. Unfortunately, much of the software we rely on is so large, and compilers so relatively slow, that binary distribution is something of a necessary evil. I'd rather see more effort towards simpler software components (that naturally compile fast) and faster compilers than portable binaries.
- vitiral 3y agoI may have misunderstood, but I'm pretty sure APE is not a "binary format" per se. It is a script that can be executed on any system. That script can then LOAD a binary. IIRC the original needed to decode it from base64 before it could be loaded. So... it's an executable binary loader
- 3y ago
- lisper 3y ago> Executables aren’t magic. Nothing in a computer is magic. It was all designed by humans, every single one of which was once a clueless noob. No one is born understanding this stuff.
- JoshuaRogers 3y ago> This does a bunch of Very Important Things that I don’t understand very well, including calling main. So I won’t explain them. Honestly, this line was the best in the whole article. It felt like at that moment I knew the person talking to me wasn't trying to prove that they were some sage (personally guilty here) but instead of was someone who wanted to show me something cool that we could both enjoy. Wonderful write up.
- infinite8s 3y ago"It is no exaggeration to regard this as the most fundamental idea in programming: The evaluator, which determines the meaning of expressions on a piece of paper, is just another piece of paper." --SICP
- latexr 3y ago> Nothing in a computer is magic. I think that’s covered by the text, in the sentence right after that one (emphasis mine): > ELF is a file format like any other!
- olsher 3y agoThe actual /behavior/ of computers, though, tends to emerge from the confluence of complex processes that humans /can't/ understand...our AGI leverages this emergence to enable problem solving in domains where complexity exceeds human capabilities.
- fragmede 3y agocat-ing a binary to the terminal is a recipe for sadness. I like | hd, which is hexdump -C, though that's just as impenetrable to the naked eye.
- heinrichhartman 3y agoI started my blog in 2012, when I shifted my academic career from Mathematics to Computer Science. This topic was literally the first thing that I studied: https://heinrichhartmann.com/archive/Dissecting-Hello-World.html https://heinrichhartmann.com/archive/Dissecting-Hello-World.... Never regretted going down this deep rabbit hole. IIRC, Julia also has a math background. Maybe it's the desire for bottom-up reasoning that leads math folks towards experiments like this. Great to see her making this approachable for a large audience.
- deleted 3y ago[deleted]
- sergejf 3y agoThe format of executable files fascinated me back in the early 90s, to the point that I spent weeks writing (in Modula 2) a DOS and Windows executable file viewer that I named VEXE, releasing it as shareware in 1991. It found a niche following among crackers, even deserving a mention in a +ORC tutorial, https://gist.github.com/callowaysutton/48bdf0245e17e72d41a15f9beee2e447 https://gist.github.com/callowaysutton/48bdf0245e17e72d41a15..., probably because it could detect various encryption and compression methods used to prevent the reverse engineering of those programs.
- randall 3y agoAmazingly helpful!
- adolph 3y agoFor folks interested in this topic who have not seen Cosmopolitan and RedBean, αcτµαlly pδrταblε εxεcµταblε (2020) is a great read too: https://justine.lol/ape.html https://justine.lol/ape.html https://redbean.dev/ https://redbean.dev/
- as1009 3y agoGreat thread
- as1009 3y agogreat thread, thank you!
- dishsoap 3y agoI think ELF should absolutely be mentioned in the title.
- setheron 3y ago(Self promotion) Check out my tool which let's you explore ELF using SQL https://github.com/fzakaria/sqlelf https://github.com/fzakaria/sqlelf
- alkh 3y agoFor a person with a heavy Python background, can anyone suggest a resource/book that would be a good applied intro to practical low-level programming? I've recently started learning Rust and realized I need to catch up on many things. I haven't taken any compiler course, so maybe that's the reason I am missing so much information. For example, I had no idea that symbols in a binary were a thing or what the difference between ELF/MACH-O was