24 ms·

I wrote Bloaty (https://github.com/google/bloaty https://github.com/google/bloaty) which involved writing a totally custom ELF file parser. Here are some epiph

I wrote Bloaty (https://github.com/google/bloaty https://github.com/google/bloaty) which involved writing a totally custom ELF file parser. Here are some epiphanies I had about ELF while writing it. ELF (and Mach-O, PE, etc) are designed to optimize the creation of a process image. The runtime loader mainly just has to mmap() a bunch of file ranges into memory with various permissions. This is quite different than loading a .jar file, .pyc, etc. which involve building a runtime heap and loading objects into that heap. ELF has two file-level tables: sections and segments (the latter are also called program headers). Things clicked for me when I realized: sections are for the linker and segments are for the runtime loader. Sections are the atomic unit of data that linkers operate on: the linker will never rearrange data within a section (it may concatenate several input sections into a single output section though). The loader doesn't even look at the section table AFAIK, everything needed to load the binary is put into segments / program headers. Only some parts of the binary are actually read/loaded when the binary is executed. Debugging info may bloat the binary but it doesn't cost any RAM at runtime because it's never loaded unless you run a debugger. Bloaty makes this clear by showing both VM size and file size: https://github.com/google/bloaty#running-bloaty https://github.com/google/bloaty#running-bloaty

monocasa 7y ago
I just want to say thanks for bloaty. I've used it all the way from 100MB backend server programs, to deeply embedded, bare metal STM32F apps measured in KB.
- haberman 7y ago
  That makes me really happy to hear. I'm glad Bloaty is useful for you!
AceJohnny2 7y ago
I've been wondering about qualitative differences between Mach-O and ELF, after hearing a Mach developer trash the ELF format, but I don't know enough about either to comment. Do you have any insight?
- monocasa 7y ago
  Not haberman, but I've written loaders for both Mach-O and ELF, and really prefer ELF. ELF is mainly structured like descriptive tables of how the relevant pieces look in memory; Mach-O is more structured like a script of commands that you run to load the binary. There's a couple places where the model breaks down for ELF, DWARF and GNU_STACK both feel more Mach-O, but if you're playing with binaries for non standard uses, ELF just feels a lot cleaner IMO. I'd love to hear the Mach developer's arguments though.
  - haberman 7y ago
    Interesting, what was your job that required writing both a Mach-O and ELF loader?
    - monocasa 7y ago
      Binary analysis and introspection tools.
- jcranmer 7y ago
  I only know the ELF format in detail, but I think the major complaint is that ELF uses a flat namespace for symbols whereas Mach-O has a two-level namespace. Furthermore, ELF lets you preload dynamic libraries such that you can override calls even to symbols provided in the same shared object.
  - saagarjha 7y ago
    Note that it's possible to force a flat namespace for Mach-O through a variety of linker flags and DYLD environment variables. And I'm not sure if it does everything you'd want it to, but you can use DYLD_INSERT_LIBRARIES to preload Mach-O dynamic libraries as well.
- haberman 7y ago
  That is a deep and interesting question. I'm not sure I can give a great answer, but here are a few thoughts. If we look at the file format itself (separate from the features/semantics of the linker and loader), I think ELF is simpler and more orthogonal. You can iterate over the section/segment tables of an ELF file without knowing anything about what each section/segment means. ELF nicely decouples the high-level "container" aspect of the file format from the lower-level semantics of how you interpret each section/segment in the linker and loader. Mach on the other hand couples these two concepts together. The top-level table is an array of "load commands", each with its own type, but you can't even parse a load command until you know what type it is. Unlike ELF, the entries of this table do not have a generic format or even a consistent size. If you haven't written code to specifically recognize a given command type, all you can do as fallback behavior is skip it. To me ELF feels like a refactoring of Mach to make it a little more general and layered. If we consider the actual semantics and features of the file formats, there are pros and cons to both. Mach-O has built-in support for fat (multi-architecture) binaries, which is kind of nifty, though I've never actually used it myself. Mach-O distinguishes between "dylib" and "bundle" for shared libraries -- for the life of me I can never remember the difference between these two -- whereas ELF just has one type of shared library. (https://docstore.mik.ua/orelly/unix3/mac/ch05_03.htm https://docstore.mik.ua/orelly/unix3/mac/ch05_03.htm). The distinction seems to add complexity and I'm not sure I understand the benefit. Mach-O has two-level namespaces (dynamic symbols are resolved by both name and the library they come from) -- colliding symbols aren't generally a problem I've seen with ELF, but maybe it's useful in some cases. ELF makes symbol interpositioning easy with LD_PRELOAD, though Mach-O seems to have its own version of this that I've never tried: https://stackoverflow.com/questions/12609728/changing-function-reference-in-mach-o-binary https://stackoverflow.com/questions/12609728/changing-functi.... Overall I prefer ELF.
IshKebab 7y ago
Sections and segments are so badly named. They may as well have called them data and different-data. I guess it is too late to rename them though.
CalChris 7y ago
I've worked with both ELF and Mach-O. They're really very similar from a loading standpoint although Mach-O is a little more wordy and precise. One difference is that Mach-O allows you to specify the initial stack (x86_thread_state64_t.rsp) and ELF (for some reason which I've never understood) doesn't; but maybe there are ELF extensions in the embedded space that I don't know about. With most Mach-O files, rsp is set to 0 and a default is used. The code is in bsd_i386.c/thread_userstack(): if (state25->rsp) { *user_stack = state25->rsp; if (customstack) *customstack = 1; } else { *user_stack = VM_USRSTACK64; if (customstack) *customstack = 0; } This can be set on the OSX ld command line with -stack_addr. OSX uses an obscenely old fork of GNU ld for Mach-O. No linker scripts. Things like zero pages are explicit (but required!) in OSX while they are implicit in ELF+Linux. Also llvm lld's ELF support is first class whereas its Mach-O support is less fleshed out. With Mach-O it helps to read the Darwin source files to figure out what it does with a file. Same to a lesser degree with Linux.

monocasa 7y ago

I just want to say thanks for bloaty. I've used it all the way from 100MB backend server programs, to deeply embedded, bare metal STM32F apps measured in KB.

haberman 7y ago
That makes me really happy to hear. I'm glad Bloaty is useful for you!

AceJohnny2 7y ago

I've been wondering about qualitative differences between Mach-O and ELF, after hearing a Mach developer trash the ELF format, but I don't know enough about either to comment. Do you have any insight?

monocasa 7y ago
Not haberman, but I've written loaders for both Mach-O and ELF, and really prefer ELF. ELF is mainly structured like descriptive tables of how the relevant pieces look in memory; Mach-O is more structured like a script of commands that you run to load the binary. There's a couple places where the model breaks down for ELF, DWARF and GNU_STACK both feel more Mach-O, but if you're playing with binaries for non standard uses, ELF just feels a lot cleaner IMO. I'd love to hear the Mach developer's arguments though.
- haberman 7y ago
  Interesting, what was your job that required writing both a Mach-O and ELF loader?
  - monocasa 7y ago
    Binary analysis and introspection tools.
jcranmer 7y ago
I only know the ELF format in detail, but I think the major complaint is that ELF uses a flat namespace for symbols whereas Mach-O has a two-level namespace. Furthermore, ELF lets you preload dynamic libraries such that you can override calls even to symbols provided in the same shared object.
- saagarjha 7y ago
  Note that it's possible to force a flat namespace for Mach-O through a variety of linker flags and DYLD environment variables. And I'm not sure if it does everything you'd want it to, but you can use DYLD_INSERT_LIBRARIES to preload Mach-O dynamic libraries as well.
haberman 7y ago
That is a deep and interesting question. I'm not sure I can give a great answer, but here are a few thoughts. If we look at the file format itself (separate from the features/semantics of the linker and loader), I think ELF is simpler and more orthogonal. You can iterate over the section/segment tables of an ELF file without knowing anything about what each section/segment means. ELF nicely decouples the high-level "container" aspect of the file format from the lower-level semantics of how you interpret each section/segment in the linker and loader. Mach on the other hand couples these two concepts together. The top-level table is an array of "load commands", each with its own type, but you can't even parse a load command until you know what type it is. Unlike ELF, the entries of this table do not have a generic format or even a consistent size. If you haven't written code to specifically recognize a given command type, all you can do as fallback behavior is skip it. To me ELF feels like a refactoring of Mach to make it a little more general and layered. If we consider the actual semantics and features of the file formats, there are pros and cons to both. Mach-O has built-in support for fat (multi-architecture) binaries, which is kind of nifty, though I've never actually used it myself. Mach-O distinguishes between "dylib" and "bundle" for shared libraries -- for the life of me I can never remember the difference between these two -- whereas ELF just has one type of shared library. (https://docstore.mik.ua/orelly/unix3/mac/ch05_03.htm https://docstore.mik.ua/orelly/unix3/mac/ch05_03.htm). The distinction seems to add complexity and I'm not sure I understand the benefit. Mach-O has two-level namespaces (dynamic symbols are resolved by both name and the library they come from) -- colliding symbols aren't generally a problem I've seen with ELF, but maybe it's useful in some cases. ELF makes symbol interpositioning easy with LD_PRELOAD, though Mach-O seems to have its own version of this that I've never tried: https://stackoverflow.com/questions/12609728/changing-function-reference-in-mach-o-binary https://stackoverflow.com/questions/12609728/changing-functi.... Overall I prefer ELF.

IshKebab 7y ago

Sections and segments are so badly named. They may as well have called them data and different-data. I guess it is too late to rename them though.

CalChris 7y ago

I've worked with both ELF and Mach-O. They're really very similar from a loading standpoint although Mach-O is a little more wordy and precise. One difference is that Mach-O allows you to specify the initial stack (x86_thread_state64_t.rsp) and ELF (for some reason which I've never understood) doesn't; but maybe there are ELF extensions in the embedded space that I don't know about. With most Mach-O files, rsp is set to 0 and a default is used. The code is in bsd_i386.c/thread_userstack(): if (state25->rsp) { *user_stack = state25->rsp; if (customstack) *customstack = 1; } else { *user_stack = VM_USRSTACK64; if (customstack) *customstack = 0; } This can be set on the OSX ld command line with -stack_addr. OSX uses an obscenely old fork of GNU ld for Mach-O. No linker scripts. Things like zero pages are explicit (but required!) in OSX while they are implicit in ELF+Linux. Also llvm lld's ELF support is first class whereas its Mach-O support is less fleshed out. With Mach-O it helps to read the Darwin source files to figure out what it does with a file. Same to a lesser degree with Linux.