7 ms·

Oh hey, a favorite topic of mine! The general rule is that program headers (containing segments) are for runtime, and section headers are for compile time. The

Oh hey, a favorite topic of mine! The general rule is that program headers (containing segments) are for runtime, and section headers are for compile time. The ELF loader and RTLD do not even look at the section headers, which is abused often in various ways: most commonly, section headers are just stripped right off, but it's entirely possible to devise a binary with valid-looking section headers that don't match the program headers. Imagine: a malicious binary which points to the malicious code in the program header, but totally innocent code in the section header. Lots of tools - including reverse-engineering tools! - blindly trust the section headers for some reason. The usual compile-time linker does look at section headers, and in particular will merge the various sections of the input .o files based on section name. Sections have more granularity than segments - for example, there is usually a .data section distinct from the .bss section, but often just a single RW segment that covers both (specifying a longer memory size than file size in order to zero-fill the BSS bits). I believe the distinction between segments and sections largely arises because the runtime (ELF loader/dynamic linker) have different needs from the compiler (static linker); however, I have not looked deeply enough into the history of the ELF format to know exactly why the distinction was made initially. I know many other formats do not make this distinction as clearly: Mach-O nests sections underneath segments hierarchically, whereas PE just has sections: much of the "metadata" that would normally live in ELF sections is instead in special "directories" pointed to by specific entries in the PE file header.

matheusmoreira 3y ago
It's turned into one of my favorite topics too. It's weird how all the tools out there are all about the ELF sections. Apparently the sections are where all the fun stuff gets stored and the program headers are just some kernel wizardry nobody really cares about. The section/segment dichotomy exists so that kernels need not load needless stuff onto memory. In practice it means things like ELF symbol tables are not available for introspection by the program. If pointers to those symbols are required, the easiest way is to get the ELF interpreter to give them to the program. If it's a static executable then there's no interpreter and things get annoying fast. Most solutions I've seen involve reading the program's own executable into memory and parsing the ELF structures, essentially duplicating the work of the kernel's ELF loader. Alternatively, one may engage in this incredibly fun activity called linker scripting. I chose to hack together an ELF patching tool in order to get Linux to automatically load arbitrary ELF file segments onto memory. Along the way I requested some linker features and actually got them! https://github.com/lone-lang/lone/blob/master/source/tools/lone-embed.c https://github.com/lone-lang/lone/blob/master/source/tools/l... With these tools I made it possible for my language's interpreter to load programs embedded inside itself. https://www.matheusmoreira.com/articles/self-contained-lone-lisp-applications https://www.matheusmoreira.com/articles/self-contained-lone-... I really want to see this feature get adopted by the mainstream languages. Apparently no one else ever thought of it.
- nneonneo 3y ago
  Python one-file-packagers such as PyInstaller typically use the technique of shoving the entire Python standard library into the executable as a zip file; Python can natively import scripts straight out of .zip files (zipimport) so you just point `sys.path` at the executable and voila, self-contained Python in a single binary. (Note that this doesn't immediately solve the problem of using native code dynamic libraries, which usually have to get unpacked anyway or use hacky custom loaders). One year, for DEF CON CTF, we had to write all of our exploits as single ELF binaries targeting a minimal OS (DARPA CGC) with only seven system calls (exit, read, write, select, malloc, free, getrandom). Since writing exploits in C sucks, I packaged Python for the OS instead, creating a "virtual filesystem" for including Python code (including standard library) as a static chunk of data and implementing open/read/write on top of the VFS. This was really quite successful for us, as we were able to write very high-level Python code for a full, functioning Python interpreter in our exploits.
  - ithkuil 3y ago
    Nice. But the beauty of GP's approach is that you don't need to issue another open syscall to open argv[0] in order to access (read or mmap) your code but instead just piggy back on what the kernel just did for your entry point anyway. Except for the beauty of the approach, I'm not sure what the practical advantages are. Are there cases where a process wouldn't have permissions to access its own executable or argv[0] value is unreliable? Can you exec on a file descriptor of a deleted file? EDIT: or would /proc/self/exe always point to something the process could open?
    - loeg 3y ago
      The benefit of GP's approach is mostly just elegance. You can reliably introspect the (mapped) program headers with getauxval AT_PHDR.
    - nneonneo 3y ago
      I believe the process always holds the binary open as a "txt" file descriptor (check lsof), so opening `/proc/self/exe` always works. You can even execve a "memfd", an in-memory "file" which is not in the filesystem (distinct from a ramdisk file, which is a file sitting on an in-memory filesystem). /proc/self/exe still works even in that case, even when the original memfd is closed. Note that argv[0] can never be relied on 100%. The vast majority of programs will set it correctly (especially since many programs will malfunction if provided bogus argv[0]), but a caller has full control over argv[0] and can set it to anything, including a NULL pointer (by simply passing an empty argv array).
xelxebar 3y ago
Very interesting. When I was hand-writing ELFs, it took me a while to grok sections vs. segments. If you ever run across a history of executable formats and the motivations for the particular decisions made with ELF, I'd love to read that. The main thing that surprised me was how loose of a format ELF is compared to the much more rigid conventions imposed by modern mainstream compilers.

matheusmoreira 3y ago

It's turned into one of my favorite topics too. It's weird how all the tools out there are all about the ELF sections. Apparently the sections are where all the fun stuff gets stored and the program headers are just some kernel wizardry nobody really cares about. The section/segment dichotomy exists so that kernels need not load needless stuff onto memory. In practice it means things like ELF symbol tables are not available for introspection by the program. If pointers to those symbols are required, the easiest way is to get the ELF interpreter to give them to the program. If it's a static executable then there's no interpreter and things get annoying fast. Most solutions I've seen involve reading the program's own executable into memory and parsing the ELF structures, essentially duplicating the work of the kernel's ELF loader. Alternatively, one may engage in this incredibly fun activity called linker scripting. I chose to hack together an ELF patching tool in order to get Linux to automatically load arbitrary ELF file segments onto memory. Along the way I requested some linker features and actually got them! https://github.com/lone-lang/lone/blob/master/source/tools/lone-embed.c https://github.com/lone-lang/lone/blob/master/source/tools/l... With these tools I made it possible for my language's interpreter to load programs embedded inside itself. https://www.matheusmoreira.com/articles/self-contained-lone-lisp-applications https://www.matheusmoreira.com/articles/self-contained-lone-... I really want to see this feature get adopted by the mainstream languages. Apparently no one else ever thought of it.

nneonneo 3y ago
Python one-file-packagers such as PyInstaller typically use the technique of shoving the entire Python standard library into the executable as a zip file; Python can natively import scripts straight out of .zip files (zipimport) so you just point `sys.path` at the executable and voila, self-contained Python in a single binary. (Note that this doesn't immediately solve the problem of using native code dynamic libraries, which usually have to get unpacked anyway or use hacky custom loaders). One year, for DEF CON CTF, we had to write all of our exploits as single ELF binaries targeting a minimal OS (DARPA CGC) with only seven system calls (exit, read, write, select, malloc, free, getrandom). Since writing exploits in C sucks, I packaged Python for the OS instead, creating a "virtual filesystem" for including Python code (including standard library) as a static chunk of data and implementing open/read/write on top of the VFS. This was really quite successful for us, as we were able to write very high-level Python code for a full, functioning Python interpreter in our exploits.
- ithkuil 3y ago
  Nice. But the beauty of GP's approach is that you don't need to issue another open syscall to open argv[0] in order to access (read or mmap) your code but instead just piggy back on what the kernel just did for your entry point anyway. Except for the beauty of the approach, I'm not sure what the practical advantages are. Are there cases where a process wouldn't have permissions to access its own executable or argv[0] value is unreliable? Can you exec on a file descriptor of a deleted file? EDIT: or would /proc/self/exe always point to something the process could open?
  - loeg 3y ago
    The benefit of GP's approach is mostly just elegance. You can reliably introspect the (mapped) program headers with getauxval AT_PHDR.
  - nneonneo 3y ago
    I believe the process always holds the binary open as a "txt" file descriptor (check lsof), so opening `/proc/self/exe` always works. You can even execve a "memfd", an in-memory "file" which is not in the filesystem (distinct from a ramdisk file, which is a file sitting on an in-memory filesystem). /proc/self/exe still works even in that case, even when the original memfd is closed. Note that argv[0] can never be relied on 100%. The vast majority of programs will set it correctly (especially since many programs will malfunction if provided bogus argv[0]), but a caller has full control over argv[0] and can set it to anything, including a NULL pointer (by simply passing an empty argv array).

xelxebar 3y ago

Very interesting. When I was hand-writing ELFs, it took me a while to grok sections vs. segments. If you ever run across a history of executable formats and the motivations for the particular decisions made with ELF, I'd love to read that. The main thing that surprised me was how loose of a format ELF is compared to the much more rigid conventions imposed by modern mainstream compilers.