7 ms·

Evolution of the ELF object file format

148 points by MaskRay 2y ago

MaskRay 2y ago
A few folks have asked me the generic ABI status (unmaintained?) and the availability of an up-to-date specification (no). I compiled “History” and “Evolution of the generic ABI” in the blog post. I have two specific questions: - Key features (symbol visibility, section groups, SHF_MERGE, etc) were all available as of April 2001. Where can we find the discussion mailing lists? Are they still available? - How does the ABI end up being “All rights reserved” by SCO? Tool Interface Standard (TIS) Portable Formats Specification, version 1.2 effectively put the specification in the public domain.
- Joker_vD 2y ago
  > How does the ABI end up being “All rights reserved” by SCO? Text of the spec is copyrighted by the SCO, they did not put it in public domain. That's what "All rights reserved" was intended to mean even though this phrase has not been meaning anything since 2000.
  - LegionMammal978 2y ago
    Yeah, it's not like they could copyright the ELF format anyway, only that particular description, or any particular implementation they had. Pretty much everyone seems to agree that file formats themselves aren't copyrightable expression, though I have trouble finding any U.S. case law to that effect. (Personally, that makes me a bit uneasy: for instance, if the copyrighted spec lists the file sections in a certain order, and your implementation happens to output them in the same order even if it doesn't have to, then have you infringed on the owner's copyright of that particular arrangement?) Meanwhile, they could have gotten a patent on (some parts of) the format, but that doesn't seem to be the case here.
    - Veserv 2y ago
      The rule in general (not just for software) is that patents are for functional elements where as copyrights are for non-functional elements; they are exclusive, a element can only be patentable or copyrightable, not both. If it is necessary for operation or inter-operation [1] then they can probably not enforce copyright or trademark. [1] https://en.wikipedia.org/wiki/Sega_v._Accolade https://en.wikipedia.org/wiki/Sega_v._Accolade
      - LegionMammal978 2y ago
        Sure, but usually in any file format, an encoder has some choice in how to produce its output, with no functional difference either way. For instance, different PNG encoders might include the necessary chunks in a different order in the file. And my concern is that some such choices, if they align with choices described in the copyrighted spec, might be protectable as a creative arrangement or similar. (E.g., a spec might list the chunks in such an order that their names form a creative acronym.) Of course, the defense against this would be to scramble (or normalize) all non-functional choices compared to anything in the spec. But you have to be careful to make sure there's nothing left of the spec's non-functional influence. At least Oracle v. Google appears to provide some ammunition here in favor of implementers: the court found the transformative use in that case enough to trump even the byte-for-byte copying of the API signatures. So perhaps interoperability could similarly trump such non-functional copying from the spec. But overall, it's still on shakier ground than I'd like.
    - MaskRay 2y ago
      I feel that The SCO Group's role in the evolution of the System V ABI seems to have been more of a curator/editor than an innovator, inheriting the System V ABI from previous entities. Given that the Tool Interface Standard (TIS) Committee has essentially released the ELF-related chapters into the public domain, and others have made changes, it's unclear what specific rights The SCO Group (and now Xinuos) could claim to reserve. (That said, their maintenance work needs to be remembered.)
  - vintagedave 2y ago
    It's concerning that a huge open source ecosystem like Linux depends on a closed specification. That is, I read it as that the spec cannot be continued, developed, and evolved or republished without full redefinition and respecification. Has anyone looked at creating a new object format for Linux? A non-open spec seems a minor issue, really, in an era when we put binary blobs in the kernel (hi Nvidia.) But the more decades I work in closed source, the more I value open source, and believe keeping _everything_ open.
    - yjftsjthsd-h 2y ago
      I wouldn't think you'd need to create a new format, just write up a spec of what all available FOSS implementations are actually doing and agree that that's the standard going forward. Between Linux/*BSD/illumos you should be able to pretty completely describe all currently-active uses of ELF. (Of course this assumes that doing this is sufficient to make it legally not derived from the original spec as far as copyright law cares about, which is beyond my non-lawyer ability to be that confident in.)
      - skissane 2y ago
        > Between Linux/*BSD/illumos you should be able to pretty completely describe all currently-active uses of ELF. OpenVMS uses ELF too. Rather niche proprietary OS but still maintained and in production use. (Not a new thing with x86-64 port, ELF was adopted during the Alpha to Iranian transition.) As do many RTOS (both open source and proprietary)
        Iwan-Zotow 2y ago
        To WHAT transition?
        yjftsjthsd-h 2y ago
        Lol. Although, Itanium does have an entertaining history of having caused a lot of positive changes even though it crashed and burned itself - it's where we got UEFI, and apparently a good time for some OSs to pick up ELF.
- jcranmer 2y ago
  One of the things I've been (very slowly) putting together is a master list of all the references one needs to build a compiler toolchain--such as processor ISA manuals, ABI specifications, language standards, even things like IEEE 754 or DWARF. While IEEE 754 is an issue because it's not freely available for most people, the only document that has truly stumped me is the ELF specification. From your blog post, it seems like my failure to find the most up-to-date specification is simply because one doesn't exist.
  - marssaxman 2y ago
    Are you familiar with the OSDev wiki? Perhaps their list of links will be helpful: https://wiki.osdev.org/ELF#External_Links https://wiki.osdev.org/ELF#External_Links
    - jcranmer 2y ago
      That wiki's list of links isn't always the most up-to-date.
      - marssaxman 2y ago
        How often does anyone change the ELF file format, though? No matter; but that's the most up-to-date list of references I know about.
        jcranmer 2y ago
        The ELF format isn't the one that I'm the most concerned about, but for example compressed sections were updated in the past decade. The ABI for x86-64 is out-of-date (actually, the site it links to is now a 404, but my recollection was that was a version around 2013)--missing things like _Float16 rules, DWARF register naming for AMX and APX registers, microarchitecture levels, new relocation types,
        khuey 2y ago
        The up to date ABI for x86-64 lives at https://gitlab.com/x86-psABIs/x86-64-ABI https://gitlab.com/x86-psABIs/x86-64-ABI
- WalterBright 2y ago
  The first thing I do when porting to a new platform is build an object file format pretty-printer. It's the only way to figure out how it really works. The written specifications for them are often a bit of a fairy tale. The real specification is whatever the local C compiler generates. The other gatekeeper is what the linker will accept. That can sometimes be a nightmare.
  - rvbissell 2y ago
    Do you have any examples of the pretty-printing that you've generated?
    - WalterBright 2y ago
      I never got around to open sourcing them. But here's part of a dump of a .o file: Section 14 .eh_frame PROGBITS,ALLOC,SIZE=0x0030(48),OFFSET=0x0040,ALIGN=8 offset = 00000000, length = 0x0014(20) CIE_id = 00000000 version = 1 augmentation = 'zR' code_alignment_factor = 1 data_alignment_factor = -8 return_address_register = 16 Augmentation_Length = 1 Address pointer encoding = DW_EH_PE_pcrel|sdata4 DW_CFA_def_cfa reg = 7, off = 00000008 DW_CFA_offset_10 off = 1 DW_CFA_nop DW_CFA_nop offset = 00000018, length = 0x0014(20) CIE_pointer = 0000001c offset 00000000 augmentation = 'zR' PC_Begin = 0 PC_Range = 6 Augmentation_Length = 0 DW_CFA_nop DW_CFA_nop DW_CFA_nop DW_CFA_nop DW_CFA_nop DW_CFA_nop DW_CFA_nop 0040: 14 0 0 0 0 0 0 0 1 7a 52 0 1 78 10 1 .........zR..x.. 0050: 1b c 7 8 90 1 0 0 14 0 0 0 1c 0 0 0 ................ 0060: 0 0 0 0 6 0 0 0 0 0 0 0 0 0 0 0 ................ Section 15 .data._D16TypeInfo_S3__C1S6__initZ PROGBITS,WRITE,ALLOC,SIZE=0x0090(144),OFFSET=0x0070,ALIGN=16 0070: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ................ 0080: 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ................ 0090: 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ................ 00a0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ................ 00b0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ................ 00c0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ................ 00d0: 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 ................ 00e0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ................ 00f0: 0 0 0 0 0 0 0 0 53 33 5f 5f 43 31 53 0 ........S3__C1S. Section 16 .rela.data._D16TypeInfo_S3__C1S6__initZ RELA,ENTRIES=3,OFFSET=0x06A4,ALIGN=8,LINK=8,INFO=15 0 offset=00000000 addend=0000000000000000 type=R_X86_64_64 sym=24 sym="_D15TypeInfo_Struct6__vtblZ" 1 offset=00000018 addend=0000000000000088 type=R_X86_64_64 sym=21 sym="_D16TypeInfo_S3__C1S6__initZ" 2 offset=00000070 addend=0000000000000000 type=R_X86_64_64 sym=25 sym="_D10TypeInfo_i6__initZ" Section 17 .bss.__bzeroBytes NOBITS,WRITE,ALLOC,SIZE=0x0080(128),OFFSET=0x0100,ALIGN=16 Section 18 .group GROUP,ENTRIES=2,OFFSET=0x0100,ALIGN=4,LINK=8,INFO=23 0100: 1 0 0 0 13 0 0 0 ........ Section 19 .text.test PROGBITS,ALLOC,EXEC,GROUP,SIZE=0x0008(8),OFFSET=0x0108,ALIGN=4 0108: b8 4 0 0 0 c3 0 0 ........
- trws 2y ago
  I fell very far down this rabbit hole trying to figure out the history of library link orders and symbol resolution. One fun outcome of which was finding that the first ever public release of a manpage for ld.so on Linux listed RPATH as deprecated. Finding mailing lists from before the source ware transition is hard, some of it exists if you follow the path back to the Solaris forums, but only some. I’ll look through my archives for the features you listed, but I don’t think I uncovered anything about them specifically. Good luck!
  - MaskRay 2y ago
    https://www.sco.com/developers/gabi/latest/revision.html https://www.sco.com/developers/gabi/latest/revision.html specifies > Second draft published May 3, 1999. > ... > New dynamic section tags DT_RUNPATH and DT_FLAGS added. Dynamic section tag DT_RPATH moved to level 2. In glibc, DT_RPATH and DT_RUNPATH have different semantics regarding precedence related to LD_LIBRARY_PATH. In FreeBSD rtld and musl, DT_RPATH and DT_RUNPATH are equivallent.
- IAmLiterallyAB 2y ago
  Just want to say, your blog has been a godsend for understanding the weird quirks of linking and ABI stuff. Thank you.
yjftsjthsd-h 2y ago
> Q18: How can you get a single binary to work identically across all these diverse systems? > Most Unix-on-Intel binary packages are already largely similar. Almost all such operating systems use the "ELF" binary 'packaging'; the various operating systems have small but significant differences, though, that make each system's ELF binary unusable on others'. Though the scope has diminished with the decline of proprietary unixen, there is a nice bright spot with APE binaries ( https://justine.lol/ape.html https://justine.lol/ape.html ) which are in some regards even more portable since they work on Darwin (which natively uses Mach-O) and NT (which natively uses PE).
- MaskRay 2y ago
  Yes! I immediately thought about APE when I read The 86open Project's FAQ. However, I feel that APE is less related to the blog post so decide not to add the link to the post.
CalChris 2y ago
Who controls the ELF standard? The best I can tell is that you submit a pull request with LLVM and maybe GCC.
- MaskRay 2y ago
  - 2003-2010 The SCO Group - 2011- Xinous, but Xinous has stopped updating https://www.sco.com/developers/gabi/latest/contents.html https://www.sco.com/developers/gabi/latest/contents.html . It seems that Xinous has moved on from System V based OpenServer/UnixWare. The newer OpenServer seems to be based on FreeBSD. They likely no longer care about the System V ABI. Nowadays, people make proposals to the generic-abi Google Group. Essentially, a proposal is considered "standardized" if it receives approval from GNU, LLVM, and Ali Bahrami (Solaris representative). Many GNU extensions are implemented by LLVM and adopted by BSD and newer ELF-based OSes. For extensions that are considered generic enough, it's recommended to propose them through the generic-abi Google Group. psABI documents generally prefer generic extensions over GNU or LLVM-specific ones.
matheusmoreira 2y ago
> Despite this uncertainty, innovation within the ELF ecosystem should continue. > In practice, achieving consensus among major toolchain vendors (GNU and LLVM) may be sufficient, even without formal approval from the generic ABI. That makes it sound easy. GNU projects seem to be very conservatively maintained. LLVM has around one trillion open pull requests. I came up with a little mechanism to get the kernel to automatically load into memory data embedded into the program. I used it in my programming language to embed code into the interpreter so that they can be automatically loaded and executed. https://www.matheusmoreira.com/articles/self-contained-lone-lisp-applications https://www.matheusmoreira.com/articles/self-contained-lone-... Only the maintainer of the relatively new mold linker cared to implement a helpful feature to make this kind of thing possible and easy. I requested the same feature in GNU ld and the idea wasn't exactly received with enthusiasm. I'm not sure LLVM ever received it at all. And that was just a linker option to add some extra PT_NULL segments for easy and efficient patching. Can't even imagine the effort it would take to actually change something about this ABI.
- Joker_vD 2y ago
  Isn't that's what NOTE segments are for? It can have arbitrary data embedded in it, that's how GCC implements most of its insane dynamic-linking-related extensions IIRC. Of course, it has to overlap with some LOAD segment to be useful. $ LANG=en_US objdump -x /bin/bash /bin/bash: file format elf64-x86-64 /bin/bash architecture: i386:x86-64, flags 0x00000150: HAS_SYMS, DYNAMIC, D_PAGED start address 0x0000000000032ef0 Program Header: PHDR off 0x0000000000000040 vaddr 0x0000000000000040 paddr 0x0000000000000040 align 2**3 filesz 0x00000000000002d8 memsz 0x00000000000002d8 flags r-- INTERP off 0x0000000000000318 vaddr 0x0000000000000318 paddr 0x0000000000000318 align 2**0 filesz 0x000000000000001c memsz 0x000000000000001c flags r-- LOAD off 0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**12 filesz 0x000000000002e188 memsz 0x000000000002e188 flags r-- LOAD off 0x000000000002f000 vaddr 0x000000000002f000 paddr 0x000000000002f000 align 2**12 filesz 0x00000000000def6d memsz 0x00000000000def6d flags r-x LOAD off 0x000000000010e000 vaddr 0x000000000010e000 paddr 0x000000000010e000 align 2**12 filesz 0x0000000000039b08 memsz 0x0000000000039b08 flags r-- LOAD off 0x0000000000148a90 vaddr 0x0000000000149a90 paddr 0x0000000000149a90 align 2**12 filesz 0x000000000000bbc0 memsz 0x0000000000016b28 flags rw- DYNAMIC off 0x000000000014b4c0 vaddr 0x000000000014c4c0 paddr 0x000000000014c4c0 align 2**3 filesz 0x0000000000000200 memsz 0x0000000000000200 flags rw- NOTE off 0x0000000000000338 vaddr 0x0000000000000338 paddr 0x0000000000000338 align 2**3 filesz 0x0000000000000030 memsz 0x0000000000000030 flags r-- NOTE off 0x0000000000000368 vaddr 0x0000000000000368 paddr 0x0000000000000368 align 2**2 filesz 0x0000000000000044 memsz 0x0000000000000044 flags r-- 0x6474e553 off 0x0000000000000338 vaddr 0x0000000000000338 paddr 0x0000000000000338 align 2**3 filesz 0x0000000000000030 memsz 0x0000000000000030 flags r-- EH_FRAME off 0x00000000001278a8 vaddr 0x00000000001278a8 paddr 0x00000000001278a8 align 2**2 filesz 0x000000000000472c memsz 0x000000000000472c flags r-- STACK off 0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**4 filesz 0x0000000000000000 memsz 0x0000000000000000 flags rw- RELRO off 0x0000000000148a90 vaddr 0x0000000000149a90 paddr 0x0000000000149a90 align 2**0 filesz 0x0000000000003570 memsz 0x0000000000003570 flags r-- ... $ hd -s 0x338 -n 0x30 /bin/bash 00000338 04 00 00 00 20 00 00 00 05 00 00 00 47 4e 55 00 |.... .......GNU.| 00000348 02 00 00 c0 04 00 00 00 03 00 00 00 00 00 00 00 |................| 00000358 02 80 00 c0 04 00 00 00 01 00 00 00 00 00 00 00 |................| 00000368 $ hd -s 0x368 -n 0x44 /bin/bash 00000368 04 00 00 00 14 00 00 00 03 00 00 00 47 4e 55 00 |............GNU.| 00000378 7a 64 08 ba 82 a2 d8 6d d9 8f 1f 75 ac 8e dc b6 |zd.....m...u....| 00000388 95 f6 fd 60 04 00 00 00 10 00 00 00 01 00 00 00 |...`............| 00000398 47 4e 55 00 00 00 00 00 03 00 00 00 02 00 00 00 |GNU.............| 000003a8 00 00 00 00 |....| 000003ac Plus, of course, you can always leave the section table in as well, with SHT_NOTE sections in it.
kibwen 2y ago
I'd be interested in reading about the history of object file formats in general. I'm wondering what motivates them being so relatively complicated for something that seems so conceptually simple; is it historical baggage or is it essential complexity?
- KerrAvon 2y ago
  Get yourself a copy of “Linkers and Loaders” — https://dl.acm.org/doi/10.5555/519563 https://dl.acm.org/doi/10.5555/519563 . Explains the history of it all.
- MaskRay 2y ago
  Linkers and Loaders I have analyzed a few object file formats in another blog post https://maskray.me/blog/2024-01-14-exploring-object-file-formats https://maskray.me/blog/2024-01-14-exploring-object-file-for... (HN discussion: https://news.ycombinator.com/item?id=38998914 https://news.ycombinator.com/item?id=38998914) ELF is technically not complicated and simpler than COFF and Mach-O.

148 points by MaskRay 2y ago

A few folks have asked me the generic ABI status (unmaintained?) and the availability of an up-to-date specification (no). I compiled “History” and “Evolution of the generic ABI” in the blog post. I have two specific questions: - Key features (symbol visibility, section groups, SHF_MERGE, etc) were all available as of April 2001. Where can we find the discussion mailing lists? Are they still available? - How does the ABI end up being “All rights reserved” by SCO? Tool Interface Standard (TIS) Portable Formats Specification, version 1.2 effectively put the specification in the public domain.

Joker_vD 2y ago
> How does the ABI end up being “All rights reserved” by SCO? Text of the spec is copyrighted by the SCO, they did not put it in public domain. That's what "All rights reserved" was intended to mean even though this phrase has not been meaning anything since 2000.
- LegionMammal978 2y ago
  Yeah, it's not like they could copyright the ELF format anyway, only that particular description, or any particular implementation they had. Pretty much everyone seems to agree that file formats themselves aren't copyrightable expression, though I have trouble finding any U.S. case law to that effect. (Personally, that makes me a bit uneasy: for instance, if the copyrighted spec lists the file sections in a certain order, and your implementation happens to output them in the same order even if it doesn't have to, then have you infringed on the owner's copyright of that particular arrangement?) Meanwhile, they could have gotten a patent on (some parts of) the format, but that doesn't seem to be the case here.
  - Veserv 2y ago
    The rule in general (not just for software) is that patents are for functional elements where as copyrights are for non-functional elements; they are exclusive, a element can only be patentable or copyrightable, not both. If it is necessary for operation or inter-operation [1] then they can probably not enforce copyright or trademark. [1] https://en.wikipedia.org/wiki/Sega_v._Accolade https://en.wikipedia.org/wiki/Sega_v._Accolade
    - LegionMammal978 2y ago
      Sure, but usually in any file format, an encoder has some choice in how to produce its output, with no functional difference either way. For instance, different PNG encoders might include the necessary chunks in a different order in the file. And my concern is that some such choices, if they align with choices described in the copyrighted spec, might be protectable as a creative arrangement or similar. (E.g., a spec might list the chunks in such an order that their names form a creative acronym.) Of course, the defense against this would be to scramble (or normalize) all non-functional choices compared to anything in the spec. But you have to be careful to make sure there's nothing left of the spec's non-functional influence. At least Oracle v. Google appears to provide some ammunition here in favor of implementers: the court found the transformative use in that case enough to trump even the byte-for-byte copying of the API signatures. So perhaps interoperability could similarly trump such non-functional copying from the spec. But overall, it's still on shakier ground than I'd like.
  - MaskRay 2y ago
    I feel that The SCO Group's role in the evolution of the System V ABI seems to have been more of a curator/editor than an innovator, inheriting the System V ABI from previous entities. Given that the Tool Interface Standard (TIS) Committee has essentially released the ELF-related chapters into the public domain, and others have made changes, it's unclear what specific rights The SCO Group (and now Xinuos) could claim to reserve. (That said, their maintenance work needs to be remembered.)
- vintagedave 2y ago
  It's concerning that a huge open source ecosystem like Linux depends on a closed specification. That is, I read it as that the spec cannot be continued, developed, and evolved or republished without full redefinition and respecification. Has anyone looked at creating a new object format for Linux? A non-open spec seems a minor issue, really, in an era when we put binary blobs in the kernel (hi Nvidia.) But the more decades I work in closed source, the more I value open source, and believe keeping _everything_ open.
  - yjftsjthsd-h 2y ago
    I wouldn't think you'd need to create a new format, just write up a spec of what all available FOSS implementations are actually doing and agree that that's the standard going forward. Between Linux/*BSD/illumos you should be able to pretty completely describe all currently-active uses of ELF. (Of course this assumes that doing this is sufficient to make it legally not derived from the original spec as far as copyright law cares about, which is beyond my non-lawyer ability to be that confident in.)
    - skissane 2y ago
      > Between Linux/*BSD/illumos you should be able to pretty completely describe all currently-active uses of ELF. OpenVMS uses ELF too. Rather niche proprietary OS but still maintained and in production use. (Not a new thing with x86-64 port, ELF was adopted during the Alpha to Iranian transition.) As do many RTOS (both open source and proprietary)
      - Iwan-Zotow 2y ago
        To WHAT transition?
        yjftsjthsd-h 2y ago
        Lol. Although, Itanium does have an entertaining history of having caused a lot of positive changes even though it crashed and burned itself - it's where we got UEFI, and apparently a good time for some OSs to pick up ELF.
jcranmer 2y ago
One of the things I've been (very slowly) putting together is a master list of all the references one needs to build a compiler toolchain--such as processor ISA manuals, ABI specifications, language standards, even things like IEEE 754 or DWARF. While IEEE 754 is an issue because it's not freely available for most people, the only document that has truly stumped me is the ELF specification. From your blog post, it seems like my failure to find the most up-to-date specification is simply because one doesn't exist.
- marssaxman 2y ago
  Are you familiar with the OSDev wiki? Perhaps their list of links will be helpful: https://wiki.osdev.org/ELF#External_Links https://wiki.osdev.org/ELF#External_Links
  - jcranmer 2y ago
    That wiki's list of links isn't always the most up-to-date.
    - marssaxman 2y ago
      How often does anyone change the ELF file format, though? No matter; but that's the most up-to-date list of references I know about.
      - jcranmer 2y ago
        The ELF format isn't the one that I'm the most concerned about, but for example compressed sections were updated in the past decade. The ABI for x86-64 is out-of-date (actually, the site it links to is now a 404, but my recollection was that was a version around 2013)--missing things like _Float16 rules, DWARF register naming for AMX and APX registers, microarchitecture levels, new relocation types,
        khuey 2y ago
        The up to date ABI for x86-64 lives at https://gitlab.com/x86-psABIs/x86-64-ABI https://gitlab.com/x86-psABIs/x86-64-ABI
WalterBright 2y ago
The first thing I do when porting to a new platform is build an object file format pretty-printer. It's the only way to figure out how it really works. The written specifications for them are often a bit of a fairy tale. The real specification is whatever the local C compiler generates. The other gatekeeper is what the linker will accept. That can sometimes be a nightmare.
- rvbissell 2y ago
  Do you have any examples of the pretty-printing that you've generated?
  - WalterBright 2y ago
    I never got around to open sourcing them. But here's part of a dump of a .o file: Section 14 .eh_frame PROGBITS,ALLOC,SIZE=0x0030(48),OFFSET=0x0040,ALIGN=8 offset = 00000000, length = 0x0014(20) CIE_id = 00000000 version = 1 augmentation = 'zR' code_alignment_factor = 1 data_alignment_factor = -8 return_address_register = 16 Augmentation_Length = 1 Address pointer encoding = DW_EH_PE_pcrel|sdata4 DW_CFA_def_cfa reg = 7, off = 00000008 DW_CFA_offset_10 off = 1 DW_CFA_nop DW_CFA_nop offset = 00000018, length = 0x0014(20) CIE_pointer = 0000001c offset 00000000 augmentation = 'zR' PC_Begin = 0 PC_Range = 6 Augmentation_Length = 0 DW_CFA_nop DW_CFA_nop DW_CFA_nop DW_CFA_nop DW_CFA_nop DW_CFA_nop DW_CFA_nop 0040: 14 0 0 0 0 0 0 0 1 7a 52 0 1 78 10 1 .........zR..x.. 0050: 1b c 7 8 90 1 0 0 14 0 0 0 1c 0 0 0 ................ 0060: 0 0 0 0 6 0 0 0 0 0 0 0 0 0 0 0 ................ Section 15 .data._D16TypeInfo_S3__C1S6__initZ PROGBITS,WRITE,ALLOC,SIZE=0x0090(144),OFFSET=0x0070,ALIGN=16 0070: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ................ 0080: 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ................ 0090: 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ................ 00a0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ................ 00b0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ................ 00c0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ................ 00d0: 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 ................ 00e0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ................ 00f0: 0 0 0 0 0 0 0 0 53 33 5f 5f 43 31 53 0 ........S3__C1S. Section 16 .rela.data._D16TypeInfo_S3__C1S6__initZ RELA,ENTRIES=3,OFFSET=0x06A4,ALIGN=8,LINK=8,INFO=15 0 offset=00000000 addend=0000000000000000 type=R_X86_64_64 sym=24 sym="_D15TypeInfo_Struct6__vtblZ" 1 offset=00000018 addend=0000000000000088 type=R_X86_64_64 sym=21 sym="_D16TypeInfo_S3__C1S6__initZ" 2 offset=00000070 addend=0000000000000000 type=R_X86_64_64 sym=25 sym="_D10TypeInfo_i6__initZ" Section 17 .bss.__bzeroBytes NOBITS,WRITE,ALLOC,SIZE=0x0080(128),OFFSET=0x0100,ALIGN=16 Section 18 .group GROUP,ENTRIES=2,OFFSET=0x0100,ALIGN=4,LINK=8,INFO=23 0100: 1 0 0 0 13 0 0 0 ........ Section 19 .text.test PROGBITS,ALLOC,EXEC,GROUP,SIZE=0x0008(8),OFFSET=0x0108,ALIGN=4 0108: b8 4 0 0 0 c3 0 0 ........
trws 2y ago
I fell very far down this rabbit hole trying to figure out the history of library link orders and symbol resolution. One fun outcome of which was finding that the first ever public release of a manpage for ld.so on Linux listed RPATH as deprecated. Finding mailing lists from before the source ware transition is hard, some of it exists if you follow the path back to the Solaris forums, but only some. I’ll look through my archives for the features you listed, but I don’t think I uncovered anything about them specifically. Good luck!
- MaskRay 2y ago
  https://www.sco.com/developers/gabi/latest/revision.html https://www.sco.com/developers/gabi/latest/revision.html specifies > Second draft published May 3, 1999. > ... > New dynamic section tags DT_RUNPATH and DT_FLAGS added. Dynamic section tag DT_RPATH moved to level 2. In glibc, DT_RPATH and DT_RUNPATH have different semantics regarding precedence related to LD_LIBRARY_PATH. In FreeBSD rtld and musl, DT_RPATH and DT_RUNPATH are equivallent.
IAmLiterallyAB 2y ago
Just want to say, your blog has been a godsend for understanding the weird quirks of linking and ABI stuff. Thank you.

yjftsjthsd-h 2y ago

> Q18: How can you get a single binary to work identically across all these diverse systems? > Most Unix-on-Intel binary packages are already largely similar. Almost all such operating systems use the "ELF" binary 'packaging'; the various operating systems have small but significant differences, though, that make each system's ELF binary unusable on others'. Though the scope has diminished with the decline of proprietary unixen, there is a nice bright spot with APE binaries ( https://justine.lol/ape.html https://justine.lol/ape.html ) which are in some regards even more portable since they work on Darwin (which natively uses Mach-O) and NT (which natively uses PE).

MaskRay 2y ago
Yes! I immediately thought about APE when I read The 86open Project's FAQ. However, I feel that APE is less related to the blog post so decide not to add the link to the post.

CalChris 2y ago

Who controls the ELF standard? The best I can tell is that you submit a pull request with LLVM and maybe GCC.

MaskRay 2y ago
- 2003-2010 The SCO Group - 2011- Xinous, but Xinous has stopped updating https://www.sco.com/developers/gabi/latest/contents.html https://www.sco.com/developers/gabi/latest/contents.html . It seems that Xinous has moved on from System V based OpenServer/UnixWare. The newer OpenServer seems to be based on FreeBSD. They likely no longer care about the System V ABI. Nowadays, people make proposals to the generic-abi Google Group. Essentially, a proposal is considered "standardized" if it receives approval from GNU, LLVM, and Ali Bahrami (Solaris representative). Many GNU extensions are implemented by LLVM and adopted by BSD and newer ELF-based OSes. For extensions that are considered generic enough, it's recommended to propose them through the generic-abi Google Group. psABI documents generally prefer generic extensions over GNU or LLVM-specific ones.

matheusmoreira 2y ago

> Despite this uncertainty, innovation within the ELF ecosystem should continue. > In practice, achieving consensus among major toolchain vendors (GNU and LLVM) may be sufficient, even without formal approval from the generic ABI. That makes it sound easy. GNU projects seem to be very conservatively maintained. LLVM has around one trillion open pull requests. I came up with a little mechanism to get the kernel to automatically load into memory data embedded into the program. I used it in my programming language to embed code into the interpreter so that they can be automatically loaded and executed. https://www.matheusmoreira.com/articles/self-contained-lone-lisp-applications https://www.matheusmoreira.com/articles/self-contained-lone-... Only the maintainer of the relatively new mold linker cared to implement a helpful feature to make this kind of thing possible and easy. I requested the same feature in GNU ld and the idea wasn't exactly received with enthusiasm. I'm not sure LLVM ever received it at all. And that was just a linker option to add some extra PT_NULL segments for easy and efficient patching. Can't even imagine the effort it would take to actually change something about this ABI.

Joker_vD 2y ago
Isn't that's what NOTE segments are for? It can have arbitrary data embedded in it, that's how GCC implements most of its insane dynamic-linking-related extensions IIRC. Of course, it has to overlap with some LOAD segment to be useful. $ LANG=en_US objdump -x /bin/bash /bin/bash: file format elf64-x86-64 /bin/bash architecture: i386:x86-64, flags 0x00000150: HAS_SYMS, DYNAMIC, D_PAGED start address 0x0000000000032ef0 Program Header: PHDR off 0x0000000000000040 vaddr 0x0000000000000040 paddr 0x0000000000000040 align 2**3 filesz 0x00000000000002d8 memsz 0x00000000000002d8 flags r-- INTERP off 0x0000000000000318 vaddr 0x0000000000000318 paddr 0x0000000000000318 align 2**0 filesz 0x000000000000001c memsz 0x000000000000001c flags r-- LOAD off 0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**12 filesz 0x000000000002e188 memsz 0x000000000002e188 flags r-- LOAD off 0x000000000002f000 vaddr 0x000000000002f000 paddr 0x000000000002f000 align 2**12 filesz 0x00000000000def6d memsz 0x00000000000def6d flags r-x LOAD off 0x000000000010e000 vaddr 0x000000000010e000 paddr 0x000000000010e000 align 2**12 filesz 0x0000000000039b08 memsz 0x0000000000039b08 flags r-- LOAD off 0x0000000000148a90 vaddr 0x0000000000149a90 paddr 0x0000000000149a90 align 2**12 filesz 0x000000000000bbc0 memsz 0x0000000000016b28 flags rw- DYNAMIC off 0x000000000014b4c0 vaddr 0x000000000014c4c0 paddr 0x000000000014c4c0 align 2**3 filesz 0x0000000000000200 memsz 0x0000000000000200 flags rw- NOTE off 0x0000000000000338 vaddr 0x0000000000000338 paddr 0x0000000000000338 align 2**3 filesz 0x0000000000000030 memsz 0x0000000000000030 flags r-- NOTE off 0x0000000000000368 vaddr 0x0000000000000368 paddr 0x0000000000000368 align 2**2 filesz 0x0000000000000044 memsz 0x0000000000000044 flags r-- 0x6474e553 off 0x0000000000000338 vaddr 0x0000000000000338 paddr 0x0000000000000338 align 2**3 filesz 0x0000000000000030 memsz 0x0000000000000030 flags r-- EH_FRAME off 0x00000000001278a8 vaddr 0x00000000001278a8 paddr 0x00000000001278a8 align 2**2 filesz 0x000000000000472c memsz 0x000000000000472c flags r-- STACK off 0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**4 filesz 0x0000000000000000 memsz 0x0000000000000000 flags rw- RELRO off 0x0000000000148a90 vaddr 0x0000000000149a90 paddr 0x0000000000149a90 align 2**0 filesz 0x0000000000003570 memsz 0x0000000000003570 flags r-- ... $ hd -s 0x338 -n 0x30 /bin/bash 00000338 04 00 00 00 20 00 00 00 05 00 00 00 47 4e 55 00 |.... .......GNU.| 00000348 02 00 00 c0 04 00 00 00 03 00 00 00 00 00 00 00 |................| 00000358 02 80 00 c0 04 00 00 00 01 00 00 00 00 00 00 00 |................| 00000368 $ hd -s 0x368 -n 0x44 /bin/bash 00000368 04 00 00 00 14 00 00 00 03 00 00 00 47 4e 55 00 |............GNU.| 00000378 7a 64 08 ba 82 a2 d8 6d d9 8f 1f 75 ac 8e dc b6 |zd.....m...u....| 00000388 95 f6 fd 60 04 00 00 00 10 00 00 00 01 00 00 00 |...`............| 00000398 47 4e 55 00 00 00 00 00 03 00 00 00 02 00 00 00 |GNU.............| 000003a8 00 00 00 00 |....| 000003ac Plus, of course, you can always leave the section table in as well, with SHT_NOTE sections in it.

kibwen 2y ago

I'd be interested in reading about the history of object file formats in general. I'm wondering what motivates them being so relatively complicated for something that seems so conceptually simple; is it historical baggage or is it essential complexity?

KerrAvon 2y ago
Get yourself a copy of “Linkers and Loaders” — https://dl.acm.org/doi/10.5555/519563 https://dl.acm.org/doi/10.5555/519563 . Explains the history of it all.
MaskRay 2y ago
Linkers and Loaders I have analyzed a few object file formats in another blog post https://maskray.me/blog/2024-01-14-exploring-object-file-formats https://maskray.me/blog/2024-01-14-exploring-object-file-for... (HN discussion: https://news.ycombinator.com/item?id=38998914 https://news.ycombinator.com/item?id=38998914) ELF is technically not complicated and simpler than COFF and Mach-O.