4 ms·

One of the things I've been (very slowly) putting together is a master list of all the references one needs to build a compiler toolchain--such as processor ISA

One of the things I've been (very slowly) putting together is a master list of all the references one needs to build a compiler toolchain--such as processor ISA manuals, ABI specifications, language standards, even things like IEEE 754 or DWARF. While IEEE 754 is an issue because it's not freely available for most people, the only document that has truly stumped me is the ELF specification. From your blog post, it seems like my failure to find the most up-to-date specification is simply because one doesn't exist.

marssaxman 2y ago
Are you familiar with the OSDev wiki? Perhaps their list of links will be helpful: https://wiki.osdev.org/ELF#External_Links https://wiki.osdev.org/ELF#External_Links
- jcranmer 2y ago
  That wiki's list of links isn't always the most up-to-date.
  - marssaxman 2y ago
    How often does anyone change the ELF file format, though? No matter; but that's the most up-to-date list of references I know about.
    - jcranmer 2y ago
      The ELF format isn't the one that I'm the most concerned about, but for example compressed sections were updated in the past decade. The ABI for x86-64 is out-of-date (actually, the site it links to is now a 404, but my recollection was that was a version around 2013)--missing things like _Float16 rules, DWARF register naming for AMX and APX registers, microarchitecture levels, new relocation types,
      - khuey 2y ago
        The up to date ABI for x86-64 lives at https://gitlab.com/x86-psABIs/x86-64-ABI https://gitlab.com/x86-psABIs/x86-64-ABI
        jcranmer 2y ago
        I'm aware of that (and peeved that they've got runners set up for the x86-64 ABI but not the i386 ABI, so there's no nice PDF version of the i386 version I can use).
agent281 2y ago
Are you going to make this list of resources available online or is it for personal consumption only?
- jcranmer 2y ago
  So far, the list of resources is a collection of PDFs I've downloaded plus a script to download the latest version of every relevant specification. (Although it does seem that Intel likes breaking its links to the latest ISA PDF every now and then). I don't have a great solution for things that are essentially web-only pages, which is how the Itanium ABI is handled (and unlike C, C++ makes it hard to find the relevant working draft copies, instead preferring to use the HTML-based https://eel.is/c++draft https://eel.is/c++draft for that). I probably will make it publicly available eventually, it just hasn't been a priority for me.
  - int_19h 2y ago
    The most widespread standard for archiving collections of web pages is WARC - it is a publicly available ISO standard so it's very future-proof. WACZ (https://specs.webrecorder.net/wacz/1.1.1/ https://specs.webrecorder.net/wacz/1.1.1/) is a more recent take that is basically zipped WARC + index + metadata, and Internet Archive can generate them when archiving is requested (https://web.archive.org/save/ https://web.archive.org/save/). In addition to that, I would also suggest using https://zimit.kiwix.org https://zimit.kiwix.org. It's a very compact representation that has full text index support for convenient searching, the format is open and documented, and first-party readers are available on all major platforms. And for stuff like this, Kiwix might actually be interested in adding them to their official library at https://library.kiwix.org/ https://library.kiwix.org/ if you contact them.
MaskRay 2y ago
Thank you! This will be very useful. Yes, for C/C++ one needs: * ISA manual * ELF (generic ABI, psABI (e.g. x86-64-psABI), OSABI) * DWARF * Floating-point * Language standards * Itanium C++ ABI and probably a few other stuff. > it seems like my failure to find the most up-to-date specification is simply because one doesn't exist. While the up-to-date specification is unavailable, hopefully it is not too bad because all essential features have been completed as of 2001:) RELR is a linker and loader feature, not on the compiler side. Compressed debug sections are a pretty natural extension.
- jcranmer 2y ago
  I file things like linkers, loaders, and debuggers into the same bucket as compilers, so all of the ELF extensions are interesting to me, even if compilers directly don't interact with them. I'm also interested in things like the loader-debugger rendezvous interface. Basically any document anyone who touches any project in llvm-project might need to read. And then there are things like the layout of .gcc_except_table, which are undocumented except for random blog posts.
  - WalterBright 2y ago
    > there are things like the layout of .gcc_except_table, which are undocumented except for random blog posts. The way I dealt with it was write an ELF dumper for the tables, and then run various C++ code snippets through g++ and look at how the tables were constructed.
    - gumby 2y ago
      You could just have called objdump — that’s what it’s for.
      - WalterBright 2y ago
        I like mine better :-/ The same executable will also detect and dump Elf, Mach, MScoff, and OMF object files. Besides, one doesn't learn anything about how to decode the format by using an existing tool that presents it in decoded form.
zorgmonkey 2y ago
I wish their was a better option, but people can at least get the IEEE 754 standard by putting the DOI for it into sci-hub.

marssaxman 2y ago

Are you familiar with the OSDev wiki? Perhaps their list of links will be helpful: https://wiki.osdev.org/ELF#External_Links https://wiki.osdev.org/ELF#External_Links

jcranmer 2y ago
That wiki's list of links isn't always the most up-to-date.
- marssaxman 2y ago
  How often does anyone change the ELF file format, though? No matter; but that's the most up-to-date list of references I know about.
  - jcranmer 2y ago
    The ELF format isn't the one that I'm the most concerned about, but for example compressed sections were updated in the past decade. The ABI for x86-64 is out-of-date (actually, the site it links to is now a 404, but my recollection was that was a version around 2013)--missing things like _Float16 rules, DWARF register naming for AMX and APX registers, microarchitecture levels, new relocation types,
    - khuey 2y ago
      The up to date ABI for x86-64 lives at https://gitlab.com/x86-psABIs/x86-64-ABI https://gitlab.com/x86-psABIs/x86-64-ABI
      - jcranmer 2y ago
        I'm aware of that (and peeved that they've got runners set up for the x86-64 ABI but not the i386 ABI, so there's no nice PDF version of the i386 version I can use).

agent281 2y ago

Are you going to make this list of resources available online or is it for personal consumption only?

jcranmer 2y ago
So far, the list of resources is a collection of PDFs I've downloaded plus a script to download the latest version of every relevant specification. (Although it does seem that Intel likes breaking its links to the latest ISA PDF every now and then). I don't have a great solution for things that are essentially web-only pages, which is how the Itanium ABI is handled (and unlike C, C++ makes it hard to find the relevant working draft copies, instead preferring to use the HTML-based https://eel.is/c++draft https://eel.is/c++draft for that). I probably will make it publicly available eventually, it just hasn't been a priority for me.
- int_19h 2y ago
  The most widespread standard for archiving collections of web pages is WARC - it is a publicly available ISO standard so it's very future-proof. WACZ (https://specs.webrecorder.net/wacz/1.1.1/ https://specs.webrecorder.net/wacz/1.1.1/) is a more recent take that is basically zipped WARC + index + metadata, and Internet Archive can generate them when archiving is requested (https://web.archive.org/save/ https://web.archive.org/save/). In addition to that, I would also suggest using https://zimit.kiwix.org https://zimit.kiwix.org. It's a very compact representation that has full text index support for convenient searching, the format is open and documented, and first-party readers are available on all major platforms. And for stuff like this, Kiwix might actually be interested in adding them to their official library at https://library.kiwix.org/ https://library.kiwix.org/ if you contact them.

MaskRay 2y ago

Thank you! This will be very useful. Yes, for C/C++ one needs: * ISA manual * ELF (generic ABI, psABI (e.g. x86-64-psABI), OSABI) * DWARF * Floating-point * Language standards * Itanium C++ ABI and probably a few other stuff. > it seems like my failure to find the most up-to-date specification is simply because one doesn't exist. While the up-to-date specification is unavailable, hopefully it is not too bad because all essential features have been completed as of 2001:) RELR is a linker and loader feature, not on the compiler side. Compressed debug sections are a pretty natural extension.

jcranmer 2y ago
I file things like linkers, loaders, and debuggers into the same bucket as compilers, so all of the ELF extensions are interesting to me, even if compilers directly don't interact with them. I'm also interested in things like the loader-debugger rendezvous interface. Basically any document anyone who touches any project in llvm-project might need to read. And then there are things like the layout of .gcc_except_table, which are undocumented except for random blog posts.
- WalterBright 2y ago
  > there are things like the layout of .gcc_except_table, which are undocumented except for random blog posts. The way I dealt with it was write an ELF dumper for the tables, and then run various C++ code snippets through g++ and look at how the tables were constructed.
  - gumby 2y ago
    You could just have called objdump — that’s what it’s for.
    - WalterBright 2y ago
      I like mine better :-/ The same executable will also detect and dump Elf, Mach, MScoff, and OMF object files. Besides, one doesn't learn anything about how to decode the format by using an existing tool that presents it in decoded form.

zorgmonkey 2y ago

I wish their was a better option, but people can at least get the IEEE 754 standard by putting the DOI for it into sci-hub.