4 ms·

With all the work on compilers and programming languages, doesn't it seem like linker and loader technology is ripe for a major overhaul? Loaders? N

With all the work on compilers and programming languages, doesn't it seem like linker and loader technology is ripe for a major overhaul? Loaders? Not really that just puts _text_ (raw executable machine code, and binary data) at certain locations in memory and updates some other _text_ to ensure branches/functions point to the right location. This isn't rocket science it works well. All in all loaders are very fast. Linkers? I would like to think so. It is just a damn complex problem. The real issue here is legacy support. Object files were a _hack_ to make compilers faster, use less resources, and give old school incremental-recompilation. To make linkers faster, better, etc. one really needs to move away from opaque object files. I like what the LLVM is doing with LTO but this _kind of_ assumes you have all the rs/c/cpp files your compiling so you can see everything in llvm-ir format. The real issue is build tools. Modern C/C++ have a high level tool orchestrate compiling/moving object files/linking object files. In some total you need half a dozen different programs to do all these task (cc, cp, rm, make, sed/awk, ln). Each tool has no clue what everything else is doing. You need to have that context for the linker to do it's job better. But now the linker has to do the job of all those tools :|

dfox 10y ago
What strikes me as weird is that I know of exactly one compiler system that got the incrementally compiled modules problem (almost) right and at the same time is AOT compiler without huge runtime: Turbo Pascal. The intermediate files contained both compiler readable definitions as well as object code, the linker was able to link only the required parts without resorting to hacks such as having each function in separate source file (see glibc) and the toolchain was able to generate missing intermediate files even without having to explicitly name them on command line. FPC implements the same mechanism by splitting the compiler metadata and object file into separate on-disk files, which seems like good compromise. The problem with such approach is that the compiler frontends have to know about each other and produce/consume compatible metadata. For C (and Fortran and...) you an get away without sharing any metadata between separate compilation units and when you cannot (constants, structure layouts) CPP is workable and simple solution, this ignores the problem of inline functions defined in headers, but there is not that much C code that does that.
- valarauca1 10y ago
  What strikes me as weird is that I know of exactly one compiler system that got it right (...) Turbo Pascal. Long Live Turbo Pascal. when you cannot (...) CPP I really see no way forward to be able to modernize CPP build systems. CPP has invented it's own paradigm for how a build system should work. Unless square 1 is "support CPP" making something support cpp is almost impossible.
  - dfox 10y ago
    cpp(1) is not where the paradigm stems from, the same paradigm was there before and was there even before C itself. The whole idea of preprocessor stems from assembly programming and the idea that having one file with lines like "PORTB .equ 0x17" and reusing it is useful, in the same spirit the way how ld(1) works was/is useful for assembly programming (you can jump arbitrarily between notionally different functions, and in fact GCC is perfectly capable of generating object files that do that from C/C++ code). On the other hand, cpp(1) is useful for more things than just C/C++, there are even programs which read their configuration by means of popen("cpp /etc/foo"). And as for supporting ld(1) and unix toolchain in general, see my remark above about how FPC works, IIRC MSVC compiller does something similar for C++, albeit in more limited manner (stdaux.h and such stuff).
- rwmj 10y ago
  OCaml does much the same, but it embeds the meta-information in a .cmx file which sits alongside the .o file. It allows cross-module inlining of single functions for example. OTOH OCaml has another problem which is that the format of the .cmx file is just a dump of internal OCaml compiler structures so it changes on every compiler version. (You do get an error message, not a crash, but you still end up having to compile everything with the same compiler version)
rwmj 10y ago
> Not really that just puts _text_ (raw executable machine code, and binary data) at certain locations in memory and updates some other _text_ to ensure branches/functions point to the right location. This isn't rocket science it works well. All in all loaders are very fast. Really, they are not. In fact resolving symbols in ELF shared libraries is inherently complex and slow. Pick any program which is linked to lots of shared libraries and try running 'time program --version'. 'qemu-system-x86_64 -version' takes 60ms on my laptop.
- valarauca1 10y ago
  I have in fact written my own ELF parser. I was speaking more to static libraries/binaries which once parsed are only a handful of pointer jumps/memset calls. Dynamic segments are very complex :| The load time for Dynamic libraries isn't really on the _loader_. You need to do disk searches/file IO to load the libraries parsing MANY additional files. Starting a program that has 20 dynamic links isn't starting a program. It is starting 20. The bottle neck is not parsing, it is File IO.
  - vetinari 10y ago
    There is another thing that slows down the symbol resolution: the symbols are not scoped per library or soname, they are global and so every symbol is searched in every linked library. Link few C++ libraries, the symbol count explodes and the runtime linking slows downs. Large C++ apps, like OpenOffice.org, used to have a problem with this. The I/O may or may not be a bottleneck. The libraries are being mmaped and if some other process is already using them, chances are, that the interesting parts are already paged in.
    - brucedawson 10y ago
      Whenever I hear about how loaders resolve symbols on Linux I am amazed that it works at all, and that it runs reasonably fast. The loader on Windows has a much simpler job - it imports specific symbols from specific DLLs. In many cases the importing is done by ordinal instead of by name, to make it even faster. I'm used to the Windows system so I don't miss the extra functionality that the Linux loader provides - in fact I find it scary.
barrkel 10y ago
There's not much difference between a linker and a loader; they do almost the same thing, one persisting the result to disk and the other to memory. I do notice that your definition of a loader does not support shared libraries, which is most of what a loader does outside of a toy OS. Linkers are also similar to garbage collectors; smart linkers effectively eliminate bloat by garbage collecting the code.

dfox 10y ago

What strikes me as weird is that I know of exactly one compiler system that got the incrementally compiled modules problem (almost) right and at the same time is AOT compiler without huge runtime: Turbo Pascal. The intermediate files contained both compiler readable definitions as well as object code, the linker was able to link only the required parts without resorting to hacks such as having each function in separate source file (see glibc) and the toolchain was able to generate missing intermediate files even without having to explicitly name them on command line. FPC implements the same mechanism by splitting the compiler metadata and object file into separate on-disk files, which seems like good compromise. The problem with such approach is that the compiler frontends have to know about each other and produce/consume compatible metadata. For C (and Fortran and...) you an get away without sharing any metadata between separate compilation units and when you cannot (constants, structure layouts) CPP is workable and simple solution, this ignores the problem of inline functions defined in headers, but there is not that much C code that does that.

valarauca1 10y ago
What strikes me as weird is that I know of exactly one compiler system that got it right (...) Turbo Pascal. Long Live Turbo Pascal. when you cannot (...) CPP I really see no way forward to be able to modernize CPP build systems. CPP has invented it's own paradigm for how a build system should work. Unless square 1 is "support CPP" making something support cpp is almost impossible.
- dfox 10y ago
  cpp(1) is not where the paradigm stems from, the same paradigm was there before and was there even before C itself. The whole idea of preprocessor stems from assembly programming and the idea that having one file with lines like "PORTB .equ 0x17" and reusing it is useful, in the same spirit the way how ld(1) works was/is useful for assembly programming (you can jump arbitrarily between notionally different functions, and in fact GCC is perfectly capable of generating object files that do that from C/C++ code). On the other hand, cpp(1) is useful for more things than just C/C++, there are even programs which read their configuration by means of popen("cpp /etc/foo"). And as for supporting ld(1) and unix toolchain in general, see my remark above about how FPC works, IIRC MSVC compiller does something similar for C++, albeit in more limited manner (stdaux.h and such stuff).
rwmj 10y ago
OCaml does much the same, but it embeds the meta-information in a .cmx file which sits alongside the .o file. It allows cross-module inlining of single functions for example. OTOH OCaml has another problem which is that the format of the .cmx file is just a dump of internal OCaml compiler structures so it changes on every compiler version. (You do get an error message, not a crash, but you still end up having to compile everything with the same compiler version)

rwmj 10y ago

> Not really that just puts _text_ (raw executable machine code, and binary data) at certain locations in memory and updates some other _text_ to ensure branches/functions point to the right location. This isn't rocket science it works well. All in all loaders are very fast. Really, they are not. In fact resolving symbols in ELF shared libraries is inherently complex and slow. Pick any program which is linked to lots of shared libraries and try running 'time program --version'. 'qemu-system-x86_64 -version' takes 60ms on my laptop.

valarauca1 10y ago
I have in fact written my own ELF parser. I was speaking more to static libraries/binaries which once parsed are only a handful of pointer jumps/memset calls. Dynamic segments are very complex :| The load time for Dynamic libraries isn't really on the _loader_. You need to do disk searches/file IO to load the libraries parsing MANY additional files. Starting a program that has 20 dynamic links isn't starting a program. It is starting 20. The bottle neck is not parsing, it is File IO.
- vetinari 10y ago
  There is another thing that slows down the symbol resolution: the symbols are not scoped per library or soname, they are global and so every symbol is searched in every linked library. Link few C++ libraries, the symbol count explodes and the runtime linking slows downs. Large C++ apps, like OpenOffice.org, used to have a problem with this. The I/O may or may not be a bottleneck. The libraries are being mmaped and if some other process is already using them, chances are, that the interesting parts are already paged in.
  - brucedawson 10y ago
    Whenever I hear about how loaders resolve symbols on Linux I am amazed that it works at all, and that it runs reasonably fast. The loader on Windows has a much simpler job - it imports specific symbols from specific DLLs. In many cases the importing is done by ordinal instead of by name, to make it even faster. I'm used to the Windows system so I don't miss the extra functionality that the Linux loader provides - in fact I find it scary.

barrkel 10y ago

There's not much difference between a linker and a loader; they do almost the same thing, one persisting the result to disk and the other to memory. I do notice that your definition of a loader does not support shared libraries, which is most of what a loader does outside of a toy OS. Linkers are also similar to garbage collectors; smart linkers effectively eliminate bloat by garbage collecting the code.