4 ms·

Link time optimizations were done in the 1980s if I recall correctly. I never tried to implement them, finding it easier and more effective for the compiler to

Link time optimizations were done in the 1980s if I recall correctly. I never tried to implement them, finding it easier and more effective for the compiler to simply compile all the source files at the same time. The D compiler is designed to be able to build one object file per source file at a time, or one object file which combines all of the source files. Most people choose the one object file.

srean 1y ago
I think MLton does it this way. http://mlton.org/WholeProgramOptimization http://mlton.org/WholeProgramOptimization Dynamically linked and dynamically loaded libraries are useful though (paid for with its problems of course)
tester756 1y ago
Yea, generating many object files seems like weird thing. Maybe it was good thing decades ago, but now? Because then you need to link them, thus you need some kind of linker. Just generate one output file and skip the linker
- WalterBright 1y ago
  I've considered many times doing just that.
  - tester756 1y ago
    And what was the result/conclusion of such considerations?
    - WalterBright 1y ago
      Not worth the effort. 1. linkers have increased enormously in complexity 2. little commonality between linkers for different platforms 3. compatibility with the standalone linkers 4. trying to keep up with constant enhancement of existing linkers
- yencabulator 1y ago
  Not maybe. Sufficient RAM for compilation was a serious issue back in the day.
- kazinator 1y ago
  Sure, and if any file is touched, just process them all.
  - adrian_b 1y ago
    Some compilers had incremental compilation to handle this during development builds. Then only the functions touched inside some file would be recompiled, not the remainder of the file or other files. Obviously, choosing incremental compilation inhibited some optimizations.
- adrian_b 1y ago
  Generating many object files is pointless for building an executable or a dynamic library, but it remains the desired behavior for building a static library. Many software projects that must generate multiple executables are better structured as a static library plus one source file with the "main" function for each executable.
  - WalterBright 1y ago
    One thing the D compiler does is it can generate a library in one step (no need to use the librarian). Give a bunch of source files and object files on the command line, specify a library as the output, and boom! library created directly (compiling the source files, and adding the object files). I haven't used a librarian program for maybe a decade.
senkora 1y ago
In C++, there is a trick to get this behavior called "unity builds", where you include all of your source files into a single file and then invoke the compiler on that file. Of course, being C++, this subtly changes behavior and must be done carefully. I like this article that explains the ins and outs of using unity builds: https://austinmorlan.com/posts/unity_jumbo_build/ https://austinmorlan.com/posts/unity_jumbo_build/
- WalterBright 1y ago
  > this subtly changes behavior The D module design ensures that module imports are independent of each other and are independent of the importer.
YorickPeterse 1y ago
For Inko (https://inko-lang.org/ https://inko-lang.org/) I went a step further: it generates an object file for each type, instead of per source file or per project. The idea is that if e.g. a generic type is specialized into a new instance (or has some methods added to it), only the object file for that type needs to be re-generated. This in turn should allow for much more fine-grained incremental compilation. The downside is that you can end up with thousands of object files, but for modern linkers that isn't a problem.
dooglius 1y ago
It sounds like this would prevent the inherit concurrency you would get out of handling files separately?
- WalterBright 1y ago
  It's complicated and not at all clear. For example, most modules import other modules. With separate compilation, most of the modules need to be compiled multiple times, with all-together, it's only once. On the other hand, the optimizer and code generator can be run concurrently in multiple processes/threads.

srean 1y ago

I think MLton does it this way. http://mlton.org/WholeProgramOptimization http://mlton.org/WholeProgramOptimization Dynamically linked and dynamically loaded libraries are useful though (paid for with its problems of course)

tester756 1y ago

Yea, generating many object files seems like weird thing. Maybe it was good thing decades ago, but now? Because then you need to link them, thus you need some kind of linker. Just generate one output file and skip the linker

WalterBright 1y ago
I've considered many times doing just that.
- tester756 1y ago
  And what was the result/conclusion of such considerations?
  - WalterBright 1y ago
    Not worth the effort. 1. linkers have increased enormously in complexity 2. little commonality between linkers for different platforms 3. compatibility with the standalone linkers 4. trying to keep up with constant enhancement of existing linkers
yencabulator 1y ago
Not maybe. Sufficient RAM for compilation was a serious issue back in the day.
kazinator 1y ago
Sure, and if any file is touched, just process them all.
- adrian_b 1y ago
  Some compilers had incremental compilation to handle this during development builds. Then only the functions touched inside some file would be recompiled, not the remainder of the file or other files. Obviously, choosing incremental compilation inhibited some optimizations.
adrian_b 1y ago
Generating many object files is pointless for building an executable or a dynamic library, but it remains the desired behavior for building a static library. Many software projects that must generate multiple executables are better structured as a static library plus one source file with the "main" function for each executable.
- WalterBright 1y ago
  One thing the D compiler does is it can generate a library in one step (no need to use the librarian). Give a bunch of source files and object files on the command line, specify a library as the output, and boom! library created directly (compiling the source files, and adding the object files). I haven't used a librarian program for maybe a decade.

senkora 1y ago

In C++, there is a trick to get this behavior called "unity builds", where you include all of your source files into a single file and then invoke the compiler on that file. Of course, being C++, this subtly changes behavior and must be done carefully. I like this article that explains the ins and outs of using unity builds: https://austinmorlan.com/posts/unity_jumbo_build/ https://austinmorlan.com/posts/unity_jumbo_build/

WalterBright 1y ago
> this subtly changes behavior The D module design ensures that module imports are independent of each other and are independent of the importer.

YorickPeterse 1y ago

For Inko (https://inko-lang.org/ https://inko-lang.org/) I went a step further: it generates an object file for each type, instead of per source file or per project. The idea is that if e.g. a generic type is specialized into a new instance (or has some methods added to it), only the object file for that type needs to be re-generated. This in turn should allow for much more fine-grained incremental compilation. The downside is that you can end up with thousands of object files, but for modern linkers that isn't a problem.

dooglius 1y ago

It sounds like this would prevent the inherit concurrency you would get out of handling files separately?

WalterBright 1y ago
It's complicated and not at all clear. For example, most modules import other modules. With separate compilation, most of the modules need to be compiled multiple times, with all-together, it's only once. On the other hand, the optimizer and code generator can be run concurrently in multiple processes/threads.