4 ms·

The SQLite Amalgamation

76 points by burglins 2y ago

layla5alive 2y ago
This appears to be a hand-rolled Unity build..
rtpg 2y ago
Xonsh, a Python-based shell, used to do this for startup speed reasons, but got rid of it on account of stack traces being a mess afterwards.
stevefan1999 2y ago
Unreal Engine also do something called a Unity build (the irony) which also put multiple cpp files in one big file by using #include
- forrestthewoods 2y ago
  That's not really the same thing. Unreal's "Unity" build is more of a build time optimization. It doesn't output actual source files you can drop into a project like SQLite's amalgamation. Perhaps similar in structure. But different in both goal and output.
  - lifthrasiir 2y ago
    Both do share a minor but significant performance advantage. Also unity builds don't really improve build time much by itself, it only makes sense because many unchanged files are still too many to link in incremental settings.
    - forrestthewoods 2y ago
      I’m pretty sure the primary reason Unity builds exist is to make builds faster by eliminating massively duplicated header compiles…
      - lifthrasiir 2y ago
        You have probably confused unity builds from precompiled header files, which can be combined in the same fashion. Unity builds just mean the combined compilation of otherwise multiple files and therefore the reduction of output files to be further processed. An unqualified "unity build" typically means unity builds where inputs are source files and outputs are object files to be linked. This is most evident by considering how would you use corresponding header files from your project: you would still include the same set of header files. PCH unity builds would instead force you to include a single dedicated header file, like `#include "pch.h"`. I never heard them to be called just "unity builds" in my experience, in fact it was more likely that "PCH" implied a unity build of PCH files.
        fanf2 2y ago
        This has a good comparison of compile times with unity builds and PCH https://alexanderhoughton.co.uk/blog/faster-compiling-visual-studio-unity-jumbo-builds/ https://alexanderhoughton.co.uk/blog/faster-compiling-visual...
        lifthrasiir 2y ago
        That page correctly mentions what is really happening: the same set of headers was used for all source files, so they were very easy to eliminate either by a single compile process with a shared cache or by having headers pre-compiled. The test header file also contains <windows.h> and <iostream>, which are known to be very large and only sparingly used compared to others, so unity builds were unexpectedly close to PCH in terms of performance here. (Also typical mid-sized C++ projects that benefit from unity builds are not that fast to compile ;-) In any case, the distinction between unity builds and PCH should remain clear.
matheusmoreira 2y ago
> And because all code is in a single translation unit, compilers can do better inter-procedure and inlining optimization resulting in machine code that is between 5% and 10% faster. It's weird that this is still true even though compilers are now capable of whole program optimization at link time. Enabling LTO should be equivalent to compiling everything as a single translation unit but apparently it isn't. I wonder what exactly inhibits the aforementioned inter-procedural and code inlining optimizations in LTO builds.
- lifthrasiir 2y ago
  Because LTO is a global decision while amalgamation (or unity builds) is a local decision. As a global decision, LTO can't be too slow and is subject to heavier trade-offs. Combining files in this way is a local decision that you can just do if you know it would be a-okay and doesn't otherwise impact the whole build.
Tiberium 2y ago
FWIW it's possible to amalgamate almost any C program automatically with the help of https://github.com/goblint/cil https://github.com/goblint/cil (a maintained fork of CIL): https://goblint.github.io/cil/merger.html https://goblint.github.io/cil/merger.html All you have to do is to make your CC this: cilly --noPrintLn --merge --keepmerged And in the end after compilation there will be a file named yourproject_comb.c
deleted 2y ago
[deleted]

76 points by burglins 2y ago

This appears to be a hand-rolled Unity build..

rtpg 2y ago

Xonsh, a Python-based shell, used to do this for startup speed reasons, but got rid of it on account of stack traces being a mess afterwards.

stevefan1999 2y ago

Unreal Engine also do something called a Unity build (the irony) which also put multiple cpp files in one big file by using #include

forrestthewoods 2y ago
That's not really the same thing. Unreal's "Unity" build is more of a build time optimization. It doesn't output actual source files you can drop into a project like SQLite's amalgamation. Perhaps similar in structure. But different in both goal and output.
- lifthrasiir 2y ago
  Both do share a minor but significant performance advantage. Also unity builds don't really improve build time much by itself, it only makes sense because many unchanged files are still too many to link in incremental settings.
  - forrestthewoods 2y ago
    I’m pretty sure the primary reason Unity builds exist is to make builds faster by eliminating massively duplicated header compiles…
    - lifthrasiir 2y ago
      You have probably confused unity builds from precompiled header files, which can be combined in the same fashion. Unity builds just mean the combined compilation of otherwise multiple files and therefore the reduction of output files to be further processed. An unqualified "unity build" typically means unity builds where inputs are source files and outputs are object files to be linked. This is most evident by considering how would you use corresponding header files from your project: you would still include the same set of header files. PCH unity builds would instead force you to include a single dedicated header file, like `#include "pch.h"`. I never heard them to be called just "unity builds" in my experience, in fact it was more likely that "PCH" implied a unity build of PCH files.
      - fanf2 2y ago
        This has a good comparison of compile times with unity builds and PCH https://alexanderhoughton.co.uk/blog/faster-compiling-visual-studio-unity-jumbo-builds/ https://alexanderhoughton.co.uk/blog/faster-compiling-visual...
        lifthrasiir 2y ago
        That page correctly mentions what is really happening: the same set of headers was used for all source files, so they were very easy to eliminate either by a single compile process with a shared cache or by having headers pre-compiled. The test header file also contains <windows.h> and <iostream>, which are known to be very large and only sparingly used compared to others, so unity builds were unexpectedly close to PCH in terms of performance here. (Also typical mid-sized C++ projects that benefit from unity builds are not that fast to compile ;-) In any case, the distinction between unity builds and PCH should remain clear.

matheusmoreira 2y ago

> And because all code is in a single translation unit, compilers can do better inter-procedure and inlining optimization resulting in machine code that is between 5% and 10% faster. It's weird that this is still true even though compilers are now capable of whole program optimization at link time. Enabling LTO should be equivalent to compiling everything as a single translation unit but apparently it isn't. I wonder what exactly inhibits the aforementioned inter-procedural and code inlining optimizations in LTO builds.

lifthrasiir 2y ago
Because LTO is a global decision while amalgamation (or unity builds) is a local decision. As a global decision, LTO can't be too slow and is subject to heavier trade-offs. Combining files in this way is a local decision that you can just do if you know it would be a-okay and doesn't otherwise impact the whole build.

Tiberium 2y ago

FWIW it's possible to amalgamate almost any C program automatically with the help of https://github.com/goblint/cil https://github.com/goblint/cil (a maintained fork of CIL): https://goblint.github.io/cil/merger.html https://goblint.github.io/cil/merger.html All you have to do is to make your CC this: cilly --noPrintLn --merge --keepmerged And in the end after compilation there will be a file named yourproject_comb.c

deleted 2y ago

[deleted]