4 ms·

"Lastly, we started combining files, and we found out that combining all of our 200 models into one file decreased the compilation time from 1min35sec, to just

"Lastly, we started combining files, and we found out that combining all of our 200 models into one file decreased the compilation time from 1min35sec, to just 17sec. So we are like, "Hold on, this is interesting, combining everything into one makes it much faster." The reason for this is that, as much as I know, that a compiler does type checking for every single file. So if you spawn 200 processes of Swift compilers, it needs to 200x check all the other files and make sure that you're using the correct types. So combining everything into one makes it much faster. " Good to know

flamedoge 9y ago
It's probably time spent in the linker. The advice of using WPO without opt is verrry circumstantial.
- slavapestov 9y ago
  No, it's valid advice. Non-WMO starts a frontend job for each file, WMO runs one job. The frontend jobs do not share state, so each one parses all the files in the module. In general the parser is very fast, and the type checker tries to only type check declarations in files other than the primary file when absolutely necessary, so it's not always O(n^2). But there are pathological cases you can construct today where the type checker ends up doing too much work.
  - flamedoge 9y ago
    It just feels hacky to do it the way author suggested. You can just invoke all the files in cmdline, swift *.swift to same effect I think.
    - slavapestov 9y ago
      `swiftc * .swift` spawns one frontend job per file and then runs the linker to link together the .o's. `swiftc -whole-module-optimization * .swift` compiles all files in a single frontend job. Note that -O is independent of -whole-module-optimization, which is perhaps a bit confusing.
symisc_devel 9y ago
The technique is known as Amalgamation and was first introduced in the SQLite2 source tree. In my company were we do embedded software[1], this shown to be a powerful technique. Not only, the compilation is extremely fast (A 1.3 megabytes of C code, took 9 seconds to compile on a core I3), but a modern compiler will be able to do additional optimizations on code when it is contained with in a single translation unit. [1]: https://unqlite.org https://unqlite.org, http://ph7.symisc.net http://ph7.symisc.net
- lanna 9y ago
  Why don't compilers do amalgamation themselves as a pre-compilation step?
  - monocasa 9y ago
    You could argue that's the point of link time optimization.
  - kornish 9y ago
    My guess is that it's to prevent compilation of code which is unused. If code which isn't actually called anywhere is amalgamated and compiled, compilation time could actually increase instead of decrease. Seems easily avoidable though through basic dependency analysis, though.
    - klodolph 9y ago
      That's not actually true. You have to pass special linker flags to tell the linker to avoid including code which isn't used. With GNU Binutils and GCC, those flags are -Wl,--as-needed and -Wl,--gc-sections. Normally, unused code is only excluded if it is part of a static library. Even then it is only excluded or included an entire file at a time, unless you split files into multiple sections with the compiler (which has drawbacks—the compiler can do certain optimizations if it knows that two pieces of code or some code and data end up in the same section).
      - 9y ago

flamedoge 9y ago

It's probably time spent in the linker. The advice of using WPO without opt is verrry circumstantial.

slavapestov 9y ago
No, it's valid advice. Non-WMO starts a frontend job for each file, WMO runs one job. The frontend jobs do not share state, so each one parses all the files in the module. In general the parser is very fast, and the type checker tries to only type check declarations in files other than the primary file when absolutely necessary, so it's not always O(n^2). But there are pathological cases you can construct today where the type checker ends up doing too much work.
- flamedoge 9y ago
  It just feels hacky to do it the way author suggested. You can just invoke all the files in cmdline, swift *.swift to same effect I think.
  - slavapestov 9y ago
    `swiftc * .swift` spawns one frontend job per file and then runs the linker to link together the .o's. `swiftc -whole-module-optimization * .swift` compiles all files in a single frontend job. Note that -O is independent of -whole-module-optimization, which is perhaps a bit confusing.

symisc_devel 9y ago

The technique is known as Amalgamation and was first introduced in the SQLite2 source tree. In my company were we do embedded software[1], this shown to be a powerful technique. Not only, the compilation is extremely fast (A 1.3 megabytes of C code, took 9 seconds to compile on a core I3), but a modern compiler will be able to do additional optimizations on code when it is contained with in a single translation unit. [1]: https://unqlite.org https://unqlite.org, http://ph7.symisc.net http://ph7.symisc.net

lanna 9y ago
Why don't compilers do amalgamation themselves as a pre-compilation step?
- monocasa 9y ago
  You could argue that's the point of link time optimization.
- kornish 9y ago
  My guess is that it's to prevent compilation of code which is unused. If code which isn't actually called anywhere is amalgamated and compiled, compilation time could actually increase instead of decrease. Seems easily avoidable though through basic dependency analysis, though.
  - klodolph 9y ago
    That's not actually true. You have to pass special linker flags to tell the linker to avoid including code which isn't used. With GNU Binutils and GCC, those flags are -Wl,--as-needed and -Wl,--gc-sections. Normally, unused code is only excluded if it is part of a static library. Even then it is only excluded or included an entire file at a time, unless you split files into multiple sections with the compiler (which has drawbacks—the compiler can do certain optimizations if it knows that two pieces of code or some code and data end up in the same section).
    - 9y ago