5 ms·

I am not sure faster compilation is the way to go. Cutting 30 sec to 15 helps little and the acceleration of compilation can't catch up with the growth of packa

I am not sure faster compilation is the way to go. Cutting 30 sec to 15 helps little and the acceleration of compilation can't catch up with the growth of packages in the long run. Also, multi-threaded precompilation sounds a bad idea on managed clusters where resource usage should be controlled. I wonder how much Julia devs have done on caching precompiled binaries. That, though non-trivial to implement properly, is the ultimate solution to slow compilation.

ChrisRackauckas 6y ago
We recently acquired new hardware and plan to look into it. Invalidations can get in the way, but there are some things that can be done.
- deleted 6y ago
  [deleted]
fanf2 6y ago
The manual says “Large modules can take several seconds to load because executing all of the statements in a module often involves compiling a large amount of code. Julia creates precompiled caches of the module to reduce this time.” https://docs.julialang.org/en/v1/manual/modules/#Module-initialization-and-precompilation https://docs.julialang.org/en/v1/manual/modules/#Module-init...
- attractivechaos 6y ago
  This sounds great in theory, but in practice julia-1.5.2 still takes 10 sec to start when I use a medium-sized package including its dependencies (16 compiled files in total). If julia is loading cached files, I would expect something below 1 sec.
  - roywiggins 6y ago
    You can manually precompile a system image with libraries you find yourself using often, like Plots.jl. The usability of it stinks, but... you can do it, and it speeds common things up. https://julialang.github.io/PackageCompiler.jl/dev/examples/plots/ https://julialang.github.io/PackageCompiler.jl/dev/examples/...
    - ChrisRackauckas 6y ago
      Its usability is being integrated into the package manager so that it automatically gives warnings and triggers when you need to redo the sysimage. Not in v1.6, but there's been a lot of discussions for doing it shortly after. That is something I'm excited for.
    - attractivechaos 6y ago
      Thanks for the tip. This does look promising.
- celrod 6y ago
  Just to clarify, those precompiled caches are (currently) not of binaries. At the very least, LLVM still needs to generate code after loading, and often everything up to inference still needs to be run as well. Julia 1.6 involved a lot of work reducing invalidations, where old methods are invalidated by new definitions and thus forced to recompile. This has the obvious benefit of reducing the total amount of compilation needed, but the secondary benefit of making compiling to binaries more profitable; it's less likely that work will be wasted when someone loads another package.

ChrisRackauckas 6y ago

We recently acquired new hardware and plan to look into it. Invalidations can get in the way, but there are some things that can be done.

deleted 6y ago
[deleted]

fanf2 6y ago

The manual says “Large modules can take several seconds to load because executing all of the statements in a module often involves compiling a large amount of code. Julia creates precompiled caches of the module to reduce this time.” https://docs.julialang.org/en/v1/manual/modules/#Module-initialization-and-precompilation https://docs.julialang.org/en/v1/manual/modules/#Module-init...

attractivechaos 6y ago
This sounds great in theory, but in practice julia-1.5.2 still takes 10 sec to start when I use a medium-sized package including its dependencies (16 compiled files in total). If julia is loading cached files, I would expect something below 1 sec.
- roywiggins 6y ago
  You can manually precompile a system image with libraries you find yourself using often, like Plots.jl. The usability of it stinks, but... you can do it, and it speeds common things up. https://julialang.github.io/PackageCompiler.jl/dev/examples/plots/ https://julialang.github.io/PackageCompiler.jl/dev/examples/...
  - ChrisRackauckas 6y ago
    Its usability is being integrated into the package manager so that it automatically gives warnings and triggers when you need to redo the sysimage. Not in v1.6, but there's been a lot of discussions for doing it shortly after. That is something I'm excited for.
  - attractivechaos 6y ago
    Thanks for the tip. This does look promising.
celrod 6y ago
Just to clarify, those precompiled caches are (currently) not of binaries. At the very least, LLVM still needs to generate code after loading, and often everything up to inference still needs to be run as well. Julia 1.6 involved a lot of work reducing invalidations, where old methods are invalidated by new definitions and thus forced to recompile. This has the obvious benefit of reducing the total amount of compilation needed, but the secondary benefit of making compiling to binaries more profitable; it's less likely that work will be wasted when someone loads another package.