5 ms·

>> A new tool has been added to provide insight into the way that loading dependencies contribute to the load time of a package. That doesn't sound like an imp

by wiz21c 4y ago

>> A new tool has been added to provide insight into the way that loading dependencies contribute to the load time of a package. That doesn't sound like an improvement to the dependencies loading... Which for me happens every time I change a struct and must reload Julia (which happens a lot when I'm prototyping code, which is what I do 90% of the time)...

adammarples 4y ago
This is very annoying but they recommend working in the REPL
- wiz21c 4y ago
  Maybe I'm wrong, but I do work in the REPL. Each time I update a struct, I need to restart the REPL, hence reload dependencies... Of course, I could make my own sysimage, but it's a bit painful... Is Julia "doomed" on this issue or is it just that they prefer to work on other use cases first ?
  - celrod 4y ago
    Julia's 1.8's load times are dramatically better than Julia 1.0's. It keeps getting a bit better with every release. But there is a project to cache native code (without needing to build a sysimage) that may make a huge difference...
  - socialdemocrat 4y ago
    No, you can use a macro which makes your structs slower but changeable. There are several Julia packages offering this functionality.
  - borodi 4y ago
    Yes and no. There are ways around that for development like https://github.com/BeastyBlacksmith/ProtoStructs.jl https://github.com/BeastyBlacksmith/ProtoStructs.jl. But changing the definition of actual structs isn't very feasable because what happens with the structs that use the old definition, i.e an array of them.
  - jakobnissen 4y ago
    It's not doomed, no. It's possible to have a low-latency Julia. It's just been happening slower that one could have hoped, especially for the users who started using Julia and could tolerate the latency because they expected it would be better soon. * It's been reduced by like 75% (roughly, not sure about the real number) since Julia 1.0, so material progress has been made. * In the future (I'm guessing 1-2 years, of course uncertain), native code caching could make a huge difference. * On a similar timescale, Julia could allow compilation into static binaries. This could possibly be done for some dependencies that does not need to be generic, which would take another chunk of the compilation need off.
  - eigenspace 4y ago
    my go-to strategy is to just make a module and write code there, and then if I need to change a struct definition, I just change the name of the module and re-run it. ReplMaker.jl can make this process more convenient, but in version 1.9, there’ll be a facility for that built into the regular REPL.
krull10 4y ago
It is useful in determining which packages are dominating a given package’s load times, allowing for a better understanding of where to invest effort to reduce runtimes. This has already helped in directing efforts towards a number of common dependencies that increase load times…
jarbus 4y ago
I saw this beautiful hack to get around this: ``` struct MyClass1 ... end MyClass = MyClass1 ``` Then, whenever you need to change MyClass, just add 1 to the version number of MyClass in those two or three instances, and the repl will just compile MyClass2 as a new class to be used for all future calls in the repl.
- freemint 4y ago
  Not good if you want to also know stuff about the algorithms performance as that assignment is now accessing a global vs accessing a const.
t6jvcereio 4y ago
Had it ever occurred to you - if this was a real issue how would anyone ever get any work done? The answer is modules. module X struct etc... end end #module Change without reloading.
- bobbylarrybobby 4y ago
  But doesn't defining a new module lead to code recompilation because all the functions in the module are new? Or does Revise.jl have some way around this?
  - adgjlsfhk1 4y ago
    yes, but it will only recompile the code in the module. All the dependents will already be compiled.
amval 4y ago
That is not dependency loading. About your issue, things you can do: - Put the struct inside another module. - Use this package: https://github.com/BeastyBlacksmith/ProtoStructs.jl https://github.com/BeastyBlacksmith/ProtoStructs.jl . It gives you a simple @proto macro that you can prepend to the struct definition and makes all changes immediate.
- wiz21c 4y ago
  It seems to address my problem of constantly restarting REPL (and yes, it triggers dependencies reloads; although not an issue with dependencies directly). If it was stackoverflow, I would have accepted this answer with a kind comment :-) Thanks !
sundarurfriend 4y ago
> That doesn't sound like an improvement to the dependencies loading... This gives the package authors a tool to basically "profile" the loading time of their package, which will help them optimize the loading time. So there _will_ be downstream improvement to package loading for us users too. > Which for me happens every time I change a struct and must reload Julia (which happens a lot when I'm prototyping code, which is what I do 90% of the time) The workaround involving `MyClass = MyClass1` that a sibling comment mentions comes from [this Revise.jl manual page](https://timholy.github.io/Revise.jl/stable/limitations/ https://timholy.github.io/Revise.jl/stable/limitations/). Revise in general enables a few different ways to work around this, and it's useful to integrate into your workflow (as much as it feels weird at first to go from full REPL to half REPL-half editor).
socialdemocrat 4y ago
You can use a macro for your structs to make them changeable while prototyping. I think it is called @prototype or soemthing. You need a package for it. Google it.
- leephillips 4y ago
  There is RedefStructs: https://juliahub.com/ui/Packages/RedefStructs/nrk8Q/0.1.2 https://juliahub.com/ui/Packages/RedefStructs/nrk8Q/0.1.2 and ProtoStructs: https://github.com/BeastyBlacksmith/ProtoStructs.jl https://github.com/BeastyBlacksmith/ProtoStructs.jl I’ve never tried them, but they’re advertised as doing the job.
  - ssivark 4y ago
    +1 for these packages; I’ve used both at different points. Addendum for those who are new to this, the idea is to use these macros during development, and then get rid of them once the types stabilize (since they tend to have a performance cost).
thetwentyone 4y ago
Right, but there is a separate new feature which caches more lowered code. https://julialang.org/blog/2022/08/julia-1.8-highlights/#improved_precompilation https://julialang.org/blog/2022/08/julia-1.8-highlights/#imp... The core devs also said that they are aiming to cache native code in the next year or so.
ChrisRackauckas 4y ago
> > This gives the package authors a tool to basically "profile" the loading time of their package, which will help them optimize the loading time. So there _will_ be downstream improvement to package loading for us users too. It lead to https://github.com/SciML/RecursiveArrayTools.jl/pull/217 https://github.com/SciML/RecursiveArrayTools.jl/pull/217 . 6228.5 ms to 292.7 ms isn't too shabby.

by wiz21c 4y ago

adammarples 4y ago

This is very annoying but they recommend working in the REPL

wiz21c 4y ago
Maybe I'm wrong, but I do work in the REPL. Each time I update a struct, I need to restart the REPL, hence reload dependencies... Of course, I could make my own sysimage, but it's a bit painful... Is Julia "doomed" on this issue or is it just that they prefer to work on other use cases first ?
- celrod 4y ago
  Julia's 1.8's load times are dramatically better than Julia 1.0's. It keeps getting a bit better with every release. But there is a project to cache native code (without needing to build a sysimage) that may make a huge difference...
- socialdemocrat 4y ago
  No, you can use a macro which makes your structs slower but changeable. There are several Julia packages offering this functionality.
- borodi 4y ago
  Yes and no. There are ways around that for development like https://github.com/BeastyBlacksmith/ProtoStructs.jl https://github.com/BeastyBlacksmith/ProtoStructs.jl. But changing the definition of actual structs isn't very feasable because what happens with the structs that use the old definition, i.e an array of them.
- jakobnissen 4y ago
  It's not doomed, no. It's possible to have a low-latency Julia. It's just been happening slower that one could have hoped, especially for the users who started using Julia and could tolerate the latency because they expected it would be better soon. * It's been reduced by like 75% (roughly, not sure about the real number) since Julia 1.0, so material progress has been made. * In the future (I'm guessing 1-2 years, of course uncertain), native code caching could make a huge difference. * On a similar timescale, Julia could allow compilation into static binaries. This could possibly be done for some dependencies that does not need to be generic, which would take another chunk of the compilation need off.
- eigenspace 4y ago
  my go-to strategy is to just make a module and write code there, and then if I need to change a struct definition, I just change the name of the module and re-run it. ReplMaker.jl can make this process more convenient, but in version 1.9, there’ll be a facility for that built into the regular REPL.

krull10 4y ago

It is useful in determining which packages are dominating a given package’s load times, allowing for a better understanding of where to invest effort to reduce runtimes. This has already helped in directing efforts towards a number of common dependencies that increase load times…

jarbus 4y ago

I saw this beautiful hack to get around this: ``` struct MyClass1 ... end MyClass = MyClass1 ``` Then, whenever you need to change MyClass, just add 1 to the version number of MyClass in those two or three instances, and the repl will just compile MyClass2 as a new class to be used for all future calls in the repl.

freemint 4y ago
Not good if you want to also know stuff about the algorithms performance as that assignment is now accessing a global vs accessing a const.

t6jvcereio 4y ago

Had it ever occurred to you - if this was a real issue how would anyone ever get any work done? The answer is modules. module X struct etc... end end #module Change without reloading.

bobbylarrybobby 4y ago
But doesn't defining a new module lead to code recompilation because all the functions in the module are new? Or does Revise.jl have some way around this?
- adgjlsfhk1 4y ago
  yes, but it will only recompile the code in the module. All the dependents will already be compiled.

amval 4y ago

That is not dependency loading. About your issue, things you can do: - Put the struct inside another module. - Use this package: https://github.com/BeastyBlacksmith/ProtoStructs.jl https://github.com/BeastyBlacksmith/ProtoStructs.jl . It gives you a simple @proto macro that you can prepend to the struct definition and makes all changes immediate.

wiz21c 4y ago
It seems to address my problem of constantly restarting REPL (and yes, it triggers dependencies reloads; although not an issue with dependencies directly). If it was stackoverflow, I would have accepted this answer with a kind comment :-) Thanks !

sundarurfriend 4y ago

> That doesn't sound like an improvement to the dependencies loading... This gives the package authors a tool to basically "profile" the loading time of their package, which will help them optimize the loading time. So there _will_ be downstream improvement to package loading for us users too. > Which for me happens every time I change a struct and must reload Julia (which happens a lot when I'm prototyping code, which is what I do 90% of the time) The workaround involving `MyClass = MyClass1` that a sibling comment mentions comes from [this Revise.jl manual page](https://timholy.github.io/Revise.jl/stable/limitations/ https://timholy.github.io/Revise.jl/stable/limitations/). Revise in general enables a few different ways to work around this, and it's useful to integrate into your workflow (as much as it feels weird at first to go from full REPL to half REPL-half editor).

socialdemocrat 4y ago

You can use a macro for your structs to make them changeable while prototyping. I think it is called @prototype or soemthing. You need a package for it. Google it.

leephillips 4y ago
There is RedefStructs: https://juliahub.com/ui/Packages/RedefStructs/nrk8Q/0.1.2 https://juliahub.com/ui/Packages/RedefStructs/nrk8Q/0.1.2 and ProtoStructs: https://github.com/BeastyBlacksmith/ProtoStructs.jl https://github.com/BeastyBlacksmith/ProtoStructs.jl I’ve never tried them, but they’re advertised as doing the job.
- ssivark 4y ago
  +1 for these packages; I’ve used both at different points. Addendum for those who are new to this, the idea is to use these macros during development, and then get rid of them once the types stabilize (since they tend to have a performance cost).

thetwentyone 4y ago

Right, but there is a separate new feature which caches more lowered code. https://julialang.org/blog/2022/08/julia-1.8-highlights/#improved_precompilation https://julialang.org/blog/2022/08/julia-1.8-highlights/#imp... The core devs also said that they are aiming to cache native code in the next year or so.

ChrisRackauckas 4y ago

> > This gives the package authors a tool to basically "profile" the loading time of their package, which will help them optimize the loading time. So there _will_ be downstream improvement to package loading for us users too. It lead to https://github.com/SciML/RecursiveArrayTools.jl/pull/217 https://github.com/SciML/RecursiveArrayTools.jl/pull/217 . 6228.5 ms to 292.7 ms isn't too shabby.