3 ms·

Can I ask what about your use case would involve starting Julia and importing packages over and over? At least for my workloads, I often write a small .jl scri

Can I ask what about your use case would involve starting Julia and importing packages over and over? At least for my workloads, I often write a small .jl script that imports everything I want and then runs some functions on a list of files, for instance, or I just do my work in a Jupyter notebook, and I pay the load cost once for hours or days of computation.. Annuitized over the entire time I might use the kernel the startup time is basically nothing. I'm not saying the load time should be a lot lot faster! That would be awesome! But I see folks complain about this and I'm wondering if they really need to be booting a new kernel constantly or if they could adapt how they work a little bit and get their desired result. (For instance, by running a .jl script instead of a bash script that calls Julia or something.)

waffletower 4y ago
Currently, the adaptation cost is too high. The ecosystem that I work in has many small, distributed, fast-startup applications. While I could create a fairly complex client server application written in Julia to work with its limitations, and write compact fast startup applications to communicate with it, the complexity and time investment is undesirable. Again, to use Julia here one must bend to the ecosystem's weaknesses. I think the community truly doesn't understand what a general purpose language is. If you compile with LLVM startup really ought to be a non-issue. Look at the startup of GraalVM, and they began with Java bytecode.
- joshjob42 4y ago
  Ah okay well yes I see how that might be an issue. I basically do everything in Julia, including calling out to shell commands, running scripts in other languages, Python packages, etc. etc. It's effectively my shell. I think the problem is that there needs to be a way to store the compiled code to disk and reimport it, which I think there are some plans toward that path. There were some developments discussed [here](https://www.youtube.com/watch?v=GnsONc9DYg0 https://www.youtube.com/watch?v=GnsONc9DYg0) to help with this, but the holy grail of fully compiled function caching across Julia kernels is not there yet.
jakobnissen 4y ago
In my last job, I built a data pipeline with Snakemake. Snakemake is a workflow manager where each command (often hundreds) is called from the shell. That was excruciating to do with Julia, and I can think of no alternative.

waffletower 4y ago

Currently, the adaptation cost is too high. The ecosystem that I work in has many small, distributed, fast-startup applications. While I could create a fairly complex client server application written in Julia to work with its limitations, and write compact fast startup applications to communicate with it, the complexity and time investment is undesirable. Again, to use Julia here one must bend to the ecosystem's weaknesses. I think the community truly doesn't understand what a general purpose language is. If you compile with LLVM startup really ought to be a non-issue. Look at the startup of GraalVM, and they began with Java bytecode.

joshjob42 4y ago
Ah okay well yes I see how that might be an issue. I basically do everything in Julia, including calling out to shell commands, running scripts in other languages, Python packages, etc. etc. It's effectively my shell. I think the problem is that there needs to be a way to store the compiled code to disk and reimport it, which I think there are some plans toward that path. There were some developments discussed [here](https://www.youtube.com/watch?v=GnsONc9DYg0 https://www.youtube.com/watch?v=GnsONc9DYg0) to help with this, but the holy grail of fully compiled function caching across Julia kernels is not there yet.

jakobnissen 4y ago

In my last job, I built a data pipeline with Snakemake. Snakemake is a workflow manager where each command (often hundreds) is called from the shell. That was excruciating to do with Julia, and I can think of no alternative.