3 ms·

good news is 1.10 already is shaping up to be about 2x faster.

shakow 3y ago
So ~0.85"? Python is not compiled and start & load pandas (which is comparable to the libraries loaded in the article) in ~0.4" on my computer, and that's a notoriously slow language. If I were to use e.g. Rust with polars, load time would be virtually none. And when I have to process ~50k different datasets, I can't afford 0.85" per file, which would translate to ~11 hours of overhead.
- Nanana909 3y ago
  > If I were to use e.g. Rust with polars, load time would be virtually none. Because you're compiling... And if you need to do the same in Julia, you should also pre-compile or some other method like https://github.com/dmolina/DaemonMode.jl https://github.com/dmolina/DaemonMode.jl (their demo shows loading a database, with subsequent loads after the first one taking roughly ~0.2% of the first)
- adgjlsfhk1 3y ago
  It's not 0.85 per file. It's 0.85 for the 50k files combined.
  - shakow 3y ago
    No, because the program has to be run by slurm over several compute nodes, so it can't process them all at once.
    - adgjlsfhk1 3y ago
      as long as you don't have 50k computers, it still should be 0.85 seconds per node which is still tiny.
      - shakow 3y ago
        > it still should be 0.85 seconds per node No it's not, because I will not architecture my whole pipeline & program around Julia inability to start in maybe a second in a year or 1.7" now, I will just use another language.
        pjmlp 3y ago
        In good company, that is what Python folks do all the time.
        shakow 3y ago
        Python does not take 1.7" to load pandas.
        pjmlp 3y ago
        > I will just use another language. Hello C, C++ and Fortran.
ksec 3y ago
When is 1.10 expected? Just wondering I did a quick google and it seems a lot of the 1.9 improvement were back ported from 1.10? Or is that really 2x on top of current 1.9?
- adgjlsfhk1 3y ago
  1.10 feature freeze is going to be in the next few weeks. after that it will be a few months depending on how much is required to fix all the bugs that have probably been introduced. 1.9 mostly doesn't shorten loading times (although weak dependencies end up helping a bit). 1.10 has had a bunch of load time optimization which became a lot more obvious once 1.9 got rid of all the stupid stuff. the exact speedups are package dependent, but 2x is a good estimate. some packages get a lot more, some are about the same.
  - ksec 3y ago
    Thanks. Exciting times for Julia!

So ~0.85"? Python is not compiled and start & load pandas (which is comparable to the libraries loaded in the article) in ~0.4" on my computer, and that's a notoriously slow language. If I were to use e.g. Rust with polars, load time would be virtually none. And when I have to process ~50k different datasets, I can't afford 0.85" per file, which would translate to ~11 hours of overhead.

Nanana909 3y ago
> If I were to use e.g. Rust with polars, load time would be virtually none. Because you're compiling... And if you need to do the same in Julia, you should also pre-compile or some other method like https://github.com/dmolina/DaemonMode.jl https://github.com/dmolina/DaemonMode.jl (their demo shows loading a database, with subsequent loads after the first one taking roughly ~0.2% of the first)
adgjlsfhk1 3y ago
It's not 0.85 per file. It's 0.85 for the 50k files combined.
- shakow 3y ago
  No, because the program has to be run by slurm over several compute nodes, so it can't process them all at once.
  - adgjlsfhk1 3y ago
    as long as you don't have 50k computers, it still should be 0.85 seconds per node which is still tiny.
    - shakow 3y ago
      > it still should be 0.85 seconds per node No it's not, because I will not architecture my whole pipeline & program around Julia inability to start in maybe a second in a year or 1.7" now, I will just use another language.
      - pjmlp 3y ago
        In good company, that is what Python folks do all the time.
        shakow 3y ago
        Python does not take 1.7" to load pandas.
        pjmlp 3y ago
        > I will just use another language. Hello C, C++ and Fortran.

ksec 3y ago

When is 1.10 expected? Just wondering I did a quick google and it seems a lot of the 1.9 improvement were back ported from 1.10? Or is that really 2x on top of current 1.9?

adgjlsfhk1 3y ago
1.10 feature freeze is going to be in the next few weeks. after that it will be a few months depending on how much is required to fix all the bugs that have probably been introduced. 1.9 mostly doesn't shorten loading times (although weak dependencies end up helping a bit). 1.10 has had a bunch of load time optimization which became a lot more obvious once 1.9 got rid of all the stupid stuff. the exact speedups are package dependent, but 2x is a good estimate. some packages get a lot more, some are about the same.
- ksec 3y ago
  Thanks. Exciting times for Julia!