3 ms·
So ~0.85"? Python is not compiled and start & load pandas (which is comparable to the libraries loaded in the article) in ~0.4" on my computer, and that's a no
by shakow 3y ago
So ~0.85"?
Python is not compiled and start & load pandas (which is comparable to the libraries loaded in the article) in ~0.4" on my computer, and that's a notoriously slow language.
If I were to use e.g. Rust with polars, load time would be virtually none. And when I have to process ~50k different datasets, I can't afford 0.85" per file, which would translate to ~11 hours of overhead.
- Nanana909 3y ago> If I were to use e.g. Rust with polars, load time would be virtually none. Because you're compiling... And if you need to do the same in Julia, you should also pre-compile or some other method like https://github.com/dmolina/DaemonMode.jl https://github.com/dmolina/DaemonMode.jl (their demo shows loading a database, with subsequent loads after the first one taking roughly ~0.2% of the first)
- adgjlsfhk1 3y agoIt's not 0.85 per file. It's 0.85 for the 50k files combined.
- shakow 3y agoNo, because the program has to be run by slurm over several compute nodes, so it can't process them all at once.
- adgjlsfhk1 3y agoas long as you don't have 50k computers, it still should be 0.85 seconds per node which is still tiny.
- shakow 3y ago> it still should be 0.85 seconds per node No it's not, because I will not architecture my whole pipeline & program around Julia inability to start in maybe a second in a year or 1.7" now, I will just use another language.