56 ms·

I hope Julia succeeds and replaces the clunky R and numpy+python "ecosystems". Every few months I try to decide to do all my computing on Julia and quit foolin

I hope Julia succeeds and replaces the clunky R and numpy+python "ecosystems". Every few months I try to decide to do all my computing on Julia and quit fooling around with lesser environments. And every time I get stuck at the same point: the slow startup time. I want to draw 100 plots per second, by calling the same Julia script on a bash loop 100 times; but this is utterly impossible. Of course, the Julia community has a standard answer to this concern: this is not how you are supposed to use the language. But I don't listen to them, for the best tools are those that perform well doing tasks they were not designed to do. I feel the same dismay when I get stuck at slow loops in Python, and the Python people tell me that I'm not supposed to use loops. Well, this is the main reason that I want to move away from Python and into Julia. I'm not interested in the implementation details of the language interpreter. The language is already very good, and the libraries are excellent. My main point of friction against using Julia is the slow startup time that forbids its use in a wide variety of contexts. I feel like the best usage of resources for the Julia community would be to spend all available money in hiring a Mike Pall-esque figure to advise them on JIT. Even if it was a part-time or a one-shot hire.

dunefox 6y ago
Start up time for the interpreter is 0.13s for me: ~ time julia -E "1+1" What takes time is precompilation of packages and functions - with Julia 1.6 the precompilation is much faster now than before. Your bash script that calls Julia 100 times is indeed not something that Julia was made for. It excels in many other areas and that's quite fine. I'm okay with plotting in a bash script in Python if it means that I can use Julia for everything else.
- michaericalribo 6y ago
  Your last sentence seems to ignore the reality that plotting is only one step in a larger pipeline. It sounds miserable to need to write analytics code twice, once in Julia “for everything else” and again in Python just for the plotting. I’ll just write the whole thing in Python and save myself the headache.
  - socialdemocrat 6y ago
    So why not write the whole thing in Julia? That is the whole problem here, that the whole thing was NOT written in Julia. Why manage two different languages? Julia is a better shell programming language than bash anyway.
    - michaericalribo 6y ago
      I mean, the comment I was replying to answers your question: > not something Julia was made for
      - dunefox 6y ago
        What I meant with this is that Julia isn't intended as a bash replacement. You can write your code in Julia and circumvent the overhead of having to start up the interpreter every time. But if you try to execute it 100 times per second then of course the overhead will add up.
        michaericalribo 6y ago
        Ah, I understand what you mean now. And you may be right that there’s a Better Way to do it natively in Julia. But, there’s lots of friction to adopting entirely new dev practices, and I’m inclined to just stick w what tried and true methods I’m already familiar with—old habits die hard! And that’s a big friction against Julia adoption (IMO).
        dunefox 6y ago
        There's nothing wrong with using the tools you know. But IMO it's quite interesting to use languages that might just be a big improvement over how things have been done so far. I think Julia is such a language when compared to Python (excluding the ecosystem, of course). Also, if you come back to Julia sometime there's this: https://github.com/JuliaPy/PyCall.jl https://github.com/JuliaPy/PyCall.jl https://github.com/JuliaInterop/RCall.jl https://github.com/JuliaInterop/RCall.jl
socialdemocrat 6y ago
This just seems like such an artificially constructed problem. You just move that loop into Julia and the problem is solved. It is not hard to write shell code in Julia. I have rewritten complex build scripts in bash into Julia code. It was not very hard and it made everything run way faster despite mostly shelling out to external programs. I don't know how secret/protected this shell script of yours is, but I'd be willing to have a go at rewriting the loop part in Julia if you showed it to me. Anyway I do think the startup speed problem can be solved in Julia and they don't need a rockstar advisor to do it. I think the solutions are already pretty well known. It is more of an issue of manpower. Somebody has to put in the hours to do it. And that is already being done. First time to plot will be twice as fast as it used to be in the next Julia release. Further improvements can be done, but again that requires somebody to put in the hours. Julia does not have access to Google level resources.
- enriquto 6y ago
  > the startup speed problem can be solved in Julia and they don't need a rockstar advisor to do it. I think the solutions are already pretty well known. > And that is already being done. First time to plot will be twice as fast as it used to be in the next Julia release. Further improvements can be done I'm really, really happy to hear that! I hope the Julia runtime becomes more and more streamlined in the (near) future. See, I was just stating my use case, without pretense that it is representative at all. Yet I received 20 upvotes in a few minutes, and Julia is maybe the only language where "time to first plot" is a thing. So I'm not completely alone in my (admittedly minoritary) concern. You say that rewriting everything in Julia would solve my problem. I'm sure that this is the case, but this is not at all my point. Some of us do not want a shell replacement, we want a bc replacement, and julia is a nearly perfect one, if it wasn't for the outrageously slow startup time. I have zero interest in the julia REPL, I just write julia scripts (among scripts in other languages) and I'm not willing to change that.
  - dTal 6y ago
    Julia is a compiled language, like C. Using it in this way is basically like writing a shell script in C and calling GCC on every invocation. You can't expect good performance from that, and it's really a testament to Julia that it feels so dynamic that you feel as it it should do that. If you AOT-compiled your Julia "script" (program) to a binary and invoked that, your startup time problems would go away. Julia's "application deployment" stack is underdeveloped compared to its REPL experience, but it's still possible to do this today with PackageCompiler.jl and will only get easier with time. I think that will prove to be the "right" way to solve this problem in the long run. (Or, if you don't care about performance, you can just turn off the compiler and interpret everything, and you basically have Python-but-in-Julia. Fast startup, slow running. Just run it with julia --compile=no)
deleted 6y ago
[deleted]
classified 6y ago
> I get stuck at the same point: the slow startup time… The language maintainers steadfastly refuse to include ahead-of-time compilation. They seem super focused on their narrow use case scenario and ignore everything outside that.
- socialdemocrat 6y ago
  It is not nice to lie about people like that. They have never refused to do that and in fact you can already do ahead-of-time compilation in Julia. Many of us have already done it. It is not great yet, but it is an ongoing problem, which they constantly work on and improve. Claiming they refuse to do it is either ignorant or a flat out lie.
  - classified 6y ago
    > It is not great yet, but it is an ongoing problem, which they constantly work on and improve. That's just it. This effort has been dragging on for years now, slow as molasses in winter. If it were a regular goal it would have been a solved problem long ago. That hacks and workarounds for this have been deemed acceptable for so long just goes to show that it's not on the list.
- dunefox 6y ago
  Please show me a source where they outright refuse AOT compilation.
  - classified 6y ago
    They don't have to outright say, "we refuse". Not doing this obvious step for years and years and years clearly shows their priorities lie elsewhere.
    - Sukera 6y ago
      I'm sorry if I'm blunt, but the last year of compiler improvements have been nothing but targeted for this exact purpose? That's half of all spent time since 1.0! The issue tracker is filled to the brim with PRs and issues about making _everything_ faster. How does this constitute "refusal of a obvious step" for you?
krull10 6y ago
If you are going to use Julia in one of the absolute worst workflows for how it is designed, you shouldn’t be surprised it doesn’t work well... That said, have you tried using PackageCompiler to add your needed libraries to the system image? This seems to show a factor of 100 speed up for the time to first plot: https://julialang.github.io/PackageCompiler.jl/dev/examples/plots/ https://julialang.github.io/PackageCompiler.jl/dev/examples/...
nojito 6y ago
Calling R clunky is definitely a reach. Parameterized reporting in R/knitr is unmatched in the industry
celrod 6y ago
While moving the loop into Julia (as others suggested) is probably the better option, an alternative you could consider is DaemonMode: https://github.com/dmolina/DaemonMode.jl https://github.com/dmolina/DaemonMode.jl I.e., have a background Julia process so that you only have to pay the precompile cost once.
tpoacher 6y ago
One thing to be aware of is that you're not running a bash script which simply causes julia to "include" stuff, effectively recompiling everything each time the intepreter is run. As long as you make sure that all the custom code you want to run is in the form of a precompiled module, I think the time required for the interpreter to launch per se shouldn't be that much of a problem.
- jjoonathan 6y ago
  Yes, but if the road to cached builds or running in interpreted mode is not low friction or no friction, that matters and it's the fault of the language / ecosystem.
AuthorizedCust 6y ago
Why do you find R’s ecosystem “clunky”? The Tidyverse is unequaled for its elegance. I come from the CS world, so I’m supposed to like languages like Python, but I really, really like R, mainly for its elegance.
- enriquto 6y ago
  I agree that for statistics and data exploration R is certainly not clunky. My use case is more discrete PDE, where the R capabilities for sparse matrices and advanced linear algebra are a bit limited (but this may be just because I'm more used to the annoyances of numpy).
- veddox 6y ago
  The Tidyverse is great, but vanilla R is a monstrosity. After five years of heavy use, I still don't really understand the random idiosyncracies of its various types. Arrays and lists and dataframes and tibbles are confusingly named, and operations that work on one type often balk at the others, without telling you what's wrong. I have lost many many nerves with it.
newswasboring 6y ago
If this is your big setback did you try using a sysimage? The VSCode extension even has a build task for it. To make the sysimage just be in the base environment and then Ctrl+shift+B and select the Juild build sysimage task. The terminal will tell you where its saving the sysimage. It reduced my startup time to unnoticeable (at least to a person used to MATLAB/Python). I am not a bash guru so I don't know how you do it on command line but its a parameter to julia interpreter.

dunefox 6y ago

Start up time for the interpreter is 0.13s for me: ~ time julia -E "1+1" What takes time is precompilation of packages and functions - with Julia 1.6 the precompilation is much faster now than before. Your bash script that calls Julia 100 times is indeed not something that Julia was made for. It excels in many other areas and that's quite fine. I'm okay with plotting in a bash script in Python if it means that I can use Julia for everything else.

michaericalribo 6y ago
Your last sentence seems to ignore the reality that plotting is only one step in a larger pipeline. It sounds miserable to need to write analytics code twice, once in Julia “for everything else” and again in Python just for the plotting. I’ll just write the whole thing in Python and save myself the headache.
- socialdemocrat 6y ago
  So why not write the whole thing in Julia? That is the whole problem here, that the whole thing was NOT written in Julia. Why manage two different languages? Julia is a better shell programming language than bash anyway.
  - michaericalribo 6y ago
    I mean, the comment I was replying to answers your question: > not something Julia was made for
    - dunefox 6y ago
      What I meant with this is that Julia isn't intended as a bash replacement. You can write your code in Julia and circumvent the overhead of having to start up the interpreter every time. But if you try to execute it 100 times per second then of course the overhead will add up.
      - michaericalribo 6y ago
        Ah, I understand what you mean now. And you may be right that there’s a Better Way to do it natively in Julia. But, there’s lots of friction to adopting entirely new dev practices, and I’m inclined to just stick w what tried and true methods I’m already familiar with—old habits die hard! And that’s a big friction against Julia adoption (IMO).
        dunefox 6y ago
        There's nothing wrong with using the tools you know. But IMO it's quite interesting to use languages that might just be a big improvement over how things have been done so far. I think Julia is such a language when compared to Python (excluding the ecosystem, of course). Also, if you come back to Julia sometime there's this: https://github.com/JuliaPy/PyCall.jl https://github.com/JuliaPy/PyCall.jl https://github.com/JuliaInterop/RCall.jl https://github.com/JuliaInterop/RCall.jl

socialdemocrat 6y ago

This just seems like such an artificially constructed problem. You just move that loop into Julia and the problem is solved. It is not hard to write shell code in Julia. I have rewritten complex build scripts in bash into Julia code. It was not very hard and it made everything run way faster despite mostly shelling out to external programs. I don't know how secret/protected this shell script of yours is, but I'd be willing to have a go at rewriting the loop part in Julia if you showed it to me. Anyway I do think the startup speed problem can be solved in Julia and they don't need a rockstar advisor to do it. I think the solutions are already pretty well known. It is more of an issue of manpower. Somebody has to put in the hours to do it. And that is already being done. First time to plot will be twice as fast as it used to be in the next Julia release. Further improvements can be done, but again that requires somebody to put in the hours. Julia does not have access to Google level resources.

enriquto 6y ago
> the startup speed problem can be solved in Julia and they don't need a rockstar advisor to do it. I think the solutions are already pretty well known. > And that is already being done. First time to plot will be twice as fast as it used to be in the next Julia release. Further improvements can be done I'm really, really happy to hear that! I hope the Julia runtime becomes more and more streamlined in the (near) future. See, I was just stating my use case, without pretense that it is representative at all. Yet I received 20 upvotes in a few minutes, and Julia is maybe the only language where "time to first plot" is a thing. So I'm not completely alone in my (admittedly minoritary) concern. You say that rewriting everything in Julia would solve my problem. I'm sure that this is the case, but this is not at all my point. Some of us do not want a shell replacement, we want a bc replacement, and julia is a nearly perfect one, if it wasn't for the outrageously slow startup time. I have zero interest in the julia REPL, I just write julia scripts (among scripts in other languages) and I'm not willing to change that.
- dTal 6y ago
  Julia is a compiled language, like C. Using it in this way is basically like writing a shell script in C and calling GCC on every invocation. You can't expect good performance from that, and it's really a testament to Julia that it feels so dynamic that you feel as it it should do that. If you AOT-compiled your Julia "script" (program) to a binary and invoked that, your startup time problems would go away. Julia's "application deployment" stack is underdeveloped compared to its REPL experience, but it's still possible to do this today with PackageCompiler.jl and will only get easier with time. I think that will prove to be the "right" way to solve this problem in the long run. (Or, if you don't care about performance, you can just turn off the compiler and interpret everything, and you basically have Python-but-in-Julia. Fast startup, slow running. Just run it with julia --compile=no)

deleted 6y ago

[deleted]

classified 6y ago

> I get stuck at the same point: the slow startup time… The language maintainers steadfastly refuse to include ahead-of-time compilation. They seem super focused on their narrow use case scenario and ignore everything outside that.

socialdemocrat 6y ago
It is not nice to lie about people like that. They have never refused to do that and in fact you can already do ahead-of-time compilation in Julia. Many of us have already done it. It is not great yet, but it is an ongoing problem, which they constantly work on and improve. Claiming they refuse to do it is either ignorant or a flat out lie.
- classified 6y ago
  > It is not great yet, but it is an ongoing problem, which they constantly work on and improve. That's just it. This effort has been dragging on for years now, slow as molasses in winter. If it were a regular goal it would have been a solved problem long ago. That hacks and workarounds for this have been deemed acceptable for so long just goes to show that it's not on the list.
dunefox 6y ago
Please show me a source where they outright refuse AOT compilation.
- classified 6y ago
  They don't have to outright say, "we refuse". Not doing this obvious step for years and years and years clearly shows their priorities lie elsewhere.
  - Sukera 6y ago
    I'm sorry if I'm blunt, but the last year of compiler improvements have been nothing but targeted for this exact purpose? That's half of all spent time since 1.0! The issue tracker is filled to the brim with PRs and issues about making _everything_ faster. How does this constitute "refusal of a obvious step" for you?

krull10 6y ago

If you are going to use Julia in one of the absolute worst workflows for how it is designed, you shouldn’t be surprised it doesn’t work well... That said, have you tried using PackageCompiler to add your needed libraries to the system image? This seems to show a factor of 100 speed up for the time to first plot: https://julialang.github.io/PackageCompiler.jl/dev/examples/plots/ https://julialang.github.io/PackageCompiler.jl/dev/examples/...

nojito 6y ago

Calling R clunky is definitely a reach. Parameterized reporting in R/knitr is unmatched in the industry

celrod 6y ago

While moving the loop into Julia (as others suggested) is probably the better option, an alternative you could consider is DaemonMode: https://github.com/dmolina/DaemonMode.jl https://github.com/dmolina/DaemonMode.jl I.e., have a background Julia process so that you only have to pay the precompile cost once.

tpoacher 6y ago

One thing to be aware of is that you're not running a bash script which simply causes julia to "include" stuff, effectively recompiling everything each time the intepreter is run. As long as you make sure that all the custom code you want to run is in the form of a precompiled module, I think the time required for the interpreter to launch per se shouldn't be that much of a problem.

jjoonathan 6y ago
Yes, but if the road to cached builds or running in interpreted mode is not low friction or no friction, that matters and it's the fault of the language / ecosystem.

AuthorizedCust 6y ago

Why do you find R’s ecosystem “clunky”? The Tidyverse is unequaled for its elegance. I come from the CS world, so I’m supposed to like languages like Python, but I really, really like R, mainly for its elegance.

enriquto 6y ago
I agree that for statistics and data exploration R is certainly not clunky. My use case is more discrete PDE, where the R capabilities for sparse matrices and advanced linear algebra are a bit limited (but this may be just because I'm more used to the annoyances of numpy).
veddox 6y ago
The Tidyverse is great, but vanilla R is a monstrosity. After five years of heavy use, I still don't really understand the random idiosyncracies of its various types. Arrays and lists and dataframes and tibbles are confusingly named, and operations that work on one type often balk at the others, without telling you what's wrong. I have lost many many nerves with it.

newswasboring 6y ago

If this is your big setback did you try using a sysimage? The VSCode extension even has a build task for it. To make the sysimage just be in the base environment and then Ctrl+shift+B and select the Juild build sysimage task. The terminal will tell you where its saving the sysimage. It reduced my startup time to unnoticeable (at least to a person used to MATLAB/Python). I am not a bash guru so I don't know how you do it on command line but its a parameter to julia interpreter.