3 ms·

I don't mean that running in Bash should be THE workflow. But I think it should be supported and it's used widely and lack of support for it is a major reason w

I don't mean that running in Bash should be THE workflow. But I think it should be supported and it's used widely and lack of support for it is a major reason why Julia is not taking off. I'm not proposing taking REPL/stateful-notebook away from those who want to use it (although I'll advise them against it if asked). I don't find my plt.plot(x,y); plt.show() in a script ideal either. Something like Pluto.jl (given it's easy to modularize and preferably can be run without a notebook environment) is most likely better for a typical data analysis workflow (and will likely lead to more reproducible results and better tracking of result provenance).

sanderjd 3y ago
But there isn't "lack of support" for running cli scripts, there's just a few seconds of startup time (or less than that; it depends a lot on what dependencies the project has, in my experience). Like, I don't want to diminish that this amount of latency matters, but I just feel like you're being hyperbolic. I've used tools with more startup time for most of my career - the jvm, ruby when running any non-trivial rails app, essentially every ahead-of-time compiled language, etc. - it's not a good thing, and all those tools have invested massively in improving their startup times because of that, but I just really don't see how it can be considered a deal-breaker... (And for what it's worth, most non-trivial python codebases I work with also spend a ton of time starting up, though this can be fixed with investment into it.)
- jampekka 3y ago
  Especially for data analysis the latency is crucial. I run the code practically on every time I change a line. It's not that different from how people run REPLs, but I just keep the lines in files that I run. Would people be just fine if each REPL line would have tens of seconds of latency overhead? For different cases, e.g. application development, the latency is not that critical. Although I find it quite insane that we accept so slow code iterations this day and age.
  - sanderjd 3y ago
    Again, I'm not trying to downplay how much it matters, I just think you're being pretty hyperbolic about it! You've said stuff like it's "impossible" and "not supported", and I think these are just way stronger than "very inconvenient" or "extremely frustrating". Maybe it's a minor point, I dunno, but like, if we were talking about a five minute compile cycle or a language that doesn't even have a way to run a script at the command line, then sure, I'd agree with what you're saying, but as it is, it just feels a bit like a tempest in a teapot to me...
    - ChrisRackauckas 3y ago
      Indeed it's something in the middle. For data analysis, many people are used to using RStudio and just keeping a running REPL going, and Julia works quite well for these kinds of tasks in the same mode because that effectively removes the startup time from being a concern. However, there are some folks that are used to other workflows and we could grow the audience by improving the ability to support such workflows better.
      - sanderjd 3y ago
        For what it's worth, my workflow is actually much closer to the parent's; I have a python batch job that spins up a julia runtime in a thread at startup and waits on a `Pkg.instantiate()` of a project with pretty substantial dependencies before it can start sending method calls into it. I think it takes like 3-5 seconds to start up. I badly want it to be a lot faster than that, but it's also fine. I'm not happy with it, but if I sat here thinking about it for a few minutes, I'm certain I could list a hundred things in our system that annoy me more than this.
        jampekka 3y ago
        That's probably something close to DaemonMode.jl? If the problem was just running a daemon, it would be an inconvenience at worst. The problem is that changes to the code tend to trigger a recompile, can't be made (e.g. structs can't be redefined if I recall correctly) and most crucially the session can get "silently" into some unexpected confused state. If you don't need to change the code (and it's structured well enough not to hold a state between calls), then running in a process is no problem. But for that you can do just plain old AOT. Aside from my personal gripes, which I have probably aired here more than enough for everybody, I think the design that more or less forces REPL/notebook style is detrimental to quality of scientific code. Scientists don't learn to modularize their code and don't learn to understand the basic control flow. I see this all the time with my colleagues and the "pedagogical problem" of notebooks/REPL is very common when I teach programming. E.g. a common problem is that students don't understand variable assignment because with REPL/notebook it doesn't necessarily follow the normal program flow. I'm kind of amazed how little the scientific programming scene is concerned about this. From software development perspective (I'm an ex software developer turned scientist) having clearly defined state transitions, independent modules of code and the code doing what it says are really really fundamental things, without which you have no chance in hell in making anything more than a few lines to not turn into an unmanageable and buggy mess. Maybe there's a (mis)conception that scientific code is somehow fundamental different beast than "general programming". But it's not. It's just a relic of having bad systems languages like C and C++ and bad special purpose languages like MATLAB and R. That Python can be and is used for both is an existence proof that there is no such divide. But we need a better Python for both general and scientific programming.

sanderjd 3y ago

But there isn't "lack of support" for running cli scripts, there's just a few seconds of startup time (or less than that; it depends a lot on what dependencies the project has, in my experience). Like, I don't want to diminish that this amount of latency matters, but I just feel like you're being hyperbolic. I've used tools with more startup time for most of my career - the jvm, ruby when running any non-trivial rails app, essentially every ahead-of-time compiled language, etc. - it's not a good thing, and all those tools have invested massively in improving their startup times because of that, but I just really don't see how it can be considered a deal-breaker... (And for what it's worth, most non-trivial python codebases I work with also spend a ton of time starting up, though this can be fixed with investment into it.)

jampekka 3y ago
Especially for data analysis the latency is crucial. I run the code practically on every time I change a line. It's not that different from how people run REPLs, but I just keep the lines in files that I run. Would people be just fine if each REPL line would have tens of seconds of latency overhead? For different cases, e.g. application development, the latency is not that critical. Although I find it quite insane that we accept so slow code iterations this day and age.
- sanderjd 3y ago
  Again, I'm not trying to downplay how much it matters, I just think you're being pretty hyperbolic about it! You've said stuff like it's "impossible" and "not supported", and I think these are just way stronger than "very inconvenient" or "extremely frustrating". Maybe it's a minor point, I dunno, but like, if we were talking about a five minute compile cycle or a language that doesn't even have a way to run a script at the command line, then sure, I'd agree with what you're saying, but as it is, it just feels a bit like a tempest in a teapot to me...
  - ChrisRackauckas 3y ago
    Indeed it's something in the middle. For data analysis, many people are used to using RStudio and just keeping a running REPL going, and Julia works quite well for these kinds of tasks in the same mode because that effectively removes the startup time from being a concern. However, there are some folks that are used to other workflows and we could grow the audience by improving the ability to support such workflows better.
    - sanderjd 3y ago
      For what it's worth, my workflow is actually much closer to the parent's; I have a python batch job that spins up a julia runtime in a thread at startup and waits on a `Pkg.instantiate()` of a project with pretty substantial dependencies before it can start sending method calls into it. I think it takes like 3-5 seconds to start up. I badly want it to be a lot faster than that, but it's also fine. I'm not happy with it, but if I sat here thinking about it for a few minutes, I'm certain I could list a hundred things in our system that annoy me more than this.
      - jampekka 3y ago
        That's probably something close to DaemonMode.jl? If the problem was just running a daemon, it would be an inconvenience at worst. The problem is that changes to the code tend to trigger a recompile, can't be made (e.g. structs can't be redefined if I recall correctly) and most crucially the session can get "silently" into some unexpected confused state. If you don't need to change the code (and it's structured well enough not to hold a state between calls), then running in a process is no problem. But for that you can do just plain old AOT. Aside from my personal gripes, which I have probably aired here more than enough for everybody, I think the design that more or less forces REPL/notebook style is detrimental to quality of scientific code. Scientists don't learn to modularize their code and don't learn to understand the basic control flow. I see this all the time with my colleagues and the "pedagogical problem" of notebooks/REPL is very common when I teach programming. E.g. a common problem is that students don't understand variable assignment because with REPL/notebook it doesn't necessarily follow the normal program flow. I'm kind of amazed how little the scientific programming scene is concerned about this. From software development perspective (I'm an ex software developer turned scientist) having clearly defined state transitions, independent modules of code and the code doing what it says are really really fundamental things, without which you have no chance in hell in making anything more than a few lines to not turn into an unmanageable and buggy mess. Maybe there's a (mis)conception that scientific code is somehow fundamental different beast than "general programming". But it's not. It's just a relic of having bad systems languages like C and C++ and bad special purpose languages like MATLAB and R. That Python can be and is used for both is an existence proof that there is no such divide. But we need a better Python for both general and scientific programming.