3 ms·

Especially for data analysis the latency is crucial. I run the code practically on every time I change a line. It's not that different from how people run REPLs

Especially for data analysis the latency is crucial. I run the code practically on every time I change a line. It's not that different from how people run REPLs, but I just keep the lines in files that I run. Would people be just fine if each REPL line would have tens of seconds of latency overhead? For different cases, e.g. application development, the latency is not that critical. Although I find it quite insane that we accept so slow code iterations this day and age.

sanderjd 3y ago
Again, I'm not trying to downplay how much it matters, I just think you're being pretty hyperbolic about it! You've said stuff like it's "impossible" and "not supported", and I think these are just way stronger than "very inconvenient" or "extremely frustrating". Maybe it's a minor point, I dunno, but like, if we were talking about a five minute compile cycle or a language that doesn't even have a way to run a script at the command line, then sure, I'd agree with what you're saying, but as it is, it just feels a bit like a tempest in a teapot to me...
- ChrisRackauckas 3y ago
  Indeed it's something in the middle. For data analysis, many people are used to using RStudio and just keeping a running REPL going, and Julia works quite well for these kinds of tasks in the same mode because that effectively removes the startup time from being a concern. However, there are some folks that are used to other workflows and we could grow the audience by improving the ability to support such workflows better.
  - sanderjd 3y ago
    For what it's worth, my workflow is actually much closer to the parent's; I have a python batch job that spins up a julia runtime in a thread at startup and waits on a `Pkg.instantiate()` of a project with pretty substantial dependencies before it can start sending method calls into it. I think it takes like 3-5 seconds to start up. I badly want it to be a lot faster than that, but it's also fine. I'm not happy with it, but if I sat here thinking about it for a few minutes, I'm certain I could list a hundred things in our system that annoy me more than this.
    - jampekka 3y ago
      That's probably something close to DaemonMode.jl? If the problem was just running a daemon, it would be an inconvenience at worst. The problem is that changes to the code tend to trigger a recompile, can't be made (e.g. structs can't be redefined if I recall correctly) and most crucially the session can get "silently" into some unexpected confused state. If you don't need to change the code (and it's structured well enough not to hold a state between calls), then running in a process is no problem. But for that you can do just plain old AOT. Aside from my personal gripes, which I have probably aired here more than enough for everybody, I think the design that more or less forces REPL/notebook style is detrimental to quality of scientific code. Scientists don't learn to modularize their code and don't learn to understand the basic control flow. I see this all the time with my colleagues and the "pedagogical problem" of notebooks/REPL is very common when I teach programming. E.g. a common problem is that students don't understand variable assignment because with REPL/notebook it doesn't necessarily follow the normal program flow. I'm kind of amazed how little the scientific programming scene is concerned about this. From software development perspective (I'm an ex software developer turned scientist) having clearly defined state transitions, independent modules of code and the code doing what it says are really really fundamental things, without which you have no chance in hell in making anything more than a few lines to not turn into an unmanageable and buggy mess. Maybe there's a (mis)conception that scientific code is somehow fundamental different beast than "general programming". But it's not. It's just a relic of having bad systems languages like C and C++ and bad special purpose languages like MATLAB and R. That Python can be and is used for both is an existence proof that there is no such divide. But we need a better Python for both general and scientific programming.
      - sanderjd 3y ago
        > Aside from my personal gripes, which I have probably aired here more than enough for everybody, I think the design that more or less forces REPL/notebook style is detrimental to quality of scientific code. So, two things here: First of all, I have greatly appreciated the discussion, so don't feel like you've over-aired your thoughts! But also, here's this, in my view, hyperbole again, I just don't see how "more or less forces" is the right way to describe the state of choosing whether or not to use a notebook/REPL style for julia. It works perfectly fine to develop with a different style. There's a little bit of startup time, but it's just not nearly a big enough deal to say it "forces" a different development process. I think your criticisms of notebook driven programming make sense, but are also slightly overblown; in my experience, scientists are perfectly capable of writing good modular code, despite preferring to work in notebooks most of the time. It's not not a problem, I just don't think it's been a very big problem, in my experience. But I very much agree with your last two paragraphs. I just don't think notebooks / REPLs are a big part of the reason things have ended up this way. I think it's more of a cultural thing that gets passed down from generation to generation, and it has to do with respect; I think scientists have never taught one another that code deserves respect and professionalism. I think they have always thought of it as "some throwaway stuff that generates the data for the paper"; the paper is the thing deserving of respect, not the tools used to create it. But I think this has been slowly changing as more and more scientists are digitally native and learn increasingly more software skills as a first class concern.

sanderjd 3y ago

Again, I'm not trying to downplay how much it matters, I just think you're being pretty hyperbolic about it! You've said stuff like it's "impossible" and "not supported", and I think these are just way stronger than "very inconvenient" or "extremely frustrating". Maybe it's a minor point, I dunno, but like, if we were talking about a five minute compile cycle or a language that doesn't even have a way to run a script at the command line, then sure, I'd agree with what you're saying, but as it is, it just feels a bit like a tempest in a teapot to me...

ChrisRackauckas 3y ago
Indeed it's something in the middle. For data analysis, many people are used to using RStudio and just keeping a running REPL going, and Julia works quite well for these kinds of tasks in the same mode because that effectively removes the startup time from being a concern. However, there are some folks that are used to other workflows and we could grow the audience by improving the ability to support such workflows better.
- sanderjd 3y ago
  For what it's worth, my workflow is actually much closer to the parent's; I have a python batch job that spins up a julia runtime in a thread at startup and waits on a `Pkg.instantiate()` of a project with pretty substantial dependencies before it can start sending method calls into it. I think it takes like 3-5 seconds to start up. I badly want it to be a lot faster than that, but it's also fine. I'm not happy with it, but if I sat here thinking about it for a few minutes, I'm certain I could list a hundred things in our system that annoy me more than this.
  - jampekka 3y ago
    That's probably something close to DaemonMode.jl? If the problem was just running a daemon, it would be an inconvenience at worst. The problem is that changes to the code tend to trigger a recompile, can't be made (e.g. structs can't be redefined if I recall correctly) and most crucially the session can get "silently" into some unexpected confused state. If you don't need to change the code (and it's structured well enough not to hold a state between calls), then running in a process is no problem. But for that you can do just plain old AOT. Aside from my personal gripes, which I have probably aired here more than enough for everybody, I think the design that more or less forces REPL/notebook style is detrimental to quality of scientific code. Scientists don't learn to modularize their code and don't learn to understand the basic control flow. I see this all the time with my colleagues and the "pedagogical problem" of notebooks/REPL is very common when I teach programming. E.g. a common problem is that students don't understand variable assignment because with REPL/notebook it doesn't necessarily follow the normal program flow. I'm kind of amazed how little the scientific programming scene is concerned about this. From software development perspective (I'm an ex software developer turned scientist) having clearly defined state transitions, independent modules of code and the code doing what it says are really really fundamental things, without which you have no chance in hell in making anything more than a few lines to not turn into an unmanageable and buggy mess. Maybe there's a (mis)conception that scientific code is somehow fundamental different beast than "general programming". But it's not. It's just a relic of having bad systems languages like C and C++ and bad special purpose languages like MATLAB and R. That Python can be and is used for both is an existence proof that there is no such divide. But we need a better Python for both general and scientific programming.
    - sanderjd 3y ago
      > Aside from my personal gripes, which I have probably aired here more than enough for everybody, I think the design that more or less forces REPL/notebook style is detrimental to quality of scientific code. So, two things here: First of all, I have greatly appreciated the discussion, so don't feel like you've over-aired your thoughts! But also, here's this, in my view, hyperbole again, I just don't see how "more or less forces" is the right way to describe the state of choosing whether or not to use a notebook/REPL style for julia. It works perfectly fine to develop with a different style. There's a little bit of startup time, but it's just not nearly a big enough deal to say it "forces" a different development process. I think your criticisms of notebook driven programming make sense, but are also slightly overblown; in my experience, scientists are perfectly capable of writing good modular code, despite preferring to work in notebooks most of the time. It's not not a problem, I just don't think it's been a very big problem, in my experience. But I very much agree with your last two paragraphs. I just don't think notebooks / REPLs are a big part of the reason things have ended up this way. I think it's more of a cultural thing that gets passed down from generation to generation, and it has to do with respect; I think scientists have never taught one another that code deserves respect and professionalism. I think they have always thought of it as "some throwaway stuff that generates the data for the paper"; the paper is the thing deserving of respect, not the tools used to create it. But I think this has been slowly changing as more and more scientists are digitally native and learn increasingly more software skills as a first class concern.