5 ms·

If it were 20 seconds for every plot, this would be a major problem for me, as I tend to make lots of plots. But it's only the first plot where this is an issue

by DNF2 6y ago

If it were 20 seconds for every plot, this would be a major problem for me, as I tend to make lots of plots. But it's only the first plot where this is an issue. Surely you're not in 'the zone' that soon? Seems to me like people are making a mountain out of a molehill.

jampekka 6y ago
If you don't use REPL (and for many good reasons you shouldn't) or some other such horror, every plot is the first plot. And it's pain. And not just plots really. Doing anything in Julia is pain if you try to use it as a programming language instead of an app for buggy, unreproducible and misunderstood ad-hoc analyses. Seeing these answers makes me think Julia will never be fixed. I forecast Julia will be back in a niche within five years if they don't get their act together. And it's sad, because the alternatives are fundamentally broken. Julia isn't fundamentally broken, but the devs and the community seem to insist on superficial breakage.
- eigenspace 6y ago
  > (and for many good reasons you shouldn't) Could you elaborate on this? I'd say repl based interactive programming is one of julia's greatest strengths, and avoiding the repl is probably setting yourself up for pain. That said, if you do find yourself running lots of scripts and paying this penalty all the time, I'd suggest https://github.com/dmolina/DaemonMode.jl https://github.com/dmolina/DaemonMode.jl as a great way around these pains.
  - jampekka 6y ago
    > Could you elaborate on this? I'd say repl based interactive programming is one of julia's greatest strengths, and avoiding the repl is probably setting yourself up for pain. With REPL you have an invisible global state, can't reproduce what you have done, changes earlier in code path don't propagate to results, you don't have documentation of what you did. It's for me really like trying to write a book by dictating. Except that you're dictating to somebody who's gonna give an independent summary of it to a third party and never gonna write down what you dictated. It boggles my mind how people can work like this, but they probably get hooked to REPL from the first tutorials and just don't know better. I'll look into DaemonMode.jl. Not a fan of using a daemon (and I'm guessing there will be problems with e.g. interactive plots), but in the short term I'll take anything that could make Julia programming tolerable.
    - eigenspace 6y ago
      > With REPL you have an invisible global state, can't reproduce what you have done, changes earlier in code path don't propagate to results, you don't have documentation of what you did. Mhm, that's fair. I think Pluto.jl has a really neat approach to this, using reactivity (and technically even more state) to actually eliminate that experienced state. If I could use it from emacs it might even be my goto way to interact with julia, but I also don't mind the statefulnes and find it manageable. For me, the most important thing is that when I'm writing serious code, I create a local package. Then, in the REPL I load that package and have Revise.jl active so that it can watch the the package source ode and constantly do hot code reloading for me so that I'm never stuck with old versions of code running. Then I do all my interactive analysis in the REPL, and plumbing in the package module. This eliminates a lot of statefulness, but keeps restarts to a minimum.
      - jampekka 6y ago
        I briefly looked at Pluto.jl, and I think it's probably a good way. As I understood it it's a "notebook" that always runs the whole file. Like e.g. RMarkdown or sweave. All good. The state is fine too if it's explicit. But I'm fine with just CLI and print and occasional plot, which should be a lot simpler use case for development. I create a "local package", meaning a file from which I relatively import. During development/analysis it's hard to foresee what the package structure is gonna be, so it's quite pointless to go through the whole packaging ceremony at this point. FromFile works fine for this. As a temporary hack I could use REPL to call my "main" function and let Revise.jl update automatically. (In long term this is bad for interoperability with rest of the system). But in my experience Revise.jl tends to break a lot. Julia breakage is hard to analyze by itself, and Revise.jl often makes this more or less impossible. I have to repeat that I really don't see how caching of the compilation results is even close the complications that Revise.jl or Pluto.jl have to do.
        eigenspace 6y ago
        > I briefly looked at Pluto.jl, and I think it's probably a good way. As I understood it it's a "notebook" that always runs the whole file. Not quite. It builds a dependancy graph of your code and can figure out what definitions depend on others. So depending on what you change, maybe only one or two cells need to be rerun. Or in other circumstances, the whole notebook will have to re-run. It just depends on what changes. > I have to repeat that I really don't see how caching of the compilation results is even close the complications that Revise.jl or Pluto.jl have to do. I think the main trouble with the caching is that the native code you cache can depend very strongly on the exact combination of packages you have loaded. This means you can hit a combinatorial explosion of different methods to cache pretty quickly, so you'd need to find a very clever way to find the right methods to keep and which ones to delete once the cache gets too big. I think there's also other potential issues that I understand less. This is being actively worked on though.
  - orbifold 6y ago
    I and probably many others intend to develop reproducible figures. The way to do that in python / matlab is to have a script which loads data from disk and then produces a figure (a png / pdf). You then execute that file many times each time tweaking one aspect of the figure. Julia makes that workflow almost impossibly slow.
    - cbkeller 6y ago
      Then write a function that does everything your script would do in the clean local scope of that function, and call it many times as needed. I mean heck, that’s a more elegant solution even if script latency wasn’t in the equation.
      - jampekka 6y ago
        I usually do a clean local scope of the function anyway. The problem is the reloading. Revise.jl works sometimes, but sometimes doesn't, and it makes debugging more difficult (this is difficult enough in Julia as is in my experience). Another problem is having to use the REPL that doesn't integrate as nicely with the rest of the OS as shell. I don't see why REPLing it is a more elegant solution. With that solution I can call the function form the REPL if I want, but also from the shell if I want. With shell I get the elegance of having a persistent, complete and reproducible description of the state all the time, which can be e.g. version controlled.
    - eigenspace 6y ago
      Why not just write a function? You can do this without restarting tour julia session every time...
      - orbifold 6y ago
        Well to be more precise the script contains functions of course , each of which produces one of the figures and takes the path to the data / data as input. In any case the REPL has no place in such a workflow because I want to be able to check in the script at the end and have a reliable way of reproducing any of the figures I put into the paper later.

by DNF2 6y ago

jampekka 6y ago

If you don't use REPL (and for many good reasons you shouldn't) or some other such horror, every plot is the first plot. And it's pain. And not just plots really. Doing anything in Julia is pain if you try to use it as a programming language instead of an app for buggy, unreproducible and misunderstood ad-hoc analyses. Seeing these answers makes me think Julia will never be fixed. I forecast Julia will be back in a niche within five years if they don't get their act together. And it's sad, because the alternatives are fundamentally broken. Julia isn't fundamentally broken, but the devs and the community seem to insist on superficial breakage.

eigenspace 6y ago
> (and for many good reasons you shouldn't) Could you elaborate on this? I'd say repl based interactive programming is one of julia's greatest strengths, and avoiding the repl is probably setting yourself up for pain. That said, if you do find yourself running lots of scripts and paying this penalty all the time, I'd suggest https://github.com/dmolina/DaemonMode.jl https://github.com/dmolina/DaemonMode.jl as a great way around these pains.
- jampekka 6y ago
  > Could you elaborate on this? I'd say repl based interactive programming is one of julia's greatest strengths, and avoiding the repl is probably setting yourself up for pain. With REPL you have an invisible global state, can't reproduce what you have done, changes earlier in code path don't propagate to results, you don't have documentation of what you did. It's for me really like trying to write a book by dictating. Except that you're dictating to somebody who's gonna give an independent summary of it to a third party and never gonna write down what you dictated. It boggles my mind how people can work like this, but they probably get hooked to REPL from the first tutorials and just don't know better. I'll look into DaemonMode.jl. Not a fan of using a daemon (and I'm guessing there will be problems with e.g. interactive plots), but in the short term I'll take anything that could make Julia programming tolerable.
  - eigenspace 6y ago
    > With REPL you have an invisible global state, can't reproduce what you have done, changes earlier in code path don't propagate to results, you don't have documentation of what you did. Mhm, that's fair. I think Pluto.jl has a really neat approach to this, using reactivity (and technically even more state) to actually eliminate that experienced state. If I could use it from emacs it might even be my goto way to interact with julia, but I also don't mind the statefulnes and find it manageable. For me, the most important thing is that when I'm writing serious code, I create a local package. Then, in the REPL I load that package and have Revise.jl active so that it can watch the the package source ode and constantly do hot code reloading for me so that I'm never stuck with old versions of code running. Then I do all my interactive analysis in the REPL, and plumbing in the package module. This eliminates a lot of statefulness, but keeps restarts to a minimum.
    - jampekka 6y ago
      I briefly looked at Pluto.jl, and I think it's probably a good way. As I understood it it's a "notebook" that always runs the whole file. Like e.g. RMarkdown or sweave. All good. The state is fine too if it's explicit. But I'm fine with just CLI and print and occasional plot, which should be a lot simpler use case for development. I create a "local package", meaning a file from which I relatively import. During development/analysis it's hard to foresee what the package structure is gonna be, so it's quite pointless to go through the whole packaging ceremony at this point. FromFile works fine for this. As a temporary hack I could use REPL to call my "main" function and let Revise.jl update automatically. (In long term this is bad for interoperability with rest of the system). But in my experience Revise.jl tends to break a lot. Julia breakage is hard to analyze by itself, and Revise.jl often makes this more or less impossible. I have to repeat that I really don't see how caching of the compilation results is even close the complications that Revise.jl or Pluto.jl have to do.
      - eigenspace 6y ago
        > I briefly looked at Pluto.jl, and I think it's probably a good way. As I understood it it's a "notebook" that always runs the whole file. Not quite. It builds a dependancy graph of your code and can figure out what definitions depend on others. So depending on what you change, maybe only one or two cells need to be rerun. Or in other circumstances, the whole notebook will have to re-run. It just depends on what changes. > I have to repeat that I really don't see how caching of the compilation results is even close the complications that Revise.jl or Pluto.jl have to do. I think the main trouble with the caching is that the native code you cache can depend very strongly on the exact combination of packages you have loaded. This means you can hit a combinatorial explosion of different methods to cache pretty quickly, so you'd need to find a very clever way to find the right methods to keep and which ones to delete once the cache gets too big. I think there's also other potential issues that I understand less. This is being actively worked on though.
- orbifold 6y ago
  I and probably many others intend to develop reproducible figures. The way to do that in python / matlab is to have a script which loads data from disk and then produces a figure (a png / pdf). You then execute that file many times each time tweaking one aspect of the figure. Julia makes that workflow almost impossibly slow.
  - cbkeller 6y ago
    Then write a function that does everything your script would do in the clean local scope of that function, and call it many times as needed. I mean heck, that’s a more elegant solution even if script latency wasn’t in the equation.
    - jampekka 6y ago
      I usually do a clean local scope of the function anyway. The problem is the reloading. Revise.jl works sometimes, but sometimes doesn't, and it makes debugging more difficult (this is difficult enough in Julia as is in my experience). Another problem is having to use the REPL that doesn't integrate as nicely with the rest of the OS as shell. I don't see why REPLing it is a more elegant solution. With that solution I can call the function form the REPL if I want, but also from the shell if I want. With shell I get the elegance of having a persistent, complete and reproducible description of the state all the time, which can be e.g. version controlled.
  - eigenspace 6y ago
    Why not just write a function? You can do this without restarting tour julia session every time...
    - orbifold 6y ago
      Well to be more precise the script contains functions of course , each of which produces one of the figures and takes the path to the data / data as input. In any case the REPL has no place in such a workflow because I want to be able to check in the script at the end and have a reliable way of reproducing any of the figures I put into the paper later.