4 ms·

This just seems like such an artificially constructed problem. You just move that loop into Julia and the problem is solved. It is not hard to write shell code

This just seems like such an artificially constructed problem. You just move that loop into Julia and the problem is solved. It is not hard to write shell code in Julia. I have rewritten complex build scripts in bash into Julia code. It was not very hard and it made everything run way faster despite mostly shelling out to external programs. I don't know how secret/protected this shell script of yours is, but I'd be willing to have a go at rewriting the loop part in Julia if you showed it to me. Anyway I do think the startup speed problem can be solved in Julia and they don't need a rockstar advisor to do it. I think the solutions are already pretty well known. It is more of an issue of manpower. Somebody has to put in the hours to do it. And that is already being done. First time to plot will be twice as fast as it used to be in the next Julia release. Further improvements can be done, but again that requires somebody to put in the hours. Julia does not have access to Google level resources.

enriquto 6y ago
> the startup speed problem can be solved in Julia and they don't need a rockstar advisor to do it. I think the solutions are already pretty well known. > And that is already being done. First time to plot will be twice as fast as it used to be in the next Julia release. Further improvements can be done I'm really, really happy to hear that! I hope the Julia runtime becomes more and more streamlined in the (near) future. See, I was just stating my use case, without pretense that it is representative at all. Yet I received 20 upvotes in a few minutes, and Julia is maybe the only language where "time to first plot" is a thing. So I'm not completely alone in my (admittedly minoritary) concern. You say that rewriting everything in Julia would solve my problem. I'm sure that this is the case, but this is not at all my point. Some of us do not want a shell replacement, we want a bc replacement, and julia is a nearly perfect one, if it wasn't for the outrageously slow startup time. I have zero interest in the julia REPL, I just write julia scripts (among scripts in other languages) and I'm not willing to change that.
- dTal 6y ago
  Julia is a compiled language, like C. Using it in this way is basically like writing a shell script in C and calling GCC on every invocation. You can't expect good performance from that, and it's really a testament to Julia that it feels so dynamic that you feel as it it should do that. If you AOT-compiled your Julia "script" (program) to a binary and invoked that, your startup time problems would go away. Julia's "application deployment" stack is underdeveloped compared to its REPL experience, but it's still possible to do this today with PackageCompiler.jl and will only get easier with time. I think that will prove to be the "right" way to solve this problem in the long run. (Or, if you don't care about performance, you can just turn off the compiler and interpret everything, and you basically have Python-but-in-Julia. Fast startup, slow running. Just run it with julia --compile=no)
  - enriquto 6y ago
    > writing a shell script in C and calling GCC on every invocation. You can't expect good performance from that Of course you can. Have you ever used the "-run" option of the TCC compiler? It's blazingly fast. With gcc it's a bit slower, but still orders of magnitude faster than julia. You can use pre-compiled libraries and the linking to your freshly compiled code is extremely fast. The fact is that compiling and linking C code is much faster that just launching the Julia environment with some packages. There's no fundamental reason for that enormous disparity in running time. I agree that it is a completely irrelevant nuisance for most people; but still, for some workflows not blessed by the Julia developers, it is the main point of friction. EDIT: if you want to try it yourself, write the following text into a .c file, chmod +x it, and you can run it like a C script on most unix systems: //usr/bin/gcc -O0 "$0" -lpng && exec ./a.out "$@" #include <png.h> int main(int c, char *v[]) { // do stuff with png images return 0; }
    - dTal 6y ago
      Point taken, but I think you overreach a bit with: >There's no fundamental reason for that enormous disparity in running time I mean... Julia does full type inference, which it uses to present a dynamic type interface. It's not necessarily possible to statically compile a module ahead of time, because the module is designed to be generic and will generate different code if fed different types, which is how Julia attains its extraordinary composability. In other words, it's a much nicer language than C, and correspondingly much harder to compile. I'd call that a pretty fundamental reason. Perhaps C was a poor example for me to pick. C++, maybe? p.s. your "script" example clobbers any file named "a.out" in the current directory.
- deleted 6y ago
  [deleted]
CJefferson 6y ago
Often the reason I have a bash script is because I'm running some AI tool to solve lots of problems in parallel, gather/filter results, and then gathering results, then finally doing plotting and things. Personally, I never really like solutions which are "you can't do 5% of stuff in X, you have to do everything in X". I like trying new things out, but I'm not willing to 100% invest everything in Julia up front, rather than just trying out some small bits. Also, I think my collaborators would get annoyed if I rewrote all our bash scripts in Julia -- I can't expect them all to learn Julia.
cygx 6y ago
A tool with slow startup time that forces me to structure the workflow around its idiosyncrasies is less convenient than a tool with fast startup time that can be seamlessly integrated into pre-existing workflows (eg driving pipelines via Makefiles).

enriquto 6y ago

> the startup speed problem can be solved in Julia and they don't need a rockstar advisor to do it. I think the solutions are already pretty well known. > And that is already being done. First time to plot will be twice as fast as it used to be in the next Julia release. Further improvements can be done I'm really, really happy to hear that! I hope the Julia runtime becomes more and more streamlined in the (near) future. See, I was just stating my use case, without pretense that it is representative at all. Yet I received 20 upvotes in a few minutes, and Julia is maybe the only language where "time to first plot" is a thing. So I'm not completely alone in my (admittedly minoritary) concern. You say that rewriting everything in Julia would solve my problem. I'm sure that this is the case, but this is not at all my point. Some of us do not want a shell replacement, we want a bc replacement, and julia is a nearly perfect one, if it wasn't for the outrageously slow startup time. I have zero interest in the julia REPL, I just write julia scripts (among scripts in other languages) and I'm not willing to change that.

dTal 6y ago
Julia is a compiled language, like C. Using it in this way is basically like writing a shell script in C and calling GCC on every invocation. You can't expect good performance from that, and it's really a testament to Julia that it feels so dynamic that you feel as it it should do that. If you AOT-compiled your Julia "script" (program) to a binary and invoked that, your startup time problems would go away. Julia's "application deployment" stack is underdeveloped compared to its REPL experience, but it's still possible to do this today with PackageCompiler.jl and will only get easier with time. I think that will prove to be the "right" way to solve this problem in the long run. (Or, if you don't care about performance, you can just turn off the compiler and interpret everything, and you basically have Python-but-in-Julia. Fast startup, slow running. Just run it with julia --compile=no)
- enriquto 6y ago
  > writing a shell script in C and calling GCC on every invocation. You can't expect good performance from that Of course you can. Have you ever used the "-run" option of the TCC compiler? It's blazingly fast. With gcc it's a bit slower, but still orders of magnitude faster than julia. You can use pre-compiled libraries and the linking to your freshly compiled code is extremely fast. The fact is that compiling and linking C code is much faster that just launching the Julia environment with some packages. There's no fundamental reason for that enormous disparity in running time. I agree that it is a completely irrelevant nuisance for most people; but still, for some workflows not blessed by the Julia developers, it is the main point of friction. EDIT: if you want to try it yourself, write the following text into a .c file, chmod +x it, and you can run it like a C script on most unix systems: //usr/bin/gcc -O0 "$0" -lpng && exec ./a.out "$@" #include <png.h> int main(int c, char *v[]) { // do stuff with png images return 0; }
  - dTal 6y ago
    Point taken, but I think you overreach a bit with: >There's no fundamental reason for that enormous disparity in running time I mean... Julia does full type inference, which it uses to present a dynamic type interface. It's not necessarily possible to statically compile a module ahead of time, because the module is designed to be generic and will generate different code if fed different types, which is how Julia attains its extraordinary composability. In other words, it's a much nicer language than C, and correspondingly much harder to compile. I'd call that a pretty fundamental reason. Perhaps C was a poor example for me to pick. C++, maybe? p.s. your "script" example clobbers any file named "a.out" in the current directory.
deleted 6y ago
[deleted]

CJefferson 6y ago

Often the reason I have a bash script is because I'm running some AI tool to solve lots of problems in parallel, gather/filter results, and then gathering results, then finally doing plotting and things. Personally, I never really like solutions which are "you can't do 5% of stuff in X, you have to do everything in X". I like trying new things out, but I'm not willing to 100% invest everything in Julia up front, rather than just trying out some small bits. Also, I think my collaborators would get annoyed if I rewrote all our bash scripts in Julia -- I can't expect them all to learn Julia.

cygx 6y ago

A tool with slow startup time that forces me to structure the workflow around its idiosyncrasies is less convenient than a tool with fast startup time that can be seamlessly integrated into pre-existing workflows (eg driving pipelines via Makefiles).