9 ms·

> For example, a trivial hello world program in Julia runs ~27x slower than Python’s version and ~187x slower than the one in C. I don't think it makes any sen

by ken 7y ago

> For example, a trivial hello world program in Julia runs ~27x slower than Python’s version and ~187x slower than the one in C. I don't think it makes any sense to speak of "__x slower" for hello-world. Clearly, this is just a benchmark of startup time, so you only pay it once per program. It should be reported as "__ms slower". Julia startup (according to this post) takes 371ms. That's 357ms slower than Python, and 369ms slower than C. Faster is always better, but this doesn't seem so bad to me. For comparison, on my old workstation here, starting a Swift repl takes 2724ms, and starting a Clojure repl takes 4792ms.

JoelMcCracken 7y ago
It’s relevant if you are discussing cli scripts. It’s often something I write and usually something I consider.
- seanmcdirmid 7y ago
  Is Julia really meant for CLI scripting?
  - leiroigh 7y ago
    It's not. But people who like the language will invariably use beyond its core competency. Hence it is important to ensure that julia is "kinda mediocre" for CLI scripting (big step up from "absolutely terrible"). Personally, my familiarity with julia and its features outweighs the fact that python/perl/bash would be the better tool for many CLI scripts.
    - snackematician 7y ago
      I think it's disappointing that Julia is not meant for CLI scripting. IMO it has failed to live up to its manifesto (https://julialang.org/blog/2012/02/why-we-created-julia https://julialang.org/blog/2012/02/why-we-created-julia) of being a general-purpose language that can "have it all". My own bioinformatics work involves parsing massive amounts of text data, and Julia is excellent for this -- it is very fast, yet high-level and easy to write. However, bash pipelines are also a huge part of bioinformatics, and Julia scripts are not very good for this due to the long startup time, which is a shame.
      - elcritch 7y ago
        > I think it's disappointing that Julia is not meant for CLI scripting. IMO it has failed to live up to its manifesto (https://julialang.org/blog/2012/02/why-we-created-julia https://julialang.org/blog/2012/02/why-we-created-julia) of being a general-purpose language that can "have it all". I think it’s still early in the languages development to say that. There isn’t any fundamental reasons Julia couldn’t be tuned to be better at CLI scripting. Currently it’s what, 400ms to startup, which isn’t too terrible but could be made faster. I could see using the Julia debugger as an interpreter for CLI scripts. Or if you run a script a lot it’s possible to have Julia compile an executable. Personally I use it in mainly via notebooks or a repl.
        dnautics 7y ago
        400ms to startup? > time julia -e "exit()" real 0m0.164s user 0m0.106s sys 0m0.047s This is on a relatively old laptop: Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz according to my lscpu
        manwe150 7y ago
        I helped work on getting that number from ~400ms to ~150ms maybe 18 month ago for the v1.0 release. FWIW, a big part of it was shedding a couple excess C libraries with bad load times (usually now lazy loading them when required instead). The next big jump will take more internal effort, but I don’t think there’s any serious showstopper. The bigger short-term interest though has been towards reducing the latency of loading external (user) code/libraries. For an example of what I mean by the latter, python seems to be pretty fast initially, but then seems to take a huge hit from just trying to get numpy loaded. $ time python -c '0' real0m0.024s user0m0.016s sys0m0.008s $ time python -c 'import numpy' real0m0.165s user0m1.508s sys0m2.312s $ time ./julia -e 0 real0m0.215s user0m0.240s sys0m0.144s
        elcritch 7y ago
        Excellent! I was mainly going off the numbers given by another comment. Well ~150ms seems within useable CLI times for me. One note, I’d reckon to have a fair comparison to the Numpy timings the Julia example would need to use an array type to do something.
    - ChrisFoster 7y ago
      Julia has been excellent for shelling out since almost the beginning. See https://julialang.org/blog/2012/03/shelling-out-sucks https://julialang.org/blog/2012/03/shelling-out-sucks
      - leiroigh 7y ago
        Nobody complains about julia for "shelling out". The imo justified complaint is that julia sucks at "shelling in". (OK, the pipe buffering rules still annoy me) Similar with FFI: Julia has awesome support for calling out to other languages; support for getting called by other languages is suboptimal.
kace91 7y ago
> It should be reported as "__ms slower". That would be dependent on hardware, while the relation will be constant.
- bogdanoff_2 7y ago
  Hardware is so complex and there is so much variation in it that this is "obviously" not true. As a simple example, if two computers are comparable in speed but one has a bigger cpu cache or faster ram, they might run programs with low memory footprint at the same speed, but one will be much faster when running a program that uses a lot of memory.
amelius 7y ago
To be fair, the article says: > If you ignore startup time, Julia might have good performance for simple array/matrix operations and loops, but we already know how to make them fast in Python and other languages. > And it’s not just scripts, Julia’s REPL which should ideally be optimized for responsiveness takes long to start and has noticeable JIT (?) lags. What’s even more worrying is that there doesn’t seem to be much progress there. The REPL was a pain to use a year ago and it still is. > In addition to that, Julia programs have excessive memory consumption. The above hello world example in Julia uses 18x more memory than Python and 92x more memory than the C version.
- ken 7y ago
  > The above hello world example in Julia uses 18x more memory than Python and 92x more memory than the C version. Again, I'm left wondering if this is proportional, or simply an X megabyte overhead of the runtime. Maybe it's even shared among all running programs, as some runtimes are. Or it looks like maybe stdio could be just particularly bad in Julia, and in practice I've never written a program for any environment where that was my performance bottleneck. Performance is not easy to measure, and not easy to report. A single number makes a good headline but it's really not enough information.
  - StefanKarpinski 7y ago
    It’s a fixed overhead like the JVM, CLR or any substantial runtime that includes a JIT.
    - cestith 7y ago
      Which, by the way, only applies to Julia programs under JIT. The AoT compiler will handle nearly 100% of vast the JIT or REPL will and vice versa, with a goal of 100%.
  - igouy 7y ago
    > … wondering if this is proportional, or simply an X megabyte overhead of the runtime… Which is why the benchmarks game measures at more than one workload — https://benchmarksgame-team.pages.debian.net/benchmarksgame/measurements/julia.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
deleted 7y ago
[deleted]
z3t4 7y ago
Developing tools for devs suck as they will use "hello world" as benchmark. And the tech stack will be decided by the non technical founder. Devs are more concerned about what tools others use rather then the pro et contra. /rant
- rini17 7y ago
  I haven't understood it as benchmark, rather a smell "they did not bother to optimize this?". The computers are so fast today, that instant startup time is low hanging fruit for a language.
  - Gibbon1 7y ago
    I think a lot of people that set out to design languages don't start out with the top three design goals as, fast compile times, fast startup, and robust hooks for tooling. The problem is if you don't start out with those as primary design goals you're going to be totally hosed once they become a problem later.
  - rickyrick_s 7y ago
    But it's an entirely 100% meaningless point of discussion though. My peers and I use Julia to run numerical heavy code where the large majority of the time is spent inverting a large matrix to solve the equation Ax = b for x. It's a lot more complicated than this, but essentially it can be boiled down to this main equation. When my code takes hours, pushing onto days, to run, why should I care about 300 ms of startup time at all? I stopped reading the article because their first point (the hello world benchmark) is entirely meaningless and is just noise in the wind. Their second point is a matter of personal opinion. After these two I didn't bother reading the rest.
- the8472 7y ago
  I think the language benchmarks game or the techempower benchmarks are fairly widely known and are a step or two above comparing hello world performance.
  - 7y ago
old-gregg 7y ago
Sure... and it means that Swift and Clojure are just as useless as Julia for numerous use cases where start-up times matter, like piping CLI commands together, in Julia's case you're looking at pathetic 3 execs per second. Moreover, even in some server-side applications it's super neat to have the luxury to spawn a new process to service certain requests, not having to worry about memory leaks. It's a perfect "API" which allows multiple languages to interact together. It always bothered me when people dismiss the start-up time by adding "just" in front of it. We're not talking about web frameworks, these are _general purpose_ programming languages, and horrendous start-up time automatically disqualifies them from being general purpose and places them into a niche category, in my humble opinion.
- 8ytecoder 7y ago
  Not necessarily. It's entirely possible to keep a daemon running with the loaded runtime to speed up start time. I'm not arguing if and whether these languages are suitable for scripting rather startup time alone need not be a disqualifying factor. If you evaluate holistically and arrive at a language you'll always have one thing or the other that is not as what you want.
  - skissane 7y ago
    A few years ago, as a hobby project, I designed a programming language and wrote an interpreter for it in Java. (I've never released it or shared it with anyone; I was never quite happy with it and eventually moved on to other things.) As you can imagine, an interpreted language with the interpreter written in Java is going to be rather slow to start. So, I implemented exactly the solution you describe here. I made my interpreter run as a daemon and listen for requests on a Unix domain socket (with a custom binary protocol). I then wrote a client in C, which opened that Unix domain socket and sent it a script to run. The C program also redirected stdin/stdout/stderr to/from the interpreter through the Unix domain socket, if connected to a tty it could switch to/from raw mode, it notified the server if certain signals occurred, etc. Using the interpreter this way, startup time was a lot faster. This approach can be applied to any language which supports runtime evaluation of code.
svnpenn 7y ago
its a systemic problem, not just startup time: https://github.com/JuliaLang/julia/issues/32198 https://github.com/JuliaLang/julia/issues/32198
- StefanKarpinski 7y ago
  That’s totally unrelated to startup or compilation time. It’s due to a bug in the crufty old Windows CMD.exe terminal—if you use a real terminal it doesn’t happen.
  - nneonneo 7y ago
    I’m now actually curious - what’s the bug in cmd.exe and why doesn’t that happen with other REPLs? I’m well aware that cmd.exe has its fair share of weird bugs, but on Windows it’s sometimes the only thing available. Plus, every third party terminal basically has to interface with the Windows GUI command-line program via screen-scraping, and so they inherit many of the same bugs. (The Windows Console architecture is fascinating - see https://devblogs.microsoft.com/commandline/windows-command-line-inside-the-windows-console/ https://devblogs.microsoft.com/commandline/windows-command-l... for a rundown!)
    - StefanKarpinski 7y ago
      If we knew that it would be fixed by now. Julia's REPL only supports VT100 terminals and its descendants. My guess is that CMD.exe on older Windowses fails to emulate VT100 correctly somehow.
  - svnpenn 7y ago
    that doesnt explain why every other interpreted language ive ever used doesnt have this problem. its a julia issue that the devs refuse to acknowledge or even test
    - ChrisRackauckas 7y ago
      Julia isn't interpreted. There is a package that implements an interpreter though and it doesn't have these issues.
dTal 7y ago
I want to know what business anything running on a modern computer has taking that much time to start up. Aside from the fact that 5 seconds is an insane amount of time, this is before any user code has been loaded, so presumably your Clojure repl is executing the same 5 seconds of work every time it starts. Can't that be cached?
- harperlee 7y ago
  I think this is leiningen and not clojure itself, by the way.
jnbiche 7y ago
You can't compare starting up a statically-typed, compiled Swift repl to those other dynamic, interpreted languages -- it's doing a lot of things that running a normal, compiled Swift program wouldn't do, and that dynamically-typed Clojure repl doesn't do (even though it's slower). The Swift repl is essentially a debugger running a compile-run cycle on each entered expression. Some statically-compiled languages have slow compilers but extremely fast runtime execution (Rust, and to a lesser extent C++, come to mind).

by ken 7y ago

JoelMcCracken 7y ago

It’s relevant if you are discussing cli scripts. It’s often something I write and usually something I consider.

seanmcdirmid 7y ago
Is Julia really meant for CLI scripting?
- leiroigh 7y ago
  It's not. But people who like the language will invariably use beyond its core competency. Hence it is important to ensure that julia is "kinda mediocre" for CLI scripting (big step up from "absolutely terrible"). Personally, my familiarity with julia and its features outweighs the fact that python/perl/bash would be the better tool for many CLI scripts.
  - snackematician 7y ago
    I think it's disappointing that Julia is not meant for CLI scripting. IMO it has failed to live up to its manifesto (https://julialang.org/blog/2012/02/why-we-created-julia https://julialang.org/blog/2012/02/why-we-created-julia) of being a general-purpose language that can "have it all". My own bioinformatics work involves parsing massive amounts of text data, and Julia is excellent for this -- it is very fast, yet high-level and easy to write. However, bash pipelines are also a huge part of bioinformatics, and Julia scripts are not very good for this due to the long startup time, which is a shame.
    - elcritch 7y ago
      > I think it's disappointing that Julia is not meant for CLI scripting. IMO it has failed to live up to its manifesto (https://julialang.org/blog/2012/02/why-we-created-julia https://julialang.org/blog/2012/02/why-we-created-julia) of being a general-purpose language that can "have it all". I think it’s still early in the languages development to say that. There isn’t any fundamental reasons Julia couldn’t be tuned to be better at CLI scripting. Currently it’s what, 400ms to startup, which isn’t too terrible but could be made faster. I could see using the Julia debugger as an interpreter for CLI scripts. Or if you run a script a lot it’s possible to have Julia compile an executable. Personally I use it in mainly via notebooks or a repl.
      - dnautics 7y ago
        400ms to startup? > time julia -e "exit()" real 0m0.164s user 0m0.106s sys 0m0.047s This is on a relatively old laptop: Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz according to my lscpu
        manwe150 7y ago
        I helped work on getting that number from ~400ms to ~150ms maybe 18 month ago for the v1.0 release. FWIW, a big part of it was shedding a couple excess C libraries with bad load times (usually now lazy loading them when required instead). The next big jump will take more internal effort, but I don’t think there’s any serious showstopper. The bigger short-term interest though has been towards reducing the latency of loading external (user) code/libraries. For an example of what I mean by the latter, python seems to be pretty fast initially, but then seems to take a huge hit from just trying to get numpy loaded. $ time python -c '0' real0m0.024s user0m0.016s sys0m0.008s $ time python -c 'import numpy' real0m0.165s user0m1.508s sys0m2.312s $ time ./julia -e 0 real0m0.215s user0m0.240s sys0m0.144s
        elcritch 7y ago
        Excellent! I was mainly going off the numbers given by another comment. Well ~150ms seems within useable CLI times for me. One note, I’d reckon to have a fair comparison to the Numpy timings the Julia example would need to use an array type to do something.
  - ChrisFoster 7y ago
    Julia has been excellent for shelling out since almost the beginning. See https://julialang.org/blog/2012/03/shelling-out-sucks https://julialang.org/blog/2012/03/shelling-out-sucks
    - leiroigh 7y ago
      Nobody complains about julia for "shelling out". The imo justified complaint is that julia sucks at "shelling in". (OK, the pipe buffering rules still annoy me) Similar with FFI: Julia has awesome support for calling out to other languages; support for getting called by other languages is suboptimal.

kace91 7y ago

> It should be reported as "__ms slower". That would be dependent on hardware, while the relation will be constant.

bogdanoff_2 7y ago
Hardware is so complex and there is so much variation in it that this is "obviously" not true. As a simple example, if two computers are comparable in speed but one has a bigger cpu cache or faster ram, they might run programs with low memory footprint at the same speed, but one will be much faster when running a program that uses a lot of memory.

amelius 7y ago

To be fair, the article says: > If you ignore startup time, Julia might have good performance for simple array/matrix operations and loops, but we already know how to make them fast in Python and other languages. > And it’s not just scripts, Julia’s REPL which should ideally be optimized for responsiveness takes long to start and has noticeable JIT (?) lags. What’s even more worrying is that there doesn’t seem to be much progress there. The REPL was a pain to use a year ago and it still is. > In addition to that, Julia programs have excessive memory consumption. The above hello world example in Julia uses 18x more memory than Python and 92x more memory than the C version.

ken 7y ago
> The above hello world example in Julia uses 18x more memory than Python and 92x more memory than the C version. Again, I'm left wondering if this is proportional, or simply an X megabyte overhead of the runtime. Maybe it's even shared among all running programs, as some runtimes are. Or it looks like maybe stdio could be just particularly bad in Julia, and in practice I've never written a program for any environment where that was my performance bottleneck. Performance is not easy to measure, and not easy to report. A single number makes a good headline but it's really not enough information.
- StefanKarpinski 7y ago
  It’s a fixed overhead like the JVM, CLR or any substantial runtime that includes a JIT.
  - cestith 7y ago
    Which, by the way, only applies to Julia programs under JIT. The AoT compiler will handle nearly 100% of vast the JIT or REPL will and vice versa, with a goal of 100%.
- igouy 7y ago
  > … wondering if this is proportional, or simply an X megabyte overhead of the runtime… Which is why the benchmarks game measures at more than one workload — https://benchmarksgame-team.pages.debian.net/benchmarksgame/measurements/julia.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

deleted 7y ago

[deleted]

z3t4 7y ago

Developing tools for devs suck as they will use "hello world" as benchmark. And the tech stack will be decided by the non technical founder. Devs are more concerned about what tools others use rather then the pro et contra. /rant

rini17 7y ago
I haven't understood it as benchmark, rather a smell "they did not bother to optimize this?". The computers are so fast today, that instant startup time is low hanging fruit for a language.
- Gibbon1 7y ago
  I think a lot of people that set out to design languages don't start out with the top three design goals as, fast compile times, fast startup, and robust hooks for tooling. The problem is if you don't start out with those as primary design goals you're going to be totally hosed once they become a problem later.
- rickyrick_s 7y ago
  But it's an entirely 100% meaningless point of discussion though. My peers and I use Julia to run numerical heavy code where the large majority of the time is spent inverting a large matrix to solve the equation Ax = b for x. It's a lot more complicated than this, but essentially it can be boiled down to this main equation. When my code takes hours, pushing onto days, to run, why should I care about 300 ms of startup time at all? I stopped reading the article because their first point (the hello world benchmark) is entirely meaningless and is just noise in the wind. Their second point is a matter of personal opinion. After these two I didn't bother reading the rest.
the8472 7y ago
I think the language benchmarks game or the techempower benchmarks are fairly widely known and are a step or two above comparing hello world performance.
- 7y ago

old-gregg 7y ago

Sure... and it means that Swift and Clojure are just as useless as Julia for numerous use cases where start-up times matter, like piping CLI commands together, in Julia's case you're looking at pathetic 3 execs per second. Moreover, even in some server-side applications it's super neat to have the luxury to spawn a new process to service certain requests, not having to worry about memory leaks. It's a perfect "API" which allows multiple languages to interact together. It always bothered me when people dismiss the start-up time by adding "just" in front of it. We're not talking about web frameworks, these are _general purpose_ programming languages, and horrendous start-up time automatically disqualifies them from being general purpose and places them into a niche category, in my humble opinion.

8ytecoder 7y ago
Not necessarily. It's entirely possible to keep a daemon running with the loaded runtime to speed up start time. I'm not arguing if and whether these languages are suitable for scripting rather startup time alone need not be a disqualifying factor. If you evaluate holistically and arrive at a language you'll always have one thing or the other that is not as what you want.
- skissane 7y ago
  A few years ago, as a hobby project, I designed a programming language and wrote an interpreter for it in Java. (I've never released it or shared it with anyone; I was never quite happy with it and eventually moved on to other things.) As you can imagine, an interpreted language with the interpreter written in Java is going to be rather slow to start. So, I implemented exactly the solution you describe here. I made my interpreter run as a daemon and listen for requests on a Unix domain socket (with a custom binary protocol). I then wrote a client in C, which opened that Unix domain socket and sent it a script to run. The C program also redirected stdin/stdout/stderr to/from the interpreter through the Unix domain socket, if connected to a tty it could switch to/from raw mode, it notified the server if certain signals occurred, etc. Using the interpreter this way, startup time was a lot faster. This approach can be applied to any language which supports runtime evaluation of code.

svnpenn 7y ago

its a systemic problem, not just startup time: https://github.com/JuliaLang/julia/issues/32198 https://github.com/JuliaLang/julia/issues/32198

StefanKarpinski 7y ago
That’s totally unrelated to startup or compilation time. It’s due to a bug in the crufty old Windows CMD.exe terminal—if you use a real terminal it doesn’t happen.
- nneonneo 7y ago
  I’m now actually curious - what’s the bug in cmd.exe and why doesn’t that happen with other REPLs? I’m well aware that cmd.exe has its fair share of weird bugs, but on Windows it’s sometimes the only thing available. Plus, every third party terminal basically has to interface with the Windows GUI command-line program via screen-scraping, and so they inherit many of the same bugs. (The Windows Console architecture is fascinating - see https://devblogs.microsoft.com/commandline/windows-command-line-inside-the-windows-console/ https://devblogs.microsoft.com/commandline/windows-command-l... for a rundown!)
  - StefanKarpinski 7y ago
    If we knew that it would be fixed by now. Julia's REPL only supports VT100 terminals and its descendants. My guess is that CMD.exe on older Windowses fails to emulate VT100 correctly somehow.
- svnpenn 7y ago
  that doesnt explain why every other interpreted language ive ever used doesnt have this problem. its a julia issue that the devs refuse to acknowledge or even test
  - ChrisRackauckas 7y ago
    Julia isn't interpreted. There is a package that implements an interpreter though and it doesn't have these issues.

dTal 7y ago

I want to know what business anything running on a modern computer has taking that much time to start up. Aside from the fact that 5 seconds is an insane amount of time, this is before any user code has been loaded, so presumably your Clojure repl is executing the same 5 seconds of work every time it starts. Can't that be cached?

harperlee 7y ago
I think this is leiningen and not clojure itself, by the way.

jnbiche 7y ago

You can't compare starting up a statically-typed, compiled Swift repl to those other dynamic, interpreted languages -- it's doing a lot of things that running a normal, compiled Swift program wouldn't do, and that dynamically-typed Clojure repl doesn't do (even though it's slower). The Swift repl is essentially a debugger running a compile-run cycle on each entered expression. Some statically-compiled languages have slow compilers but extremely fast runtime execution (Rust, and to a lesser extent C++, come to mind).