39 ms·

I am curious: What is Julia's current niche? I mean, I tried it a few times a decade ago, but the environment was not mature, and performance improvements were

by stared 3y ago

I am curious: What is Julia's current niche? I mean, I tried it a few times a decade ago, but the environment was not mature, and performance improvements were not in the places I needed. Though, I guess, things moved from there. Python has the richest environment for data science. For statistics, there are some tools in R, such as the stunning ggplot2. For numerics-heavy arrays, we have PyTorch. For custom low-level numerics, Rust rocks.

the__alchemist 3y ago
ODE solving? Best in class. I tried to use it for quick scripts and calculator-like-use, but the JIT compile times made it unfeasible. (Julia's math syntax is the best I've seen) For more complex and performance-intensive programs, I don't see the case for choosing it over a general languages like Rust and C++.
- tombert 3y ago
  > For more complex and performance-intensive programs, I don't see the case for choosing it over a general languages like Rust and C++. I mean, the fact that it's getting comparable speeds to C++ or Rust, while having full garbage collection and nice built-in concurrency support is a selling point to me. I'm not saying you couldn't write code faster in C++ or Rust, but I'm going to argue that Julia will often be just as fast or faster from "default implement".
  - CyberDildonics 3y ago
    The garbage collection will cause hiccups and hangs. It isn't faster if you are leaning on garbage collection. Just like any other garbage collected language, you want to avoid the garbage collection.
    - tombert 3y ago
      Before I say anything, I think your username is pretty funny. Props for snagging that one. > It isn't faster if you are leaning on garbage collection. Just like any other garbage collected language, you want to avoid the garbage collection. I agree in theory but not in practice. Absolutely, if you are good at properly freeing memory, and knowing when to free memory, and how much to free and all that fun stuff, you can definitely get much better performance than a GC, with fewer (or zero) pauses and better throughput. However, I don't think I agree that that is the average case. A lot of people will simply malloc and free at the beginning and ending of the function, or worse they will do things incorrectly and so the data won't actually free, or it frees before it's completed. A GC does incur some overhead, but they can also be extremely optimized and and have the advantage of doing things basically correctly, or at least safely. If you through multiple threads into the mix, then the problems get much harder. Julia's doesn't do this, but GC in languages like Go also basically don't cause hiccups or hangs, at least not long ones. The pause times are sub-millisecond [1], and it's pretty impressive. Even Java, the language that initially gave GC's a bad name, has the Shenandoah GC in some builds of OpenJDK which also gets sub-millisecond pauses [2]. Personally, I tend to believe that an application that stops working is generally "slower" than one that keeps working. A lot of the time savings you get from avoiding a GC will be eliminated if your program crashes and restarts. (That said, I will acknowledge that Rust probably makes this much better; I haven't done anything "real" with Rust so I cannot comment on it.) What I was trying to say was that Julia, at least in benchmarks, does get speeds getting close to something like C++, even with GC enabled, and it will certainly run circles around something like Python. If you want something that gets around at least 80+% of the speed of C++, while also giving all the prettiness and easiness of Python, I think Julia is a very good choice, and I would happily choose it over something like Go. [1] https://groups.google.com/g/golang-dev/c/Ab1sFeoZg_8 https://groups.google.com/g/golang-dev/c/Ab1sFeoZg_8 [2] https://wiki.openjdk.org/display/shenandoah/Main#Main-PerformanceGuidelinesandDiagnostics https://wiki.openjdk.org/display/shenandoah/Main#Main-Perfor...
      - CyberDildonics 3y ago
        A lot of people will simply malloc and free at the beginning and ending of the function, or worse they will do things incorrectly and so the data won't actually free, or it frees before it's completed I don't think programming mistakes in other languages make a point when talking about garbage collection performance. but they can also be extremely optimized Every language I've ever seen with garbage collection has gone through decades of "now the garbage collection is better" or "just wait until the next version, garbage collection will be better". I've seen it in java, D, julia and every scripting language. It's always a thorn people have to work around when talking about performance interactivity. Julia is right in the middle of this common problem and that is fine for what it is because it isn't a typical systems language and avoiding the gc is not difficult, but to pretend that it is the same as C++ is just not true. A lot of the time savings you get from avoiding a GC will be eliminated if your program crashes and restarts. This is besides the point of performance and no longer talking about reality, it's just FUD from a "what if" future. Julia, at least in benchmarks, does get speeds getting close to something like C++, even with GC enabled Right, but you get it by avoiding allocation and avoiding the garbage collector the same way avoiding allocation in C++ is important, but in julia it won't be woven in to the performance, it will cause big pauses.
        tombert 3y ago
        > Every language I've ever seen with garbage collection has gone through decades of "now the garbage collection is better" or "just wait until the next version, garbage collection will be better". Ok but the Go example I linked is already in production, right now, you can use it. This isn't a "it will get better in two releases" situation, Go's GC as of today has pause times that are sub-millisecond. The Java Shenandoah example I linked is still mostly in beta, but it's also something you can use right now, though admittedly it'll probably be awhile before it's in a mainline release. > This is besides the point of performance and no longer talking about reality, it's just FUD from a "what if" future. It's not "just FUD", there are dozens of reported security issues that have happened because of bad manual memory management problems. Off the top of my head, Heartbleed was a famous case. This isn't me badmouthing anyone; manual memory management is hard to get right, even for very smart people. > Right, but you get it by avoiding allocation and avoiding the garbage collector the same way avoiding allocation in C++ is important, but in julia it won't be woven in to the performance, it will cause big pauses. Fair enough, I did look at the code for the official benchmarks (https://github.com/JuliaLang/Microbenchmarks/blob/master/perf.jl#L25 https://github.com/JuliaLang/Microbenchmarks/blob/master/per...) and outside of the integer parsing code it does indeed seem to avoid dynamic allocations so I will concede that the benchmarks might be a bit more skewed compared to real-world code. I still get a hunch that if you compared it allocation-heavy Julia to malloc+free-heavy C++ the differences wouldn't really be that far off, but that's just a hunch and I don't have data to back that up; might be a fun test to write though, so maybe I'll try that this weekend. ----- Sort of tangential, but I also do think that there's value in having decent concurrency constructs built into the language. With C++, if you stick to built-ins you are basically stuck with mutexes and despite what people like to pretend, getting correct code with mutexes is really really hard to get right, and very easy to screw up in a non-obvious way. If you allow yourself to use libraries, then you have stuff like ZeroMQ and OpenMP and stuff, so it's really not that dire realistically. However, I think there's value in having nice, easy to use concurrency constructs in the language other than mutexes, and I do wonder if as a result of that it encourages people to utilize multiple threads more frequently, because they don't have to worry about weird deadlock situations as much. Again, I believe Rust actually does address this because of the single-owner-enforced-at-compile-time stuff, but I haven't used it enough to really draw a conclusion on it.
- samatman 3y ago
  > the JIT compile times made it unfeasible Worth trying again unless that was in the last 18 months. Precompilation has gotten dramatically faster, like 50x improvement in load times faster. So has startup time in general, `time julia -e "return 5"` gave 161 milliseconds of user time on my computer. I don't actually use Julia for simple script munging, simply because I'm used to Python in that role and it works fine, but it's suitable for it. Calculator-like use? Julia, all the way. Of course I keep the REPL open at all times. Gotta use tools the way they're intended, y'know?
  - the__alchemist 3y ago
    I'll give it a shot! I've been tracking a steady stream of published improvements this area over the years (decade?) I think the most striking moment I had was when I realized through experiment that rust, which has a notoriously slow compiler, was faster to compile and run a small plotting program than Julia. That was ~5 years ago though!
    - samatman 3y ago
      Entirely different language when it comes to compile times, they were tremendously successful in the time-to-first-plot project. 1.6 was the release in 2021 that made the biggest difference, but with steady improvement since. The big push now is to add everything needed so that making object/dylib code in Julia of comparable size to the usual systems languages is possible. The team has a track record of finishing what they start, I'm optimistic that they'll get that right as well.
sampo 3y ago
> What is Julia's current niche? Python, but for engineers and scientists. Syntax is friendlier for doing mathematics, barrier to entry to first calculation script is lower a = [1 2 3] b = [4 5 6] a + b vs import numpy as np a = np.array([1, 2, 3]) b = np.array([4, 5, 6]) a + b Also you can learn Julia step by step, and improve your code to run faster. Whereas in Python, the eventual road to fast code is to write the library in C and call it from Python. Python is for software people. Matlab, Julia and R are for people who only write code because they want answers.
- sampo 3y ago
  > Python is for software people. Matlab, Julia and R are for people who only write code because they want answers. Or, Julia can be for both. Julia lets you start from scientific calculations scripts and grow your code to a larger software package. Same as Python grew popular 20 years ago because it let you grow your small sysadmin scripts to larger software. Matlab and R are too limited in their "real programming language" constructs, and it gets painful when your codebase grows larger. Julia has all "real programming language" constructs a software engineer would need.
- the__alchemist 3y ago
  This syntax is my favorite feature of Julia's. I wish I could get something like this in python or rust, without a macro mess.
staplung 3y ago
Scientific computing. Of the languages you cited, none (other than Julia) have dynamic dispatch, a feature that turns out to be unexpectedly powerful and allows lots of packages to hang together much more effectively than you would expect. Also, Rust specifically doesn't have built-in rational numbers, complex numbers, bigints or proper matrixes. There are lots of crates that implement them but that's precisely what prevents a cohesive ecosystem from developing.
- abdullahkhalids 3y ago
  I don't know anything about this, but the wiki article [1] says that Python, Rust and Julia all have Dynamic dispatch. However, it seems like Python has single dispatch, but Julia has multiple dispatch. Is that what you meant? [1] https://en.wikipedia.org/wiki/Dynamic_dispatch https://en.wikipedia.org/wiki/Dynamic_dispatch
  - kruxigt 3y ago
    [dead]
jakobnissen 3y ago
I use it for high-performance scientific code. Code that needs to run fast, be written fast, and frequently extended and modified, with a mix of high and low level computing. This could be achieved by a mix of Python and Rust, but it's just too annoying to write scripts in two languages, where one is compiled and the other has terrible package management. Much easier to write it in Julia and have it be fast out of the box

by stared 3y ago

the__alchemist 3y ago

ODE solving? Best in class. I tried to use it for quick scripts and calculator-like-use, but the JIT compile times made it unfeasible. (Julia's math syntax is the best I've seen) For more complex and performance-intensive programs, I don't see the case for choosing it over a general languages like Rust and C++.

tombert 3y ago
> For more complex and performance-intensive programs, I don't see the case for choosing it over a general languages like Rust and C++. I mean, the fact that it's getting comparable speeds to C++ or Rust, while having full garbage collection and nice built-in concurrency support is a selling point to me. I'm not saying you couldn't write code faster in C++ or Rust, but I'm going to argue that Julia will often be just as fast or faster from "default implement".
- CyberDildonics 3y ago
  The garbage collection will cause hiccups and hangs. It isn't faster if you are leaning on garbage collection. Just like any other garbage collected language, you want to avoid the garbage collection.
  - tombert 3y ago
    Before I say anything, I think your username is pretty funny. Props for snagging that one. > It isn't faster if you are leaning on garbage collection. Just like any other garbage collected language, you want to avoid the garbage collection. I agree in theory but not in practice. Absolutely, if you are good at properly freeing memory, and knowing when to free memory, and how much to free and all that fun stuff, you can definitely get much better performance than a GC, with fewer (or zero) pauses and better throughput. However, I don't think I agree that that is the average case. A lot of people will simply malloc and free at the beginning and ending of the function, or worse they will do things incorrectly and so the data won't actually free, or it frees before it's completed. A GC does incur some overhead, but they can also be extremely optimized and and have the advantage of doing things basically correctly, or at least safely. If you through multiple threads into the mix, then the problems get much harder. Julia's doesn't do this, but GC in languages like Go also basically don't cause hiccups or hangs, at least not long ones. The pause times are sub-millisecond [1], and it's pretty impressive. Even Java, the language that initially gave GC's a bad name, has the Shenandoah GC in some builds of OpenJDK which also gets sub-millisecond pauses [2]. Personally, I tend to believe that an application that stops working is generally "slower" than one that keeps working. A lot of the time savings you get from avoiding a GC will be eliminated if your program crashes and restarts. (That said, I will acknowledge that Rust probably makes this much better; I haven't done anything "real" with Rust so I cannot comment on it.) What I was trying to say was that Julia, at least in benchmarks, does get speeds getting close to something like C++, even with GC enabled, and it will certainly run circles around something like Python. If you want something that gets around at least 80+% of the speed of C++, while also giving all the prettiness and easiness of Python, I think Julia is a very good choice, and I would happily choose it over something like Go. [1] https://groups.google.com/g/golang-dev/c/Ab1sFeoZg_8 https://groups.google.com/g/golang-dev/c/Ab1sFeoZg_8 [2] https://wiki.openjdk.org/display/shenandoah/Main#Main-PerformanceGuidelinesandDiagnostics https://wiki.openjdk.org/display/shenandoah/Main#Main-Perfor...
    - CyberDildonics 3y ago
      A lot of people will simply malloc and free at the beginning and ending of the function, or worse they will do things incorrectly and so the data won't actually free, or it frees before it's completed I don't think programming mistakes in other languages make a point when talking about garbage collection performance. but they can also be extremely optimized Every language I've ever seen with garbage collection has gone through decades of "now the garbage collection is better" or "just wait until the next version, garbage collection will be better". I've seen it in java, D, julia and every scripting language. It's always a thorn people have to work around when talking about performance interactivity. Julia is right in the middle of this common problem and that is fine for what it is because it isn't a typical systems language and avoiding the gc is not difficult, but to pretend that it is the same as C++ is just not true. A lot of the time savings you get from avoiding a GC will be eliminated if your program crashes and restarts. This is besides the point of performance and no longer talking about reality, it's just FUD from a "what if" future. Julia, at least in benchmarks, does get speeds getting close to something like C++, even with GC enabled Right, but you get it by avoiding allocation and avoiding the garbage collector the same way avoiding allocation in C++ is important, but in julia it won't be woven in to the performance, it will cause big pauses.
      - tombert 3y ago
        > Every language I've ever seen with garbage collection has gone through decades of "now the garbage collection is better" or "just wait until the next version, garbage collection will be better". Ok but the Go example I linked is already in production, right now, you can use it. This isn't a "it will get better in two releases" situation, Go's GC as of today has pause times that are sub-millisecond. The Java Shenandoah example I linked is still mostly in beta, but it's also something you can use right now, though admittedly it'll probably be awhile before it's in a mainline release. > This is besides the point of performance and no longer talking about reality, it's just FUD from a "what if" future. It's not "just FUD", there are dozens of reported security issues that have happened because of bad manual memory management problems. Off the top of my head, Heartbleed was a famous case. This isn't me badmouthing anyone; manual memory management is hard to get right, even for very smart people. > Right, but you get it by avoiding allocation and avoiding the garbage collector the same way avoiding allocation in C++ is important, but in julia it won't be woven in to the performance, it will cause big pauses. Fair enough, I did look at the code for the official benchmarks (https://github.com/JuliaLang/Microbenchmarks/blob/master/perf.jl#L25 https://github.com/JuliaLang/Microbenchmarks/blob/master/per...) and outside of the integer parsing code it does indeed seem to avoid dynamic allocations so I will concede that the benchmarks might be a bit more skewed compared to real-world code. I still get a hunch that if you compared it allocation-heavy Julia to malloc+free-heavy C++ the differences wouldn't really be that far off, but that's just a hunch and I don't have data to back that up; might be a fun test to write though, so maybe I'll try that this weekend. ----- Sort of tangential, but I also do think that there's value in having decent concurrency constructs built into the language. With C++, if you stick to built-ins you are basically stuck with mutexes and despite what people like to pretend, getting correct code with mutexes is really really hard to get right, and very easy to screw up in a non-obvious way. If you allow yourself to use libraries, then you have stuff like ZeroMQ and OpenMP and stuff, so it's really not that dire realistically. However, I think there's value in having nice, easy to use concurrency constructs in the language other than mutexes, and I do wonder if as a result of that it encourages people to utilize multiple threads more frequently, because they don't have to worry about weird deadlock situations as much. Again, I believe Rust actually does address this because of the single-owner-enforced-at-compile-time stuff, but I haven't used it enough to really draw a conclusion on it.
samatman 3y ago
> the JIT compile times made it unfeasible Worth trying again unless that was in the last 18 months. Precompilation has gotten dramatically faster, like 50x improvement in load times faster. So has startup time in general, `time julia -e "return 5"` gave 161 milliseconds of user time on my computer. I don't actually use Julia for simple script munging, simply because I'm used to Python in that role and it works fine, but it's suitable for it. Calculator-like use? Julia, all the way. Of course I keep the REPL open at all times. Gotta use tools the way they're intended, y'know?
- the__alchemist 3y ago
  I'll give it a shot! I've been tracking a steady stream of published improvements this area over the years (decade?) I think the most striking moment I had was when I realized through experiment that rust, which has a notoriously slow compiler, was faster to compile and run a small plotting program than Julia. That was ~5 years ago though!
  - samatman 3y ago
    Entirely different language when it comes to compile times, they were tremendously successful in the time-to-first-plot project. 1.6 was the release in 2021 that made the biggest difference, but with steady improvement since. The big push now is to add everything needed so that making object/dylib code in Julia of comparable size to the usual systems languages is possible. The team has a track record of finishing what they start, I'm optimistic that they'll get that right as well.

sampo 3y ago

> What is Julia's current niche? Python, but for engineers and scientists. Syntax is friendlier for doing mathematics, barrier to entry to first calculation script is lower a = [1 2 3] b = [4 5 6] a + b vs import numpy as np a = np.array([1, 2, 3]) b = np.array([4, 5, 6]) a + b Also you can learn Julia step by step, and improve your code to run faster. Whereas in Python, the eventual road to fast code is to write the library in C and call it from Python. Python is for software people. Matlab, Julia and R are for people who only write code because they want answers.

sampo 3y ago
> Python is for software people. Matlab, Julia and R are for people who only write code because they want answers. Or, Julia can be for both. Julia lets you start from scientific calculations scripts and grow your code to a larger software package. Same as Python grew popular 20 years ago because it let you grow your small sysadmin scripts to larger software. Matlab and R are too limited in their "real programming language" constructs, and it gets painful when your codebase grows larger. Julia has all "real programming language" constructs a software engineer would need.
the__alchemist 3y ago
This syntax is my favorite feature of Julia's. I wish I could get something like this in python or rust, without a macro mess.

staplung 3y ago

Scientific computing. Of the languages you cited, none (other than Julia) have dynamic dispatch, a feature that turns out to be unexpectedly powerful and allows lots of packages to hang together much more effectively than you would expect. Also, Rust specifically doesn't have built-in rational numbers, complex numbers, bigints or proper matrixes. There are lots of crates that implement them but that's precisely what prevents a cohesive ecosystem from developing.

abdullahkhalids 3y ago
I don't know anything about this, but the wiki article [1] says that Python, Rust and Julia all have Dynamic dispatch. However, it seems like Python has single dispatch, but Julia has multiple dispatch. Is that what you meant? [1] https://en.wikipedia.org/wiki/Dynamic_dispatch https://en.wikipedia.org/wiki/Dynamic_dispatch
- kruxigt 3y ago
  [dead]

jakobnissen 3y ago

I use it for high-performance scientific code. Code that needs to run fast, be written fast, and frequently extended and modified, with a mix of high and low level computing. This could be achieved by a mix of Python and Rust, but it's just too annoying to write scripts in two languages, where one is compiled and the other has terrible package management. Much easier to write it in Julia and have it be fast out of the box