6 ms·

Romeo and Julia, Where Romeo Is Basic Statistics

116 points by xgdgsc 3y ago

deleted 3y ago
[deleted]
naniwaduni 3y ago
With a title like that, one might've expected a case study in how the two aren't suited for one another, or something of the sort...
- mmahemoff 3y ago
  A Bayesian tragedy
  - perihelions 3y ago
    Shakespeare, Marlowe, et al.: "Retrospective analysis of false positives in medical diagnosis of death"
  - erikerikson 3y ago
    Frequently
- aredox 3y ago
  They are suited for each other, but their families are at war with each other
- dvfjsdhgfv 3y ago
  It basically depends if you view the story as an example of strong love or as a great drama that had to end badly.
a1o 3y ago
This title is great, kudos for the creativity. > So I guess it’s gonna be about a programming language named Julia and its usage for basic statistics. This writing is weird, you are the author right?
- Zhyl 3y ago
  It's not unheard of for programming books to affect a wacky style. See: http://poignant.guide/book/chapter-2.html http://poignant.guide/book/chapter-2.html https://learnyouahaskell.com/introduction https://learnyouahaskell.com/introduction
clircle 3y ago
Considering that the book ends by saying that the reader should validate Julia code by comparing it against R, i think i will continue to use R.
- bb1234 3y ago
  That is a good practice for any statistical software you are using. You should validate your R code with Python or Julia.
  - Fomite 3y ago
    Agreed. Whenever I'm using new statistical software, I validate it against one I know.
- affinepplan 3y ago
  I validate my C++ code by comparing against a Python implementation this is just a normal sanity check and is not a strike against Julia (or R)
- jakobnissen 3y ago
  Isn't it always a good idea to test your implementations against a known correct implementation? Like, if I were testing, say, a SHA implementation, I would also test against the results from an independent implementation. How is this an issue with Julia the language?
- lolinder 3y ago
  The way you frame it makes it sound like the author was saying Julia is a less trustworthy language, which isn't what they're saying at all. Here's the quote in context: > Still, if you are new to (Julia) programming and statistics then most likely you should calibrate your tools first. Before you run some statistical analysis you may want to try it out on an example from a textbook written by an expert (not me though) and see if you get the same (or at least comparable) result on your own. Although this is a sound approach, I suspect you are more prone to visit some statistical blog or internet forum and go with the examples that are contained there. One such option is rseek.org, i.e. a search engine for the R programming language. > ... Once I got both outputs that are similar enough I can be fairly sure I did right. Otherwise I should investigate where the differences come from and possibly make some necessary adjustments. It doesn't say to compare it against R, it says that if you are new to programming and statistics you should check your work against known-good answers or against a second implementation before rushing ahead, and it gives R as an example of a place to find code that you can use to check your work. This isn't advice about one language being better, it's the usual advice to solve the same problem in two different ways to make sure you got it right!
empyrrhicist 3y ago
Is anyone aware of a YouTube or tutorial series that goes deep into the graphics stacks for Julia? I can make some plots, but I feel like I'm missing the deeper understanding of what's going on that I have with R base graphics.
- krastanov 3y ago
  The presentations and workshops about Makie.jl from the last few JuliaCons are pretty good. All recordings from JuliaCon are available at the official Julia youtube channel.
- nextos 3y ago
  For simple 2D graphics, Gadfly.jl is pretty good and its backend is also quite elegant. See [1,2]. [1] https://gadflyjl.org/stable/dev/pipeline https://gadflyjl.org/stable/dev/pipeline [2] https://giovineitalia.github.io/Compose.jl https://giovineitalia.github.io/Compose.jl
- deleted 3y ago
  [deleted]
- kruxigt 3y ago
  [dead]
richrichie 3y ago
I used Julia quite a bit, even did a couple of professional projects. I quite like the all native code autodiff + ML set up. Since the LLM breakout on Huggingface, I am back in Python world.
stared 3y ago
I am curious: What is Julia's current niche? I mean, I tried it a few times a decade ago, but the environment was not mature, and performance improvements were not in the places I needed. Though, I guess, things moved from there. Python has the richest environment for data science. For statistics, there are some tools in R, such as the stunning ggplot2. For numerics-heavy arrays, we have PyTorch. For custom low-level numerics, Rust rocks.
- the__alchemist 3y ago
  ODE solving? Best in class. I tried to use it for quick scripts and calculator-like-use, but the JIT compile times made it unfeasible. (Julia's math syntax is the best I've seen) For more complex and performance-intensive programs, I don't see the case for choosing it over a general languages like Rust and C++.
  - tombert 3y ago
    > For more complex and performance-intensive programs, I don't see the case for choosing it over a general languages like Rust and C++. I mean, the fact that it's getting comparable speeds to C++ or Rust, while having full garbage collection and nice built-in concurrency support is a selling point to me. I'm not saying you couldn't write code faster in C++ or Rust, but I'm going to argue that Julia will often be just as fast or faster from "default implement".
    - CyberDildonics 3y ago
      The garbage collection will cause hiccups and hangs. It isn't faster if you are leaning on garbage collection. Just like any other garbage collected language, you want to avoid the garbage collection.
      - tombert 3y ago
        Before I say anything, I think your username is pretty funny. Props for snagging that one. > It isn't faster if you are leaning on garbage collection. Just like any other garbage collected language, you want to avoid the garbage collection. I agree in theory but not in practice. Absolutely, if you are good at properly freeing memory, and knowing when to free memory, and how much to free and all that fun stuff, you can definitely get much better performance than a GC, with fewer (or zero) pauses and better throughput. However, I don't think I agree that that is the average case. A lot of people will simply malloc and free at the beginning and ending of the function, or worse they will do things incorrectly and so the data won't actually free, or it frees before it's completed. A GC does incur some overhead, but they can also be extremely optimized and and have the advantage of doing things basically correctly, or at least safely. If you through multiple threads into the mix, then the problems get much harder. Julia's doesn't do this, but GC in languages like Go also basically don't cause hiccups or hangs, at least not long ones. The pause times are sub-millisecond [1], and it's pretty impressive. Even Java, the language that initially gave GC's a bad name, has the Shenandoah GC in some builds of OpenJDK which also gets sub-millisecond pauses [2]. Personally, I tend to believe that an application that stops working is generally "slower" than one that keeps working. A lot of the time savings you get from avoiding a GC will be eliminated if your program crashes and restarts. (That said, I will acknowledge that Rust probably makes this much better; I haven't done anything "real" with Rust so I cannot comment on it.) What I was trying to say was that Julia, at least in benchmarks, does get speeds getting close to something like C++, even with GC enabled, and it will certainly run circles around something like Python. If you want something that gets around at least 80+% of the speed of C++, while also giving all the prettiness and easiness of Python, I think Julia is a very good choice, and I would happily choose it over something like Go. [1] https://groups.google.com/g/golang-dev/c/Ab1sFeoZg_8 https://groups.google.com/g/golang-dev/c/Ab1sFeoZg_8 [2] https://wiki.openjdk.org/display/shenandoah/Main#Main-PerformanceGuidelinesandDiagnostics https://wiki.openjdk.org/display/shenandoah/Main#Main-Perfor...
tombert 3y ago
My dad recently had me help him port some Matlab code to a web service. He eventually ported it over to Octave so I could run it, since I don't have any kind of license for Matlab. I couldn't find any good web server implementations for Octave, so I was looking for free "matlab-like" languages that might, and I found Julia. The code wasn't stats-heavy, but it was heavily number-crunchy, making heavy use of matrices and tensors and stuff that admittedly still kind of goes over my head. I've not done a lot with "numeric" programming, but I was able to pick up Julia really quickly, and I was able to do a more-or-less straight port of my dad's code to Julia. What I was extremely impressed with was how much faster it went out of the box; for some of the more crunchy bits, running on the same hardware, the Julia code ran quite literally 150x faster, and this was without any Julia-specific optimizations other than adding types to the function arguments. Since then I've learned a lot more about Julia, to a point where it has sort of become my go-to language for starting new projects (assuming I don't need to use a specific JVM library or something). It's basically guaranteed to give me fast code out of the box, it has most of the concurrency constructs that I actually care about, it has a better-than-average macro system (at least compared to something like C++), and personally I just think that the language looks pretty once you get into some more of the details. I wish it were a more popular language, because the only thing that has held me back from using it more has been the lack of library support. Most of the libraries are fine, but for example I had a lot of trouble getting the Bcrypt library working, which is something I wouldn't really have a problem with on a JVM language or C++ or something. Of course, this is (hopefully) a temporary problem, as the language slowly grows in popularity the library support should just get better. ETA: I haven't done proper benchmarks on this so this is purely speculation, but that said I think that the biggest boost comes from the fact that Julia properly optimizes loops. My dad is a smart dude, but not being a software engineer in 2024, didn't really know that using loop constructs in most higher-level language is considered a code smell. I don't now about Octave, but I know in Python it's generally considered way better to try and avoid vanilla "loops" and try and convert looping things into some kind of vector operation. I think the JIT part of Julia is smart enough to optimize loops out of the box, and so my straight port of my dad's code just "magically" got faster, and I was extremely impressed, but I suspect that if my dad had utilized a lot of the vector operations of Octave and/or Matlab, the results wouldn't be nearly as impressive. I suspect Julia would still have performed better (if nothing else because I was able to add type information), but I suspect it wouldn't be the 100+X speed increase that I was getting.

116 points by xgdgsc 3y ago

deleted 3y ago

[deleted]

With a title like that, one might've expected a case study in how the two aren't suited for one another, or something of the sort...

mmahemoff 3y ago
A Bayesian tragedy
- perihelions 3y ago
  Shakespeare, Marlowe, et al.: "Retrospective analysis of false positives in medical diagnosis of death"
- erikerikson 3y ago
  Frequently
aredox 3y ago
They are suited for each other, but their families are at war with each other
dvfjsdhgfv 3y ago
It basically depends if you view the story as an example of strong love or as a great drama that had to end badly.

a1o 3y ago

This title is great, kudos for the creativity. > So I guess it’s gonna be about a programming language named Julia and its usage for basic statistics. This writing is weird, you are the author right?

Zhyl 3y ago
It's not unheard of for programming books to affect a wacky style. See: http://poignant.guide/book/chapter-2.html http://poignant.guide/book/chapter-2.html https://learnyouahaskell.com/introduction https://learnyouahaskell.com/introduction

clircle 3y ago

Considering that the book ends by saying that the reader should validate Julia code by comparing it against R, i think i will continue to use R.

bb1234 3y ago
That is a good practice for any statistical software you are using. You should validate your R code with Python or Julia.
- Fomite 3y ago
  Agreed. Whenever I'm using new statistical software, I validate it against one I know.
affinepplan 3y ago
I validate my C++ code by comparing against a Python implementation this is just a normal sanity check and is not a strike against Julia (or R)
jakobnissen 3y ago
Isn't it always a good idea to test your implementations against a known correct implementation? Like, if I were testing, say, a SHA implementation, I would also test against the results from an independent implementation. How is this an issue with Julia the language?
lolinder 3y ago
The way you frame it makes it sound like the author was saying Julia is a less trustworthy language, which isn't what they're saying at all. Here's the quote in context: > Still, if you are new to (Julia) programming and statistics then most likely you should calibrate your tools first. Before you run some statistical analysis you may want to try it out on an example from a textbook written by an expert (not me though) and see if you get the same (or at least comparable) result on your own. Although this is a sound approach, I suspect you are more prone to visit some statistical blog or internet forum and go with the examples that are contained there. One such option is rseek.org, i.e. a search engine for the R programming language. > ... Once I got both outputs that are similar enough I can be fairly sure I did right. Otherwise I should investigate where the differences come from and possibly make some necessary adjustments. It doesn't say to compare it against R, it says that if you are new to programming and statistics you should check your work against known-good answers or against a second implementation before rushing ahead, and it gives R as an example of a place to find code that you can use to check your work. This isn't advice about one language being better, it's the usual advice to solve the same problem in two different ways to make sure you got it right!

empyrrhicist 3y ago

Is anyone aware of a YouTube or tutorial series that goes deep into the graphics stacks for Julia? I can make some plots, but I feel like I'm missing the deeper understanding of what's going on that I have with R base graphics.

krastanov 3y ago
The presentations and workshops about Makie.jl from the last few JuliaCons are pretty good. All recordings from JuliaCon are available at the official Julia youtube channel.
nextos 3y ago
For simple 2D graphics, Gadfly.jl is pretty good and its backend is also quite elegant. See [1,2]. [1] https://gadflyjl.org/stable/dev/pipeline https://gadflyjl.org/stable/dev/pipeline [2] https://giovineitalia.github.io/Compose.jl https://giovineitalia.github.io/Compose.jl
deleted 3y ago
[deleted]
kruxigt 3y ago
[dead]

richrichie 3y ago

I used Julia quite a bit, even did a couple of professional projects. I quite like the all native code autodiff + ML set up. Since the LLM breakout on Huggingface, I am back in Python world.

stared 3y ago

I am curious: What is Julia's current niche? I mean, I tried it a few times a decade ago, but the environment was not mature, and performance improvements were not in the places I needed. Though, I guess, things moved from there. Python has the richest environment for data science. For statistics, there are some tools in R, such as the stunning ggplot2. For numerics-heavy arrays, we have PyTorch. For custom low-level numerics, Rust rocks.

the__alchemist 3y ago
ODE solving? Best in class. I tried to use it for quick scripts and calculator-like-use, but the JIT compile times made it unfeasible. (Julia's math syntax is the best I've seen) For more complex and performance-intensive programs, I don't see the case for choosing it over a general languages like Rust and C++.
- tombert 3y ago
  > For more complex and performance-intensive programs, I don't see the case for choosing it over a general languages like Rust and C++. I mean, the fact that it's getting comparable speeds to C++ or Rust, while having full garbage collection and nice built-in concurrency support is a selling point to me. I'm not saying you couldn't write code faster in C++ or Rust, but I'm going to argue that Julia will often be just as fast or faster from "default implement".
  - CyberDildonics 3y ago
    The garbage collection will cause hiccups and hangs. It isn't faster if you are leaning on garbage collection. Just like any other garbage collected language, you want to avoid the garbage collection.
    - tombert 3y ago
      Before I say anything, I think your username is pretty funny. Props for snagging that one. > It isn't faster if you are leaning on garbage collection. Just like any other garbage collected language, you want to avoid the garbage collection. I agree in theory but not in practice. Absolutely, if you are good at properly freeing memory, and knowing when to free memory, and how much to free and all that fun stuff, you can definitely get much better performance than a GC, with fewer (or zero) pauses and better throughput. However, I don't think I agree that that is the average case. A lot of people will simply malloc and free at the beginning and ending of the function, or worse they will do things incorrectly and so the data won't actually free, or it frees before it's completed. A GC does incur some overhead, but they can also be extremely optimized and and have the advantage of doing things basically correctly, or at least safely. If you through multiple threads into the mix, then the problems get much harder. Julia's doesn't do this, but GC in languages like Go also basically don't cause hiccups or hangs, at least not long ones. The pause times are sub-millisecond [1], and it's pretty impressive. Even Java, the language that initially gave GC's a bad name, has the Shenandoah GC in some builds of OpenJDK which also gets sub-millisecond pauses [2]. Personally, I tend to believe that an application that stops working is generally "slower" than one that keeps working. A lot of the time savings you get from avoiding a GC will be eliminated if your program crashes and restarts. (That said, I will acknowledge that Rust probably makes this much better; I haven't done anything "real" with Rust so I cannot comment on it.) What I was trying to say was that Julia, at least in benchmarks, does get speeds getting close to something like C++, even with GC enabled, and it will certainly run circles around something like Python. If you want something that gets around at least 80+% of the speed of C++, while also giving all the prettiness and easiness of Python, I think Julia is a very good choice, and I would happily choose it over something like Go. [1] https://groups.google.com/g/golang-dev/c/Ab1sFeoZg_8 https://groups.google.com/g/golang-dev/c/Ab1sFeoZg_8 [2] https://wiki.openjdk.org/display/shenandoah/Main#Main-PerformanceGuidelinesandDiagnostics https://wiki.openjdk.org/display/shenandoah/Main#Main-Perfor...

tombert 3y ago

My dad recently had me help him port some Matlab code to a web service. He eventually ported it over to Octave so I could run it, since I don't have any kind of license for Matlab. I couldn't find any good web server implementations for Octave, so I was looking for free "matlab-like" languages that might, and I found Julia. The code wasn't stats-heavy, but it was heavily number-crunchy, making heavy use of matrices and tensors and stuff that admittedly still kind of goes over my head. I've not done a lot with "numeric" programming, but I was able to pick up Julia really quickly, and I was able to do a more-or-less straight port of my dad's code to Julia. What I was extremely impressed with was how much faster it went out of the box; for some of the more crunchy bits, running on the same hardware, the Julia code ran quite literally 150x faster, and this was without any Julia-specific optimizations other than adding types to the function arguments. Since then I've learned a lot more about Julia, to a point where it has sort of become my go-to language for starting new projects (assuming I don't need to use a specific JVM library or something). It's basically guaranteed to give me fast code out of the box, it has most of the concurrency constructs that I actually care about, it has a better-than-average macro system (at least compared to something like C++), and personally I just think that the language looks pretty once you get into some more of the details. I wish it were a more popular language, because the only thing that has held me back from using it more has been the lack of library support. Most of the libraries are fine, but for example I had a lot of trouble getting the Bcrypt library working, which is something I wouldn't really have a problem with on a JVM language or C++ or something. Of course, this is (hopefully) a temporary problem, as the language slowly grows in popularity the library support should just get better. ETA: I haven't done proper benchmarks on this so this is purely speculation, but that said I think that the biggest boost comes from the fact that Julia properly optimizes loops. My dad is a smart dude, but not being a software engineer in 2024, didn't really know that using loop constructs in most higher-level language is considered a code smell. I don't now about Octave, but I know in Python it's generally considered way better to try and avoid vanilla "loops" and try and convert looping things into some kind of vector operation. I think the JIT part of Julia is smart enough to optimize loops out of the box, and so my straight port of my dad's code just "magically" got faster, and I was extremely impressed, but I suspect that if my dad had utilized a lot of the vector operations of Octave and/or Matlab, the results wouldn't be nearly as impressive. I suspect Julia would still have performed better (if nothing else because I was able to add type information), but I suspect it wouldn't be the 100+X speed increase that I was getting.