17 ms·

Julia 1.6: what has changed since Julia 1.0?

262 points by Sukera 6y ago

npr11 6y ago
Really nice to see so many little useability improvements -- like easy temporary envs, syntax highlighting in dependency conflict errors, and more partially-applied functions -- as well as more significant language development on threading, stack allocations, and reducing latency.
- oscardssmith 6y ago
  In a lot of ways, I feel like 1.0 was a backend LTS while 1.6 is more of a front end one. 1.0 had most of the basics nailed down, but it's taken a while for it to become as seemless as it now is.
klmadfejno 6y ago
> People often complain about the “Time To First Plot” (TTFP) in Julia. I personally have never minded it – by the time I am plotting something, I have done minutes of thinking so 20 seconds of compilation is nothing. This amuses me. I hadn't really considered the author's perspective, and now I think it aligns with my take on it pretty well.
- oblio 6y ago
  That won't stop these people. It's the same discussion as the interpreted languages vs compiled languages or the editor flame wars. On one hand you have people that say "thinking takes a lot longer than waiting a bit for compilation or actually editing source code" (I'm in this camp) and people that go "I don't want to wait for compilation and I want my editing to be hyper-efficient even if I have to invest hundreds and thousands of hours into it, so that I'm always in the zone/in the flow". People are just different but every camp thinks They're Right and The Others Are Dumb and Stupid And Dangerous. My personal guess is that besides the split in personalities/workflows, there's also a difference in projects. People who work on existing projects tend to read a ton more code/docs/team comms/architectural diagrams and edit/compile less so they care less about these issues. People who constantly create tons of mini-projects with short lifecycles care more about them.
  - simias 6y ago
    I don't know how Julia fares but personally what bothers me with long compile times is when I can't context switch, waiting for the compiler's output. What I mean in practice is that if you take Rust for instance, the compile times can be fairly long but the type checking occurs early on and is quite fast. Therefore once I know that this step succeeded I can usually let the compilation continue in the background while I focus my attention elsewhere. If on the other hand if I need to wait a lot longer to confirm that my code is actually valid I find myself just staring at the output window, not willing to let go of my short term memory until I get a confirmation that my code was accepted. The problem is not adding 20s to your overall dev time, it's to have a 20s interruption while you're "in the zone".
    - dm3 6y ago
      That's right. However, the 20s delay only happens once during the first plot action in the fresh REPL. That's when Julia compiles all of the functions not present in the system image. All of the repeated plotting will not require recompilation of the "base" plotting libraries.
    - DNF2 6y ago
      If it were 20 seconds for every plot, this would be a major problem for me, as I tend to make lots of plots. But it's only the first plot where this is an issue. Surely you're not in 'the zone' that soon? Seems to me like people are making a mountain out of a molehill.
      - jampekka 6y ago
        If you don't use REPL (and for many good reasons you shouldn't) or some other such horror, every plot is the first plot. And it's pain. And not just plots really. Doing anything in Julia is pain if you try to use it as a programming language instead of an app for buggy, unreproducible and misunderstood ad-hoc analyses. Seeing these answers makes me think Julia will never be fixed. I forecast Julia will be back in a niche within five years if they don't get their act together. And it's sad, because the alternatives are fundamentally broken. Julia isn't fundamentally broken, but the devs and the community seem to insist on superficial breakage.
- linspace 6y ago
  It is nevertheless im portant because a simple plot is a nice test if you are considering using a language for scientific computing. Criticism about plotting libraries quality is even more valid. I cannot believe that plotting an scatter plot of a few million points is so slow while AAA videogames render millions of pixels in real time. On the long term? I think Julia is a better language than Python, Matlab or R for several reasons like modularity, package management and performance. But these are things that require at least 10 hours of use (to say something) instead of 10 minutes. With so many languages promising enlightent and transcendence to a next power level you cannot expect people to make that kind of investment.
  - eigenspace 6y ago
    Makie.jl [1] does it's plotting on the GPU (like a video game engine), so can handle millions of datapoints just fine. Also, note that for people to whom plotting is really important, it's quite easy nowadays to just AOT compile your plotting library to your sysimage with PackageCompiler.jl [2] for instant plots. [1] https://github.com/JuliaPlots/Makie.jl https://github.com/JuliaPlots/Makie.jl [2] https://github.com/JuliaLang/PackageCompiler.jl https://github.com/JuliaLang/PackageCompiler.jl
  - The_rationalist 6y ago
    What does Julia bring over Python Poetry package manager ? https://github.com/python-poetry/poetry https://github.com/python-poetry/poetry What about modularity, concretely?
    - krastanov 6y ago
      Julia's model is based around multimethods, which enables pretty crazy levels of interoperability (imagine being able to use scipy and tensorflow together, with tensorflow's autograd just magically working on most of scipy's functions). More about it here https://m.youtube.com/watch?v=kc9HwsxE1OY https://m.youtube.com/watch?v=kc9HwsxE1OY
    - linspace 6y ago
      I don't use poetry but I will risk a comment. You will have to judge: At the language level Julia separates files from modules. Modules are just a language construct, another object. They can span several files, have several them in the same file or even declare them in the REPL. This last point is important for me since I can (re)evaluate code in the REPL without polluting it. Usually Python feels more interactive but this is a point where Julia wins at the REPL. Julia's package manager is just another library and you have support integrated for it inside the REPL. I'm not devops, I don't think I can judge the relative merits of Julia's package manager design relative to Python's but for me "it feels" better. You will have to judge for yourself reading the docs and this things about federated package management. I have not tried it but Julia allows to build a sysimage with all dependencies included. I know there are similar things to build standalone Python apps but last time I tried (long ago) it was a pain. Finally, Python depends a lot on external C libraries to achieve performance. This obviously complicates deployment and is a reason why I use Ubuntu: I have binary wheels for almost everything. Julia provides performance without external tools and packages are usually pure Julia code. I think Julia's is vastly superior concerning packaging but of course for 99.9% of people this is not enough to make a switch yet (me included, for the moment I use it for hobby projects). Regarding modularity is amazing and not obvious at first why Julia's performance increases modularity. The reason is that in Python since you need to use C/C++ for performance your data structures need to be shaped appropriately when they cross this interface. This rigidity propagates through your program and makes you build big frameworks. I have in mind for example PyTorch or Tensorflow. So you have Numpy arrays, PyTorch tensor... you have of course almost transparent conversion between them since numpy is a standard. But all of this is achieved because a behemoth like Facebook or Google are behind injecting money and manpower. It's pyramid building: some engineering but a lot of work. Even then you are stuck with arrays and contorting your program to vectorized operations. There are of course some dark spots for Julia but I have the feeling they will be solved. I don't consider myself a fan boy or an early adopter but I think it has a future. So I thought for Python 20 years ago :)
- vanderZwan 6y ago
  If that's what you're used to it's fine probably, you'll often have no choice but to adjust your workflow to it and make it work. But if you've ever experienced an environment that instantly shows you results and lets you get into a fast iterative loop to explore your data, it can be hard to give that up again. Source: my current and previous job were basically data viz programming jobs which were all about optimizing said iterative loop for scientists. Going from minute-long to sub-second rendering speeds is a game-changer for many. EDIT: having said that, this kind of reminds me of what the biggest difference between analog and digital photography is for me, namely whether or not you get instant feedback. I do remember from my art school days that in my experience film was a much better option for training the skill of observation and composition than digital, because it forces you to essentially picture the photograph before you take it. However, once you get somewhat decent at that... I'd switch to digital and reap all the benefits it has ;). The same logic might apply to learning how to plot your data.
  - stjohnswarts 6y ago
    I used to thing c++/rust was bad. Then I did some FPGA work and it was quite painful waiting for rebuilds :) . I even appreciated c++/rust build times a bit more after that. Waiting an hour before being able to test your code will definitely make you cognizant of dotting your i's and crossing your t's in new code. We had a design that filled almost 80% of the chip and had pretty strict timing (near the limit of the chip) and it took it quite a while for the fitter to meet the timing requirements.
  - cbkeller 6y ago
    Totally fair, though note that these slow times are only for _first_ plot - I leave my repl open all day, and all but the first plot are sub-second
    - vanderZwan 6y ago
      Ah, then I suppose it's just a good excuse to go fetch another cup of coffee or tea. I can imagine it being a bit irritating to some users though.
  - 6y ago
hpcjoe 6y ago
TIL of OhMyREPL. And tried it. Its a keeper.
- moelf 6y ago
  any chance you use fzf for history search too? ;)
  - hpcjoe 6y ago
    Now I must research what fzf is ... :D
tomrod 6y ago
I come back every so often to check out Julia again. I have hopes for it. Some questions still in mind since I reviewed previously (1) How is its database connectivity? (2) Is there something like python's `requests` lib? (3) Are the features mature enough that I don't anticipate major rewrites for code each year?
- Sukera 6y ago
  About (2), I've extensively used HTTP.jl and am pretty happy with it. I don't know about exact differences to e.g. requests, but I've found it sufficient for my uses. Regarding (3), there's a daily CI job called "PkgEval" (which also runs before a new release is made) checking for regressions of julia vs. all registered packages, seeing if any break. This identifies misuse of internal APIs (which are allowed to break under semver) and actually breaking changes (which are then either undone or the packages are fixed). Additionally, you can [compat] bound julia itself in the Project.toml of your code. Those two combined should mak sure you don't have to rewrite your code. All 1.x versions are backwards compatible after all.
- leephillips 6y ago
  I don’t know about (1) and (2), but since v. 1.0 there has been practically no need to rewrite. I believe there is a commitment to not introducing breaking changes, or at least a strong reluctance to do so. EDIT after actually looking at the article: the “no breaking changes” commitment is right at the top.
- oxinabox 6y ago
  > (1) How is its database connectivity? Databases are not really my thing but: From what I hear: It is ok. Not amazing. But decent. LibPQ.jl is very mature (we run it in production). MySQL.jl exists, I hear about SQLite.jl being used pretty often. I know people use ODBC.jl, and JDBC.jl, though only because they complaint about things. I suspect there are a fair few people using them without complain that i never hear from. Though I haven't heard any mention really of JDBC in a while. While there is nothing like SQLAlchemy, a nice thing about DataBases in julia is they all conform to Tables.jl tables. So very easy to take your DataFrame library of choice (or CSV reader, or Arrow.jl or a dozen other formats), and use that is the input or output from a database query.
swagonomixxx 6y ago
As someone who knows very little but wants to learn: what's the best way to get up and running with Julia? On the website, they link a lot of videos, but I prefer textual formats. Is there something like the Rust Book for Julia?
- nextos 6y ago
  The official manual is very good, and reads more or less like a textbook: https://docs.julialang.org/en/v1/ https://docs.julialang.org/en/v1/
  - cbkeller 6y ago
    Yeah, this is probably actually the closest equivalent to the Rust Book, but I’ll also second the suggestion from the other comment of “Think Julia” (beginner) and “Design Patterns and Best Practices with Julia” (intermediate /advanced). For me, the most important thing to grasp when coming from another language was that Julia’s multiple dispatch brings with it effectively a whole paradigm of “dispatch-centric programming” that you have to embrace to really get the most out of Julia, including the c-like speed (have to strictly avoid type-instability for that) and the composability that everyone talks about.
- thetwentyone 6y ago
  - [JuliaLang.org](https://julialang.org/ https://julialang.org/), the home site with the downloads to get started, and links to learning resources. - [JuliaHub](https://juliahub.com/ui/Home https://juliahub.com/ui/Home) indexes open-source Julia packages and makes the entire ecosystem and documentation searchable from one place. - [JuliaAcademy](https://juliaacademy.com/courses https://juliaacademy.com/courses), which has free short courses in Data Science, Introduction to Julia, DataFrames.jl, Machine Learning, and more. - [Data Science Tutorials](https://alan-turing-institute.github.io/DataScienceTutorials.jl/ https://alan-turing-institute.github.io/DataScienceTutorials...) from the Alan Turing Institute. - [Learn Julia in Y minutes](https://learnxinyminutes.com/docs/julia/ https://learnxinyminutes.com/docs/julia/), a great quick-start if you are already comfortable with coding. - [Think Julia](https://benlauwens.github.io/ThinkJulia.jl/latest/book.html https://benlauwens.github.io/ThinkJulia.jl/latest/book.html), a free e-book (or paid print edition) book which introduces programming from the start and teaches you valuable ways of thinking. - [Design Patterns and Best Practices](https://www.packtpub.com/application-development/hands-design-patterns-julia-10 https://www.packtpub.com/application-development/hands-desig...), a book that will help you as you transition from smaller, one-off scripts to designing larger packages and projects. - Lots more topical books (Statistics, Optimization, etc) if looking for a Julia-oriented subject matter
saiojd 6y ago
I've tried Julia and really liked it, but the user experience was pretty bad. The language really needs faster interactivity or a strong type checker. I found myself waiting after than compiler a lot more than in some AOT compiled language like Rust... I fear the language suffers from being overused by people who are familiar with Matlab and Python and draws too much inspiration from them, much like how Rust draws too much inspiration from C++.
- cambalache 6y ago
  What boggles my mind is that a language oriented to scientific programming has such a lousy time to first plot. I know it has been improving, it is still not acceptable. Not for me as a user, not acceptable for a language who wants to become mainstream.
  - oscardssmith 6y ago
    It's a better time to first plot than matlab, which is one of the other major contenders. On my computer it is about 3 seconds, which is noticable, but far from disqualifing.
    - systemvoltage 6y ago
      Julia oversells itself as a general purpose language which I find absolutely out of line. Their marketing needs to be a lot more humble until they figure out the kinks. Also, my guess would be Python and not Matlab as it’s main competitors.
      - eigenspace 6y ago
        In what way is marketing julia as a general purpose programming language 'way out of line'? People use julia to make webservers, write programming languages, create plotting libraries, do scientific analysis, do compiler research, make video games, do HPC, etc. Julia has a design that's indeed strongly informed by scientific computing, but in order to actually meet the needs of the various people using it for scientific and technical purposes, it ended up needing to become a flexible enough language to be useful for anything.
kitsune_ 6y ago
It takes 180s to import Statistics, PlutoUI, Images and OffsetArrays in Pluto on this Macbook Pro with 32GB that is a couple of years old, while the fans are going crazy. If I'm unlucky a worker process will seg fault. All in all the interactive / REPL part (even with stuff like Revise) is kind of a let down thanks to the slow precompilation / lack of caching? I like it, but I really don't see how it is ever going to replace Matlab, R or Python if these issues aren't addressed.
- eigenspace 6y ago
  I assume that's including precompilation, otherwise something is seriously wrong with your computer. Precompilation can take a while, but it's cached between sessions, so it should be much faster the second time. Also, in version 1.6, we have multithreaded precompilation which happens at installation time instead of when you first try to load the package. This makes it much faster.
  - ku-man 6y ago
    Julia people say "in version 1.6... precompilation... is much faster". But they said the same when version 1.2 was released, same when 1.3, same with 1.4, and with 1.5, do you see a trend here?. It's clear the precompilation issue was never solved, otherwise it wouldn't be "fixed" in every single release.
- Acur 6y ago
  Just checked. On my slightly dated laptop the import of these packages in Pluto takes 5 seconds with Julia 1.6. In general 1.6 is a huge improvements in compilation times and usability for me.
- pkofod 6y ago
  What does "32GB" have to do with it? Why would a worker process segfault in this scenario?
the__alchemist 6y ago
> Plotting, it turns out, is basically a really hard thing for a compiler. It is many, many, small methods, most of which are only called once. And unlike most Julia code, it doesn’t actually benefit all that much from Julia’s JIT. Julia’s JIT is normally specializing code, and running a ton of optimizations. But plotting itself isn’t in the hot-loop – optimizing the code takes longer than running it the few dozen times it might be used unoptimized. To make a long-story short, plotting is the poster child example for Julia needing to compile things before it can run them. This is misleading. There's no reason you need to to compile the plotting library every time you load a REPL or program. Python, a "slow" interpreted language handles this quickly, as does Rust, a slow-to-compile Lang - You can compile and run a Rust program that plots more quickly than in Julia, since it doesn't need to compile the plotting lib after it's initially installed.
- oxinabox 6y ago
  > You don't need to compile the plotting library every time. You don't need to but Julia does. Its definately a issue, and I know it is being worked on. Julia doesn't store compiled binary code between sessions. Unless you compile it into a sysimage. There are apparently reasons why caching compiled binary like this is complicated in Julia. But once that is solved, wow things are going to be nice.
  - CyberDildonics 6y ago
    > There are apparently reasons why caching compiled binary like this is complicated in Julia. But once that is solved, wow things are going to be nice. I put a lot of hours into julia five years ago and all the same things were being said. I don't know why the compilation is so slow or why the caching is so bad, but it was the main complaint then and still is. The solutions are all 'just around the corner'. It reminds me of java two decades ago. Actually most languages that get a lot of use seem to go through this. The big problems for some reason have solutions "just around the corner" but they remain giant problems. I think what really happens is that people work on what they want. Solving their hard problems is not fun and no one holds anyone's feet to the flames. C++ has had problems with compile time and template errors, but there has been real commercial pressure to making progress on those. Julia's problems are the same as they were half a decade ago. Start working and wait an enormous amount of time for the exact things to compile that you compiled yesterday when you started it up.
baldfat 6y ago
I loved the idea of Julia but R and specifically the tiddyverse https://www.tidyverse.org/ https://www.tidyverse.org/ Just makes everything else seem not as elegant to my humble eyes.
- phillc73 6y ago
  Have you tried Query.jl or DataFramesMeta.jl?
  - baldfat 6y ago
    I don't like working with DATA TABLES UNLESS it is a HUGE data frames. Then if it is huge I'll go towards sparks. I normally am working with under a million objects which with today's computers is not that big. Edit I meant to say that DATA TABLES library in R reminds me more of Query.jl then tiddyverse
    - wodenokoto 6y ago
      What are you doing in the tidyverse that is not related to dataframes (or tibbles as the subclass of data.frame, that tidyverse uses is called)?
    - phillc73 6y ago
      Query.jl supports two different paradigms, one inspired directly by LINQ, the other by dplyr.[1] I actually prefer data.table in R, over dplyr, and Query.jl is really quite different. [1] http://www.queryverse.org/Query.jl/stable/ http://www.queryverse.org/Query.jl/stable/
  - cwyers 6y ago
    Very much not the parent, but as a heavy R user, I don't think either of them quite nail the way dplyr and the tidyverse work. The thing about dplyr is... it's just functions. Okay, so, it's functions that leverage features R has (notably lazy evaluation and non-standard evaluation). But it's just functions. All you need is a function that takes a data frame and returns a data frame. So you can take a function out of the R standard library, you can take a function from a package written before dplyr came around, you can take a function from a recent non-tidyverse package, you can write your own function... it's all just functions. In DataFramesMeta.jl, though, you have a macro, and everything runs inside that macro. So if you want to take something that isn't a part of DataFramesMeta.jl... here's an example. Let's say you want to take the popular mtcars dataset, and get the five cars with the best gas milage. In dplyr, that goes mtcars %>% arrange(mpg) %>% head(5) arrange is a function from the dplyr package, head is a function from the standard library, but they both work seamlessly together. DataFramesMeta.jl lets you work in a pipe-forward fashion, but (at last I knew, at least, it's been a while since I played with it), you couldn't use the Julia head function within a DataFramesMeta.jl pipeline. You have to do your data transformations, assign to a variable, and then get the head of that variable. Which, okay, probably doesn't sound like a big deal. But I think it gets at the heart of what efforts to do something Tidyverse-like in other languages (Python and Julia, mostly) really miss. The key value proposition of the Tidyverse in R is that it is very composable and very extensible. That means, if you are trying to solve something in a Tidyverse way, you can probably find something that works for you. If you are doing financial analysis? Get tidyquant. If you're doing time series analysis, the tidyverts packages are for you. And it all works because there is so little friction involved in writing your own functions that extend the functionality of Tidyverse packages. Yes, dplyr is a useful querying DSL in its own right, but you can find a bunch of SQLish query languages, and they're all some degree of fine. Query.jl or DataFramesMeta.jl might expose a useful querying DSL for data frames, but they don't seem to me to be built to support building a whole ecosystem like dplyr and the Tidyverse are.
295310e0 6y ago
Can someone comment on the ease of distributing Julia code? Can I easily take a bit of code I write (w/o external libraries) and produce a single binary I can ship to someone or do they require a full Julia environment to run it?
- jakobnissen 6y ago
  They require the full Julia environment to run it - and it's a heavy environment. It is possible to compile a binary that includes the environment and the compiler, but IIRC, that will result in a >400 MB hello-world script taking 150 MB of RAM to run. The core devs have mentioned they are going to add the capacity to compile to actual static binaries, but that does not seem to be a top priority, so I wouldn't hold my breadth waiting for it.
  - adgjlsfhk1 6y ago
    Note that this feature doesn't necessary require work by the core devs. It is completely feasible (at least in theory) to write a library that can output static exectuables.
    - ChrisRackauckas 6y ago
      It's actually done all of the time. GPUCompiler.jl, the core of the CUDA and AMD GPU stack, builds static binaries (compiled to .ptx by LLVM for CUDA for example, but the choice is just a switch), then stashes those binaries to use with a ccall in a Julia function. You could in theory use that stack to statically-compile anything that's GPU compliable, and it's really well-tested.
- eigenspace 6y ago
  Doing this without the user having Julia installed would require using PackageCompiler.jl to make a standalone executable. https://julialang.github.io/PackageCompiler.jl/dev/apps/ https://julialang.github.io/PackageCompiler.jl/dev/apps/ This is a pretty stable well established process now, but the binaries it produces are huge because they actually have the full Julia runtime in them. Active work is happening on small binary static compilation. We already do it for GPUs, we just have to repurpose our GPU AOT compilation pipeline for the CPU. There are proofs of concept that currently work, but something more polished is feeling like it’ll probably be another year or so.
celrod 6y ago
> (* Technically not all mutable objects live on the heap, because some never live at all, as they are optimized away so are never allocated in the first place.) The compiler will often stack allocate mutable objects in Julia. This is not the same as "never existed in the first place", because the stack pointer gets incremented and underlying data layout is you load from it is the same as that of the mutable object you allocated. Here is one example where that's very obviously what's happening: https://discourse.julialang.org/t/why-is-svector-faster-than-mvector/55174/10?u=elrod https://discourse.julialang.org/t/why-is-svector-faster-than... But it can happen now generally with mutable structures that can't practically just live in registers.
- oxinabox 6y ago
  fixed, to be more limitted in the claim
  - celrod 6y ago
    Cool, and fantastic summary! I enjoyed reading it.
- calebwinston 6y ago
  If they are optimized away, are their destructors/finalizers still called? I'm really hoping the answer is yes...
  - StefanKarpinski 6y ago
    Yes, that is guaranteed.
aliceryhl 6y ago
I really wish the language doesn't force me to use the REPL.
- moelf 6y ago
  most of the users use an IDE or a notebook, no need to only use REPL. If you're a old school editor -> terminal run kind of person, checkout https://github.com/dmolina/DaemonMode.jl https://github.com/dmolina/DaemonMode.jl
  - aliceryhl 6y ago
    Thanks, I'll have to check this out. I really wish these things were more discoverable. It took me a month of using Julia until I figured out that the compile-times were even avoidable on the REPL by using Revise.jl.
- krastanov 6y ago
  How is it forcing you!?
  - ecnahc515 6y ago
    I'm assuming you may not be familiar with Julia, but basically every "tool" you use in Julia is a library. It requires importing it in the Julia REPL or activating a new contextual REPL within the Julia REPL. You can then start calling the functions from the imported module. You don't really get standalone Julia tools, it's all libraries and you're always starting the REPL, importing something, then doing stuff from inside the REPL. For example, installing a package in Julia is the following steps: julia ] # keybinding which activates pkg mode add StaticArrays You could do it also like this: julia -e 'use Pkg; Pkg.add("StaticArrays")' There is no `julia-pkg add StaticArrays`, unfortunately.
    - gugagore 6y ago
      Why do you want `julia-pkg add StaticArrays` when you can do `julia -e 'use Pkg; Pkg.add("StaticArrays")'`, though?

262 points by Sukera 6y ago

Really nice to see so many little useability improvements -- like easy temporary envs, syntax highlighting in dependency conflict errors, and more partially-applied functions -- as well as more significant language development on threading, stack allocations, and reducing latency.

oscardssmith 6y ago
In a lot of ways, I feel like 1.0 was a backend LTS while 1.6 is more of a front end one. 1.0 had most of the basics nailed down, but it's taken a while for it to become as seemless as it now is.

klmadfejno 6y ago

> People often complain about the “Time To First Plot” (TTFP) in Julia. I personally have never minded it – by the time I am plotting something, I have done minutes of thinking so 20 seconds of compilation is nothing. This amuses me. I hadn't really considered the author's perspective, and now I think it aligns with my take on it pretty well.

oblio 6y ago
That won't stop these people. It's the same discussion as the interpreted languages vs compiled languages or the editor flame wars. On one hand you have people that say "thinking takes a lot longer than waiting a bit for compilation or actually editing source code" (I'm in this camp) and people that go "I don't want to wait for compilation and I want my editing to be hyper-efficient even if I have to invest hundreds and thousands of hours into it, so that I'm always in the zone/in the flow". People are just different but every camp thinks They're Right and The Others Are Dumb and Stupid And Dangerous. My personal guess is that besides the split in personalities/workflows, there's also a difference in projects. People who work on existing projects tend to read a ton more code/docs/team comms/architectural diagrams and edit/compile less so they care less about these issues. People who constantly create tons of mini-projects with short lifecycles care more about them.
- simias 6y ago
  I don't know how Julia fares but personally what bothers me with long compile times is when I can't context switch, waiting for the compiler's output. What I mean in practice is that if you take Rust for instance, the compile times can be fairly long but the type checking occurs early on and is quite fast. Therefore once I know that this step succeeded I can usually let the compilation continue in the background while I focus my attention elsewhere. If on the other hand if I need to wait a lot longer to confirm that my code is actually valid I find myself just staring at the output window, not willing to let go of my short term memory until I get a confirmation that my code was accepted. The problem is not adding 20s to your overall dev time, it's to have a 20s interruption while you're "in the zone".
  - dm3 6y ago
    That's right. However, the 20s delay only happens once during the first plot action in the fresh REPL. That's when Julia compiles all of the functions not present in the system image. All of the repeated plotting will not require recompilation of the "base" plotting libraries.
  - DNF2 6y ago
    If it were 20 seconds for every plot, this would be a major problem for me, as I tend to make lots of plots. But it's only the first plot where this is an issue. Surely you're not in 'the zone' that soon? Seems to me like people are making a mountain out of a molehill.
    - jampekka 6y ago
      If you don't use REPL (and for many good reasons you shouldn't) or some other such horror, every plot is the first plot. And it's pain. And not just plots really. Doing anything in Julia is pain if you try to use it as a programming language instead of an app for buggy, unreproducible and misunderstood ad-hoc analyses. Seeing these answers makes me think Julia will never be fixed. I forecast Julia will be back in a niche within five years if they don't get their act together. And it's sad, because the alternatives are fundamentally broken. Julia isn't fundamentally broken, but the devs and the community seem to insist on superficial breakage.
linspace 6y ago
It is nevertheless im portant because a simple plot is a nice test if you are considering using a language for scientific computing. Criticism about plotting libraries quality is even more valid. I cannot believe that plotting an scatter plot of a few million points is so slow while AAA videogames render millions of pixels in real time. On the long term? I think Julia is a better language than Python, Matlab or R for several reasons like modularity, package management and performance. But these are things that require at least 10 hours of use (to say something) instead of 10 minutes. With so many languages promising enlightent and transcendence to a next power level you cannot expect people to make that kind of investment.
- eigenspace 6y ago
  Makie.jl [1] does it's plotting on the GPU (like a video game engine), so can handle millions of datapoints just fine. Also, note that for people to whom plotting is really important, it's quite easy nowadays to just AOT compile your plotting library to your sysimage with PackageCompiler.jl [2] for instant plots. [1] https://github.com/JuliaPlots/Makie.jl https://github.com/JuliaPlots/Makie.jl [2] https://github.com/JuliaLang/PackageCompiler.jl https://github.com/JuliaLang/PackageCompiler.jl
- The_rationalist 6y ago
  What does Julia bring over Python Poetry package manager ? https://github.com/python-poetry/poetry https://github.com/python-poetry/poetry What about modularity, concretely?
  - krastanov 6y ago
    Julia's model is based around multimethods, which enables pretty crazy levels of interoperability (imagine being able to use scipy and tensorflow together, with tensorflow's autograd just magically working on most of scipy's functions). More about it here https://m.youtube.com/watch?v=kc9HwsxE1OY https://m.youtube.com/watch?v=kc9HwsxE1OY
  - linspace 6y ago
    I don't use poetry but I will risk a comment. You will have to judge: At the language level Julia separates files from modules. Modules are just a language construct, another object. They can span several files, have several them in the same file or even declare them in the REPL. This last point is important for me since I can (re)evaluate code in the REPL without polluting it. Usually Python feels more interactive but this is a point where Julia wins at the REPL. Julia's package manager is just another library and you have support integrated for it inside the REPL. I'm not devops, I don't think I can judge the relative merits of Julia's package manager design relative to Python's but for me "it feels" better. You will have to judge for yourself reading the docs and this things about federated package management. I have not tried it but Julia allows to build a sysimage with all dependencies included. I know there are similar things to build standalone Python apps but last time I tried (long ago) it was a pain. Finally, Python depends a lot on external C libraries to achieve performance. This obviously complicates deployment and is a reason why I use Ubuntu: I have binary wheels for almost everything. Julia provides performance without external tools and packages are usually pure Julia code. I think Julia's is vastly superior concerning packaging but of course for 99.9% of people this is not enough to make a switch yet (me included, for the moment I use it for hobby projects). Regarding modularity is amazing and not obvious at first why Julia's performance increases modularity. The reason is that in Python since you need to use C/C++ for performance your data structures need to be shaped appropriately when they cross this interface. This rigidity propagates through your program and makes you build big frameworks. I have in mind for example PyTorch or Tensorflow. So you have Numpy arrays, PyTorch tensor... you have of course almost transparent conversion between them since numpy is a standard. But all of this is achieved because a behemoth like Facebook or Google are behind injecting money and manpower. It's pyramid building: some engineering but a lot of work. Even then you are stuck with arrays and contorting your program to vectorized operations. There are of course some dark spots for Julia but I have the feeling they will be solved. I don't consider myself a fan boy or an early adopter but I think it has a future. So I thought for Python 20 years ago :)
vanderZwan 6y ago
If that's what you're used to it's fine probably, you'll often have no choice but to adjust your workflow to it and make it work. But if you've ever experienced an environment that instantly shows you results and lets you get into a fast iterative loop to explore your data, it can be hard to give that up again. Source: my current and previous job were basically data viz programming jobs which were all about optimizing said iterative loop for scientists. Going from minute-long to sub-second rendering speeds is a game-changer for many. EDIT: having said that, this kind of reminds me of what the biggest difference between analog and digital photography is for me, namely whether or not you get instant feedback. I do remember from my art school days that in my experience film was a much better option for training the skill of observation and composition than digital, because it forces you to essentially picture the photograph before you take it. However, once you get somewhat decent at that... I'd switch to digital and reap all the benefits it has ;). The same logic might apply to learning how to plot your data.
- stjohnswarts 6y ago
  I used to thing c++/rust was bad. Then I did some FPGA work and it was quite painful waiting for rebuilds :) . I even appreciated c++/rust build times a bit more after that. Waiting an hour before being able to test your code will definitely make you cognizant of dotting your i's and crossing your t's in new code. We had a design that filled almost 80% of the chip and had pretty strict timing (near the limit of the chip) and it took it quite a while for the fitter to meet the timing requirements.
- cbkeller 6y ago
  Totally fair, though note that these slow times are only for _first_ plot - I leave my repl open all day, and all but the first plot are sub-second
  - vanderZwan 6y ago
    Ah, then I suppose it's just a good excuse to go fetch another cup of coffee or tea. I can imagine it being a bit irritating to some users though.
- 6y ago

hpcjoe 6y ago

TIL of OhMyREPL. And tried it. Its a keeper.

moelf 6y ago
any chance you use fzf for history search too? ;)
- hpcjoe 6y ago
  Now I must research what fzf is ... :D

tomrod 6y ago

I come back every so often to check out Julia again. I have hopes for it. Some questions still in mind since I reviewed previously (1) How is its database connectivity? (2) Is there something like python's `requests` lib? (3) Are the features mature enough that I don't anticipate major rewrites for code each year?

Sukera 6y ago
About (2), I've extensively used HTTP.jl and am pretty happy with it. I don't know about exact differences to e.g. requests, but I've found it sufficient for my uses. Regarding (3), there's a daily CI job called "PkgEval" (which also runs before a new release is made) checking for regressions of julia vs. all registered packages, seeing if any break. This identifies misuse of internal APIs (which are allowed to break under semver) and actually breaking changes (which are then either undone or the packages are fixed). Additionally, you can [compat] bound julia itself in the Project.toml of your code. Those two combined should mak sure you don't have to rewrite your code. All 1.x versions are backwards compatible after all.
leephillips 6y ago
I don’t know about (1) and (2), but since v. 1.0 there has been practically no need to rewrite. I believe there is a commitment to not introducing breaking changes, or at least a strong reluctance to do so. EDIT after actually looking at the article: the “no breaking changes” commitment is right at the top.
oxinabox 6y ago
> (1) How is its database connectivity? Databases are not really my thing but: From what I hear: It is ok. Not amazing. But decent. LibPQ.jl is very mature (we run it in production). MySQL.jl exists, I hear about SQLite.jl being used pretty often. I know people use ODBC.jl, and JDBC.jl, though only because they complaint about things. I suspect there are a fair few people using them without complain that i never hear from. Though I haven't heard any mention really of JDBC in a while. While there is nothing like SQLAlchemy, a nice thing about DataBases in julia is they all conform to Tables.jl tables. So very easy to take your DataFrame library of choice (or CSV reader, or Arrow.jl or a dozen other formats), and use that is the input or output from a database query.

swagonomixxx 6y ago

As someone who knows very little but wants to learn: what's the best way to get up and running with Julia? On the website, they link a lot of videos, but I prefer textual formats. Is there something like the Rust Book for Julia?

nextos 6y ago
The official manual is very good, and reads more or less like a textbook: https://docs.julialang.org/en/v1/ https://docs.julialang.org/en/v1/
- cbkeller 6y ago
  Yeah, this is probably actually the closest equivalent to the Rust Book, but I’ll also second the suggestion from the other comment of “Think Julia” (beginner) and “Design Patterns and Best Practices with Julia” (intermediate /advanced). For me, the most important thing to grasp when coming from another language was that Julia’s multiple dispatch brings with it effectively a whole paradigm of “dispatch-centric programming” that you have to embrace to really get the most out of Julia, including the c-like speed (have to strictly avoid type-instability for that) and the composability that everyone talks about.
thetwentyone 6y ago
- [JuliaLang.org](https://julialang.org/ https://julialang.org/), the home site with the downloads to get started, and links to learning resources. - [JuliaHub](https://juliahub.com/ui/Home https://juliahub.com/ui/Home) indexes open-source Julia packages and makes the entire ecosystem and documentation searchable from one place. - [JuliaAcademy](https://juliaacademy.com/courses https://juliaacademy.com/courses), which has free short courses in Data Science, Introduction to Julia, DataFrames.jl, Machine Learning, and more. - [Data Science Tutorials](https://alan-turing-institute.github.io/DataScienceTutorials.jl/ https://alan-turing-institute.github.io/DataScienceTutorials...) from the Alan Turing Institute. - [Learn Julia in Y minutes](https://learnxinyminutes.com/docs/julia/ https://learnxinyminutes.com/docs/julia/), a great quick-start if you are already comfortable with coding. - [Think Julia](https://benlauwens.github.io/ThinkJulia.jl/latest/book.html https://benlauwens.github.io/ThinkJulia.jl/latest/book.html), a free e-book (or paid print edition) book which introduces programming from the start and teaches you valuable ways of thinking. - [Design Patterns and Best Practices](https://www.packtpub.com/application-development/hands-design-patterns-julia-10 https://www.packtpub.com/application-development/hands-desig...), a book that will help you as you transition from smaller, one-off scripts to designing larger packages and projects. - Lots more topical books (Statistics, Optimization, etc) if looking for a Julia-oriented subject matter

saiojd 6y ago

I've tried Julia and really liked it, but the user experience was pretty bad. The language really needs faster interactivity or a strong type checker. I found myself waiting after than compiler a lot more than in some AOT compiled language like Rust... I fear the language suffers from being overused by people who are familiar with Matlab and Python and draws too much inspiration from them, much like how Rust draws too much inspiration from C++.

cambalache 6y ago
What boggles my mind is that a language oriented to scientific programming has such a lousy time to first plot. I know it has been improving, it is still not acceptable. Not for me as a user, not acceptable for a language who wants to become mainstream.
- oscardssmith 6y ago
  It's a better time to first plot than matlab, which is one of the other major contenders. On my computer it is about 3 seconds, which is noticable, but far from disqualifing.
  - systemvoltage 6y ago
    Julia oversells itself as a general purpose language which I find absolutely out of line. Their marketing needs to be a lot more humble until they figure out the kinks. Also, my guess would be Python and not Matlab as it’s main competitors.
    - eigenspace 6y ago
      In what way is marketing julia as a general purpose programming language 'way out of line'? People use julia to make webservers, write programming languages, create plotting libraries, do scientific analysis, do compiler research, make video games, do HPC, etc. Julia has a design that's indeed strongly informed by scientific computing, but in order to actually meet the needs of the various people using it for scientific and technical purposes, it ended up needing to become a flexible enough language to be useful for anything.

kitsune_ 6y ago

It takes 180s to import Statistics, PlutoUI, Images and OffsetArrays in Pluto on this Macbook Pro with 32GB that is a couple of years old, while the fans are going crazy. If I'm unlucky a worker process will seg fault. All in all the interactive / REPL part (even with stuff like Revise) is kind of a let down thanks to the slow precompilation / lack of caching? I like it, but I really don't see how it is ever going to replace Matlab, R or Python if these issues aren't addressed.

eigenspace 6y ago
I assume that's including precompilation, otherwise something is seriously wrong with your computer. Precompilation can take a while, but it's cached between sessions, so it should be much faster the second time. Also, in version 1.6, we have multithreaded precompilation which happens at installation time instead of when you first try to load the package. This makes it much faster.
- ku-man 6y ago
  Julia people say "in version 1.6... precompilation... is much faster". But they said the same when version 1.2 was released, same when 1.3, same with 1.4, and with 1.5, do you see a trend here?. It's clear the precompilation issue was never solved, otherwise it wouldn't be "fixed" in every single release.
Acur 6y ago
Just checked. On my slightly dated laptop the import of these packages in Pluto takes 5 seconds with Julia 1.6. In general 1.6 is a huge improvements in compilation times and usability for me.
pkofod 6y ago
What does "32GB" have to do with it? Why would a worker process segfault in this scenario?

the__alchemist 6y ago

> Plotting, it turns out, is basically a really hard thing for a compiler. It is many, many, small methods, most of which are only called once. And unlike most Julia code, it doesn’t actually benefit all that much from Julia’s JIT. Julia’s JIT is normally specializing code, and running a ton of optimizations. But plotting itself isn’t in the hot-loop – optimizing the code takes longer than running it the few dozen times it might be used unoptimized. To make a long-story short, plotting is the poster child example for Julia needing to compile things before it can run them. This is misleading. There's no reason you need to to compile the plotting library every time you load a REPL or program. Python, a "slow" interpreted language handles this quickly, as does Rust, a slow-to-compile Lang - You can compile and run a Rust program that plots more quickly than in Julia, since it doesn't need to compile the plotting lib after it's initially installed.

oxinabox 6y ago
> You don't need to compile the plotting library every time. You don't need to but Julia does. Its definately a issue, and I know it is being worked on. Julia doesn't store compiled binary code between sessions. Unless you compile it into a sysimage. There are apparently reasons why caching compiled binary like this is complicated in Julia. But once that is solved, wow things are going to be nice.
- CyberDildonics 6y ago
  > There are apparently reasons why caching compiled binary like this is complicated in Julia. But once that is solved, wow things are going to be nice. I put a lot of hours into julia five years ago and all the same things were being said. I don't know why the compilation is so slow or why the caching is so bad, but it was the main complaint then and still is. The solutions are all 'just around the corner'. It reminds me of java two decades ago. Actually most languages that get a lot of use seem to go through this. The big problems for some reason have solutions "just around the corner" but they remain giant problems. I think what really happens is that people work on what they want. Solving their hard problems is not fun and no one holds anyone's feet to the flames. C++ has had problems with compile time and template errors, but there has been real commercial pressure to making progress on those. Julia's problems are the same as they were half a decade ago. Start working and wait an enormous amount of time for the exact things to compile that you compiled yesterday when you started it up.

baldfat 6y ago

I loved the idea of Julia but R and specifically the tiddyverse https://www.tidyverse.org/ https://www.tidyverse.org/ Just makes everything else seem not as elegant to my humble eyes.

phillc73 6y ago
Have you tried Query.jl or DataFramesMeta.jl?
- baldfat 6y ago
  I don't like working with DATA TABLES UNLESS it is a HUGE data frames. Then if it is huge I'll go towards sparks. I normally am working with under a million objects which with today's computers is not that big. Edit I meant to say that DATA TABLES library in R reminds me more of Query.jl then tiddyverse
  - wodenokoto 6y ago
    What are you doing in the tidyverse that is not related to dataframes (or tibbles as the subclass of data.frame, that tidyverse uses is called)?
  - phillc73 6y ago
    Query.jl supports two different paradigms, one inspired directly by LINQ, the other by dplyr.[1] I actually prefer data.table in R, over dplyr, and Query.jl is really quite different. [1] http://www.queryverse.org/Query.jl/stable/ http://www.queryverse.org/Query.jl/stable/
- cwyers 6y ago
  Very much not the parent, but as a heavy R user, I don't think either of them quite nail the way dplyr and the tidyverse work. The thing about dplyr is... it's just functions. Okay, so, it's functions that leverage features R has (notably lazy evaluation and non-standard evaluation). But it's just functions. All you need is a function that takes a data frame and returns a data frame. So you can take a function out of the R standard library, you can take a function from a package written before dplyr came around, you can take a function from a recent non-tidyverse package, you can write your own function... it's all just functions. In DataFramesMeta.jl, though, you have a macro, and everything runs inside that macro. So if you want to take something that isn't a part of DataFramesMeta.jl... here's an example. Let's say you want to take the popular mtcars dataset, and get the five cars with the best gas milage. In dplyr, that goes mtcars %>% arrange(mpg) %>% head(5) arrange is a function from the dplyr package, head is a function from the standard library, but they both work seamlessly together. DataFramesMeta.jl lets you work in a pipe-forward fashion, but (at last I knew, at least, it's been a while since I played with it), you couldn't use the Julia head function within a DataFramesMeta.jl pipeline. You have to do your data transformations, assign to a variable, and then get the head of that variable. Which, okay, probably doesn't sound like a big deal. But I think it gets at the heart of what efforts to do something Tidyverse-like in other languages (Python and Julia, mostly) really miss. The key value proposition of the Tidyverse in R is that it is very composable and very extensible. That means, if you are trying to solve something in a Tidyverse way, you can probably find something that works for you. If you are doing financial analysis? Get tidyquant. If you're doing time series analysis, the tidyverts packages are for you. And it all works because there is so little friction involved in writing your own functions that extend the functionality of Tidyverse packages. Yes, dplyr is a useful querying DSL in its own right, but you can find a bunch of SQLish query languages, and they're all some degree of fine. Query.jl or DataFramesMeta.jl might expose a useful querying DSL for data frames, but they don't seem to me to be built to support building a whole ecosystem like dplyr and the Tidyverse are.

295310e0 6y ago

Can someone comment on the ease of distributing Julia code? Can I easily take a bit of code I write (w/o external libraries) and produce a single binary I can ship to someone or do they require a full Julia environment to run it?

jakobnissen 6y ago
They require the full Julia environment to run it - and it's a heavy environment. It is possible to compile a binary that includes the environment and the compiler, but IIRC, that will result in a >400 MB hello-world script taking 150 MB of RAM to run. The core devs have mentioned they are going to add the capacity to compile to actual static binaries, but that does not seem to be a top priority, so I wouldn't hold my breadth waiting for it.
- adgjlsfhk1 6y ago
  Note that this feature doesn't necessary require work by the core devs. It is completely feasible (at least in theory) to write a library that can output static exectuables.
  - ChrisRackauckas 6y ago
    It's actually done all of the time. GPUCompiler.jl, the core of the CUDA and AMD GPU stack, builds static binaries (compiled to .ptx by LLVM for CUDA for example, but the choice is just a switch), then stashes those binaries to use with a ccall in a Julia function. You could in theory use that stack to statically-compile anything that's GPU compliable, and it's really well-tested.
eigenspace 6y ago
Doing this without the user having Julia installed would require using PackageCompiler.jl to make a standalone executable. https://julialang.github.io/PackageCompiler.jl/dev/apps/ https://julialang.github.io/PackageCompiler.jl/dev/apps/ This is a pretty stable well established process now, but the binaries it produces are huge because they actually have the full Julia runtime in them. Active work is happening on small binary static compilation. We already do it for GPUs, we just have to repurpose our GPU AOT compilation pipeline for the CPU. There are proofs of concept that currently work, but something more polished is feeling like it’ll probably be another year or so.

celrod 6y ago

> (* Technically not all mutable objects live on the heap, because some never live at all, as they are optimized away so are never allocated in the first place.) The compiler will often stack allocate mutable objects in Julia. This is not the same as "never existed in the first place", because the stack pointer gets incremented and underlying data layout is you load from it is the same as that of the mutable object you allocated. Here is one example where that's very obviously what's happening: https://discourse.julialang.org/t/why-is-svector-faster-than-mvector/55174/10?u=elrod https://discourse.julialang.org/t/why-is-svector-faster-than... But it can happen now generally with mutable structures that can't practically just live in registers.

oxinabox 6y ago
fixed, to be more limitted in the claim
- celrod 6y ago
  Cool, and fantastic summary! I enjoyed reading it.
calebwinston 6y ago
If they are optimized away, are their destructors/finalizers still called? I'm really hoping the answer is yes...
- StefanKarpinski 6y ago
  Yes, that is guaranteed.

aliceryhl 6y ago

I really wish the language doesn't force me to use the REPL.

moelf 6y ago
most of the users use an IDE or a notebook, no need to only use REPL. If you're a old school editor -> terminal run kind of person, checkout https://github.com/dmolina/DaemonMode.jl https://github.com/dmolina/DaemonMode.jl
- aliceryhl 6y ago
  Thanks, I'll have to check this out. I really wish these things were more discoverable. It took me a month of using Julia until I figured out that the compile-times were even avoidable on the REPL by using Revise.jl.
krastanov 6y ago
How is it forcing you!?
- ecnahc515 6y ago
  I'm assuming you may not be familiar with Julia, but basically every "tool" you use in Julia is a library. It requires importing it in the Julia REPL or activating a new contextual REPL within the Julia REPL. You can then start calling the functions from the imported module. You don't really get standalone Julia tools, it's all libraries and you're always starting the REPL, importing something, then doing stuff from inside the REPL. For example, installing a package in Julia is the following steps: julia ] # keybinding which activates pkg mode add StaticArrays You could do it also like this: julia -e 'use Pkg; Pkg.add("StaticArrays")' There is no `julia-pkg add StaticArrays`, unfortunately.
  - gugagore 6y ago
    Why do you want `julia-pkg add StaticArrays` when you can do `julia -e 'use Pkg; Pkg.add("StaticArrays")'`, though?