9 ms·

It's mainly targeting MATLAB, and to a slightly lesser extent scientific and numeric programming in Python and R. It's a well thought out language that allows y

It's mainly targeting MATLAB, and to a slightly lesser extent scientific and numeric programming in Python and R. It's a well thought out language that allows you to write MATLAB-like high level code with an easy gradient for progressive typing and optimization to near C level performance. It's a bit harder to sell versus Python, since Python has enormous value in the ecosystem, community, and ubiquity. Also, because it primary targets MATLAB a lot of the standard libraries try to have similar ergonomics, which is a bit of a waste of a great tool to recreate a poor interface.

fareesh 8y ago
I often wonder what inspires folks to start from scratch in the face of a gigantic ecosystem like the one that Python brings with it, which will also keep improving.
- 3JPLW 8y ago
  In the case of Julia, you can check out the original motivation back in 2012 or this recent answer on the message board. https://julialang.org/blog/2012/02/why-we-created-julia https://julialang.org/blog/2012/02/why-we-created-julia https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/10 https://discourse.julialang.org/t/julia-motivation-why-weren...
  - Gravityloss 8y ago
    If you have some Matlab background, working with Python is frustrating. It is hard to explain, but vectors and matrices should be the primary concepts, with absolutely minimum extra glue needed.
    - prestonh 8y ago
      I don't have much experience with Matlab, so forgive me if this is incorrect, but numpy should be able to do everything that matlab can at comparable speeds. No one performing matrix/vector-like operations is doing so with standard Python lists if numpy is available.
      - Gravityloss 8y ago
        I'm not talking about execution speed but the human interface. The syntax of Python just is not nice and using libraries just adds more and more boilerplate. I don't expect anyone who has not spent a lot of time with Matlab to "get" it.
        acdha 8y ago
        It's interesting because those were the reasons I heard from MATLAB users for switching to Python: moving to a language with a cleaner, less ad-hoc design and less boilerplate / copy-paste code made a big difference once you had more than a little code. Has the language improved dramatically in the last few decade?
        Gravityloss 8y ago
        Matlab is good for non-software engineers and scientists to explore and solve problems. Once you have a solution there, write it up in some proper language. I don't think it is likely for a software engineer to understand Matlab's niche and effectiveness. It comes from a different direction. A ball point pen, a paint brush and a piece of drafting graphite all have their uses.
        twtw 8y ago
        I'm an electrical engineer oriented towards signal processing and controls, and I'm also incapable of understanding Matlab's niche and effectiveness. I wish Matlab could just vanish. It's a glorified calculator that has mutated over the years into a crap programming language, with random features just bolted on and the strangest semantics of all time. It also has the drawback that most of its users generate write-only code, so everyone that learns it also learns to write code that way.
        geoalchimista 8y ago
        I second this. And to make matters worse, a lot of MATLAB users are not aware of coding style. Poorly readable MATLAB code stinks. I sometimes rather wish to read Fortran 90 instead of MATLAB code.
    - goatlover 8y ago
      That's definitely a thing. I like Pandas, but the syntax is a bit cumbersome compared to R or Julia.
- kgwgk 8y ago
  Hubris. Which is one of the three great virtues of a programmer.
  - simonbyrne 8y ago
    I'm pretty sure the other two were as well.
- ScottPJones 8y ago
  Sometimes languages get stuck by their history, and (without really becoming a rather different incompatible language) the only way forward is to start from scratch. Also, Julia is good at letting you use those old ecosystems, C and Fortran libraries, Python, Java, R, all from the comfort of home (Julia)
- cdsousa 8y ago
  In part, the fact that there are major problems moving the "giant" to where one would like (speed), e.g., unladen swallow, pypy, pyston, ...
- ChrisRackauckas 8y ago
  Well, there are big areas where the ecosystem is quite underdeveloped, like differential equations, which have a lot of holes that need new algorithms and improvement. It would be extremely difficult to develop all of the necessary algorithms in C/C++/Fortran, and pretty much impossible in Python/MATLAB (I tried at first), but it's a breeze to tackle this in Julia. So for these kinds of scientific computing areas where there's tons of work with few people with the necessary expertise, Julia is a great way to start getting some good implementations out there for people to use.
bunderbunder 8y ago
Right now it’s a hard sell vs Python, but I can imagine Python running out of runway soon. A lot of it’s existing scientific and statistical computing stack is built around the assumption that you’ll be working with data that conveniently fits in memory. Once you’ve sized out of pandas/scipy/scikit, your next major option is Spark, which is certainly powerful, but is also unwieldy. I could see something like Julia earning a lot of mindshare if it had a really polished solution for the space between, “my data is hundreds of megabytes”, and, “my data is hundreds of gigabytes”.
- mattip 8y ago
  Dask is another choice for distributed out-of-memory data structures but still within the python ecosystem
- costrouc 8y ago
  Python does not make these assumptions. There are Python tools that exist to solve these problems that are equally as powerful as other language's solutions. The two that I believe right now address these problems best are dask and mpi4py. Mpi4py can achieve very low latencies but given that it's based on MPI it can be complex to use. dask is the most user friendly and as a Python user is clearly easier to use than spark. Paired with numba you can get equivalent performance to distributed C programs.
  - deleted 8y ago
    [deleted]
- throwawaymath 8y ago
  Speaking as someone who uses Python with half a terabyte of memory, I think you're underestimating how much memory these labs will use. In my experience most HPC architecture is optimized first by rewriting the code in the same (already fast) language or library, then by increasing hardware resources (especially among distributed nodes), then by seeking a new library in the same ecosystem, and finally by moving to a new language if they have to. Moving to a new language has more friction than basically anything else unless there's a real language feature missing or the budget doesn't allow for more compute hardware. Hundreds of gigabytes is well below where academic and industry labs will start having to think about these problems. It's going to be really tough to displace Python with anything equally as general purpose. This is all to say that I buy that Julia can shine more than Python for I/O bound HPC, but it really shouldn't be I/O bound until you have terabytes of data (and likely tens of terabytes). And aside from that, the Python numerical computing ecosystem includes a lot more than just Numpy and Pandas. As other commenters have mentioned, you can use Dask if your hot data has grown into the terabyte range. Anaconda includes a lot of libraries which can bail you out of situations once you've left the familiar world of Pandas data frames.
stabbles 8y ago
It's definitely targeting Python and R to the same extent as MATLAB, in the sense that it claims to solve the two-language problem that is so apparent in these languages. MATLAB, Python and R are easy scripting languages, but as soon as you have to do heavy computations, you're forced to call C / FORTRAN libraries. Julia on the other hand is prove that we can have a high-level scripting language that runs as fast as C and Fortran. Combine this with Julia's generic programming and type system, and you can easily run your algorithm with floats, complex numbers, arbitrary precision, etc etc. Even if Julia wraps a library like Tensorflow, its API is looking really nice compared to Python [1]: using TensorFlow sess = TensorFlow.Session() x = TensorFlow.constant(Float64[1,2]) y = TensorFlow.Variable(Float64[3,4]) z = TensorFlow.placeholder(Float64) w = exp(x + z + -y) run(sess, TensorFlow.global_variables_initializer()) res = run(sess, w, Dict(z=>Float64[1,2])) Base.Test.@test res[1] ≈ exp(-1) [1] https://github.com/malmaud/TensorFlow.jl https://github.com/malmaud/TensorFlow.jl
- blablablerg 8y ago
  To compete with R it needs something like the tidyverse.
  - elsherbini 8y ago
    I agree - declarative in memory dataframe manipulation is extremely powerful. And the composability of plotting in the tidyverse is really nice as well. It looks like there are the beginnings of both of these in Julia: [0] http://gadflyjl.org/stable/ http://gadflyjl.org/stable/ [1] https://github.com/JuliaStats/DataFramesMeta.jl https://github.com/JuliaStats/DataFramesMeta.jl
  - psandersen 8y ago
    Agree completely with this, tidyverse is whats keeping me in R when I'd prefer to mostly use Python.
  - ViralBShah 8y ago
    I often ask myself the question, how can Julia do things that R cannot do. After all, when something is good at doing something, why replace it? Part of this is why we did JuliaDB: http://juliadb.org/ http://juliadb.org/ and continue to try push the boundaries on parallelism, missing data, OnlineStats.jl and making data manipulation and modeling that much easier.
metaobject 8y ago
Language implementation-wise, can anyone explain why/how Julia is able to get close to C-level performance? Is it doing some extra steps under the hood (JIT compilation?) that Python and R aren't doing?
- jabl 8y ago
  Yes, it's using JIT compilation (last I checked, they are using LLVM as the backend). Combined with a language design that takes JIT compilation into account from the get-go, making the problem much easier than trying to use a JIT later on (see e.g. PyPy).
- improbable22 8y ago
  Yes, it's JIT compiled. And my (very crude) understanding is that the stronger type system makes this much easier than in Python. The compiled version of any function is specific to the types of its inputs, and thus need not contain any further checks: simple functions often end up with literally the same assembly as C would produce.
  - skolemtotem 8y ago
    If anyone wants to do more research on this, the keyword is "monomorphization".
  - dnautics 8y ago
    It's "extremely lazy ahead of time compiled", is one way I've described the compilation model, since you're basically never executing code in an interpreted fashion (usually jits let you do either). Also, typically jit's choice of when to but may be non-deterministic, or deterministic but difficult to understand. When Julia chooses to compile is pretty easy to understand
    - ScottPJones 8y ago
      I believe though that there is some work being done on actually directly interpreting the AST, in cases where going through all the work of generating LLVM IR and compiling that to native code is unnecessary, particularly when it is code that is only run once when a package is compiled the first time.

fareesh 8y ago

I often wonder what inspires folks to start from scratch in the face of a gigantic ecosystem like the one that Python brings with it, which will also keep improving.

3JPLW 8y ago
In the case of Julia, you can check out the original motivation back in 2012 or this recent answer on the message board. https://julialang.org/blog/2012/02/why-we-created-julia https://julialang.org/blog/2012/02/why-we-created-julia https://discourse.julialang.org/t/julia-motivation-why-werent-numpy-scipy-numba-good-enough/2236/10 https://discourse.julialang.org/t/julia-motivation-why-weren...
- Gravityloss 8y ago
  If you have some Matlab background, working with Python is frustrating. It is hard to explain, but vectors and matrices should be the primary concepts, with absolutely minimum extra glue needed.
  - prestonh 8y ago
    I don't have much experience with Matlab, so forgive me if this is incorrect, but numpy should be able to do everything that matlab can at comparable speeds. No one performing matrix/vector-like operations is doing so with standard Python lists if numpy is available.
    - Gravityloss 8y ago
      I'm not talking about execution speed but the human interface. The syntax of Python just is not nice and using libraries just adds more and more boilerplate. I don't expect anyone who has not spent a lot of time with Matlab to "get" it.
      - acdha 8y ago
        It's interesting because those were the reasons I heard from MATLAB users for switching to Python: moving to a language with a cleaner, less ad-hoc design and less boilerplate / copy-paste code made a big difference once you had more than a little code. Has the language improved dramatically in the last few decade?
        Gravityloss 8y ago
        Matlab is good for non-software engineers and scientists to explore and solve problems. Once you have a solution there, write it up in some proper language. I don't think it is likely for a software engineer to understand Matlab's niche and effectiveness. It comes from a different direction. A ball point pen, a paint brush and a piece of drafting graphite all have their uses.
        twtw 8y ago
        I'm an electrical engineer oriented towards signal processing and controls, and I'm also incapable of understanding Matlab's niche and effectiveness. I wish Matlab could just vanish. It's a glorified calculator that has mutated over the years into a crap programming language, with random features just bolted on and the strangest semantics of all time. It also has the drawback that most of its users generate write-only code, so everyone that learns it also learns to write code that way.
        geoalchimista 8y ago
        I second this. And to make matters worse, a lot of MATLAB users are not aware of coding style. Poorly readable MATLAB code stinks. I sometimes rather wish to read Fortran 90 instead of MATLAB code.
  - goatlover 8y ago
    That's definitely a thing. I like Pandas, but the syntax is a bit cumbersome compared to R or Julia.
kgwgk 8y ago
Hubris. Which is one of the three great virtues of a programmer.
- simonbyrne 8y ago
  I'm pretty sure the other two were as well.
ScottPJones 8y ago
Sometimes languages get stuck by their history, and (without really becoming a rather different incompatible language) the only way forward is to start from scratch. Also, Julia is good at letting you use those old ecosystems, C and Fortran libraries, Python, Java, R, all from the comfort of home (Julia)
cdsousa 8y ago
In part, the fact that there are major problems moving the "giant" to where one would like (speed), e.g., unladen swallow, pypy, pyston, ...
ChrisRackauckas 8y ago
Well, there are big areas where the ecosystem is quite underdeveloped, like differential equations, which have a lot of holes that need new algorithms and improvement. It would be extremely difficult to develop all of the necessary algorithms in C/C++/Fortran, and pretty much impossible in Python/MATLAB (I tried at first), but it's a breeze to tackle this in Julia. So for these kinds of scientific computing areas where there's tons of work with few people with the necessary expertise, Julia is a great way to start getting some good implementations out there for people to use.

bunderbunder 8y ago

Right now it’s a hard sell vs Python, but I can imagine Python running out of runway soon. A lot of it’s existing scientific and statistical computing stack is built around the assumption that you’ll be working with data that conveniently fits in memory. Once you’ve sized out of pandas/scipy/scikit, your next major option is Spark, which is certainly powerful, but is also unwieldy. I could see something like Julia earning a lot of mindshare if it had a really polished solution for the space between, “my data is hundreds of megabytes”, and, “my data is hundreds of gigabytes”.

mattip 8y ago
Dask is another choice for distributed out-of-memory data structures but still within the python ecosystem
costrouc 8y ago
Python does not make these assumptions. There are Python tools that exist to solve these problems that are equally as powerful as other language's solutions. The two that I believe right now address these problems best are dask and mpi4py. Mpi4py can achieve very low latencies but given that it's based on MPI it can be complex to use. dask is the most user friendly and as a Python user is clearly easier to use than spark. Paired with numba you can get equivalent performance to distributed C programs.
- deleted 8y ago
  [deleted]
throwawaymath 8y ago
Speaking as someone who uses Python with half a terabyte of memory, I think you're underestimating how much memory these labs will use. In my experience most HPC architecture is optimized first by rewriting the code in the same (already fast) language or library, then by increasing hardware resources (especially among distributed nodes), then by seeking a new library in the same ecosystem, and finally by moving to a new language if they have to. Moving to a new language has more friction than basically anything else unless there's a real language feature missing or the budget doesn't allow for more compute hardware. Hundreds of gigabytes is well below where academic and industry labs will start having to think about these problems. It's going to be really tough to displace Python with anything equally as general purpose. This is all to say that I buy that Julia can shine more than Python for I/O bound HPC, but it really shouldn't be I/O bound until you have terabytes of data (and likely tens of terabytes). And aside from that, the Python numerical computing ecosystem includes a lot more than just Numpy and Pandas. As other commenters have mentioned, you can use Dask if your hot data has grown into the terabyte range. Anaconda includes a lot of libraries which can bail you out of situations once you've left the familiar world of Pandas data frames.

stabbles 8y ago

It's definitely targeting Python and R to the same extent as MATLAB, in the sense that it claims to solve the two-language problem that is so apparent in these languages. MATLAB, Python and R are easy scripting languages, but as soon as you have to do heavy computations, you're forced to call C / FORTRAN libraries. Julia on the other hand is prove that we can have a high-level scripting language that runs as fast as C and Fortran. Combine this with Julia's generic programming and type system, and you can easily run your algorithm with floats, complex numbers, arbitrary precision, etc etc. Even if Julia wraps a library like Tensorflow, its API is looking really nice compared to Python [1]: using TensorFlow sess = TensorFlow.Session() x = TensorFlow.constant(Float64[1,2]) y = TensorFlow.Variable(Float64[3,4]) z = TensorFlow.placeholder(Float64) w = exp(x + z + -y) run(sess, TensorFlow.global_variables_initializer()) res = run(sess, w, Dict(z=>Float64[1,2])) Base.Test.@test res[1] ≈ exp(-1) [1] https://github.com/malmaud/TensorFlow.jl https://github.com/malmaud/TensorFlow.jl

blablablerg 8y ago
To compete with R it needs something like the tidyverse.
- elsherbini 8y ago
  I agree - declarative in memory dataframe manipulation is extremely powerful. And the composability of plotting in the tidyverse is really nice as well. It looks like there are the beginnings of both of these in Julia: [0] http://gadflyjl.org/stable/ http://gadflyjl.org/stable/ [1] https://github.com/JuliaStats/DataFramesMeta.jl https://github.com/JuliaStats/DataFramesMeta.jl
- psandersen 8y ago
  Agree completely with this, tidyverse is whats keeping me in R when I'd prefer to mostly use Python.
- ViralBShah 8y ago
  I often ask myself the question, how can Julia do things that R cannot do. After all, when something is good at doing something, why replace it? Part of this is why we did JuliaDB: http://juliadb.org/ http://juliadb.org/ and continue to try push the boundaries on parallelism, missing data, OnlineStats.jl and making data manipulation and modeling that much easier.

metaobject 8y ago

Language implementation-wise, can anyone explain why/how Julia is able to get close to C-level performance? Is it doing some extra steps under the hood (JIT compilation?) that Python and R aren't doing?

jabl 8y ago
Yes, it's using JIT compilation (last I checked, they are using LLVM as the backend). Combined with a language design that takes JIT compilation into account from the get-go, making the problem much easier than trying to use a JIT later on (see e.g. PyPy).
improbable22 8y ago
Yes, it's JIT compiled. And my (very crude) understanding is that the stronger type system makes this much easier than in Python. The compiled version of any function is specific to the types of its inputs, and thus need not contain any further checks: simple functions often end up with literally the same assembly as C would produce.
- skolemtotem 8y ago
  If anyone wants to do more research on this, the keyword is "monomorphization".
- dnautics 8y ago
  It's "extremely lazy ahead of time compiled", is one way I've described the compilation model, since you're basically never executing code in an interpreted fashion (usually jits let you do either). Also, typically jit's choice of when to but may be non-deterministic, or deterministic but difficult to understand. When Julia chooses to compile is pretty easy to understand
  - ScottPJones 8y ago
    I believe though that there is some work being done on actually directly interpreting the AST, in cases where going through all the work of generating LLVM IR and compiling that to native code is unnecessary, particularly when it is code that is only run once when a package is compiled the first time.