11 ms·

Julia's Arraypocalypse Release

62 points by ViralBShah 11y ago

jsnathan 11y ago
Could someone maybe tl;dr what is going on here? I've tried out Julia a few years back, but didn't find it ready for serious use (yet). Am I right to assume that this means the language is still in serious flux?
- ViralBShah 11y ago
  Compared to a few years back, Julia is in seriously good shape. The long and short of this is that while the language has largely been stable, the array library in Julia will get an upgrade to become a lot more modern and clean. For most users, this should not change much. However, it gives us the ability to do some really amazing things with our array infrastructure going forward.
  - Lofkin 11y ago
    "However, it gives us the ability to do some really amazing things with our array infrastructure going forward." What kind of things? Super curious.
    - ViralBShah 11y ago
      The github issue linked actually has a detailed list of all the things that are planned - native bounds checking and removal, reshaped arrays, a lot more consistency in indexing and concatenation, slices as views, etc.
  - Lofkin 11y ago
    Also, any plans for the stats ecosystem? (dataframe, dashboards, imputation, data cleaning etc) It is still a bit wanting.
    - johnmyleswhite 11y ago
      I've been doing a lot of work behind the scenes there, but it's still going to take a while to offer a mature product. What we really need are more qualified engineers to speed up the process, but there's a vanishingly small group of people who are interested in doing eng work on statistics libraries.
      - Lofkin 11y ago
        Makes sense. Thanks for both your open source work and your reply!
    - ViralBShah 11y ago
      A lot of interesting things are going on in the DataFrames area - especially with the recent work on NullableArrays by David Gold and John Myles White. A blog post will be out soon describing it all. In my opinion, Escher.jl makes it really easy to build amazing dashboards and is progressing well. But, a lot more remains to be done, and contributions would be really welcome in these areas that take Julia's statistical computing to the next level.
      - elcritch 11y ago
        I'd like to add a note about my experience with doing this; I ended up forking Distributions.jl and added a fitting function for the Weibull distribution. Still need to clean it up and do a pull request,. The amazing thing about Julia I've come to enjoy is how clean the source code, how easy it is to find functionality, and it's all written IN Julia. That means I can modify libraries that are normally C or C++ for perf reasons. It's changed the prospect of needing to add functionality from one of hesitation to feeling that it's actually enjoyable.
        KenoFischer 11y ago
        I'm really glad to hear this! I'd like to think that the community and inspectability argument for Julia is actually one of the strongest ones out there, but it's not easy convincing people who haven't had that experience.
  - ViralBShah 11y ago
    I must say that while some of this work may not change much for users, like array slices as views, for many - the entire set of the arraypocalypse work is certainly a huge change. The key is to get it all done sooner so that we can keep getting that much closer to 1.0.
- 3JPLW 11y ago
  There are many many tomes of discussion that have led to this point (I'm mbauman). In short, there is an inherent disconnect between arrays-as-containers and matrices-as-linear-algebra-operators. There are many ways that you can define array operations such that the universe is closed and self-consistent, but there always seems to be some awkwardness. Right now, Julia uses some Matlab-ish semantics (sometimes ignoring trailing singleton dimensions), but there is an interaction with Julia's types that makes the status quo a little strange. So there are "just" two major breaking changes proposed here: changing our array semantics universe to be more APL-like (instead of Matlabish), and making indexing return a view instead of a copy.
  - ViralBShah 11y ago
    A little bit of history here. Back when we started Julia 6 years ago, I personally was in a much more of a Matlab/Octave frame of mind from my own experience. That is why the early array implementation followed that style. We had no idea where Julia would go. In some sense, it has been a personal journey of maturity in my own thinking, and if I can take the liberty to say so, the same is true for the community as well.
    - exDM69 11y ago
      Has your stance on one-based array indexing changed in those 6 years? How many off-by-one issues have you encountered? I'm not from a Matlab/Octave background, so one-based indexing feels really foreign to me.
      - jules 11y ago
        For what it's worth, I've ran into this a couple of times. Maybe it's just because I'm more familiar with 0-based indexing, but I think it's not just that. In every case the 0-based indexing had fewer (+1)'s and (-1)'s than the 1-based code. For instance k%n returns a number in the range [0,n), or in Julia notation [0:n-1]. When you want to do array indexing with that, you need (k%n)+1 in Julia. Btw, Matlab also uses 1-based indexing, no? I thought that Julia chose 1-based indexing to get brain backwards compatibility with Matlab.
johnmyleswhite 11y ago
Probably worth noting that the GitHub issue in question relates to work being done on Julia 0.5, a future release that is not yet ready to ship and won't ship for several months. Within a few weeks, Julia 0.4 will be shipping, which is not focused on changes to arrays.
deleted 11y ago
[deleted]
east2west 11y ago
I love Julia for its expressiveness but I am somewhat confused about its dynamic typing and its powerful type system. Are we to write mostly Python-like function and rely on multiple dispatch and JIT code generation for speed? Duck typing is one reason I stopped using Python for technical computing. I tried Julia out a few months ago and found that type inference has some holes that make type annotation brittle for function parameters. I finally gave up when I found out functions have no return type in Julia. Hopefully Julia can work out all the language features and mature into a good technical language. It would also help if there is a guide about the canonical programming style. I prefer static typing but thought using type annotation I could come close to static typing while still retain the freedom of quick-and-dirty code. I believe for the current Julia I was wrong.
- johnmyleswhite 11y ago
  If you'd like to know canonical programming style, you should read the code in Julia Base. Learning how to write good Julia is somewhat difficult because the language occupies an unfamiliar position between traditional static languages and traditional dynamic languages. Can you provide some concrete examples of the functionality you're looking for? What exactly does "quick-and-dirty" code look like to you?
  - east2west 11y ago
    I don't use "quick-and-dirty" as a derogatory term, so mostly I mean no error checking for rare occurrences or not ensuring correct values are being passed through. An example would be in Python if I just write MatrixA * MatrixB without any checking. I should check that they are both matrices if that is what I want, and that is what I did for reusable code. For one-off data analysis or computation, no-checking is certainly quick.
- StefanKarpinski 11y ago
  This StackOverflow answer may help clarify things: http://stackoverflow.com/a/28096079/659248 http://stackoverflow.com/a/28096079/659248. Making return types work well with multiple dispatch is not so simple; we feel that it's better to wait to add that feature and get it right than to rush things and have to fix it later. Some background, context and tentative plans for return type annotations: https://github.com/JuliaLang/julia/issues/1090 https://github.com/JuliaLang/julia/issues/1090.
jbssm 11y ago
Is there any progress when it comes to the slow package loading times of Julia? That made it quite unusable as a data exploration language when I last tried it about 1 year ago, but perhaps there was progress in that regard.
- johnmyleswhite 11y ago
  That issue is largely resolved in 0.4. Some functions are still a little slow, but most packages will now load much more quickly.
- ViralBShah 11y ago
  Julia 0.4, already in the RC phase, has precompilation thanks to Jameson Nash's work. It has been enabled for several packages, and gives sub-second load times.
- jbssm 11y ago
  Thank you both, there are very good news. I'll give it a try as soon as 0.4 is launched.
StefanKarpinski 11y ago
The "arraypocalypse" name is somewhat tongue in cheek, referring to the fact that this change will include some breaking changes. To provide some very brief motivation for the two most significant changes that are planned: 1. Array slices will create views rather than copies. This is just a matter of performance – although possible, it is currently way too annoying to write copy-free array code. This change will make it the default. 2. Slicing will behave more like APL: the rank of a slice of an array will be the sum of the ranks of the slices. This is a regular and powerful scheme, but sometimes at odds with how slicing works in Matlab. These changes will make the language easier to write high-performance, general code, while, with some care not making linear algebra harder.
- masklinn 11y ago
  > Array slices will create views rather than copies. This is just a matter of performance Except for breaking any existing code which mutates slices (unless slices are COW?), and implicit sharing can have disastrous effects on memory usage.
  - StefanKarpinski 11y ago
    How can implicit sharing have "disasterous effects on memory usage"? It will certainly break some code, hence the large amount of warning and the "pocalypse" appellation. Note that this is how array slices work in NumPy and that ecosystem has not collapsed.
    - masklinn 11y ago
      > How can implicit sharing have "disasterous effects on memory usage"? A small long-lived slices keeping a original arrays alive implicitly. Oracle recently undid the same optimisation in Java's strings because of that exact issue (which obviously broke code the other way around, code relying on cheap substrings would now take O(n) instead of O(1)). Erlang developers are also warned early and often about that behaviour in binaries.
      - 11y ago
Xcelerate 11y ago
I'll be glad once this is all implemented. I recently rewrote most of my functions that took abstract vectors and matrices to instead take concrete arrays along with a few index parameters. I did this because I realized SubArrays were making huge memory allocations inside the tightest parts of my loops for some unknown reason. Since I didn't want to spend days debugging that weirdness, I just took the simpler (but cruder) option of passing in the "kitchen sink" to each function.

62 points by ViralBShah 11y ago

Could someone maybe tl;dr what is going on here? I've tried out Julia a few years back, but didn't find it ready for serious use (yet). Am I right to assume that this means the language is still in serious flux?

ViralBShah 11y ago
Compared to a few years back, Julia is in seriously good shape. The long and short of this is that while the language has largely been stable, the array library in Julia will get an upgrade to become a lot more modern and clean. For most users, this should not change much. However, it gives us the ability to do some really amazing things with our array infrastructure going forward.
- Lofkin 11y ago
  "However, it gives us the ability to do some really amazing things with our array infrastructure going forward." What kind of things? Super curious.
  - ViralBShah 11y ago
    The github issue linked actually has a detailed list of all the things that are planned - native bounds checking and removal, reshaped arrays, a lot more consistency in indexing and concatenation, slices as views, etc.
- Lofkin 11y ago
  Also, any plans for the stats ecosystem? (dataframe, dashboards, imputation, data cleaning etc) It is still a bit wanting.
  - johnmyleswhite 11y ago
    I've been doing a lot of work behind the scenes there, but it's still going to take a while to offer a mature product. What we really need are more qualified engineers to speed up the process, but there's a vanishingly small group of people who are interested in doing eng work on statistics libraries.
    - Lofkin 11y ago
      Makes sense. Thanks for both your open source work and your reply!
  - ViralBShah 11y ago
    A lot of interesting things are going on in the DataFrames area - especially with the recent work on NullableArrays by David Gold and John Myles White. A blog post will be out soon describing it all. In my opinion, Escher.jl makes it really easy to build amazing dashboards and is progressing well. But, a lot more remains to be done, and contributions would be really welcome in these areas that take Julia's statistical computing to the next level.
    - elcritch 11y ago
      I'd like to add a note about my experience with doing this; I ended up forking Distributions.jl and added a fitting function for the Weibull distribution. Still need to clean it up and do a pull request,. The amazing thing about Julia I've come to enjoy is how clean the source code, how easy it is to find functionality, and it's all written IN Julia. That means I can modify libraries that are normally C or C++ for perf reasons. It's changed the prospect of needing to add functionality from one of hesitation to feeling that it's actually enjoyable.
      - KenoFischer 11y ago
        I'm really glad to hear this! I'd like to think that the community and inspectability argument for Julia is actually one of the strongest ones out there, but it's not easy convincing people who haven't had that experience.
- ViralBShah 11y ago
  I must say that while some of this work may not change much for users, like array slices as views, for many - the entire set of the arraypocalypse work is certainly a huge change. The key is to get it all done sooner so that we can keep getting that much closer to 1.0.
3JPLW 11y ago
There are many many tomes of discussion that have led to this point (I'm mbauman). In short, there is an inherent disconnect between arrays-as-containers and matrices-as-linear-algebra-operators. There are many ways that you can define array operations such that the universe is closed and self-consistent, but there always seems to be some awkwardness. Right now, Julia uses some Matlab-ish semantics (sometimes ignoring trailing singleton dimensions), but there is an interaction with Julia's types that makes the status quo a little strange. So there are "just" two major breaking changes proposed here: changing our array semantics universe to be more APL-like (instead of Matlabish), and making indexing return a view instead of a copy.
- ViralBShah 11y ago
  A little bit of history here. Back when we started Julia 6 years ago, I personally was in a much more of a Matlab/Octave frame of mind from my own experience. That is why the early array implementation followed that style. We had no idea where Julia would go. In some sense, it has been a personal journey of maturity in my own thinking, and if I can take the liberty to say so, the same is true for the community as well.
  - exDM69 11y ago
    Has your stance on one-based array indexing changed in those 6 years? How many off-by-one issues have you encountered? I'm not from a Matlab/Octave background, so one-based indexing feels really foreign to me.
    - jules 11y ago
      For what it's worth, I've ran into this a couple of times. Maybe it's just because I'm more familiar with 0-based indexing, but I think it's not just that. In every case the 0-based indexing had fewer (+1)'s and (-1)'s than the 1-based code. For instance k%n returns a number in the range [0,n), or in Julia notation [0:n-1]. When you want to do array indexing with that, you need (k%n)+1 in Julia. Btw, Matlab also uses 1-based indexing, no? I thought that Julia chose 1-based indexing to get brain backwards compatibility with Matlab.

johnmyleswhite 11y ago

Probably worth noting that the GitHub issue in question relates to work being done on Julia 0.5, a future release that is not yet ready to ship and won't ship for several months. Within a few weeks, Julia 0.4 will be shipping, which is not focused on changes to arrays.

deleted 11y ago

[deleted]

east2west 11y ago

I love Julia for its expressiveness but I am somewhat confused about its dynamic typing and its powerful type system. Are we to write mostly Python-like function and rely on multiple dispatch and JIT code generation for speed? Duck typing is one reason I stopped using Python for technical computing. I tried Julia out a few months ago and found that type inference has some holes that make type annotation brittle for function parameters. I finally gave up when I found out functions have no return type in Julia. Hopefully Julia can work out all the language features and mature into a good technical language. It would also help if there is a guide about the canonical programming style. I prefer static typing but thought using type annotation I could come close to static typing while still retain the freedom of quick-and-dirty code. I believe for the current Julia I was wrong.

johnmyleswhite 11y ago
If you'd like to know canonical programming style, you should read the code in Julia Base. Learning how to write good Julia is somewhat difficult because the language occupies an unfamiliar position between traditional static languages and traditional dynamic languages. Can you provide some concrete examples of the functionality you're looking for? What exactly does "quick-and-dirty" code look like to you?
- east2west 11y ago
  I don't use "quick-and-dirty" as a derogatory term, so mostly I mean no error checking for rare occurrences or not ensuring correct values are being passed through. An example would be in Python if I just write MatrixA * MatrixB without any checking. I should check that they are both matrices if that is what I want, and that is what I did for reusable code. For one-off data analysis or computation, no-checking is certainly quick.
StefanKarpinski 11y ago
This StackOverflow answer may help clarify things: http://stackoverflow.com/a/28096079/659248 http://stackoverflow.com/a/28096079/659248. Making return types work well with multiple dispatch is not so simple; we feel that it's better to wait to add that feature and get it right than to rush things and have to fix it later. Some background, context and tentative plans for return type annotations: https://github.com/JuliaLang/julia/issues/1090 https://github.com/JuliaLang/julia/issues/1090.

jbssm 11y ago

Is there any progress when it comes to the slow package loading times of Julia? That made it quite unusable as a data exploration language when I last tried it about 1 year ago, but perhaps there was progress in that regard.

johnmyleswhite 11y ago
That issue is largely resolved in 0.4. Some functions are still a little slow, but most packages will now load much more quickly.
ViralBShah 11y ago
Julia 0.4, already in the RC phase, has precompilation thanks to Jameson Nash's work. It has been enabled for several packages, and gives sub-second load times.
jbssm 11y ago
Thank you both, there are very good news. I'll give it a try as soon as 0.4 is launched.

StefanKarpinski 11y ago

The "arraypocalypse" name is somewhat tongue in cheek, referring to the fact that this change will include some breaking changes. To provide some very brief motivation for the two most significant changes that are planned: 1. Array slices will create views rather than copies. This is just a matter of performance – although possible, it is currently way too annoying to write copy-free array code. This change will make it the default. 2. Slicing will behave more like APL: the rank of a slice of an array will be the sum of the ranks of the slices. This is a regular and powerful scheme, but sometimes at odds with how slicing works in Matlab. These changes will make the language easier to write high-performance, general code, while, with some care not making linear algebra harder.

masklinn 11y ago
> Array slices will create views rather than copies. This is just a matter of performance Except for breaking any existing code which mutates slices (unless slices are COW?), and implicit sharing can have disastrous effects on memory usage.
- StefanKarpinski 11y ago
  How can implicit sharing have "disasterous effects on memory usage"? It will certainly break some code, hence the large amount of warning and the "pocalypse" appellation. Note that this is how array slices work in NumPy and that ecosystem has not collapsed.
  - masklinn 11y ago
    > How can implicit sharing have "disasterous effects on memory usage"? A small long-lived slices keeping a original arrays alive implicitly. Oracle recently undid the same optimisation in Java's strings because of that exact issue (which obviously broke code the other way around, code relying on cheap substrings would now take O(n) instead of O(1)). Erlang developers are also warned early and often about that behaviour in binaries.
    - 11y ago

Xcelerate 11y ago

I'll be glad once this is all implemented. I recently rewrote most of my functions that took abstract vectors and matrices to instead take concrete arrays along with a few index parameters. I did this because I realized SubArrays were making huge memory allocations inside the tightest parts of my loops for some unknown reason. Since I didn't want to spend days debugging that weirdness, I just took the simpler (but cruder) option of passing in the "kitchen sink" to each function.