6 ms·

Could someone maybe tl;dr what is going on here? I've tried out Julia a few years back, but didn't find it ready for serious use (yet). Am I right to assume th

Could someone maybe tl;dr what is going on here? I've tried out Julia a few years back, but didn't find it ready for serious use (yet). Am I right to assume that this means the language is still in serious flux?

ViralBShah 11y ago
Compared to a few years back, Julia is in seriously good shape. The long and short of this is that while the language has largely been stable, the array library in Julia will get an upgrade to become a lot more modern and clean. For most users, this should not change much. However, it gives us the ability to do some really amazing things with our array infrastructure going forward.
- Lofkin 11y ago
  "However, it gives us the ability to do some really amazing things with our array infrastructure going forward." What kind of things? Super curious.
  - ViralBShah 11y ago
    The github issue linked actually has a detailed list of all the things that are planned - native bounds checking and removal, reshaped arrays, a lot more consistency in indexing and concatenation, slices as views, etc.
- Lofkin 11y ago
  Also, any plans for the stats ecosystem? (dataframe, dashboards, imputation, data cleaning etc) It is still a bit wanting.
  - johnmyleswhite 11y ago
    I've been doing a lot of work behind the scenes there, but it's still going to take a while to offer a mature product. What we really need are more qualified engineers to speed up the process, but there's a vanishingly small group of people who are interested in doing eng work on statistics libraries.
    - Lofkin 11y ago
      Makes sense. Thanks for both your open source work and your reply!
  - ViralBShah 11y ago
    A lot of interesting things are going on in the DataFrames area - especially with the recent work on NullableArrays by David Gold and John Myles White. A blog post will be out soon describing it all. In my opinion, Escher.jl makes it really easy to build amazing dashboards and is progressing well. But, a lot more remains to be done, and contributions would be really welcome in these areas that take Julia's statistical computing to the next level.
    - elcritch 11y ago
      I'd like to add a note about my experience with doing this; I ended up forking Distributions.jl and added a fitting function for the Weibull distribution. Still need to clean it up and do a pull request,. The amazing thing about Julia I've come to enjoy is how clean the source code, how easy it is to find functionality, and it's all written IN Julia. That means I can modify libraries that are normally C or C++ for perf reasons. It's changed the prospect of needing to add functionality from one of hesitation to feeling that it's actually enjoyable.
      - KenoFischer 11y ago
        I'm really glad to hear this! I'd like to think that the community and inspectability argument for Julia is actually one of the strongest ones out there, but it's not easy convincing people who haven't had that experience.
        elcritch 11y ago
        It really is pretty great.
    - deleted 11y ago
      [deleted]
- ViralBShah 11y ago
  I must say that while some of this work may not change much for users, like array slices as views, for many - the entire set of the arraypocalypse work is certainly a huge change. The key is to get it all done sooner so that we can keep getting that much closer to 1.0.
3JPLW 11y ago
There are many many tomes of discussion that have led to this point (I'm mbauman). In short, there is an inherent disconnect between arrays-as-containers and matrices-as-linear-algebra-operators. There are many ways that you can define array operations such that the universe is closed and self-consistent, but there always seems to be some awkwardness. Right now, Julia uses some Matlab-ish semantics (sometimes ignoring trailing singleton dimensions), but there is an interaction with Julia's types that makes the status quo a little strange. So there are "just" two major breaking changes proposed here: changing our array semantics universe to be more APL-like (instead of Matlabish), and making indexing return a view instead of a copy.
- ViralBShah 11y ago
  A little bit of history here. Back when we started Julia 6 years ago, I personally was in a much more of a Matlab/Octave frame of mind from my own experience. That is why the early array implementation followed that style. We had no idea where Julia would go. In some sense, it has been a personal journey of maturity in my own thinking, and if I can take the liberty to say so, the same is true for the community as well.
  - exDM69 11y ago
    Has your stance on one-based array indexing changed in those 6 years? How many off-by-one issues have you encountered? I'm not from a Matlab/Octave background, so one-based indexing feels really foreign to me.
    - jules 11y ago
      For what it's worth, I've ran into this a couple of times. Maybe it's just because I'm more familiar with 0-based indexing, but I think it's not just that. In every case the 0-based indexing had fewer (+1)'s and (-1)'s than the 1-based code. For instance k%n returns a number in the range [0,n), or in Julia notation [0:n-1]. When you want to do array indexing with that, you need (k%n)+1 in Julia. Btw, Matlab also uses 1-based indexing, no? I thought that Julia chose 1-based indexing to get brain backwards compatibility with Matlab.

ViralBShah 11y ago

Compared to a few years back, Julia is in seriously good shape. The long and short of this is that while the language has largely been stable, the array library in Julia will get an upgrade to become a lot more modern and clean. For most users, this should not change much. However, it gives us the ability to do some really amazing things with our array infrastructure going forward.

Lofkin 11y ago
"However, it gives us the ability to do some really amazing things with our array infrastructure going forward." What kind of things? Super curious.
- ViralBShah 11y ago
  The github issue linked actually has a detailed list of all the things that are planned - native bounds checking and removal, reshaped arrays, a lot more consistency in indexing and concatenation, slices as views, etc.
Lofkin 11y ago
Also, any plans for the stats ecosystem? (dataframe, dashboards, imputation, data cleaning etc) It is still a bit wanting.
- johnmyleswhite 11y ago
  I've been doing a lot of work behind the scenes there, but it's still going to take a while to offer a mature product. What we really need are more qualified engineers to speed up the process, but there's a vanishingly small group of people who are interested in doing eng work on statistics libraries.
  - Lofkin 11y ago
    Makes sense. Thanks for both your open source work and your reply!
- ViralBShah 11y ago
  A lot of interesting things are going on in the DataFrames area - especially with the recent work on NullableArrays by David Gold and John Myles White. A blog post will be out soon describing it all. In my opinion, Escher.jl makes it really easy to build amazing dashboards and is progressing well. But, a lot more remains to be done, and contributions would be really welcome in these areas that take Julia's statistical computing to the next level.
  - elcritch 11y ago
    I'd like to add a note about my experience with doing this; I ended up forking Distributions.jl and added a fitting function for the Weibull distribution. Still need to clean it up and do a pull request,. The amazing thing about Julia I've come to enjoy is how clean the source code, how easy it is to find functionality, and it's all written IN Julia. That means I can modify libraries that are normally C or C++ for perf reasons. It's changed the prospect of needing to add functionality from one of hesitation to feeling that it's actually enjoyable.
    - KenoFischer 11y ago
      I'm really glad to hear this! I'd like to think that the community and inspectability argument for Julia is actually one of the strongest ones out there, but it's not easy convincing people who haven't had that experience.
      - elcritch 11y ago
        It really is pretty great.
  - deleted 11y ago
    [deleted]
ViralBShah 11y ago
I must say that while some of this work may not change much for users, like array slices as views, for many - the entire set of the arraypocalypse work is certainly a huge change. The key is to get it all done sooner so that we can keep getting that much closer to 1.0.

3JPLW 11y ago

There are many many tomes of discussion that have led to this point (I'm mbauman). In short, there is an inherent disconnect between arrays-as-containers and matrices-as-linear-algebra-operators. There are many ways that you can define array operations such that the universe is closed and self-consistent, but there always seems to be some awkwardness. Right now, Julia uses some Matlab-ish semantics (sometimes ignoring trailing singleton dimensions), but there is an interaction with Julia's types that makes the status quo a little strange. So there are "just" two major breaking changes proposed here: changing our array semantics universe to be more APL-like (instead of Matlabish), and making indexing return a view instead of a copy.

ViralBShah 11y ago
A little bit of history here. Back when we started Julia 6 years ago, I personally was in a much more of a Matlab/Octave frame of mind from my own experience. That is why the early array implementation followed that style. We had no idea where Julia would go. In some sense, it has been a personal journey of maturity in my own thinking, and if I can take the liberty to say so, the same is true for the community as well.
- exDM69 11y ago
  Has your stance on one-based array indexing changed in those 6 years? How many off-by-one issues have you encountered? I'm not from a Matlab/Octave background, so one-based indexing feels really foreign to me.
  - jules 11y ago
    For what it's worth, I've ran into this a couple of times. Maybe it's just because I'm more familiar with 0-based indexing, but I think it's not just that. In every case the 0-based indexing had fewer (+1)'s and (-1)'s than the 1-based code. For instance k%n returns a number in the range [0,n), or in Julia notation [0:n-1]. When you want to do array indexing with that, you need (k%n)+1 in Julia. Btw, Matlab also uses 1-based indexing, no? I thought that Julia chose 1-based indexing to get brain backwards compatibility with Matlab.