4 ms·

The "arraypocalypse" name is somewhat tongue in cheek, referring to the fact that this change will include some breaking changes. To provide some very brief mot

The "arraypocalypse" name is somewhat tongue in cheek, referring to the fact that this change will include some breaking changes. To provide some very brief motivation for the two most significant changes that are planned: 1. Array slices will create views rather than copies. This is just a matter of performance – although possible, it is currently way too annoying to write copy-free array code. This change will make it the default. 2. Slicing will behave more like APL: the rank of a slice of an array will be the sum of the ranks of the slices. This is a regular and powerful scheme, but sometimes at odds with how slicing works in Matlab. These changes will make the language easier to write high-performance, general code, while, with some care not making linear algebra harder.

masklinn 11y ago
> Array slices will create views rather than copies. This is just a matter of performance Except for breaking any existing code which mutates slices (unless slices are COW?), and implicit sharing can have disastrous effects on memory usage.
- StefanKarpinski 11y ago
  How can implicit sharing have "disasterous effects on memory usage"? It will certainly break some code, hence the large amount of warning and the "pocalypse" appellation. Note that this is how array slices work in NumPy and that ecosystem has not collapsed.
  - masklinn 11y ago
    > How can implicit sharing have "disasterous effects on memory usage"? A small long-lived slices keeping a original arrays alive implicitly. Oracle recently undid the same optimisation in Java's strings because of that exact issue (which obviously broke code the other way around, code relying on cheap substrings would now take O(n) instead of O(1)). Erlang developers are also warned early and often about that behaviour in binaries.
    - btown 11y ago
      Very true. Less of a problem in scientific computing than web services, I think, since it's not common to keep slices around: you generally process a "frame" or an iteration of a simulation (Monte Carlo, etc.) and perhaps compare it to a previous frame, but the only history you keep beyond that are aggregates of the data, rather than individual slices that might go (say) in a global dictionary and stay around for a while. I don't think I've ever seen NumPy code where array slices were used as values in a long-living in-memory key-value store or cache... whereas you see that all the time with substrings in code for webservices. Thus it hasn't been as big a problem as in the Java community. Still something that should be in bold text in Julia 0.5's readmes, though; slicing is cheap now, but it comes with this very important caveat!
    - johnmyleswhite 11y ago
      The primary use case for Julia's coming array slices is to provide views into an array that you always intended to keep alive anyway. Iterating over the columns of an in-memory matrix, for example, is the use case that this change is meant to speed up.
      - masklinn 11y ago
        And so it was with Java's substring and is for Erlang's binaries sharing. I will acknowledge that array slices are less likely to cause problems than string slices, but it is still a possible issue for the unwary. Which was my point.
    - StefanKarpinski 11y ago
      Totally different typical usage patterns. Taking substrings and throwing away the original string is the norm for text. Taking a column of a matrix and throwing away the original matrix is not. Different domains call for different design tradeoffs – this change stems from extensive practical experience with writing numerical algorithms.
      - lqdc13 11y ago
        would there still be an easy way to make copies? Like advanced indexing (http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#advanced-indexing http://docs.scipy.org/doc/numpy/reference/arrays.indexing.ht...) in the Python world?
- sdevlin 11y ago
  The post you're responding to says explicitly in the first sentence that these are breaking changes.
  - masklinn 11y ago
    1. the comment I replied to stated — and this is the part I quoted and to which I was responding — that slicing creating views is just a matter of performance. 2. there are gradations in breaking changes, the breaking changes I outline are insidious because they are silent, they won't cause the program to fail loudly, but they will cause it to behave very differently than it used to and possibly to corrupt data.
unfamiliar 11y ago
This is great to hear. This should have been how arrays worked in the first place in my opinion. Julia is all round very good but currently, as someone looking to move from a Fortran codebase, the array implementation is lacking in many areas and the performance is bad due to implicit copies everywhere, allowing arbitrary number of indexes to get elements, temporary arrays created for array arithmetic, and temporary arrays inside functions having to be allocated and deallocated on every function call. All of which is a bit disappointing in a language which is trying to be the future of numerical computing. As it stands if you translate Fortran into Julia without a deep understanding of what is happening below the surface, your code will be much, much slower.

masklinn 11y ago

> Array slices will create views rather than copies. This is just a matter of performance Except for breaking any existing code which mutates slices (unless slices are COW?), and implicit sharing can have disastrous effects on memory usage.

StefanKarpinski 11y ago
How can implicit sharing have "disasterous effects on memory usage"? It will certainly break some code, hence the large amount of warning and the "pocalypse" appellation. Note that this is how array slices work in NumPy and that ecosystem has not collapsed.
- masklinn 11y ago
  > How can implicit sharing have "disasterous effects on memory usage"? A small long-lived slices keeping a original arrays alive implicitly. Oracle recently undid the same optimisation in Java's strings because of that exact issue (which obviously broke code the other way around, code relying on cheap substrings would now take O(n) instead of O(1)). Erlang developers are also warned early and often about that behaviour in binaries.
  - btown 11y ago
    Very true. Less of a problem in scientific computing than web services, I think, since it's not common to keep slices around: you generally process a "frame" or an iteration of a simulation (Monte Carlo, etc.) and perhaps compare it to a previous frame, but the only history you keep beyond that are aggregates of the data, rather than individual slices that might go (say) in a global dictionary and stay around for a while. I don't think I've ever seen NumPy code where array slices were used as values in a long-living in-memory key-value store or cache... whereas you see that all the time with substrings in code for webservices. Thus it hasn't been as big a problem as in the Java community. Still something that should be in bold text in Julia 0.5's readmes, though; slicing is cheap now, but it comes with this very important caveat!
  - johnmyleswhite 11y ago
    The primary use case for Julia's coming array slices is to provide views into an array that you always intended to keep alive anyway. Iterating over the columns of an in-memory matrix, for example, is the use case that this change is meant to speed up.
    - masklinn 11y ago
      And so it was with Java's substring and is for Erlang's binaries sharing. I will acknowledge that array slices are less likely to cause problems than string slices, but it is still a possible issue for the unwary. Which was my point.
  - StefanKarpinski 11y ago
    Totally different typical usage patterns. Taking substrings and throwing away the original string is the norm for text. Taking a column of a matrix and throwing away the original matrix is not. Different domains call for different design tradeoffs – this change stems from extensive practical experience with writing numerical algorithms.
    - lqdc13 11y ago
      would there still be an easy way to make copies? Like advanced indexing (http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#advanced-indexing http://docs.scipy.org/doc/numpy/reference/arrays.indexing.ht...) in the Python world?
sdevlin 11y ago
The post you're responding to says explicitly in the first sentence that these are breaking changes.
- masklinn 11y ago
  1. the comment I replied to stated — and this is the part I quoted and to which I was responding — that slicing creating views is just a matter of performance. 2. there are gradations in breaking changes, the breaking changes I outline are insidious because they are silent, they won't cause the program to fail loudly, but they will cause it to behave very differently than it used to and possibly to corrupt data.

unfamiliar 11y ago

This is great to hear. This should have been how arrays worked in the first place in my opinion. Julia is all round very good but currently, as someone looking to move from a Fortran codebase, the array implementation is lacking in many areas and the performance is bad due to implicit copies everywhere, allowing arbitrary number of indexes to get elements, temporary arrays created for array arithmetic, and temporary arrays inside functions having to be allocated and deallocated on every function call. All of which is a bit disappointing in a language which is trying to be the future of numerical computing. As it stands if you translate Fortran into Julia without a deep understanding of what is happening below the surface, your code will be much, much slower.