4 ms·

The thing I find most interesting about this language is that it's actually the original source of array/tensor broadcasting, which is now used pretty much ever

by jph00 3y ago

The thing I find most interesting about this language is that it's actually the original source of array/tensor broadcasting, which is now used pretty much everywhere (e.g. pytorch, numpy, tensorflow, etc...) https://twitter.com/jeremyphoward/status/1575312671288991744 https://twitter.com/jeremyphoward/status/1575312671288991744

bradrn 3y ago
Surely it dates back even further, to APL?
- semi-extrinsic 3y ago
  And the Numpy style of writing things goes back at least to Fortran 90: real a(32,32) real b(32) b = a(:,4)
  - jkafjanvnfaf 3y ago
    MATLAB had its first commercial releases in the mid-80s and had indexing syntax that looks pretty much the same. Since it got popular very quickly, I assume that it is the most immediate source of Fortran 90's indexing style. ALGOL 68 had some form of array slicing as well, but I'm not sure how influential it really was in this department.
- imurray 3y ago
  I don't know APL, but from what I found quickly, APL's "conformability" rules[1] only seem to be a subset of the Yorick/NumPy broadcasting? Here's a Python example of broadcasting that APL doesn't seem to support(?): subtracting arrays of shape (3,1,7,5) and (1,4,1,5) to get an array of shape (3,4,7,5). from numpy.random import randn randn(3,1,7,5) - randn(1,4,1,5) Inspired by other comments in this thread: The second line can also be run in current versions of Octave/Matlab as-is, BUT they didn't support it before NumPy, and neither did/does FORTRAN[2]. Mathworks resisted introducing this general form of broadcasting into Matlab for a long time. Octave then did it anyway (version 3.6.0 in 2012), and Matlab eventually followed (version R2016b). Before late 2016, Matlab code required `bsxfun(@minus, A, B)` instead of `A-B` to get broadcasting, and before bsxfun (introduced in R2007a) it was even more awkward (code often used `repmat`, or indexing tricks to explicitly expand arrays in memory to match shape). [1] https://aplwiki.com/wiki/Conformability https://aplwiki.com/wiki/Conformability [2] suggestion from 2022 to introduce broadcasting to Fortran: https://github.com/j3-fortran/fortran_proposals/issues/252 https://github.com/j3-fortran/fortran_proposals/issues/252
  - mlochbaum 3y ago
    There was a little discussion about this just Monday in the array programming chat[0]. Not many people commented but it doesn't look like anyone was in favor. I think the concern is that while broadcasting 1s is an easy mechanism to understand, it changes the problem from "how do I match up array axes" to "how to I get 1s in the right places in the shape", which is no easier to do, and further from the programmer's intent. The modern APL way is to use the Rank operator, which can specify any broadcasting pattern. The example above would be f⍤1⍤2 1⍤2 on shapes 3 7 5 and 4 5, although in practice these sorts of alternating broadcasts don't show up much and even two copies of Rank is rare. One quote from the chat: > a friend had written some code in—either numpy, or something else borrowing its model—and wanted to translate it to j. I could not figure out what she was trying to accomplish with the many seemingly-extraneous reshapes and transposes, until I found out what they did in numpy. The whole thing reduced to a single, trivial application of rank in j [0] https://aplwiki.com/wiki/APL_Farm https://aplwiki.com/wiki/APL_Farm, main or "Array languages" channel
    - imurray 3y ago
      Thanks for the pointer. I can believe that a language that looks so different will find that different patterns and primitives are natural for it. My experience from writing a lot of array-based code in NumPy/Matlab is that broadcasting absolutely has made it easier to write my code in those ecosystems. Axes of length 1 have often been in the right places already, or have been easy to insert. It's of course possible to create a big mess in any language; it seems likely that the NumPy code you saw could have been neater too. In machine learning there can be many array dimensions floating around: batch-dims, sequence and/or channel-dims, weight matrices, and so on. It can be necessary to expand two or more dimensions, and/or line up dimensions quite carefully. Einops[1] has emerged from that community as a tool to succinctly express many operations that involve lots of array dimensions. You're likely to bump into more and more people who've used it, and again it seems there's some overlap with what Rank does. (And again, you'll see uses of Einops in the wild that are unnecessarily convoluted.) [1] https://einops.rocks/ https://einops.rocks/ -- It works with all of the existing major array-based frameworks for Python (NumPy/PyTorch/Jax/etc), and the emerging array API standard for Python.

by jph00 3y ago

bradrn 3y ago

Surely it dates back even further, to APL?

semi-extrinsic 3y ago
And the Numpy style of writing things goes back at least to Fortran 90: real a(32,32) real b(32) b = a(:,4)
- jkafjanvnfaf 3y ago
  MATLAB had its first commercial releases in the mid-80s and had indexing syntax that looks pretty much the same. Since it got popular very quickly, I assume that it is the most immediate source of Fortran 90's indexing style. ALGOL 68 had some form of array slicing as well, but I'm not sure how influential it really was in this department.
imurray 3y ago
I don't know APL, but from what I found quickly, APL's "conformability" rules[1] only seem to be a subset of the Yorick/NumPy broadcasting? Here's a Python example of broadcasting that APL doesn't seem to support(?): subtracting arrays of shape (3,1,7,5) and (1,4,1,5) to get an array of shape (3,4,7,5). from numpy.random import randn randn(3,1,7,5) - randn(1,4,1,5) Inspired by other comments in this thread: The second line can also be run in current versions of Octave/Matlab as-is, BUT they didn't support it before NumPy, and neither did/does FORTRAN[2]. Mathworks resisted introducing this general form of broadcasting into Matlab for a long time. Octave then did it anyway (version 3.6.0 in 2012), and Matlab eventually followed (version R2016b). Before late 2016, Matlab code required `bsxfun(@minus, A, B)` instead of `A-B` to get broadcasting, and before bsxfun (introduced in R2007a) it was even more awkward (code often used `repmat`, or indexing tricks to explicitly expand arrays in memory to match shape). [1] https://aplwiki.com/wiki/Conformability https://aplwiki.com/wiki/Conformability [2] suggestion from 2022 to introduce broadcasting to Fortran: https://github.com/j3-fortran/fortran_proposals/issues/252 https://github.com/j3-fortran/fortran_proposals/issues/252
- mlochbaum 3y ago
  There was a little discussion about this just Monday in the array programming chat[0]. Not many people commented but it doesn't look like anyone was in favor. I think the concern is that while broadcasting 1s is an easy mechanism to understand, it changes the problem from "how do I match up array axes" to "how to I get 1s in the right places in the shape", which is no easier to do, and further from the programmer's intent. The modern APL way is to use the Rank operator, which can specify any broadcasting pattern. The example above would be f⍤1⍤2 1⍤2 on shapes 3 7 5 and 4 5, although in practice these sorts of alternating broadcasts don't show up much and even two copies of Rank is rare. One quote from the chat: > a friend had written some code in—either numpy, or something else borrowing its model—and wanted to translate it to j. I could not figure out what she was trying to accomplish with the many seemingly-extraneous reshapes and transposes, until I found out what they did in numpy. The whole thing reduced to a single, trivial application of rank in j [0] https://aplwiki.com/wiki/APL_Farm https://aplwiki.com/wiki/APL_Farm, main or "Array languages" channel
  - imurray 3y ago
    Thanks for the pointer. I can believe that a language that looks so different will find that different patterns and primitives are natural for it. My experience from writing a lot of array-based code in NumPy/Matlab is that broadcasting absolutely has made it easier to write my code in those ecosystems. Axes of length 1 have often been in the right places already, or have been easy to insert. It's of course possible to create a big mess in any language; it seems likely that the NumPy code you saw could have been neater too. In machine learning there can be many array dimensions floating around: batch-dims, sequence and/or channel-dims, weight matrices, and so on. It can be necessary to expand two or more dimensions, and/or line up dimensions quite carefully. Einops[1] has emerged from that community as a tool to succinctly express many operations that involve lots of array dimensions. You're likely to bump into more and more people who've used it, and again it seems there's some overlap with what Rank does. (And again, you'll see uses of Einops in the wild that are unnecessarily convoluted.) [1] https://einops.rocks/ https://einops.rocks/ -- It works with all of the existing major array-based frameworks for Python (NumPy/PyTorch/Jax/etc), and the emerging array API standard for Python.