6 ms·

DumPy: NumPy except it's OK if you're dum

174 points by RebelPotato 1y ago

the__alchemist 1y ago
Clear articulation of why Numpy syntax is provincial and difficult to learn. Perhaps the clearest part of it clicks when the author compares the triple-nested loop to the numpy-function approach. The former is IMO (And the author thinks so) much easier to understand, and more universal.
Gimpei 1y ago
I’ve known some people who didn’t want to learn the syntax of numpy and did it all in loops, and the code was not easy to read. It was harder to read. The fundamental issue is that operations on high dimensional arrays are very difficult to reason about. Numpy can probably be improved, but I don’t think loops are the answer.
- okigan 1y ago
  What’s a better syntax then?
  - tikhonj 1y ago
    The real question—to which I have absolutely no answer—is not about syntax, it's about concepts: what is a better way to think about higher-dimensional arrays rather than loops and indices? I'm convinced that something better exists and, if it existed, encoding it in a sufficiently expressive (ie probably not-Python) language would give us the corresponding syntax, but trying to come up with a better syntax without a better conceptual model won't get us very far. Then again, maybe even that is wrong! "Notation as a tool for thought" and all that. Maybe "dimension-munging" in APL really is the best way to do these things, once you really understand it.
    - bee_rider 1y ago
      Numpy seems somewhat constrained here… it grew out of the matrix ecosystem, and matrices map naturally to two-dimensional arrays (sidenote: it’s super annoying that we have n-dimensional matrices and n-dimensional arrays, but the matrix dimension maps to the width of the array). Anyway, the general problem of having an n-dimensional array and wanting to dynamically… I dunno, it is a little tricky. But, sometimes when I see the examples people pop up with, I wonder how much pressure could be relieved if we just had a nice way of expressing operations on block or partitioned matrices. Like the canonical annoying example of wanting to apply solve using a series of small-ish matrices on a series of vectors, that’s just a block diagonal matrix…
  - CamperBob2 1y ago
    English. "Write me a Python function or program that does X, Y, and Z on U and V using W." That will be the inevitable outcome of current trends, where relatively-primitive AI tools are used to write slightly more sophisticated code than would otherwise be written, which in turn is used to create slightly less-primitive AI tools. For example, I just cut-and-pasted the author's own cri de coeur into Claude: https://claude.ai/share/1d750315-bffa-434b-a7e8-fb4d739ac89a https://claude.ai/share/1d750315-bffa-434b-a7e8-fb4d739ac89a Presumably at least one of the vectorized versions it replied with will work, although none is identical to the author's version. When this cycle ends, high-level programs and functions will be as incomprehensible to most mainstream developers as assembly is today. Today's specs are tomorrow's programs. Not a bad thing, really. And overdue, as the article makes all too clear. But the transition will be a dizzying one, with plenty of collateral disruption along the way.
- willseth 1y ago
  why_not_both.gif
- breppp 1y ago
  I've read the article and it didn't seem to me the author is suggesting loops
- dahart 1y ago
  The point here is not that it’s loops per se, the point is that the indexing is explicit. It seems like a big win to me. The article’s ~10 non-trivial examples all make the code easier to read, and more importantly, to understand exactly what the code is doing. It is true that some operations are difficult to reason about, that’s where explicit indexing really helps. The article resonates with me because I do want to learn numpy syntax, I’ve written hundreds of programs with nympy, spent countless hours doing battle with it, and I feel like I’m no better off now than someone who’s brand new to it. The indexing is constantly confounding, nothing ever just works. Anytime you see “None” and “axis=“ inside an operation, it’s a tell: bound to be difficult to comprehend. I’m always having to guess how to use some combination of reshape, dstack, hstack, transpose, and five other shape changers I’m forgetting, just to get something to work and it’s difficult to read and understand later. It feels like there is no debugging, only rewriting. I keep reading the manual for einsum over again and I’ve used it, but I can’t explain how, why, or when to use it, it seems like this thing you have to resort to because no other indexing seems to work. The ability to do straightforward explicit non-clever indexing as if you were writing loops seems like a pretty big step forward.
  - collingreen 1y ago
    I involuntarily whispered "reshape" to myself near the top of your comment. Numpy is a very different way for me to think and I have similar feelings to what you're describing.
  - cl3misch 1y ago
    I could never understand why people use dstack, hstack and the like. I think plain np.stack and specifying the axis explicitely is easier to write and to read. For transposes, np.einsum can be easier to read as it let's you use (single character, admittedly) axis specifiers to "name" them.
okigan 1y ago
Fantastic article. I don’t use numpy often enough - but this explains the many WTF moments why it’s so annoying to get numpy pieces to work together.
crescit_eundo 1y ago
Dupe. Posted a number of times the past day and a half: https://news.ycombinator.com/item?id=44072775 https://news.ycombinator.com/item?id=44072775 https://news.ycombinator.com/item?id=44063553 https://news.ycombinator.com/item?id=44063553 https://news.ycombinator.com/item?id=44078019 https://news.ycombinator.com/item?id=44078019 https://news.ycombinator.com/item?id=44063490 https://news.ycombinator.com/item?id=44063490
- deleted 1y ago
  [deleted]
- homarp 1y ago
  but only https://news.ycombinator.com/item?id=44063490 https://news.ycombinator.com/item?id=44063490 has 'some' comments. so this current discussion is better
  - gus_massa 1y ago
    There is an "older" discussion with a different title: https://news.ycombinator.com/item?id=43996431 https://news.ycombinator.com/item?id=43996431 (488 points | 9 days ago | 212 comments)
    - palmtree3000 1y ago
      That's a different post.
      - gus_massa 1y ago
        You are right. It's a 1 letter difference in the URL and I missed it. Sorry for the noise.
kevmo314 1y ago
I’m not convinced by the loops proposal. TensorFlow had this kind of lazy evaluation (except I guess TF was the worst of both worlds) and it makes debugging very difficult to the point that I believe it’s the main reason PyTorch won out. Such systems are great if they work perfectly but they never do. NumPy definitely has some rough edges, I sympathize with the frustrations for sure.
mpascale00 1y ago
In the way that `ggplot2` tries to abstract those common "high dimensional" graphing components into an intuitive grammar, such that you may in many places be able to guess the sequence of commands correctly, I would love to see an equivalent ergonomic notation. This gets part way there by acknowledging the problem. Mathematical operations aren't obliged to respect the Zen of Python, but I like the thought that we could make it so that most have an obvious expression.
shiandow 1y ago
That is amazing. My main doubt would be how future proof this is. Does it wrap numpy? Or something equivalent? Does it require continuous development to keep up? Also I both understand the need for declaring the matrices up front and think it's a bit of a shame that it is not seamless. Here are some (ill-advised?) alternatives: X, Y, Z = dp.Slots() with dp.Slot() as X: ... import dp.Slot as new X = new['i','j'] = ... X = dp['i','j'] = ...
- darepublic 1y ago
  Need transpilation rather than relying on this library being present. I like the idea alot though
- dynm 1y ago
  (author here) This wraps JAX and JAX's version of NumPy. It would surely require some development to keep up, although it's quite short and simple (only 700 lines), so I don't think it would be a big burden. That said, I should be clear that my goal here is just to show that this is possible/easy, and possibly inspire existing array packages to consider adding this kind of syntax. I like your alternatives! I agree that having to write X = dp.Slot() before assigning to X is unfortunate. I settled on the current syntax mostly just because I thought it was the choice that made it most "obvious" what was happening under the hood. If you really want to, you could use the walrus operator and write (X := dp.Slot())['i','j'] = ... but this cure seems worse than the disease... Actually, doing something like X = new['i','j'] = ... could simplify the implementation. Currently, a Slot has to act both like an Array and a Slot, which is slightly awkward. If new is a special object, it could just return an Array, which would be a bit nicer.
turtletontine 1y ago
Pretty sure Numpy’s einsum[1] function allows all of this reasoning in vanilla numpy (albeit with a different interface that I assume this author likes less than theirs). Quite sure that first example of how annoying numpy can be could be written much simpler with einsum. [1]: https://numpy.org/doc/stable/reference/generated/numpy.einsum.html https://numpy.org/doc/stable/reference/generated/numpy.einsu...
- jampekka 1y ago
  Sure, but einsum needs a syntax and concepts of its own, and more importantly does not work if you need to do something else than a limited set of matrix operations.
- sottol 1y ago
  The author posted a previous article about why they don't like numpy and his problems with einsum: https://dynomight.net/numpy/ https://dynomight.net/numpy/
  - adgjlsfhk1 1y ago
    I think you could for linear solves into a "generalized Einstein notation", but the other option you have is to support more complex array types, in which case a batched linearsolve can be reframed as a single linearsolve by a block-diagonal matrix
- yorwba 1y ago
  How do you invert a matrix with einsum?
  - BeetleB 1y ago
    Why do you need to invert a matrix?
    - yorwba 1y ago
      It's what the "first example of how annoying numpy can be" does, using either np.linalg.solve in a loop or a cursed multidimensional index rearrangement.
ltbarcly3 1y ago
The kind of person with the background to need these operations, and who is working on the kinds of problems where this stuff comes up, is more than capable of learning numpys syntax. Its not that bad. Avoiding the special purpose tools designed by and used by the people who work on these problems every day is the instinct of someone who has just started needing them and wants to solve new kinds of problems with the tools they already know. But the people with experience don't do it that way (for a reason), you need to learn what the current best solution is before you can transcend it. Guys, every few months some well meaning person posts their great new library here that "fixes" something to remove all that pesky complexity. Invariably its made by some beginner with whatever it is who doesn't understand why the complexity exists, and we never hear of them or the library again. This is absolutely one of those times.
- lblume 1y ago
  I agree that this may sometimes be the case, but in this case the author does seem to have very strong points in favor of the proposal. I also think that the accusation of dynomight not understanding NumPy enough is unjustified based on the points given.
- Tarq0n 1y ago
  This is how terrible API's are kept in use, wasting professional's time and making the field way less accessible for newcomers. Compare ggplot to matplotlib. They both serve the same purpose but one has an API you can actually reason about and use without needing to consult the documentation every five minutes.
  - ltbarcly3 1y ago
    matplotlib is just matlab. its a free matlab implementation. if you don't know matlab then absolutely don't use matplotlib, its a terrible library except that one thing. This is what I'm talking about though, you are complaining about something without knowing why it is the way it is.
jampekka 1y ago
Lots to like here but I'm not so sure about this: > In DumPy, every time you index an array or assign to a dp.Slot, it checks that all indices have been included. Not having to specify all indices makes for more generic implementations. Sure, the broadcasting rules could be simpler and more consistent, but in the meantime (implicit) broadcasting is what makes NumPy so powerful and flexible. Also I think straight up vmap would be cleaner IF Python did not intentionally make lambdas/FP so restricted and cumbersome apparently due to some emotional reasons.
- cyanydeez 1y ago
  Implicir means write once easy, debug, extend, read hard.
  - jampekka 1y ago
    It also means that you don't have to reimplement the same things all over again when your dimensions change.
    - cyanydeez 1y ago
      assuming you understand how the inconsist broadcast works, which is part of the problem.
- thanhhaimai 1y ago
  For solo works, the terseness might work, but usually only in short term. Code I wrote 6 months ago looks like someone else's code. For team work, I'd prefer to be explicit if possible. It saves both my teammates' time and my time (when I eventually forget my own code 6 months from now).
  - jampekka 1y ago
    It's not (only) about terseness. It's about generality.
- deleted 1y ago
  [deleted]
netbioserror 1y ago
I think this sort of DSL construction is a perfect fit for languages with macros: Lisp, Nim, etc. I spend a lot of time on both so I might explore the possibilities. What should a higher-dimensional array indexing, looping, and broadcasting syntax even look like, if until now it's just been cludges? Is it just APL but with actual words?
- mlochbaum 1y ago
  Author of BQN here, I agree with how section "What about APL?" describes the APL family as not fundamentally better (although details like indexing are often less messy). I outlined a system with lexically-scoped named axes at https://gist.github.com/mlochbaum/401e379ff09d422e2761e16fedbcd506 https://gist.github.com/mlochbaum/401e379ff09d422e2761e16fed... . The linear algebra example would end up something like this: solve(X[i,_], Y[j,_], A[i,j,_,_]) = over i, j/+: Y * linalg_solve(A, X)
- adgjlsfhk1 1y ago
  hear me out: Julia.
bbminner 1y ago
Always wanted to experiment with a syntax like this myself, thanks to author! I completely agree with you reasoning re complexity of mental model for indexing vs broadcasting. Moreover, it appears to me that such a representation should allow for finding more optimal low level impls (something like deeper "out of order" op fusing, idk)? I've seen a paper from either nvidia or meta around five years ago doing exactly that - translating an index-based meta-language built on top of python into cuda kernels (usually several variants and picking the best), can't find the reference unfortunately.
bee_rider 1y ago
It seems like a neat idea. If it can just be layered on top of Jax pretty easily… I dunno, seems so simple it might actually get traction? I wish I could peek at the alternative universe where Numpy just didn’t include broadcasting. Broadcasting is a sort of ridiculous idea. Trying to multiply a NxM matrix by a 1x1 matrix… should return an error, not perform some other operation totally unrelated to matrix multiplication!
- jampekka 1y ago
  Broadcasting is an excellent idea. Implementations do have warts, but e.g. pytorch would be really painful without it. Broadcasting is a sort of generalization of the idea of scalar-matrix product. You could make that less "ridiculous" by requiring a Hadamard product with a constant value matrix instead.
  - bee_rider 1y ago
    Scalar-matrix multiplication doesn’t seem that weird; automatically converting a 1x1 matrix to a scalar is the weird seeming thing, IMO.
duchenne 1y ago
Looks awesome. Can we get the same thing for pytorch?
joshjob42 1y ago
I think I'll just use Julia, since it has great libraries for doing matrix operations and mapping loops etc onto GPUs and handling indices etc. And if you need some Python library it's easily available using the fantastic PyCall library. This is a big improvement over numpy, but I don't see much of a compelling reason to go back to Python.
- jampekka 1y ago
  > I don't see much of a compelling reason to go back to Python. Being able to use sane programming workflows is quite a compelling reason.
  - adgjlsfhk1 1y ago
    what sane programming workflows do you find Julia lacking?
    - jampekka 1y ago
      Running programs.
      - odo1242 1y ago
        Isn’t it just “julia script-name.jl”?
        adgjlsfhk1 1y ago
        Yes. Yes it is.
3eb7988a1663 1y ago
Minor style complaint, but the code font is really light.
credit_guy 1y ago
I don't think this is such a great solution. With regular numpy you can create functions that take arrays with arbitrary shapes. It is not easy, as the author explained in a prior post, but it is doable. But with loops this just doesn't work. The number of nested loops depends on the shape of the inputs. My solution, very, very inelegant, was to focus on the scalar case, and ask Copilot to vectorize my code. I did that just the last two days. It is not all that easy. Copilot will give you something, but you still need to tweak it. In the end, I still had to write line by line the code, and make sure I understand what it is doing.
culebron21 1y ago
I'm currently working on a tiny project relying on NumPy and agree it's too complicated, and it's hard to reason and to read your own code later. But the proposal is very questionable. I like loops instead of broadcast (apply_...), but making them this kind of black magic just adds equal amount of mental load. As for compiled languages - they do optimize for SIMD, if you're ok running on CPU. Optimizing for CUDA and GPU parallelization... that's quite a niche demand, I think, to justify inconveniences for the more common users.
cl3misch 1y ago
How does this compare to xarray, which also introduces named axes on numpy arrays? https://github.com/pydata/xarray https://github.com/pydata/xarray
cl3misch 1y ago
> In NumPy, AB works if A and B are both scalar. Or if A is 5×1×6 and B is 5×1×6×1. But not if A is 1×5×6 and B is 1×5×6×1. Huh? > So, I removed it. In DumPy you can only do AB if one of A or B is scalar or A and B have exactly the same shape. That’s it, anything else raises an error. Instead, use indices, so it’s clear what you’re doing. I also dislike the seemingly inconsistent adding of last axes with length 1. I do like the capability that existing axes of length 1 will be broadcasted to any length of the other array in the operation, and use that all the time. If I understand the text correctly, the author removed that as well.

174 points by RebelPotato 1y ago

Clear articulation of why Numpy syntax is provincial and difficult to learn. Perhaps the clearest part of it clicks when the author compares the triple-nested loop to the numpy-function approach. The former is IMO (And the author thinks so) much easier to understand, and more universal.

Gimpei 1y ago

I’ve known some people who didn’t want to learn the syntax of numpy and did it all in loops, and the code was not easy to read. It was harder to read. The fundamental issue is that operations on high dimensional arrays are very difficult to reason about. Numpy can probably be improved, but I don’t think loops are the answer.

okigan 1y ago
What’s a better syntax then?
- tikhonj 1y ago
  The real question—to which I have absolutely no answer—is not about syntax, it's about concepts: what is a better way to think about higher-dimensional arrays rather than loops and indices? I'm convinced that something better exists and, if it existed, encoding it in a sufficiently expressive (ie probably not-Python) language would give us the corresponding syntax, but trying to come up with a better syntax without a better conceptual model won't get us very far. Then again, maybe even that is wrong! "Notation as a tool for thought" and all that. Maybe "dimension-munging" in APL really is the best way to do these things, once you really understand it.
  - bee_rider 1y ago
    Numpy seems somewhat constrained here… it grew out of the matrix ecosystem, and matrices map naturally to two-dimensional arrays (sidenote: it’s super annoying that we have n-dimensional matrices and n-dimensional arrays, but the matrix dimension maps to the width of the array). Anyway, the general problem of having an n-dimensional array and wanting to dynamically… I dunno, it is a little tricky. But, sometimes when I see the examples people pop up with, I wonder how much pressure could be relieved if we just had a nice way of expressing operations on block or partitioned matrices. Like the canonical annoying example of wanting to apply solve using a series of small-ish matrices on a series of vectors, that’s just a block diagonal matrix…
- CamperBob2 1y ago
  English. "Write me a Python function or program that does X, Y, and Z on U and V using W." That will be the inevitable outcome of current trends, where relatively-primitive AI tools are used to write slightly more sophisticated code than would otherwise be written, which in turn is used to create slightly less-primitive AI tools. For example, I just cut-and-pasted the author's own cri de coeur into Claude: https://claude.ai/share/1d750315-bffa-434b-a7e8-fb4d739ac89a https://claude.ai/share/1d750315-bffa-434b-a7e8-fb4d739ac89a Presumably at least one of the vectorized versions it replied with will work, although none is identical to the author's version. When this cycle ends, high-level programs and functions will be as incomprehensible to most mainstream developers as assembly is today. Today's specs are tomorrow's programs. Not a bad thing, really. And overdue, as the article makes all too clear. But the transition will be a dizzying one, with plenty of collateral disruption along the way.
willseth 1y ago
why_not_both.gif
breppp 1y ago
I've read the article and it didn't seem to me the author is suggesting loops
dahart 1y ago
The point here is not that it’s loops per se, the point is that the indexing is explicit. It seems like a big win to me. The article’s ~10 non-trivial examples all make the code easier to read, and more importantly, to understand exactly what the code is doing. It is true that some operations are difficult to reason about, that’s where explicit indexing really helps. The article resonates with me because I do want to learn numpy syntax, I’ve written hundreds of programs with nympy, spent countless hours doing battle with it, and I feel like I’m no better off now than someone who’s brand new to it. The indexing is constantly confounding, nothing ever just works. Anytime you see “None” and “axis=“ inside an operation, it’s a tell: bound to be difficult to comprehend. I’m always having to guess how to use some combination of reshape, dstack, hstack, transpose, and five other shape changers I’m forgetting, just to get something to work and it’s difficult to read and understand later. It feels like there is no debugging, only rewriting. I keep reading the manual for einsum over again and I’ve used it, but I can’t explain how, why, or when to use it, it seems like this thing you have to resort to because no other indexing seems to work. The ability to do straightforward explicit non-clever indexing as if you were writing loops seems like a pretty big step forward.
- collingreen 1y ago
  I involuntarily whispered "reshape" to myself near the top of your comment. Numpy is a very different way for me to think and I have similar feelings to what you're describing.
- cl3misch 1y ago
  I could never understand why people use dstack, hstack and the like. I think plain np.stack and specifying the axis explicitely is easier to write and to read. For transposes, np.einsum can be easier to read as it let's you use (single character, admittedly) axis specifiers to "name" them.

okigan 1y ago

Fantastic article. I don’t use numpy often enough - but this explains the many WTF moments why it’s so annoying to get numpy pieces to work together.

crescit_eundo 1y ago

Dupe. Posted a number of times the past day and a half: https://news.ycombinator.com/item?id=44072775 https://news.ycombinator.com/item?id=44072775 https://news.ycombinator.com/item?id=44063553 https://news.ycombinator.com/item?id=44063553 https://news.ycombinator.com/item?id=44078019 https://news.ycombinator.com/item?id=44078019 https://news.ycombinator.com/item?id=44063490 https://news.ycombinator.com/item?id=44063490

deleted 1y ago
[deleted]
homarp 1y ago
but only https://news.ycombinator.com/item?id=44063490 https://news.ycombinator.com/item?id=44063490 has 'some' comments. so this current discussion is better
- gus_massa 1y ago
  There is an "older" discussion with a different title: https://news.ycombinator.com/item?id=43996431 https://news.ycombinator.com/item?id=43996431 (488 points | 9 days ago | 212 comments)
  - palmtree3000 1y ago
    That's a different post.
    - gus_massa 1y ago
      You are right. It's a 1 letter difference in the URL and I missed it. Sorry for the noise.

kevmo314 1y ago

I’m not convinced by the loops proposal. TensorFlow had this kind of lazy evaluation (except I guess TF was the worst of both worlds) and it makes debugging very difficult to the point that I believe it’s the main reason PyTorch won out. Such systems are great if they work perfectly but they never do. NumPy definitely has some rough edges, I sympathize with the frustrations for sure.

mpascale00 1y ago

In the way that `ggplot2` tries to abstract those common "high dimensional" graphing components into an intuitive grammar, such that you may in many places be able to guess the sequence of commands correctly, I would love to see an equivalent ergonomic notation. This gets part way there by acknowledging the problem. Mathematical operations aren't obliged to respect the Zen of Python, but I like the thought that we could make it so that most have an obvious expression.

shiandow 1y ago

That is amazing. My main doubt would be how future proof this is. Does it wrap numpy? Or something equivalent? Does it require continuous development to keep up? Also I both understand the need for declaring the matrices up front and think it's a bit of a shame that it is not seamless. Here are some (ill-advised?) alternatives: X, Y, Z = dp.Slots() with dp.Slot() as X: ... import dp.Slot as new X = new['i','j'] = ... X = dp['i','j'] = ...

darepublic 1y ago
Need transpilation rather than relying on this library being present. I like the idea alot though
dynm 1y ago
(author here) This wraps JAX and JAX's version of NumPy. It would surely require some development to keep up, although it's quite short and simple (only 700 lines), so I don't think it would be a big burden. That said, I should be clear that my goal here is just to show that this is possible/easy, and possibly inspire existing array packages to consider adding this kind of syntax. I like your alternatives! I agree that having to write X = dp.Slot() before assigning to X is unfortunate. I settled on the current syntax mostly just because I thought it was the choice that made it most "obvious" what was happening under the hood. If you really want to, you could use the walrus operator and write (X := dp.Slot())['i','j'] = ... but this cure seems worse than the disease... Actually, doing something like X = new['i','j'] = ... could simplify the implementation. Currently, a Slot has to act both like an Array and a Slot, which is slightly awkward. If new is a special object, it could just return an Array, which would be a bit nicer.

turtletontine 1y ago

Pretty sure Numpy’s einsum[1] function allows all of this reasoning in vanilla numpy (albeit with a different interface that I assume this author likes less than theirs). Quite sure that first example of how annoying numpy can be could be written much simpler with einsum. [1]: https://numpy.org/doc/stable/reference/generated/numpy.einsum.html https://numpy.org/doc/stable/reference/generated/numpy.einsu...

jampekka 1y ago
Sure, but einsum needs a syntax and concepts of its own, and more importantly does not work if you need to do something else than a limited set of matrix operations.
sottol 1y ago
The author posted a previous article about why they don't like numpy and his problems with einsum: https://dynomight.net/numpy/ https://dynomight.net/numpy/
- adgjlsfhk1 1y ago
  I think you could for linear solves into a "generalized Einstein notation", but the other option you have is to support more complex array types, in which case a batched linearsolve can be reframed as a single linearsolve by a block-diagonal matrix
yorwba 1y ago
How do you invert a matrix with einsum?
- BeetleB 1y ago
  Why do you need to invert a matrix?
  - yorwba 1y ago
    It's what the "first example of how annoying numpy can be" does, using either np.linalg.solve in a loop or a cursed multidimensional index rearrangement.

ltbarcly3 1y ago

The kind of person with the background to need these operations, and who is working on the kinds of problems where this stuff comes up, is more than capable of learning numpys syntax. Its not that bad. Avoiding the special purpose tools designed by and used by the people who work on these problems every day is the instinct of someone who has just started needing them and wants to solve new kinds of problems with the tools they already know. But the people with experience don't do it that way (for a reason), you need to learn what the current best solution is before you can transcend it. Guys, every few months some well meaning person posts their great new library here that "fixes" something to remove all that pesky complexity. Invariably its made by some beginner with whatever it is who doesn't understand why the complexity exists, and we never hear of them or the library again. This is absolutely one of those times.

lblume 1y ago
I agree that this may sometimes be the case, but in this case the author does seem to have very strong points in favor of the proposal. I also think that the accusation of dynomight not understanding NumPy enough is unjustified based on the points given.
Tarq0n 1y ago
This is how terrible API's are kept in use, wasting professional's time and making the field way less accessible for newcomers. Compare ggplot to matplotlib. They both serve the same purpose but one has an API you can actually reason about and use without needing to consult the documentation every five minutes.
- ltbarcly3 1y ago
  matplotlib is just matlab. its a free matlab implementation. if you don't know matlab then absolutely don't use matplotlib, its a terrible library except that one thing. This is what I'm talking about though, you are complaining about something without knowing why it is the way it is.

jampekka 1y ago

Lots to like here but I'm not so sure about this: > In DumPy, every time you index an array or assign to a dp.Slot, it checks that all indices have been included. Not having to specify all indices makes for more generic implementations. Sure, the broadcasting rules could be simpler and more consistent, but in the meantime (implicit) broadcasting is what makes NumPy so powerful and flexible. Also I think straight up vmap would be cleaner IF Python did not intentionally make lambdas/FP so restricted and cumbersome apparently due to some emotional reasons.

cyanydeez 1y ago
Implicir means write once easy, debug, extend, read hard.
- jampekka 1y ago
  It also means that you don't have to reimplement the same things all over again when your dimensions change.
  - cyanydeez 1y ago
    assuming you understand how the inconsist broadcast works, which is part of the problem.
thanhhaimai 1y ago
For solo works, the terseness might work, but usually only in short term. Code I wrote 6 months ago looks like someone else's code. For team work, I'd prefer to be explicit if possible. It saves both my teammates' time and my time (when I eventually forget my own code 6 months from now).
- jampekka 1y ago
  It's not (only) about terseness. It's about generality.
deleted 1y ago
[deleted]

netbioserror 1y ago

I think this sort of DSL construction is a perfect fit for languages with macros: Lisp, Nim, etc. I spend a lot of time on both so I might explore the possibilities. What should a higher-dimensional array indexing, looping, and broadcasting syntax even look like, if until now it's just been cludges? Is it just APL but with actual words?

mlochbaum 1y ago
Author of BQN here, I agree with how section "What about APL?" describes the APL family as not fundamentally better (although details like indexing are often less messy). I outlined a system with lexically-scoped named axes at https://gist.github.com/mlochbaum/401e379ff09d422e2761e16fedbcd506 https://gist.github.com/mlochbaum/401e379ff09d422e2761e16fed... . The linear algebra example would end up something like this: solve(X[i,_], Y[j,_], A[i,j,_,_]) = over i, j/+: Y * linalg_solve(A, X)
adgjlsfhk1 1y ago
hear me out: Julia.

bbminner 1y ago

Always wanted to experiment with a syntax like this myself, thanks to author! I completely agree with you reasoning re complexity of mental model for indexing vs broadcasting. Moreover, it appears to me that such a representation should allow for finding more optimal low level impls (something like deeper "out of order" op fusing, idk)? I've seen a paper from either nvidia or meta around five years ago doing exactly that - translating an index-based meta-language built on top of python into cuda kernels (usually several variants and picking the best), can't find the reference unfortunately.

bee_rider 1y ago

It seems like a neat idea. If it can just be layered on top of Jax pretty easily… I dunno, seems so simple it might actually get traction? I wish I could peek at the alternative universe where Numpy just didn’t include broadcasting. Broadcasting is a sort of ridiculous idea. Trying to multiply a NxM matrix by a 1x1 matrix… should return an error, not perform some other operation totally unrelated to matrix multiplication!

jampekka 1y ago
Broadcasting is an excellent idea. Implementations do have warts, but e.g. pytorch would be really painful without it. Broadcasting is a sort of generalization of the idea of scalar-matrix product. You could make that less "ridiculous" by requiring a Hadamard product with a constant value matrix instead.
- bee_rider 1y ago
  Scalar-matrix multiplication doesn’t seem that weird; automatically converting a 1x1 matrix to a scalar is the weird seeming thing, IMO.

duchenne 1y ago

Looks awesome. Can we get the same thing for pytorch?

joshjob42 1y ago

I think I'll just use Julia, since it has great libraries for doing matrix operations and mapping loops etc onto GPUs and handling indices etc. And if you need some Python library it's easily available using the fantastic PyCall library. This is a big improvement over numpy, but I don't see much of a compelling reason to go back to Python.

jampekka 1y ago
> I don't see much of a compelling reason to go back to Python. Being able to use sane programming workflows is quite a compelling reason.
- adgjlsfhk1 1y ago
  what sane programming workflows do you find Julia lacking?
  - jampekka 1y ago
    Running programs.
    - odo1242 1y ago
      Isn’t it just “julia script-name.jl”?
      - adgjlsfhk1 1y ago
        Yes. Yes it is.

3eb7988a1663 1y ago

Minor style complaint, but the code font is really light.

credit_guy 1y ago

I don't think this is such a great solution. With regular numpy you can create functions that take arrays with arbitrary shapes. It is not easy, as the author explained in a prior post, but it is doable. But with loops this just doesn't work. The number of nested loops depends on the shape of the inputs. My solution, very, very inelegant, was to focus on the scalar case, and ask Copilot to vectorize my code. I did that just the last two days. It is not all that easy. Copilot will give you something, but you still need to tweak it. In the end, I still had to write line by line the code, and make sure I understand what it is doing.

culebron21 1y ago

I'm currently working on a tiny project relying on NumPy and agree it's too complicated, and it's hard to reason and to read your own code later. But the proposal is very questionable. I like loops instead of broadcast (apply_...), but making them this kind of black magic just adds equal amount of mental load. As for compiled languages - they do optimize for SIMD, if you're ok running on CPU. Optimizing for CUDA and GPU parallelization... that's quite a niche demand, I think, to justify inconveniences for the more common users.

cl3misch 1y ago

How does this compare to xarray, which also introduces named axes on numpy arrays? https://github.com/pydata/xarray https://github.com/pydata/xarray

cl3misch 1y ago

> In NumPy, AB works if A and B are both scalar. Or if A is 5×1×6 and B is 5×1×6×1. But not if A is 1×5×6 and B is 1×5×6×1. Huh? > So, I removed it. In DumPy you can only do AB if one of A or B is scalar or A and B have exactly the same shape. That’s it, anything else raises an error. Instead, use indices, so it’s clear what you’re doing. I also dislike the seemingly inconsistent adding of last axes with length 1. I do like the capability that existing axes of length 1 will be broadcasted to any length of the other array in the operation, and use that all the time. If I understand the text correctly, the author removed that as well.