11 ms·

Numpy is absurd

102 points by moelf 4y ago

stosto88 4y ago
Finally! It had to be said. Im a huge Python fan and a huge Pandas fan and have written hundreds of scripts. But numpy is so "R" esque and technical and stiff and unintuitive for sure.
deleted 4y ago
[deleted]
joshlk 4y ago
> I always gripe about Python not having built-in array type Python does have an array type. It’s existed from at least v2.6 (released 2008). https://docs.python.org/3.10/library/array.html https://docs.python.org/3.10/library/array.html
tpoacher 4y ago
Oh no! Undefined behaviour! What ever shall we do?
hprotagonist 4y ago
> I always gripe about Python not having built-in array type from array import array it’s right there. > Numpy doesn't distinguish "vector of vector" from "matrix" numpy.matrix though to be fair, it’s on its way out.
- moelf 4y ago
  when built-in array type is not performant nor serves as an interface (abstract class), it's as if it doesn't exist. I guess the point is Numpy predates Python's array by a long shot. Compare and contrast with languages with built-in N-dimensional array for scientific computing.
  - hprotagonist 4y ago
    like matlab, that lies?
    - moelf 4y ago
      Julia
    - tkuraku 4y ago
      What do you mean?
      - hprotagonist 4y ago
        in matlab: >> size(4) ans = 1 1 which is 1. wrong and 2. evidence that matlab cannot represent numerical objects of less than rank 2. Numpy can: >> np.shape(4) ()
        ModernMech 4y ago
        I'm confused by what you mean. In Matlab, everything is a matrix. A naked integer is a 1x1 matrix, hence why the size is reported as [1 1]. Those are the semantics of the language, not a lie.
        hprotagonist 4y ago
        a naked integer is not a 1x1 matrix, it’s a 0x0. Matlab silently promotes it to one, though, which is a deliberately misleading thing to do because it’s convenient for matlab’s data model. i think it’s fair to call that a (white) lie
        ModernMech 4y ago
        > a naked integer is not a 1x1 matrix, it’s a 0x0 Do you have a link to where this is asserted in the documentation? I'm confused as to the basis for this.
  - jwilk 4y ago
    > Numpy predates Python's array by a long shot I don't think so. The array module was included even in Python 1.0. Or did you mean to use another verb?
    - eesmith 4y ago
      To be more specific, "git log arraymodule.c" reports: commit 778983b48165da25ee11fb97f6855af7c67f4ff2 Author: Guido van Rossum <guido@python.org> Date: Fri Feb 19 15:55:02 1993 +0000 Added new module "array" (for now optional) defining array objects.
cuteboy19 4y ago
Who needs this strange operation anyways? Deprecate the whole method and call it a day
- moelf 4y ago
  that's what I'd go for, throw more errors when `__contains__` is called with ambiguity. but, cough, mature and stable scientific computing foundation?
judofyr 4y ago
Here's a better resource for understanding how __conains__ works which actually looks at the source/documentation instead of trying to understand it intuitively: https://stackoverflow.com/questions/18320624/how-does-contains-work-for-ndarrays https://stackoverflow.com/questions/18320624/how-does-contai... Answer: `a in b` is the same as `(a == b).any()`. This really only makes sense when `a` is a scalar and any behavior for other types of arrays are more "accidental" than designed.
- Someone 4y ago
  Even that took me a while to understand it and, I guess, is another example of Numpy’s absurdity. My initial thought was “None of the examples have arguments that are equal to each other, so that should evaluate to false.any()”, but that would mean all should return the same value, so that can’t be true, so I figured out that Numpy overloads/changes a == b to do item-by-item comparison on arrays, and return an array of booleans. But then, the statement “This really only makes sense when `a` is a scalar” doesn’t make sense to me. That seems to indicate Numpy also overloads a == b as a comparison of each element of a with a single value b and vice versa, but that doesn’t make sense, given that arrays can contain arrays, making it impossible to discriminate between [1,2] as a two-item sequence and [1,2] as a single-item sequence where the item is a two-item sequence. Reading https://stackoverflow.com/questions/10580676/comparing-two-numpy-arrays-for-equality-element-wise https://stackoverflow.com/questions/10580676/comparing-two-n... further strengthens my impression that one should avoid using == in Numpy.
- gnulinux 4y ago
  Well yeah you're right OP could have explained this better; but this argues their point. From the definition that `a in b` is `(a == b).any()` it's trivial to see things like `np.array([1, 0, 0]) in np.array([1, 2, 3])` will return `True` too, which is absurd, as the title says. I find it hard to believe when numpy team was designing the API they missed this.
  - kzrdude 4y ago
    It's trivial to see some things and other things use broadcasting and that's not quite trivial :)
imiric 4y ago
For a language with "explicit is better than implicit" as part of its mantra[1], Python is chock full of surprising behavior like this where the programmer really has no idea what's happening under the hood. I'm happy I moved on to Go, which actually adopts this as a core design tenet. [1]: Even though PEP-20 is largely a meme.
- qbasic_forever 4y ago
  To be fair this person's gripes are with the behavior of Numpy arrays, not anything that's native to Python's core language. Numpy arrays are tricky beasts because they try to balance fast operations on huge amounts of data, with expressive Python operator syntax.
  - ModernMech 4y ago
    Although, another way of looking at what you just said is that Numpy arrays exist because core Python fails to strike a balance between being both fast and expressive.
    - eesmith 4y ago
      That does not necessarily follow. Python may be striking a balance between the needs of numerical and non-numerical computing, while NumPy arrays do not need to consider non-numerical computing. Case in point, historically NumPy has been designed for long-running programs, like hour-long calculations or notebook processing. They've chosen to import "the world" so users don't need explicit subpackage imports, allowing: >>> import numpy >>> numpy.polynomial.hermite.hermfromroots([2,2,3,4]) array([ 7.075e+01, -4.625e+01, 1.175e+01, -1.375e+00, 6.250e-02]) This has a high-startup cost which makes NumPy use inappropriate for short-lived programs like shell scripts, even in cases where Python's own startup cost is otherwise appropriate.
ThePhysicist 4y ago
I've worked with numpy for years and I don't think I ever used the 'in' operator with arrays. What would be the use case for that? For me a larger gotcha was around object multiplication (i.e. element-wise vs. matrix multiplication) and how it is different based on whether you have a "matrix" or an "array" type (or a mix of both). That caused some pretty big headache in the beginning.
- weare138 4y ago
  But that just seems like a foot gun for the poor unfortunate soul that eventually does.
  - rowanG077 4y ago
    I don't think people who use python are concerned with footguns. Numpy is very much not at fault here.
    - DangitBobby 4y ago
      Why would you think that?
      - rowanG077 4y ago
        Because the typing system used in Python is a big footgun. If the basis of the language you are using encourages footguns then it's not surprising that a large library in that ecosystem doesn't care about that either.
  - sillysaurusx 4y ago
    Meh. Footguns are underrated. Let’s return to the Wild West of the 90s. I’m only partially joking. But this was one case where they probably thought it was a good idea, turned out to be a bad idea, then couldn’t turn it off without breaking a bunch of code.
    - rightbyte 4y ago
bo1024 4y ago
Interpreting “in” as “is an element of”, all of these are type errors. It only makes sense to ask if an X is in an Array<X>. In a language that lets you ask nonsense questions, expect nonsense answers.
- kzrdude 4y ago
  Numpy has the highly useful operation of `a < b` where a and b are arrays and the answer is an array of booleans - i.e broadcasted elementwise compare. They changed the language to change what question `<` asks.
  - bo1024 4y ago
    Sure, and languages like C++ let you do this kind of thing very broadly, but it has a lot of pitfalls. One of my biggest gripes with python is that function interfaces and "type signatures" are hard to discover even when documentation is good.
    - kzrdude 4y ago
      FWIW I think it's very hard to discover this from C++ code as well. It has to be documented.
  - kevin_thibedeau 4y ago
    This gets ugly with !=. Instead of 'a != b' you usually want 'not (a == b).all()'.
    - cuteboy19 4y ago
      It really doesn't if you think about it. a==b cannot by itself be used as an if condition. You would always need to use (a==b).all(). So it would be very difficult to make the mistake you are suggesting https://stackoverflow.com/questions/10062954/valueerror-the-truth-value-of-an-array-with-more-than-one-element-is-ambiguous https://stackoverflow.com/questions/10062954/valueerror-the-...
moelf 4y ago
Op here, I'd also like to say I blame this on Python's lack of broadcasting syntax, (dot in Matlab and Julia) Which forces Numpy to be designed in a way to automatically broadcast, guessing what user wants is hard in nested container
- johnmyleswhite 4y ago
  Julia's broadcasting syntax depends on having Julia-style compilation to be efficient. It's not clear to me Python could ever provide the same behavior.
dima55 4y ago
Core numpy is really weird, and has really sloppy logic around higher-dimensional objects. I've a module to work around lots of the warts. If somebody has smarter versions of intersect() or in() or whatever, contribute! https://github.com/dkogan/numpysane https://github.com/dkogan/numpysane
- cuteboy19 4y ago
  Apologies if this was explained in the talk, but why not upstream your changes to numpy?
  - dima55 4y ago
    No real good reason, honestly. I wasn't feeling positively about numpy devs at the moment, and didn't want to negotiate getting the code in. The library is solid though, and should be merged. I need somebody to pester me about it, to actualy work on that today.
kakadzhun 4y ago
import numpy as np print(np.__version__) a = np.array([1,2,3]) b = np.array([1]) print(a in b) print(b in a) print(np.isin(a, b)) print(np.isin(b, a)) print(np.isin(a.astype(float), b)) print(np.isin(a, b.astype(float)))) 1.21.6 True True [ True False False] [ True] [ True False False] [ True False False]

102 points by moelf 4y ago

Finally! It had to be said. Im a huge Python fan and a huge Pandas fan and have written hundreds of scripts. But numpy is so "R" esque and technical and stiff and unintuitive for sure.

deleted 4y ago

[deleted]

joshlk 4y ago

> I always gripe about Python not having built-in array type Python does have an array type. It’s existed from at least v2.6 (released 2008). https://docs.python.org/3.10/library/array.html https://docs.python.org/3.10/library/array.html

tpoacher 4y ago

Oh no! Undefined behaviour! What ever shall we do?

hprotagonist 4y ago

> I always gripe about Python not having built-in array type from array import array it’s right there. > Numpy doesn't distinguish "vector of vector" from "matrix" numpy.matrix though to be fair, it’s on its way out.

moelf 4y ago
when built-in array type is not performant nor serves as an interface (abstract class), it's as if it doesn't exist. I guess the point is Numpy predates Python's array by a long shot. Compare and contrast with languages with built-in N-dimensional array for scientific computing.
- hprotagonist 4y ago
  like matlab, that lies?
  - moelf 4y ago
    Julia
  - tkuraku 4y ago
    What do you mean?
    - hprotagonist 4y ago
      in matlab: >> size(4) ans = 1 1 which is 1. wrong and 2. evidence that matlab cannot represent numerical objects of less than rank 2. Numpy can: >> np.shape(4) ()
      - ModernMech 4y ago
        I'm confused by what you mean. In Matlab, everything is a matrix. A naked integer is a 1x1 matrix, hence why the size is reported as [1 1]. Those are the semantics of the language, not a lie.
        hprotagonist 4y ago
        a naked integer is not a 1x1 matrix, it’s a 0x0. Matlab silently promotes it to one, though, which is a deliberately misleading thing to do because it’s convenient for matlab’s data model. i think it’s fair to call that a (white) lie
        ModernMech 4y ago
        > a naked integer is not a 1x1 matrix, it’s a 0x0 Do you have a link to where this is asserted in the documentation? I'm confused as to the basis for this.
- jwilk 4y ago
  > Numpy predates Python's array by a long shot I don't think so. The array module was included even in Python 1.0. Or did you mean to use another verb?
  - eesmith 4y ago
    To be more specific, "git log arraymodule.c" reports: commit 778983b48165da25ee11fb97f6855af7c67f4ff2 Author: Guido van Rossum <guido@python.org> Date: Fri Feb 19 15:55:02 1993 +0000 Added new module "array" (for now optional) defining array objects.

cuteboy19 4y ago

Who needs this strange operation anyways? Deprecate the whole method and call it a day

moelf 4y ago
that's what I'd go for, throw more errors when `__contains__` is called with ambiguity. but, cough, mature and stable scientific computing foundation?

judofyr 4y ago

Here's a better resource for understanding how __conains__ works which actually looks at the source/documentation instead of trying to understand it intuitively: https://stackoverflow.com/questions/18320624/how-does-contains-work-for-ndarrays https://stackoverflow.com/questions/18320624/how-does-contai... Answer: `a in b` is the same as `(a == b).any()`. This really only makes sense when `a` is a scalar and any behavior for other types of arrays are more "accidental" than designed.

Someone 4y ago
Even that took me a while to understand it and, I guess, is another example of Numpy’s absurdity. My initial thought was “None of the examples have arguments that are equal to each other, so that should evaluate to false.any()”, but that would mean all should return the same value, so that can’t be true, so I figured out that Numpy overloads/changes a == b to do item-by-item comparison on arrays, and return an array of booleans. But then, the statement “This really only makes sense when `a` is a scalar” doesn’t make sense to me. That seems to indicate Numpy also overloads a == b as a comparison of each element of a with a single value b and vice versa, but that doesn’t make sense, given that arrays can contain arrays, making it impossible to discriminate between [1,2] as a two-item sequence and [1,2] as a single-item sequence where the item is a two-item sequence. Reading https://stackoverflow.com/questions/10580676/comparing-two-numpy-arrays-for-equality-element-wise https://stackoverflow.com/questions/10580676/comparing-two-n... further strengthens my impression that one should avoid using == in Numpy.
gnulinux 4y ago
Well yeah you're right OP could have explained this better; but this argues their point. From the definition that `a in b` is `(a == b).any()` it's trivial to see things like `np.array([1, 0, 0]) in np.array([1, 2, 3])` will return `True` too, which is absurd, as the title says. I find it hard to believe when numpy team was designing the API they missed this.
- kzrdude 4y ago
  It's trivial to see some things and other things use broadcasting and that's not quite trivial :)

imiric 4y ago

For a language with "explicit is better than implicit" as part of its mantra[1], Python is chock full of surprising behavior like this where the programmer really has no idea what's happening under the hood. I'm happy I moved on to Go, which actually adopts this as a core design tenet. [1]: Even though PEP-20 is largely a meme.

qbasic_forever 4y ago
To be fair this person's gripes are with the behavior of Numpy arrays, not anything that's native to Python's core language. Numpy arrays are tricky beasts because they try to balance fast operations on huge amounts of data, with expressive Python operator syntax.
- ModernMech 4y ago
  Although, another way of looking at what you just said is that Numpy arrays exist because core Python fails to strike a balance between being both fast and expressive.
  - eesmith 4y ago
    That does not necessarily follow. Python may be striking a balance between the needs of numerical and non-numerical computing, while NumPy arrays do not need to consider non-numerical computing. Case in point, historically NumPy has been designed for long-running programs, like hour-long calculations or notebook processing. They've chosen to import "the world" so users don't need explicit subpackage imports, allowing: >>> import numpy >>> numpy.polynomial.hermite.hermfromroots([2,2,3,4]) array([ 7.075e+01, -4.625e+01, 1.175e+01, -1.375e+00, 6.250e-02]) This has a high-startup cost which makes NumPy use inappropriate for short-lived programs like shell scripts, even in cases where Python's own startup cost is otherwise appropriate.

ThePhysicist 4y ago

I've worked with numpy for years and I don't think I ever used the 'in' operator with arrays. What would be the use case for that? For me a larger gotcha was around object multiplication (i.e. element-wise vs. matrix multiplication) and how it is different based on whether you have a "matrix" or an "array" type (or a mix of both). That caused some pretty big headache in the beginning.

weare138 4y ago
But that just seems like a foot gun for the poor unfortunate soul that eventually does.
- rowanG077 4y ago
  I don't think people who use python are concerned with footguns. Numpy is very much not at fault here.
  - DangitBobby 4y ago
    Why would you think that?
    - rowanG077 4y ago
      Because the typing system used in Python is a big footgun. If the basis of the language you are using encourages footguns then it's not surprising that a large library in that ecosystem doesn't care about that either.
- sillysaurusx 4y ago
  Meh. Footguns are underrated. Let’s return to the Wild West of the 90s. I’m only partially joking. But this was one case where they probably thought it was a good idea, turned out to be a bad idea, then couldn’t turn it off without breaking a bunch of code.
  - rightbyte 4y ago

bo1024 4y ago

Interpreting “in” as “is an element of”, all of these are type errors. It only makes sense to ask if an X is in an Array<X>. In a language that lets you ask nonsense questions, expect nonsense answers.

kzrdude 4y ago
Numpy has the highly useful operation of `a < b` where a and b are arrays and the answer is an array of booleans - i.e broadcasted elementwise compare. They changed the language to change what question `<` asks.
- bo1024 4y ago
  Sure, and languages like C++ let you do this kind of thing very broadly, but it has a lot of pitfalls. One of my biggest gripes with python is that function interfaces and "type signatures" are hard to discover even when documentation is good.
  - kzrdude 4y ago
    FWIW I think it's very hard to discover this from C++ code as well. It has to be documented.
- kevin_thibedeau 4y ago
  This gets ugly with !=. Instead of 'a != b' you usually want 'not (a == b).all()'.
  - cuteboy19 4y ago
    It really doesn't if you think about it. a==b cannot by itself be used as an if condition. You would always need to use (a==b).all(). So it would be very difficult to make the mistake you are suggesting https://stackoverflow.com/questions/10062954/valueerror-the-truth-value-of-an-array-with-more-than-one-element-is-ambiguous https://stackoverflow.com/questions/10062954/valueerror-the-...

moelf 4y ago

Op here, I'd also like to say I blame this on Python's lack of broadcasting syntax, (dot in Matlab and Julia) Which forces Numpy to be designed in a way to automatically broadcast, guessing what user wants is hard in nested container

johnmyleswhite 4y ago
Julia's broadcasting syntax depends on having Julia-style compilation to be efficient. It's not clear to me Python could ever provide the same behavior.

dima55 4y ago

Core numpy is really weird, and has really sloppy logic around higher-dimensional objects. I've a module to work around lots of the warts. If somebody has smarter versions of intersect() or in() or whatever, contribute! https://github.com/dkogan/numpysane https://github.com/dkogan/numpysane

cuteboy19 4y ago
Apologies if this was explained in the talk, but why not upstream your changes to numpy?
- dima55 4y ago
  No real good reason, honestly. I wasn't feeling positively about numpy devs at the moment, and didn't want to negotiate getting the code in. The library is solid though, and should be merged. I need somebody to pester me about it, to actualy work on that today.

kakadzhun 4y ago

import numpy as np print(np.__version__) a = np.array([1,2,3]) b = np.array([1]) print(a in b) print(b in a) print(np.isin(a, b)) print(np.isin(b, a)) print(np.isin(a.astype(float), b)) print(np.isin(a, b.astype(float)))) 1.21.6 True True [ True False False] [ True] [ True False False] [ True False False]