10 ms·

NumPy 2.0

324 points by scoresmoke 2y ago

ck45 2y ago
Here's a link to the release notes: https://numpy.org/devdocs/release/2.0.0-notes.html https://numpy.org/devdocs/release/2.0.0-notes.html
- chmaynard 2y ago
  This is a draft. No release notes yet.
  - wartijn_ 2y ago
    The GitHub release seems to have the final notes. It has at least the placeholder texts replaced: > It is the result of 11 months of development since the last feature release and is the work of 212 contributors spread over 1078 pull requests instead of: > It is the result of X months of development since the last feature release by Y contributors https://github.com/numpy/numpy/releases/tag/v2.0.0 https://github.com/numpy/numpy/releases/tag/v2.0.0
- Kortaggio 2y ago
  Duplicate HN thread: https://news.ycombinator.com/item?id=40700259 https://news.ycombinator.com/item?id=40700259
- dang 2y ago
  Changed to that from https://pypi.org/project/numpy/ https://pypi.org/project/numpy/ above. Thanks!
fbdab103 2y ago
Any notable highlights for a consumer of Numpy who rarely interfaces directly with it? Most of my work is pandas+scipy, with occasionally dropping into the specific numpy algorithm when required. I am much more of an "upgrade when there is a X.1" release kind of guy, so my hat off to those who will bravely be testing the version on my behalf.
- scoresmoke 2y ago
  The most important changes are deprecations of certain public APIs: https://numpy.org/devdocs/release/2.0.0-notes.html#deprecations https://numpy.org/devdocs/release/2.0.0-notes.html#deprecati... One new interesting feature, though, is the support for string routines: https://numpy.org/devdocs/reference/routines.strings.html#module-numpy.strings https://numpy.org/devdocs/reference/routines.strings.html#mo...
  - ahurmazda 2y ago
    This one will be rough :| > arange’s start argument is positional-only
    - haiguise 2y ago
      Looks like that might get reverted [0]. [0] https://github.com/numpy/numpy/pull/25955 https://github.com/numpy/numpy/pull/25955
  - amelius 2y ago
    > One new interesting feature, though, is the support for string routines Sounds almost like they're building a language inside a language.
    - ssahoo 2y ago
      No. Native python ops in string suck in performance. String support is absolutely interesting and will enable abstractions for many NLP and LLM use cases without writing native C extensions.
      - topper-123 2y ago
        Yeah, operating on strings has historically been a major weak point of Numpy's. I'm looking forward seeing benchmarks for the new implementation.
      - ayhanfuat 2y ago
notatoad 2y ago
it feels like the first major release in 18 years which introduces lots of breaking changes should just be a fork rather than a version. let me do `pip install numpy2` and not have to worry about whether or not some other library in my project requires numpy<2.
- make3 2y ago
  knowing how careful the NumPy devs are, this was likely a very well pondered decision & all of these deprecations likely have been announced for a long time. Seeing knee jerk reactions like this is annoying.
  - deleted 2y ago
    [deleted]
  - kaashif 2y ago
    Do you have a link to the discussion where this was very well pondered? I can't find anything, but I'm very interested in that kind of discussion.
    - ngoldbaum 2y ago
      We had a developer meeting to discuss what should go into 2.0 in April 2023: https://github.com/numpy/archive/tree/main/2.0_developer_meeting https://github.com/numpy/archive/tree/main/2.0_developer_mee...
- vessenes 2y ago
  From a consumer (developer consumer) point of view, I hear you. From a project point of view, there are some pretty strong contra-indicators in the last 20 years of language development that make this plan suspect, or at least pretty scary — both Perl and Python had extremely rocky transitions around major versions; Perl’s ultimately failing and Python’s ultimately taking like 10 years. At least. I think the last time I needed Python 2 for something was a few months ago, and before that it had been a year or so. I’ve never needed Perl 6, but if I did I would be forced to read a lot of history while I downloaded and figured out which, if any, Perl 5 modules I’m looking for got ported. I’d imagine the numpy devs probably don’t have the resources to support what would certainly become two competing forks that each have communities with their own needs.
dahart 2y ago
The thing I want most is a more sane and more memorable way to compose non-element-wise operations. There are so many different ways to build views and multiply arrays that I can’t remember them and never know which to use, and have to relearn them every time I use numpy… broadcasting, padding, repeating, slicing, stacking, transposing, outers, inners, dots of all sorts, and half the stack overflow answers lead to the most confusing pickaxe of all: einsum. Am I alone? I love numpy, but every time I reach for it I somehow get stuck for hours on what ought to be really simple indexing problems.
- akasakahakada 2y ago
  To be honest einsum is the easiest one. You get fine control on which axis matmul to which. But I wish it can do more than matmul. The others are just messy shit. Like you got np.abs but no arr.abs, np.unique but no arr.unique. But now you have arr.mean. Sometimes you got argument name index, sometimes indices, sometimes accept (list, tuple), sometime only tuple.
  - samsartor 2y ago
    https://github.com/mcabbott/Tullio.jl https://github.com/mcabbott/Tullio.jl is my favorite idea for extending einsum notation to more stuff. Really hope numpy/torch get something comparable!
    - enkursigilo 2y ago
      Yeah, Tullio.jl is a great package. Basically, it is just a macro for generating efficient for loops. I guess, it might be hard to achieve similar feature in Python without metaprogramming.
- salamo 2y ago
  When I started out I was basically stumbling around for code that worked. Things got a lot easier for me once I sat down and actually understood broadcasting. The rules are: 1) scalars always broadcast, 2) if one vector has fewer dimensions, left pad it with 1s and 3) starting from the right, check dimension compatibility, where compatibility means the dimensions are equal or one of them is 1. Example: np.ones((2,3,1)) * np.ones((1,4)) = np.ones((2,3,4)) Once your dimensions are correct, it's a lot easier to reason your way through a problem, similar to how basic dimensional analysis in physics can verify your answer makes some sense. (I would disable broadcasting if I could, since it has caused way too many silent bugs in my experience. JAX can, but I don't feel like learning another library to do this.) Once I understood broadcasting, it was a lot easier to practice vectorizing basic algorithms.
ayhanfuat 2y ago
> The default integer type on Windows is now int64 rather than int32, matching the behavior on other platforms This was a footgun due to C long being int32 in win64. Glad that they changed it.
RandomBK 2y ago
I'm starting to see some packages break due to not pinning 1.x in their dependencies. `pip install numpy==1.*` is a quick and hacky way to work around those issues until the ecosystem catches up.
- globular-toast 2y ago
  "numpy~=1.0” Is this not common knowledge? Also, pip install? Or do you mean some requirements file?
antonoo 2y ago
> X months of work by Y contributors? Makes it look like they pressed publish before filling in their template, or is this on purpose?
fertrevino 2y ago
So apparently this is what broke my CI job since it was indirectly installed. One of the downsides of using loose version locking with requirements.txt rather than something like poetry I guess.
Kalanos 2y ago
What are the implications of the new stringdtype? If I remember correctly, string performance was a big part of the pandas switch to arrow.
tpoacher 2y ago
I wish numpy pushed their structured arrays (and thereby also improvements to their interface) more aggressively. Most people are simply unaware of them, which is why we get stuff like pandas on top of everything.
darepublic 2y ago
I would love for numpy to be ported as a typescript project personally. So I can do ml in ts. The python ecosystem feels a bit insane to me (more so than the js one). Venv helps but is still inferior to a half decent npm project imo. I feel there is no strict reason why this migration couldn't happen, only the inertia that makes it unlikely
- elialbert 2y ago
  who's stopping you
- emmanueloga_ 2y ago
  I think it is "better" to go with the flow. I can't see TS competing for the data analysis niche any time soon! Maybe try Pixi? [1] Python programming enjoyability really increased for me after using Pixi for dependencies, VSCode+Pylance [2] for editing, and Ruff [3] for formatting. Pixi can install both python and dependencies _per project_. Then, I add this to .vscode/settings.json: { "python.analysis.typeCheckingMode": "strict", "python.defaultInterpreterPath": ".pixi/envs/default/bin/python3" } and I'm all set! -- 1: https://pixi.sh https://pixi.sh 2: https://github.com/microsoft/pylance-release#readme https://github.com/microsoft/pylance-release#readme 3: https://docs.astral.sh/ruff/ https://docs.astral.sh/ruff/
TalTay 2y ago
https://stackoverflow.com/questions/78641150/a-module-that-was-compiled-using-numpy-1-x-cannot-be-run-in-numpy-2-0-0 https://stackoverflow.com/questions/78641150/a-module-that-w...

324 points by scoresmoke 2y ago

ck45 2y ago

Here's a link to the release notes: https://numpy.org/devdocs/release/2.0.0-notes.html https://numpy.org/devdocs/release/2.0.0-notes.html

chmaynard 2y ago
This is a draft. No release notes yet.
- wartijn_ 2y ago
  The GitHub release seems to have the final notes. It has at least the placeholder texts replaced: > It is the result of 11 months of development since the last feature release and is the work of 212 contributors spread over 1078 pull requests instead of: > It is the result of X months of development since the last feature release by Y contributors https://github.com/numpy/numpy/releases/tag/v2.0.0 https://github.com/numpy/numpy/releases/tag/v2.0.0
Kortaggio 2y ago
Duplicate HN thread: https://news.ycombinator.com/item?id=40700259 https://news.ycombinator.com/item?id=40700259
dang 2y ago
Changed to that from https://pypi.org/project/numpy/ https://pypi.org/project/numpy/ above. Thanks!

fbdab103 2y ago

Any notable highlights for a consumer of Numpy who rarely interfaces directly with it? Most of my work is pandas+scipy, with occasionally dropping into the specific numpy algorithm when required. I am much more of an "upgrade when there is a X.1" release kind of guy, so my hat off to those who will bravely be testing the version on my behalf.

scoresmoke 2y ago
The most important changes are deprecations of certain public APIs: https://numpy.org/devdocs/release/2.0.0-notes.html#deprecations https://numpy.org/devdocs/release/2.0.0-notes.html#deprecati... One new interesting feature, though, is the support for string routines: https://numpy.org/devdocs/reference/routines.strings.html#module-numpy.strings https://numpy.org/devdocs/reference/routines.strings.html#mo...
- ahurmazda 2y ago
  This one will be rough :| > arange’s start argument is positional-only
  - haiguise 2y ago
    Looks like that might get reverted [0]. [0] https://github.com/numpy/numpy/pull/25955 https://github.com/numpy/numpy/pull/25955
- amelius 2y ago
  > One new interesting feature, though, is the support for string routines Sounds almost like they're building a language inside a language.
  - ssahoo 2y ago
    No. Native python ops in string suck in performance. String support is absolutely interesting and will enable abstractions for many NLP and LLM use cases without writing native C extensions.
    - topper-123 2y ago
      Yeah, operating on strings has historically been a major weak point of Numpy's. I'm looking forward seeing benchmarks for the new implementation.
    - ayhanfuat 2y ago

notatoad 2y ago

it feels like the first major release in 18 years which introduces lots of breaking changes should just be a fork rather than a version. let me do `pip install numpy2` and not have to worry about whether or not some other library in my project requires numpy<2.

make3 2y ago
knowing how careful the NumPy devs are, this was likely a very well pondered decision & all of these deprecations likely have been announced for a long time. Seeing knee jerk reactions like this is annoying.
- deleted 2y ago
  [deleted]
- kaashif 2y ago
  Do you have a link to the discussion where this was very well pondered? I can't find anything, but I'm very interested in that kind of discussion.
  - ngoldbaum 2y ago
    We had a developer meeting to discuss what should go into 2.0 in April 2023: https://github.com/numpy/archive/tree/main/2.0_developer_meeting https://github.com/numpy/archive/tree/main/2.0_developer_mee...
vessenes 2y ago
From a consumer (developer consumer) point of view, I hear you. From a project point of view, there are some pretty strong contra-indicators in the last 20 years of language development that make this plan suspect, or at least pretty scary — both Perl and Python had extremely rocky transitions around major versions; Perl’s ultimately failing and Python’s ultimately taking like 10 years. At least. I think the last time I needed Python 2 for something was a few months ago, and before that it had been a year or so. I’ve never needed Perl 6, but if I did I would be forced to read a lot of history while I downloaded and figured out which, if any, Perl 5 modules I’m looking for got ported. I’d imagine the numpy devs probably don’t have the resources to support what would certainly become two competing forks that each have communities with their own needs.

dahart 2y ago

The thing I want most is a more sane and more memorable way to compose non-element-wise operations. There are so many different ways to build views and multiply arrays that I can’t remember them and never know which to use, and have to relearn them every time I use numpy… broadcasting, padding, repeating, slicing, stacking, transposing, outers, inners, dots of all sorts, and half the stack overflow answers lead to the most confusing pickaxe of all: einsum. Am I alone? I love numpy, but every time I reach for it I somehow get stuck for hours on what ought to be really simple indexing problems.

akasakahakada 2y ago
To be honest einsum is the easiest one. You get fine control on which axis matmul to which. But I wish it can do more than matmul. The others are just messy shit. Like you got np.abs but no arr.abs, np.unique but no arr.unique. But now you have arr.mean. Sometimes you got argument name index, sometimes indices, sometimes accept (list, tuple), sometime only tuple.
- samsartor 2y ago
  https://github.com/mcabbott/Tullio.jl https://github.com/mcabbott/Tullio.jl is my favorite idea for extending einsum notation to more stuff. Really hope numpy/torch get something comparable!
  - enkursigilo 2y ago
    Yeah, Tullio.jl is a great package. Basically, it is just a macro for generating efficient for loops. I guess, it might be hard to achieve similar feature in Python without metaprogramming.
salamo 2y ago
When I started out I was basically stumbling around for code that worked. Things got a lot easier for me once I sat down and actually understood broadcasting. The rules are: 1) scalars always broadcast, 2) if one vector has fewer dimensions, left pad it with 1s and 3) starting from the right, check dimension compatibility, where compatibility means the dimensions are equal or one of them is 1. Example: np.ones((2,3,1)) * np.ones((1,4)) = np.ones((2,3,4)) Once your dimensions are correct, it's a lot easier to reason your way through a problem, similar to how basic dimensional analysis in physics can verify your answer makes some sense. (I would disable broadcasting if I could, since it has caused way too many silent bugs in my experience. JAX can, but I don't feel like learning another library to do this.) Once I understood broadcasting, it was a lot easier to practice vectorizing basic algorithms.

ayhanfuat 2y ago

> The default integer type on Windows is now int64 rather than int32, matching the behavior on other platforms This was a footgun due to C long being int32 in win64. Glad that they changed it.

RandomBK 2y ago

I'm starting to see some packages break due to not pinning 1.x in their dependencies. `pip install numpy==1.*` is a quick and hacky way to work around those issues until the ecosystem catches up.

globular-toast 2y ago
"numpy~=1.0” Is this not common knowledge? Also, pip install? Or do you mean some requirements file?

antonoo 2y ago

> X months of work by Y contributors? Makes it look like they pressed publish before filling in their template, or is this on purpose?

fertrevino 2y ago

So apparently this is what broke my CI job since it was indirectly installed. One of the downsides of using loose version locking with requirements.txt rather than something like poetry I guess.

Kalanos 2y ago

What are the implications of the new stringdtype? If I remember correctly, string performance was a big part of the pandas switch to arrow.

tpoacher 2y ago

I wish numpy pushed their structured arrays (and thereby also improvements to their interface) more aggressively. Most people are simply unaware of them, which is why we get stuff like pandas on top of everything.

darepublic 2y ago

I would love for numpy to be ported as a typescript project personally. So I can do ml in ts. The python ecosystem feels a bit insane to me (more so than the js one). Venv helps but is still inferior to a half decent npm project imo. I feel there is no strict reason why this migration couldn't happen, only the inertia that makes it unlikely

elialbert 2y ago
who's stopping you
emmanueloga_ 2y ago
I think it is "better" to go with the flow. I can't see TS competing for the data analysis niche any time soon! Maybe try Pixi? [1] Python programming enjoyability really increased for me after using Pixi for dependencies, VSCode+Pylance [2] for editing, and Ruff [3] for formatting. Pixi can install both python and dependencies _per project_. Then, I add this to .vscode/settings.json: { "python.analysis.typeCheckingMode": "strict", "python.defaultInterpreterPath": ".pixi/envs/default/bin/python3" } and I'm all set! -- 1: https://pixi.sh https://pixi.sh 2: https://github.com/microsoft/pylance-release#readme https://github.com/microsoft/pylance-release#readme 3: https://docs.astral.sh/ruff/ https://docs.astral.sh/ruff/

TalTay 2y ago

https://stackoverflow.com/questions/78641150/a-module-that-was-compiled-using-numpy-1-x-cannot-be-run-in-numpy-2-0-0 https://stackoverflow.com/questions/78641150/a-module-that-w...