4 ms·

Bringing NumPy's type-completeness score to nearly 90%

59 points by todsacerdoti 1y ago

tecoholic 1y ago
> That's it! CanIndex was an unknown symbol and was probably mistyped, and replacing it with the correct SupportsIndex one brought NumPy's overall type-completeness to over 80%! Let’s take a pause here for a second - the ‘CanIndex’ and ‘SupportsIndex’ from the looks are just “int”. I have a hard time dealing with these custom types because they are so obscure. Does anyone find these useful? How should I be reading them? I understand custom types that package data like classes. But feel completely confused for this yadayava is string, floorpglooep is int style types.
- jkingsman 1y ago
  I find it especially helpful during refactors — understanding the significance/meaning of the variable and what its values can be (and are constrained to) and not just the type is great, but if typing is complete, I can also go and see everywhere that type can be ingested, knowing every surface that uses it and that might need to be refactored.
- CaliforniaKarl 1y ago
  > Let’s take a pause here for a second - the ‘CanIndex’ and ‘SupportsIndex’ from the looks are just “int”. The PR for the change is https://github.com/numpy/numpy/pull/28913 https://github.com/numpy/numpy/pull/28913 - The details of files changed[0] shows the change was made in 'numpy/__init__.pyi'. Looking at the whole file[1] shows SupportsIndex is being imported from the standard library's typing module[2]. Where are you seeing SupportsIndex being defined as an int? > I have a hard time dealing with these custom types because they are so obscure. SupportsIndex is obscure, I agree, but it's not a custom type. It's defined in stdlib's typing module[2], and was added in Python 3.8. [0]: https://github.com/numpy/numpy/pull/28913/files https://github.com/numpy/numpy/pull/28913/files [1]: https://github.com/charris/numpy/blob/c906f847f8ebfe0adec896e15d50d0c873d2dae5/numpy/__init__.pyi#L213 https://github.com/charris/numpy/blob/c906f847f8ebfe0adec896... [2]: https://docs.python.org/3/library/typing.html#typing.SupportsIndex https://docs.python.org/3/library/typing.html#typing.Support...
  - tecoholic 1y ago
    Thanks for pointing those out. Your comment and the others who responded tell me I don’t typically work with complex software. I am a web dev who also had to wrangle data at times, the data part is where I face the issues. I looked at the function in the article taking in an int and I stopped there. And then I read the docs for SupportsIndex, I just can’t think of an occasion I would have used it.
    - ameliaquining 1y ago
      SupportsIndex makes it so that, if you have a value that's not a Python int but can be implicitly and losslessly converted to one, you can use it to index into or slice a NumPy ndarray, without first explicitly converting it to a Python int. There are two particular data types that it's especially useful to be able to do this with: NumPy's fixed-width integer types (analogous to the ones in C, exposed because numeric calculations on ndarrays of them are often much faster than on Python's arbitrary-precision int), and zero-dimensional ndarrays (which can't be indexed into or sliced, and are morally equivalent to a single scalar). Programmers often wind up with one of these as the output of some operation, think of them as ints, and don't know or care that they're actually a different data type, so NumPy tries to make it so that you can treat them like ints and get the same result, this being the point of a dynamic language like Python. Using SupportsIndex instead of int as an argument type ensures that type checkers won't reject code that does this. This is admittedly fairly arcane, but only people delving into the guts of NumPy or the Python type system are expected to have to deal with it. The library code is complicated so that your code can be simple: you can just write NumPy code the way you're used to, and the type checker will accept it, without you having to know or care about the above details. That's the goal, anyway.
      - parhamn 1y ago
        > SupportsIndex makes it so that, if you have a value that's not a Python int but can be implicitly and losslessly converted to one Just to be clear for folks who care more about the general python around this: any class that has a method __index__ that returns an int can be used as an index for array access. e.g: class FetchIndex: def __index__(self): return int(requests.get("https://indextolookup.com").text.strip()) a = ["no", "lol", "wow"] print(a[FetchIndex()]) The builtin `int` also has a __index__ method that returns itself. Dunders are part of what makes typing in python so hard.
- masspro 1y ago
  * “which of the 3 big data structures in this part of the program/function/etc is this int/string key an index into?” * some arithmetic/geometry problems for example 2D layout where there are several different things that are “an integer number of pixels” but mean wildly different things In either case it can help pick apart dense code or help stay on track while writing it. It can instead become anti-helpful by causing distraction or increasing the friction to make changes.
- SAI_Peregrinus 1y ago
  Think of them like units of measurement. `°C` is a type, distinct from `°F`, and also distinct from `kilometer`, but all those are just `float` in Python's numeric types. You want separate types because they're not interchangeable. Sometimes two types are subtypes of the same parent type, `°C` and `°F` are subtypes of some generic `temperature` type, `kilometer` is a subtype of `distance`, etc. Types with the same supertype can be converted between one another, but unrelated types can't be. There's no meaningful way to convert `50.3km` into `°C`, even though both are just `float`s.
  - tecoholic 1y ago
    Thank you. I know it's been almost 2 weeks since you posted this comment. But I got to it only today and this makes a lot of sense and helps understand the type system a bit more. Thank you.
CaliforniaKarl 1y ago
I've recently been writing a Python SDK for an internal API that's maintained by a different group. I've been making sure to include typing for everything in the SDK, as well as attempting to maximize unit test coverage (using mocked API responses). It's a heck of a lot of work (easily as much work as writing the actual code!), but typing has already paid dividends as I've been starting to use the SDK for one-off scripts.
strbean 1y ago
Typing related NumPy whine: The type annotations say `float32`, `float64`, etc. are type aliases for `Floating[32]` and `Floating[64]`. In reality, they are concrete classes. This means that `isinstance(foo, float32)` works as expected, but pisses off static analysis tools.
- refactor_master 1y ago
  I hate when libraries do that, or create a `def ClassLike`. I’ve found the library `beartype` to be very good for runtime checks built into partially annotated `Annotated` types with custom validators, so instead of instance checks you rely directly on signatures and typeguards.
jofer 1y ago
One of my biggest gripes around typing in python actually revolves around things like numpy arrays and other scientific data structures. Typing in python is great if you're only using builtins or things that the typing system was designed for. But it wasn't designed with scientific data structures particularly in mind. Being able to denote dtype (e.g. uint8 array vs int array) is certainly helpful, but only one aspect. There's not a good way to say "Expects a 3D array-like" (i.e. something convertible into an array with at least 3 dimensions). Similarly, things like "At least 2 dimensional" or similar just aren't expressible in the type system and potentially could be. You wind up relying on docstrings. Personally, I think typing in docstrings is great. At least for me, IDE (vim) hinting/autocompletion/etc all work already with standard docstrings and strictly typed interpreters are a completely moot point for most scientific computing. What happens in practice is that you have the real info in the docstring and a type "stub" for typing. However, at the point that all of the relevant information about the expected type is going to have to be the docstring, is the additional typing really adding anything? In short, I'd love to see the ability to indicate expected dimensionality or dimensionality of operation in typing of numpy arrays. But with that said, I worry that typing for these use cases adds relatively little functionality at the significant expense of readability.
- efavdb 1y ago
  Would a custom decorator work for you?
  - jofer 1y ago
    Unless I'm missing something entirely, what would that add? You still can't express the core information you need in the type system.
    - efavdb 1y ago
      I meant only that you can insist a parameter has some quality when passed.
      - 1y ago
t1129437123 1y ago
"ArrayLike" is not a type. The entire Python "type system" is really hackish compared to Julia or typed array languages. Does this "type system" ensure that you can't get a runtime type error? I don't think so.
- jofer 1y ago
  Yes, python is not statically typed. It shouldn't be. Don't expect static typing behavior and typing in python is _not_ static typing in any way. It's documentation only, not static typing.

59 points by todsacerdoti 1y ago

> That's it! CanIndex was an unknown symbol and was probably mistyped, and replacing it with the correct SupportsIndex one brought NumPy's overall type-completeness to over 80%! Let’s take a pause here for a second - the ‘CanIndex’ and ‘SupportsIndex’ from the looks are just “int”. I have a hard time dealing with these custom types because they are so obscure. Does anyone find these useful? How should I be reading them? I understand custom types that package data like classes. But feel completely confused for this yadayava is string, floorpglooep is int style types.

jkingsman 1y ago
I find it especially helpful during refactors — understanding the significance/meaning of the variable and what its values can be (and are constrained to) and not just the type is great, but if typing is complete, I can also go and see everywhere that type can be ingested, knowing every surface that uses it and that might need to be refactored.
CaliforniaKarl 1y ago
> Let’s take a pause here for a second - the ‘CanIndex’ and ‘SupportsIndex’ from the looks are just “int”. The PR for the change is https://github.com/numpy/numpy/pull/28913 https://github.com/numpy/numpy/pull/28913 - The details of files changed[0] shows the change was made in 'numpy/__init__.pyi'. Looking at the whole file[1] shows SupportsIndex is being imported from the standard library's typing module[2]. Where are you seeing SupportsIndex being defined as an int? > I have a hard time dealing with these custom types because they are so obscure. SupportsIndex is obscure, I agree, but it's not a custom type. It's defined in stdlib's typing module[2], and was added in Python 3.8. [0]: https://github.com/numpy/numpy/pull/28913/files https://github.com/numpy/numpy/pull/28913/files [1]: https://github.com/charris/numpy/blob/c906f847f8ebfe0adec896e15d50d0c873d2dae5/numpy/__init__.pyi#L213 https://github.com/charris/numpy/blob/c906f847f8ebfe0adec896... [2]: https://docs.python.org/3/library/typing.html#typing.SupportsIndex https://docs.python.org/3/library/typing.html#typing.Support...
- tecoholic 1y ago
  Thanks for pointing those out. Your comment and the others who responded tell me I don’t typically work with complex software. I am a web dev who also had to wrangle data at times, the data part is where I face the issues. I looked at the function in the article taking in an int and I stopped there. And then I read the docs for SupportsIndex, I just can’t think of an occasion I would have used it.
  - ameliaquining 1y ago
    SupportsIndex makes it so that, if you have a value that's not a Python int but can be implicitly and losslessly converted to one, you can use it to index into or slice a NumPy ndarray, without first explicitly converting it to a Python int. There are two particular data types that it's especially useful to be able to do this with: NumPy's fixed-width integer types (analogous to the ones in C, exposed because numeric calculations on ndarrays of them are often much faster than on Python's arbitrary-precision int), and zero-dimensional ndarrays (which can't be indexed into or sliced, and are morally equivalent to a single scalar). Programmers often wind up with one of these as the output of some operation, think of them as ints, and don't know or care that they're actually a different data type, so NumPy tries to make it so that you can treat them like ints and get the same result, this being the point of a dynamic language like Python. Using SupportsIndex instead of int as an argument type ensures that type checkers won't reject code that does this. This is admittedly fairly arcane, but only people delving into the guts of NumPy or the Python type system are expected to have to deal with it. The library code is complicated so that your code can be simple: you can just write NumPy code the way you're used to, and the type checker will accept it, without you having to know or care about the above details. That's the goal, anyway.
    - parhamn 1y ago
      > SupportsIndex makes it so that, if you have a value that's not a Python int but can be implicitly and losslessly converted to one Just to be clear for folks who care more about the general python around this: any class that has a method __index__ that returns an int can be used as an index for array access. e.g: class FetchIndex: def __index__(self): return int(requests.get("https://indextolookup.com").text.strip()) a = ["no", "lol", "wow"] print(a[FetchIndex()]) The builtin `int` also has a __index__ method that returns itself. Dunders are part of what makes typing in python so hard.
masspro 1y ago
* “which of the 3 big data structures in this part of the program/function/etc is this int/string key an index into?” * some arithmetic/geometry problems for example 2D layout where there are several different things that are “an integer number of pixels” but mean wildly different things In either case it can help pick apart dense code or help stay on track while writing it. It can instead become anti-helpful by causing distraction or increasing the friction to make changes.
SAI_Peregrinus 1y ago
Think of them like units of measurement. `°C` is a type, distinct from `°F`, and also distinct from `kilometer`, but all those are just `float` in Python's numeric types. You want separate types because they're not interchangeable. Sometimes two types are subtypes of the same parent type, `°C` and `°F` are subtypes of some generic `temperature` type, `kilometer` is a subtype of `distance`, etc. Types with the same supertype can be converted between one another, but unrelated types can't be. There's no meaningful way to convert `50.3km` into `°C`, even though both are just `float`s.
- tecoholic 1y ago
  Thank you. I know it's been almost 2 weeks since you posted this comment. But I got to it only today and this makes a lot of sense and helps understand the type system a bit more. Thank you.

CaliforniaKarl 1y ago

I've recently been writing a Python SDK for an internal API that's maintained by a different group. I've been making sure to include typing for everything in the SDK, as well as attempting to maximize unit test coverage (using mocked API responses). It's a heck of a lot of work (easily as much work as writing the actual code!), but typing has already paid dividends as I've been starting to use the SDK for one-off scripts.

strbean 1y ago

Typing related NumPy whine: The type annotations say `float32`, `float64`, etc. are type aliases for `Floating[32]` and `Floating[64]`. In reality, they are concrete classes. This means that `isinstance(foo, float32)` works as expected, but pisses off static analysis tools.

refactor_master 1y ago
I hate when libraries do that, or create a `def ClassLike`. I’ve found the library `beartype` to be very good for runtime checks built into partially annotated `Annotated` types with custom validators, so instead of instance checks you rely directly on signatures and typeguards.

jofer 1y ago

One of my biggest gripes around typing in python actually revolves around things like numpy arrays and other scientific data structures. Typing in python is great if you're only using builtins or things that the typing system was designed for. But it wasn't designed with scientific data structures particularly in mind. Being able to denote dtype (e.g. uint8 array vs int array) is certainly helpful, but only one aspect. There's not a good way to say "Expects a 3D array-like" (i.e. something convertible into an array with at least 3 dimensions). Similarly, things like "At least 2 dimensional" or similar just aren't expressible in the type system and potentially could be. You wind up relying on docstrings. Personally, I think typing in docstrings is great. At least for me, IDE (vim) hinting/autocompletion/etc all work already with standard docstrings and strictly typed interpreters are a completely moot point for most scientific computing. What happens in practice is that you have the real info in the docstring and a type "stub" for typing. However, at the point that all of the relevant information about the expected type is going to have to be the docstring, is the additional typing really adding anything? In short, I'd love to see the ability to indicate expected dimensionality or dimensionality of operation in typing of numpy arrays. But with that said, I worry that typing for these use cases adds relatively little functionality at the significant expense of readability.

efavdb 1y ago
Would a custom decorator work for you?
- jofer 1y ago
  Unless I'm missing something entirely, what would that add? You still can't express the core information you need in the type system.
  - efavdb 1y ago
    I meant only that you can insist a parameter has some quality when passed.
    - 1y ago

t1129437123 1y ago

"ArrayLike" is not a type. The entire Python "type system" is really hackish compared to Julia or typed array languages. Does this "type system" ensure that you can't get a runtime type error? I don't think so.

jofer 1y ago
Yes, python is not statically typed. It shouldn't be. Don't expect static typing behavior and typing in python is _not_ static typing in any way. It's documentation only, not static typing.