5 ms·

Variadic Generics has the potential to enable all the scientific computing libraries to provide really nice type hints. One step closer to eliminating insidious

Variadic Generics has the potential to enable all the scientific computing libraries to provide really nice type hints. One step closer to eliminating insidious broadcasting bugs… or running training for 10 hrs only to have a validation loop fail because of a shape mismatch.

deleted 4y ago
[deleted]
MichaelMoser123 4y ago
i would say it makes sense for a typed language, not sure if you can have it both in python. Why do they keep adding features upon features to the syntax of poor python? do they want to turn it into another kind of c++?
- beisner 4y ago
  It's an optional typechecking tool. Speaking as an ML researcher who has authored my fair share of linear algebra bugs due to broadcasting, being able to optionally typecheck specific critical regions of the codebase would be a killer feature.
- bbkane 4y ago
  I think they want Python to fit into every niche, like C++, which means a lot of knobs to turn
davidatbu 4y ago
THIS. OH MY GOSH. THIS. The only issue now is that pytorch has their own "shapes" solution[0], and last I checked, were kinda reserved about participating in the standardization of variadic generics because they don't expect to use it. I really really hope that the ML community comes together to use variadic generics because I believe it will save researchers and devs so many debug cycles (as well as compute resources, tbh). [0] https://pytorch.org/docs/stable/named_tensor.html https://pytorch.org/docs/stable/named_tensor.html
aix1 4y ago
> running training for 10 hrs only to have a validation loop fail because of a shape mismatch. I agree that tracking array shapes at build time could catch certain classes of bugs (and I would love to have that capability). However, it's not as easy as it may seem. Consider the following example. It shows that even a simple slicing operation can change the number of array dimensions in ways that are only known at run time. >>> def f(arr, elem): ... return arr[elem] ... >>> arr=np.zeros((1, 1)) >>> f(arr, 0).shape (1,) >>> f(arr, ...).shape (1, 1) >>> f(arr, np.newaxis).shape (1, 1, 1)
verdverm 4y ago
Why don't you run on a very small subset of data to ensure that the code works e2e in the workflow pipeline?
- killingtime74 4y ago
  Yes that’s what I thought everyone did…
- davidatbu 4y ago
  Yes, the example @beisner gave here (the val loop crashing after the training is done) is kind of a bad example, and some frameworks (like Pytorch lightning) do do a "full workflow check" before going into training, but his overall point that variadic generics could massively impact dev/researcher productivity stands, I think.
- beisner 4y ago
  Sure, there are of course runtime checks you can do (and I now do after having been bitten by this before), but that’s not necessarily better than having a static analyzer which can guarantee correctness without executing a single line of code.
patrickkidger 4y ago
I disagree. I've had a serious attempt at array typing using variadic generics and I'm not impressed. Python's type system has numerous issues... and now they just apply to any "ArrayWithNDimensions" type as well as any "ArrayWith2Dimensions" type. Variadic protocols don't exist; many operations like stacking are inexpressible; the synatx is awful and verbose; etc. etc. I've written more about this here as part of my TorchTyping project: [0] [0] https://github.com/patrick-kidger/torchtyping/issues/37#issuecomment-1153294196 https://github.com/patrick-kidger/torchtyping/issues/37#issu...
- davidatbu 4y ago
  Thanks for linking this! And I (though not OP) totally agree with your points! I hope that these things could be solved with future iterations. For example: Variadic protocols don't exist. Hopefully they are added sometime. the syntax is awful and verbose. Hopefully we can settle on allowing `1` instead of `Literal[1]` as a type (and other similar improvements). many operations like stacking are inexpressible. These could be expressed if things like "multiplying"/"adding" literal numeric types would be supported. The variadic generics PEP was partly motivated by the ML use-case (and took input from maintainers of numpy ..etc), so I hope future iterations will also improve the usage in the ML space.
  - Mehdi2277 4y ago
    Partly motivated? The primary focus was ML space. The creators of PEP 646 work at Facebook with goal of supporting pytorch tensor dimension tracking. That was main motivation and many of the features/design of that pep were based on what's needed to type hint tensor dimensions. Pep was originally even longer and there's planned follow up peps for other tensor related type features like literal arithmetic to allow type hinting function like np.concatenate. I expect 2/3 more peps in that area in the next year or two.
    - davidatbu 4y ago
      Absolutely right about ML being the main motivation. The PEP says: The main case this PEP targets - concerns typing in numerical libraries. However, despite the authors of the PEP working at facebook, the Pytorch team, at facebook, wasn't interested at all in the PEP. This is also from the PEP: For the sake of transparency - we also reached out to folks from a third popular numerical computing library, PyTorch, but did not receive a statement of endorsement from them. Our understanding is that although they are interested in some of the same issues - e.g. static shape inference - they are currently focusing on enabling this through a DSL rather than the Python type system

deleted 4y ago

[deleted]

MichaelMoser123 4y ago

i would say it makes sense for a typed language, not sure if you can have it both in python. Why do they keep adding features upon features to the syntax of poor python? do they want to turn it into another kind of c++?

beisner 4y ago
It's an optional typechecking tool. Speaking as an ML researcher who has authored my fair share of linear algebra bugs due to broadcasting, being able to optionally typecheck specific critical regions of the codebase would be a killer feature.
bbkane 4y ago
I think they want Python to fit into every niche, like C++, which means a lot of knobs to turn

davidatbu 4y ago

THIS. OH MY GOSH. THIS. The only issue now is that pytorch has their own "shapes" solution[0], and last I checked, were kinda reserved about participating in the standardization of variadic generics because they don't expect to use it. I really really hope that the ML community comes together to use variadic generics because I believe it will save researchers and devs so many debug cycles (as well as compute resources, tbh). [0] https://pytorch.org/docs/stable/named_tensor.html https://pytorch.org/docs/stable/named_tensor.html

aix1 4y ago

> running training for 10 hrs only to have a validation loop fail because of a shape mismatch. I agree that tracking array shapes at build time could catch certain classes of bugs (and I would love to have that capability). However, it's not as easy as it may seem. Consider the following example. It shows that even a simple slicing operation can change the number of array dimensions in ways that are only known at run time. >>> def f(arr, elem): ... return arr[elem] ... >>> arr=np.zeros((1, 1)) >>> f(arr, 0).shape (1,) >>> f(arr, ...).shape (1, 1) >>> f(arr, np.newaxis).shape (1, 1, 1)

verdverm 4y ago

Why don't you run on a very small subset of data to ensure that the code works e2e in the workflow pipeline?

killingtime74 4y ago
Yes that’s what I thought everyone did…
davidatbu 4y ago
Yes, the example @beisner gave here (the val loop crashing after the training is done) is kind of a bad example, and some frameworks (like Pytorch lightning) do do a "full workflow check" before going into training, but his overall point that variadic generics could massively impact dev/researcher productivity stands, I think.
beisner 4y ago
Sure, there are of course runtime checks you can do (and I now do after having been bitten by this before), but that’s not necessarily better than having a static analyzer which can guarantee correctness without executing a single line of code.

patrickkidger 4y ago

I disagree. I've had a serious attempt at array typing using variadic generics and I'm not impressed. Python's type system has numerous issues... and now they just apply to any "ArrayWithNDimensions" type as well as any "ArrayWith2Dimensions" type. Variadic protocols don't exist; many operations like stacking are inexpressible; the synatx is awful and verbose; etc. etc. I've written more about this here as part of my TorchTyping project: [0] [0] https://github.com/patrick-kidger/torchtyping/issues/37#issuecomment-1153294196 https://github.com/patrick-kidger/torchtyping/issues/37#issu...

davidatbu 4y ago
Thanks for linking this! And I (though not OP) totally agree with your points! I hope that these things could be solved with future iterations. For example: Variadic protocols don't exist. Hopefully they are added sometime. the syntax is awful and verbose. Hopefully we can settle on allowing `1` instead of `Literal[1]` as a type (and other similar improvements). many operations like stacking are inexpressible. These could be expressed if things like "multiplying"/"adding" literal numeric types would be supported. The variadic generics PEP was partly motivated by the ML use-case (and took input from maintainers of numpy ..etc), so I hope future iterations will also improve the usage in the ML space.
- Mehdi2277 4y ago
  Partly motivated? The primary focus was ML space. The creators of PEP 646 work at Facebook with goal of supporting pytorch tensor dimension tracking. That was main motivation and many of the features/design of that pep were based on what's needed to type hint tensor dimensions. Pep was originally even longer and there's planned follow up peps for other tensor related type features like literal arithmetic to allow type hinting function like np.concatenate. I expect 2/3 more peps in that area in the next year or two.
  - davidatbu 4y ago
    Absolutely right about ML being the main motivation. The PEP says: The main case this PEP targets - concerns typing in numerical libraries. However, despite the authors of the PEP working at facebook, the Pytorch team, at facebook, wasn't interested at all in the PEP. This is also from the PEP: For the sake of transparency - we also reached out to folks from a third popular numerical computing library, PyTorch, but did not receive a statement of endorsement from them. Our understanding is that although they are interested in some of the same issues - e.g. static shape inference - they are currently focusing on enabling this through a DSL rather than the Python type system