5 ms·

Use keyword-only arguments in Python dataclasses

95 points by Bogdanp 1y ago

flakes 1y ago
You could argue the same thing (forcing kwargs) for all Python functions/methods, although, that would make using your APIs very annoying. The `__init__` method for dataclasses are just another method like any other. As a general rule of thumb, I only start forcing kwargs once I'm looking at above 4-5 arguments, or if the arguments are similar enough that forcing kwargs makes the calling code more readable. For a small number of distinct arguments, forcing kwargs as a blanket rule makes the code verbose for little gain IMO.
- masklinn 1y ago
  > You could argue the same thing (forcing kwargs) for all Python functions/methods, although, that would make using your APIs very annoying. The `__init__` method for dataclasses are just another method like any other. While that is self evident at a technical level, it is not quite so from a clarity / documentary perspective: “normal” functions and methods can often hint at their parameters through their naming but it is uncommon for types, for which the composite tends to be much more of an implementation detail. Of course neither rule is universal e.g. the composite is of prime importance for newtypes, and indeed they often use tuple-style types or have special support with no member names.
- vbezhenar 1y ago
  > that would make using your APIs very annoying For Objective C, using named parameters is the only way to call methods. I don't think I read many critique about this particular aspect. IMO it's actually a good thing and increases readability quite a bit. For JavaScript/TypeScript React codebase, using objects as a poor man's named parameters also very popular approach. Also I'd like to add, that it seems a recent trend to add feature to IDEs, where it'll add hint for every parameter, somewhat simulating named parameters. So when you write `mymethod(value)`, it'll display it as `mymethod(param:value)`. So may be not very annoying. The only little thing I'd like to borrow from JavaScript is using "shortcut", so you could replace `x=x` with `x`, if your local variable happened to have the same name, as parameter name (which happens often enough).
  - laserlight 1y ago
    To be pedantic, Objective-C doesn't have named parameters. Method names are composed of multiple parts, each corresponding to a parameter. Such design contributes to the method's readability and memorability. In contrast, Python methods have their own names, and parameter names are chosen as an afterthought. While there's no reason why Python methods couldn't be named in accordance with parameter names, unfortunately it hasn't been a part of Python's culture.
- int_19h 1y ago
  I find that anything above 2 arguments benefits from explicit keyword notation. With 4-5 arguments, especially when most of them are of the same type, it can be difficult to tell which is which.
deleted 1y ago
[deleted]
joshdavham 1y ago
> Positional arguments means a caller can use MyDataClass(1, 'foo', False), and if you remove/reorder any of these arguments, you’ll break those callers unexpectedly. By forcing callers to use MyDataClass(x=1, y='foo', z=False), you remove this risk. This is an awesome way to prevent future breaking changes! ...but unfortunately, adding this to an existing project would also likely result in breakings changes haha
- chipx86 1y ago
  That's always the challenge when iterating on interfaces that other people depend on. What we do is go through a deprecation phase. Our process is: * We provide compatibility with the old signature for 2 major releases. * We document the change and the timeline clearly in the docstring. * The function gets decorated with a helper that checks the call, and if any keyword-only arguments are provided as positional, it warns and converts them to keyword-only. * After 2 major releases, we move fully to the new signature. We buit a Python library called housekeeping (https://github.com/beanbaginc/housekeeping https://github.com/beanbaginc/housekeeping) to help with this. One of the things it contains is a decorator called `@deprecate_non_keyword_only_args`, which takes a deprecation warning class and a function using the signature we're moving to. That decorator handles the check logic and generates a suitable, consistent deprecation message. That normally looks like: @deprecate_non_keyword_only_args(MyDeprecationWarning) def my_func(*, a, b, c): ... But this is a bit more tricky with dataclasses, since `__init__()` is generated automatically. Fortunately, it can be patched after the fact. A bit less clean, but doable. So here's how we'd handle this case with dataclasses: from dataclasses import dataclass from housekeeping import BaseRemovedInWarning, deprecate_non_keyword_only_args class RemovedInMyProject20Warning(BaseRemovedInWarning): product = 'MyProject' version = '2.0' @dataclass(kw_only=True) class MyDataclass: a: int b: int c: str MyDataclass.__init__ = deprecate_non_keyword_only_args( RemovedInMyProject20Warning )(MyDataclass.__init__) Call it with some positional arguments: dc = MyDataclass(1, 2, c='hi') and you'd get: testdataclass.py:26: RemovedInMyProject20Warning: Positional arguments `a`, `b` must be passed as keyword arguments when calling `__main__.MyDataclass.__init__()`. Passing as positional arguments will be required in MyProject 2.0. dc = MyDataclass(1, 2, c='hi') We'll probably add explicit dataclass support to this soon, since we're starting to move to kw_only=True for dataclasses.
FridgeSeal 1y ago
Oh man, the aws Boto3 library does this for a huge number of calls, and it’s awful. “What parameters does this take?” you ask, “why, it takes ‘kwargs’” responds the docs and your IDE. How incredibly helpful!
- chipx86 1y ago
  That's annoying for sure. Though a different problem. All the kw_only=True argument for dataclasses does is require that you pass any fields you want to provide as keyword arguments instead of positional arguments when instantiating a dataclass. So: obj = MyDataclass(a=1, b=2, c=3) Instead of: obj = MyDataclass(1, 2, 3) # This would be an error with kw_only=True The problem you're describing in boto3 (and a lot of other API bindings, and a lot of more layered Python code) is that methods often take in **kwargs and pass them down to a common function that's handling them. From the caller's perspective, **kwargs is a black box with no details on what's in there. Without a docstring or an understanding of the call chain, it's not helpful. Python sort of has a fix for this now, which is to use a TypedDict to define all the possible values in the **kwargs, like so: from typing import TypedDict, Unpack class MyFuncKwargs(TypedDict): arg1: str arg2: str arg3: int | None def my_outer_func( **kwargs: Unpack[MyFuncKwargs], ) -> None: _my_inner_func(**kwargs) def _my_inner_func( *, arg1: str, arg2: str, arg3: int | None, ) -> None: ... By defining a TypedDict and typing **kwargs, the IDE and docs can do a better job of showing what arguments the function really takes, and validating them. Also useful when the function is just a wrapper around serializing **kwargs to JSON for an API, or something. But this feature is far from free to use. The more functions you have, the more of these you need to create and maintain. Ideally, a function could type **kwargs as something like: def my_outer_func( **kwargs: KwargsOf[_my_inner_func], ) -> None: ... And then the IDEs and other tooling can just reference that function. This would help make the problem go away for many of the cases where **kwargs is used and passed around.
  - mvieira38 1y ago
    TypedDicts are so underutilized in general. I'm using them a lot even for simpler scripts
ayhanfuat 1y ago
I like keyword-only arguments, but they become tedious too quickly - especially when the variable names already match (fn(x=x, y=y, z=z)). I wish Python had JavaScript’s shorthand property syntax. (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Object_initializer#property_definitions https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...).
- sweetgiorni 1y ago
  Python 3.14 adds that shorthand, but with a slight different (and IMO uglier) syntax: fn(x=,y=,z=) https://peps.python.org/pep-0736/ https://peps.python.org/pep-0736/ edit: nevermind, that PEP was rejected :/
  - pansa2 1y ago
    That PEP says “rejected”
    - snickerdoodle12 1y ago
      Thankfully so. As much as I want a shorthand, this syntax wasn't it.
  - deleted 1y ago
    [deleted]
  - vovavili 1y ago
    By the look of it, the feeling I get is that a decent and convenient syntax proposal has been bikeshedded into rejection. Some particular form of keyword arguments at invocation would definitely make quite a few of my scripts more readable.
- dgan 1y ago
  there should definitely be an ocaml-like equivalent of "~argument"
  - 1y ago
joshdavham 1y ago
Are there other ways to get this argument-enforcing behaviour in functions, not just data classes?
- pansa2 1y ago
  Keyword-only arguments? Yes: https://peps.python.org/pep-3102/ https://peps.python.org/pep-3102/ More recently, Python also added support for positional-only parameters: https://peps.python.org/pep-0570/ https://peps.python.org/pep-0570/
est 1y ago
I tried using python dataclasses in some projects, the only complaint is that I can not directly pass any dict to a dataclass but have to lint only supported keys. I think a better way to init a dataclass would be something like `mydataclass(my_dict, lint=True)` instead of passing values as kwargs.
- x187463 1y ago
  What do you mean pass a dict? As in kwargs? kwargs = {'arg1':'val1', 'arg2':'val2'} my_obj = MyDataclass(**kwargs) Works fine for a dataclass with arg1 and arg2 attributes.
  - never_inline 1y ago
    Doesnt work recursively?
    - maleldil 1y ago
      At that point, you probably want proper validation with something like Pydantic, where you can use MyType.model_validate(dict).
  - est 1y ago
    i mean if `arg2` wasn't defined in MyDataclass, there would an error. You have to exclude fields first.
deleted 1y ago
[deleted]
meander_water 1y ago
You can also explicitly specify which arguments need to be keyword only using the KW_ONLY sentinel type annotation: from dataclasses import KW_ONLY @dataclass class Point: x: float _: KW_ONLY y: float z: float p = Point(0, y=1.5, z=2.0)
- mixedmath 1y ago
  I've never seen this before. But yes, this does work. And that's very nice. Thank you for the tip.
- chipx86 1y ago
  This is a great feature for those who want a mix of positional and keyword-only arguments. I should have mentioned originally (and I've since updated my post) that this and the kw_only= flag both require Python 3.10 and higher, so code that works with older versions can't opt into it yet.
eachro 1y ago
Is there a reason to use data classes over pedantic base models anymore?
- gjvc 1y ago
  did you mean: "pydantic base models" ?
  - eachro 1y ago
    Yeah haha I got autocorrected
- alfons_foobar 1y ago
  I guess some prefer to stick with the stdlib instead of third party libs. Also, dataclasses feels more straightforward and less "magic" to me (in the sense that it is more or less "just" a way to avoid boilerplate for class definition, while pydantic does way more "magic" stuff like de-/serialization and validation, and adding numerous methods and attributes to the classes).
- guappa 1y ago
  speed? Not pulling in a huge dependency?
- mvieira38 1y ago
  Speed and size, mainly. If you don't need the data validation there's no reason to use pydantic, it's a huge dependency
- JimDabell 1y ago
  I’ve never really gotten along with Pydantic. Something about it just doesn’t feel ergonomic. If I need something more than dataclasses, I’ll normally go for attrs/cattrs. Dataclasses were originally based on attrs, so it’s not much of a leap.
  - LtWorf 1y ago
    I never understood why basemodel even exists. When I started to implement typedload, when types were just introduced, I supported NamedTuple, and then as more things were added, also attrs, dataclasses, typed dict… What would be the point to require migrating the whole codebase to use something different to use your library? On the other hand, if you wrote your code from scratch to use basemodel you're pretty much stuck with pydantic.
semiinfinitely 1y ago
I need another blog like this but for slots=True
spstoyanov 1y ago
Nice! Explicit is better.

95 points by Bogdanp 1y ago

You could argue the same thing (forcing kwargs) for all Python functions/methods, although, that would make using your APIs very annoying. The `__init__` method for dataclasses are just another method like any other. As a general rule of thumb, I only start forcing kwargs once I'm looking at above 4-5 arguments, or if the arguments are similar enough that forcing kwargs makes the calling code more readable. For a small number of distinct arguments, forcing kwargs as a blanket rule makes the code verbose for little gain IMO.

masklinn 1y ago
> You could argue the same thing (forcing kwargs) for all Python functions/methods, although, that would make using your APIs very annoying. The `__init__` method for dataclasses are just another method like any other. While that is self evident at a technical level, it is not quite so from a clarity / documentary perspective: “normal” functions and methods can often hint at their parameters through their naming but it is uncommon for types, for which the composite tends to be much more of an implementation detail. Of course neither rule is universal e.g. the composite is of prime importance for newtypes, and indeed they often use tuple-style types or have special support with no member names.
vbezhenar 1y ago
> that would make using your APIs very annoying For Objective C, using named parameters is the only way to call methods. I don't think I read many critique about this particular aspect. IMO it's actually a good thing and increases readability quite a bit. For JavaScript/TypeScript React codebase, using objects as a poor man's named parameters also very popular approach. Also I'd like to add, that it seems a recent trend to add feature to IDEs, where it'll add hint for every parameter, somewhat simulating named parameters. So when you write `mymethod(value)`, it'll display it as `mymethod(param:value)`. So may be not very annoying. The only little thing I'd like to borrow from JavaScript is using "shortcut", so you could replace `x=x` with `x`, if your local variable happened to have the same name, as parameter name (which happens often enough).
- laserlight 1y ago
  To be pedantic, Objective-C doesn't have named parameters. Method names are composed of multiple parts, each corresponding to a parameter. Such design contributes to the method's readability and memorability. In contrast, Python methods have their own names, and parameter names are chosen as an afterthought. While there's no reason why Python methods couldn't be named in accordance with parameter names, unfortunately it hasn't been a part of Python's culture.
int_19h 1y ago
I find that anything above 2 arguments benefits from explicit keyword notation. With 4-5 arguments, especially when most of them are of the same type, it can be difficult to tell which is which.

deleted 1y ago

[deleted]

joshdavham 1y ago

> Positional arguments means a caller can use MyDataClass(1, 'foo', False), and if you remove/reorder any of these arguments, you’ll break those callers unexpectedly. By forcing callers to use MyDataClass(x=1, y='foo', z=False), you remove this risk. This is an awesome way to prevent future breaking changes! ...but unfortunately, adding this to an existing project would also likely result in breakings changes haha

chipx86 1y ago
That's always the challenge when iterating on interfaces that other people depend on. What we do is go through a deprecation phase. Our process is: * We provide compatibility with the old signature for 2 major releases. * We document the change and the timeline clearly in the docstring. * The function gets decorated with a helper that checks the call, and if any keyword-only arguments are provided as positional, it warns and converts them to keyword-only. * After 2 major releases, we move fully to the new signature. We buit a Python library called housekeeping (https://github.com/beanbaginc/housekeeping https://github.com/beanbaginc/housekeeping) to help with this. One of the things it contains is a decorator called `@deprecate_non_keyword_only_args`, which takes a deprecation warning class and a function using the signature we're moving to. That decorator handles the check logic and generates a suitable, consistent deprecation message. That normally looks like: @deprecate_non_keyword_only_args(MyDeprecationWarning) def my_func(*, a, b, c): ... But this is a bit more tricky with dataclasses, since `__init__()` is generated automatically. Fortunately, it can be patched after the fact. A bit less clean, but doable. So here's how we'd handle this case with dataclasses: from dataclasses import dataclass from housekeeping import BaseRemovedInWarning, deprecate_non_keyword_only_args class RemovedInMyProject20Warning(BaseRemovedInWarning): product = 'MyProject' version = '2.0' @dataclass(kw_only=True) class MyDataclass: a: int b: int c: str MyDataclass.__init__ = deprecate_non_keyword_only_args( RemovedInMyProject20Warning )(MyDataclass.__init__) Call it with some positional arguments: dc = MyDataclass(1, 2, c='hi') and you'd get: testdataclass.py:26: RemovedInMyProject20Warning: Positional arguments `a`, `b` must be passed as keyword arguments when calling `__main__.MyDataclass.__init__()`. Passing as positional arguments will be required in MyProject 2.0. dc = MyDataclass(1, 2, c='hi') We'll probably add explicit dataclass support to this soon, since we're starting to move to kw_only=True for dataclasses.

FridgeSeal 1y ago

Oh man, the aws Boto3 library does this for a huge number of calls, and it’s awful. “What parameters does this take?” you ask, “why, it takes ‘kwargs’” responds the docs and your IDE. How incredibly helpful!

chipx86 1y ago
That's annoying for sure. Though a different problem. All the kw_only=True argument for dataclasses does is require that you pass any fields you want to provide as keyword arguments instead of positional arguments when instantiating a dataclass. So: obj = MyDataclass(a=1, b=2, c=3) Instead of: obj = MyDataclass(1, 2, 3) # This would be an error with kw_only=True The problem you're describing in boto3 (and a lot of other API bindings, and a lot of more layered Python code) is that methods often take in **kwargs and pass them down to a common function that's handling them. From the caller's perspective, **kwargs is a black box with no details on what's in there. Without a docstring or an understanding of the call chain, it's not helpful. Python sort of has a fix for this now, which is to use a TypedDict to define all the possible values in the **kwargs, like so: from typing import TypedDict, Unpack class MyFuncKwargs(TypedDict): arg1: str arg2: str arg3: int | None def my_outer_func( **kwargs: Unpack[MyFuncKwargs], ) -> None: _my_inner_func(**kwargs) def _my_inner_func( *, arg1: str, arg2: str, arg3: int | None, ) -> None: ... By defining a TypedDict and typing **kwargs, the IDE and docs can do a better job of showing what arguments the function really takes, and validating them. Also useful when the function is just a wrapper around serializing **kwargs to JSON for an API, or something. But this feature is far from free to use. The more functions you have, the more of these you need to create and maintain. Ideally, a function could type **kwargs as something like: def my_outer_func( **kwargs: KwargsOf[_my_inner_func], ) -> None: ... And then the IDEs and other tooling can just reference that function. This would help make the problem go away for many of the cases where **kwargs is used and passed around.
- mvieira38 1y ago
  TypedDicts are so underutilized in general. I'm using them a lot even for simpler scripts

ayhanfuat 1y ago

I like keyword-only arguments, but they become tedious too quickly - especially when the variable names already match (fn(x=x, y=y, z=z)). I wish Python had JavaScript’s shorthand property syntax. (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Object_initializer#property_definitions https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...).

sweetgiorni 1y ago
Python 3.14 adds that shorthand, but with a slight different (and IMO uglier) syntax: fn(x=,y=,z=) https://peps.python.org/pep-0736/ https://peps.python.org/pep-0736/ edit: nevermind, that PEP was rejected :/
- pansa2 1y ago
  That PEP says “rejected”
  - snickerdoodle12 1y ago
    Thankfully so. As much as I want a shorthand, this syntax wasn't it.
- deleted 1y ago
  [deleted]
- vovavili 1y ago
  By the look of it, the feeling I get is that a decent and convenient syntax proposal has been bikeshedded into rejection. Some particular form of keyword arguments at invocation would definitely make quite a few of my scripts more readable.
dgan 1y ago
there should definitely be an ocaml-like equivalent of "~argument"
- 1y ago

joshdavham 1y ago

Are there other ways to get this argument-enforcing behaviour in functions, not just data classes?

pansa2 1y ago
Keyword-only arguments? Yes: https://peps.python.org/pep-3102/ https://peps.python.org/pep-3102/ More recently, Python also added support for positional-only parameters: https://peps.python.org/pep-0570/ https://peps.python.org/pep-0570/

est 1y ago

I tried using python dataclasses in some projects, the only complaint is that I can not directly pass any dict to a dataclass but have to lint only supported keys. I think a better way to init a dataclass would be something like `mydataclass(my_dict, lint=True)` instead of passing values as kwargs.

x187463 1y ago
What do you mean pass a dict? As in kwargs? kwargs = {'arg1':'val1', 'arg2':'val2'} my_obj = MyDataclass(**kwargs) Works fine for a dataclass with arg1 and arg2 attributes.
- never_inline 1y ago
  Doesnt work recursively?
  - maleldil 1y ago
    At that point, you probably want proper validation with something like Pydantic, where you can use MyType.model_validate(dict).
- est 1y ago
  i mean if `arg2` wasn't defined in MyDataclass, there would an error. You have to exclude fields first.

deleted 1y ago

[deleted]

meander_water 1y ago

You can also explicitly specify which arguments need to be keyword only using the KW_ONLY sentinel type annotation: from dataclasses import KW_ONLY @dataclass class Point: x: float _: KW_ONLY y: float z: float p = Point(0, y=1.5, z=2.0)

mixedmath 1y ago
I've never seen this before. But yes, this does work. And that's very nice. Thank you for the tip.
chipx86 1y ago
This is a great feature for those who want a mix of positional and keyword-only arguments. I should have mentioned originally (and I've since updated my post) that this and the kw_only= flag both require Python 3.10 and higher, so code that works with older versions can't opt into it yet.

eachro 1y ago

Is there a reason to use data classes over pedantic base models anymore?

gjvc 1y ago
did you mean: "pydantic base models" ?
- eachro 1y ago
  Yeah haha I got autocorrected
alfons_foobar 1y ago
I guess some prefer to stick with the stdlib instead of third party libs. Also, dataclasses feels more straightforward and less "magic" to me (in the sense that it is more or less "just" a way to avoid boilerplate for class definition, while pydantic does way more "magic" stuff like de-/serialization and validation, and adding numerous methods and attributes to the classes).
guappa 1y ago
speed? Not pulling in a huge dependency?
mvieira38 1y ago
Speed and size, mainly. If you don't need the data validation there's no reason to use pydantic, it's a huge dependency
JimDabell 1y ago
I’ve never really gotten along with Pydantic. Something about it just doesn’t feel ergonomic. If I need something more than dataclasses, I’ll normally go for attrs/cattrs. Dataclasses were originally based on attrs, so it’s not much of a leap.
- LtWorf 1y ago
  I never understood why basemodel even exists. When I started to implement typedload, when types were just introduced, I supported NamedTuple, and then as more things were added, also attrs, dataclasses, typed dict… What would be the point to require migrating the whole codebase to use something different to use your library? On the other hand, if you wrote your code from scratch to use basemodel you're pretty much stuck with pydantic.

semiinfinitely 1y ago

I need another blog like this but for slots=True

spstoyanov 1y ago

Nice! Explicit is better.