3 ms·

It’s also critical in Python to make heavy use of metaprogramming with decorators, so the bundling / unbundling operators combined with named parameters aren’t

by mlthoughts2018 6y ago

It’s also critical in Python to make heavy use of metaprogramming with decorators, so the bundling / unbundling operators combined with named parameters aren’t just for ease of reading or annotating call sites, but rather they open up a huge range of behavior that can only work if decorators can easily manipulate parameters of the functions they are decorating. For example, suppose you have a function foo and within foo you call another function baz, where baz has a very large range of customization parameters. If you want to memoize calls to foo with a decorator, you can just pass kwargs for any “pass through” customizations of baz. The memoizing decorator can flatten kwargs out into a tuple along with any positional args, so you get call signature memoization without needing to restrict or list out all possible args of baz. In fact you can make this perfectly generic over the function being memoized so it doesn’t matter if you change baz to bar one day, with a whole new signature. You can combine this with the inspect module and with functools.wraps to ensure full and complete docstrings and introspected signatures as well, so that putting kwargs all over doesn’t hurt readability, remove info from docstrings or help messages, or make it hard to read.

benrbray 6y ago
> It’s also critical in Python to make heavy use of metaprogramming with decorators Interesting use case, thanks for sharing! Out of curiosity, when are some scenarios where it's "critical" to write code this way? I mostly interact with small pytorch / numpy codebases that don't grow large enough for this sort of thing.
- mlthoughts2018 6y ago
  Pytest as a unit test framework is a great example. Some other good ones include the contextmanager decorator from contextlib, the retrying.retry decorator, gradual type enforcement decorators, decorators for control web app routing and HTTP / REST parametrics like in Flask. Even in hardcore numerical computing this way of working has huge wins over traditional OOP designs or plain flat function designs like C. For example see numba’s jit and autojit decorators that also work this way. For cases when you want to do it yourself, think of functions that wrap complex matplotlib or seaborn plotting utilities that rely on tons of optional parameters via kwargs. I often find it super helpful to do this when wrapping pandas functions too, which also involve tons of kwargs customization. FWIW I work professionally in deep learning and other ML. I think scientific computing in Python in particular highlights the importance of this.

benrbray 6y ago

> It’s also critical in Python to make heavy use of metaprogramming with decorators Interesting use case, thanks for sharing! Out of curiosity, when are some scenarios where it's "critical" to write code this way? I mostly interact with small pytorch / numpy codebases that don't grow large enough for this sort of thing.

mlthoughts2018 6y ago
Pytest as a unit test framework is a great example. Some other good ones include the contextmanager decorator from contextlib, the retrying.retry decorator, gradual type enforcement decorators, decorators for control web app routing and HTTP / REST parametrics like in Flask. Even in hardcore numerical computing this way of working has huge wins over traditional OOP designs or plain flat function designs like C. For example see numba’s jit and autojit decorators that also work this way. For cases when you want to do it yourself, think of functions that wrap complex matplotlib or seaborn plotting utilities that rely on tons of optional parameters via kwargs. I often find it super helpful to do this when wrapping pandas functions too, which also involve tons of kwargs customization. FWIW I work professionally in deep learning and other ML. I think scientific computing in Python in particular highlights the importance of this.