4 ms·

These decorators are an absolute nightmare, sorry. - @parallel: Doesn't let the caller configure the actual amount of parallelism, and no, the only overhead is

These decorators are an absolute nightmare, sorry. - @parallel: Doesn't let the caller configure the actual amount of parallelism, and no, the only overhead is not "having to define a merge function". Creating the processes has quite a bit of overhead and calls might end up being far slower than just a regular serial execution. Also, you might not want to eat all CPUs on a single function (not to mention that cpu_count tells you the amount of CPUs in the system, not the amount that are available to your process). - @production/@deployable: Functions that sometimes return values and sometimes don't depending on the hostname? That's just a bug waiting to happen. Also, at least use an environment flag and not a hardcoded list of servers. - @redirect: Don't reinvent the wheel, use the logging module. Log messages will be more useful, you'll be able to redirect them wherever you want, enable, disable, increase the level, even use different logging modules and enable/disable them at runtime. There are even handlers for pretty IPython output. - @stacktrace/@traceable Somewhat more useful but, again, the logging module usually does this. Also, debuggers exist. If anyone is using this, be very aware of where you're using them, how and possible side effects and maintenance problems. Personally, I wouldn't let anything like this run in a somewhat serious codebase.

deleted 4y ago
[deleted]
s_Hogg 4y ago
Another classic of the genre I've seen is excessive memoization, done using precisely these sorts of decorators. By the time it has to be debugged in production, the culprit has left the building. It can even obscure the source of other bugs on top of the ones it introduces. Long story short, if you want to have a cache, have a cache. If you want to make something parallel, make it parallel. This stuff just gets in the way. The only real exceptions I make for decorators by and large are for pydantic things and similar. It's very noticeable that if you're doing stuff in pytorch lightning you don't tend to fiddle with decorators much at all but in tensorflow it's common. Cute code isn't good code, end of story
misnome 4y ago
- @redirect - logging This suggestion (logging) doesn't help if it's e.g. other code you don't have control over, but this is also available as a context manager in the standard library as "redirect_stdout": https://docs.python.org/3/library/contextlib.html#contextlib.redirect_stdout https://docs.python.org/3/library/contextlib.html#contextlib... At least the article doesn't advocate inserting them into __builtins__ (which I have seen people do with their own special custom functions before).
- gjulianm 4y ago
  > This suggestion (logging) doesn't help if it's e.g. other code you don't have control over, Most of the time, other code uses the logging module so you do have control over that output. I haven't seen code not controlled by me that uses print() without me wanting it to.
  - dragonwriter 4y ago
    > Most of the time, other code uses the logging module so you do have control over that output. I haven't seen code not controlled by me that uses print() without me wanting it to. If you are a traditional software engineer, the other people's code you deal with as an upstream depency probably looks different than the other people's code data (or many other) scientists work with.
xorcist 4y ago
Agreed, most of these would be more suitable as configuration.
gknoy 4y ago
> @stacktrace/@traceable Somewhat more useful but, again, the logging module usually does this. Also, debuggers exist. While logging is good for production code, the tracing helpers look very useful for developing or debugging. One of the first things I do when troubleshooting why something is failing (in tests, or when reproducing things locally where I can run things with a visible console) is to print out _what was passed in_, or what the mid-function state is, so that I don't have to constantly query them in the debugger. Many many lines that look like print(f">>> foo: {foo}, bar: {bar}") Being able to automate this in a more convenient way seems very useful. The icecream library [0] does this, probably in a more robust way, but I've never used it because I always forget about it. 0: https://github.com/gruns/icecream https://github.com/gruns/icecream

deleted 4y ago

[deleted]

s_Hogg 4y ago

Another classic of the genre I've seen is excessive memoization, done using precisely these sorts of decorators. By the time it has to be debugged in production, the culprit has left the building. It can even obscure the source of other bugs on top of the ones it introduces. Long story short, if you want to have a cache, have a cache. If you want to make something parallel, make it parallel. This stuff just gets in the way. The only real exceptions I make for decorators by and large are for pydantic things and similar. It's very noticeable that if you're doing stuff in pytorch lightning you don't tend to fiddle with decorators much at all but in tensorflow it's common. Cute code isn't good code, end of story

misnome 4y ago

- @redirect - logging This suggestion (logging) doesn't help if it's e.g. other code you don't have control over, but this is also available as a context manager in the standard library as "redirect_stdout": https://docs.python.org/3/library/contextlib.html#contextlib.redirect_stdout https://docs.python.org/3/library/contextlib.html#contextlib... At least the article doesn't advocate inserting them into __builtins__ (which I have seen people do with their own special custom functions before).

gjulianm 4y ago
> This suggestion (logging) doesn't help if it's e.g. other code you don't have control over, Most of the time, other code uses the logging module so you do have control over that output. I haven't seen code not controlled by me that uses print() without me wanting it to.
- dragonwriter 4y ago
  > Most of the time, other code uses the logging module so you do have control over that output. I haven't seen code not controlled by me that uses print() without me wanting it to. If you are a traditional software engineer, the other people's code you deal with as an upstream depency probably looks different than the other people's code data (or many other) scientists work with.

xorcist 4y ago

Agreed, most of these would be more suitable as configuration.

gknoy 4y ago

> @stacktrace/@traceable Somewhat more useful but, again, the logging module usually does this. Also, debuggers exist. While logging is good for production code, the tracing helpers look very useful for developing or debugging. One of the first things I do when troubleshooting why something is failing (in tests, or when reproducing things locally where I can run things with a visible console) is to print out _what was passed in_, or what the mid-function state is, so that I don't have to constantly query them in the debugger. Many many lines that look like print(f">>> foo: {foo}, bar: {bar}") Being able to automate this in a more convenient way seems very useful. The icecream library [0] does this, probably in a more robust way, but I've never used it because I always forget about it. 0: https://github.com/gruns/icecream https://github.com/gruns/icecream