12 ms·
Useful Python decorators for data scientists
- d4rkp4ttern 4y agoDecorators make it hard to step through functions in a debugger. Or hard to see what is going on.
- mrtranscendence 4y agoIs it really that bad? At the end of the day a decorator is itself just as function call, not particularly magical in any way. The alternative would be reimplementing the logic of the decorator in the function to be decorated, which isn't necessarily better.
- jstx1 4y agoIt's kind of cool that this is possible but I'm not adding that kind of complexity to my code unless I really need to. And I really never need to.
- z3c0 4y agoDecorators are one of those language features that I want to use more of, but every time I attempt to, I realize that what I'm doing could be achieved more easily another way.
- zmgsabst 4y agoI use two with any regularity: 1. @dataclass https://docs.python.org/3/library/dataclasses.html https://docs.python.org/3/library/dataclasses.html 2. @cuda.jit https://numba.pydata.org/numba-doc/latest/cuda/kernels.html https://numba.pydata.org/numba-doc/latest/cuda/kernels.html
- dharmab 4y agoAnother I use is tenacity, when I need to call unreliable external services: https://tenacity.readthedocs.io/en/latest/ https://tenacity.readthedocs.io/en/latest/
- monkeybutton 4y agoAside from @dataclass, @lru_cache is my most used.
- ConceptJunkie 4y agoI created a decorator for validating arguments for functions that are receiving arguments from user input. It eliminated _so much_ boilerplate code. I use decorators quite a bit. The feature is incredibly powerful. Once you find @lru_cache, you start realizing how many ways you can take advantage of the feature.
- pizza 4y ago@cached_property is the new lru_cache for me, much of the time :^)
- Abishek_Muthian 4y agoHave felt the same way till I used one for fastapi , One can literally implement production grade* Machine Learning pipeline in 15 lines of code using transformers in a single python file; from fastapi import FastAPI from pydantic import BaseModel, constr, conlist from typing import List from transformers import pipeline classifier = pipeline("zero-shot-classification", model="models/distilbert-base-uncased-mnli") app = FastAPI() class UserRequestIn(BaseModel): text: constr(min_length=1) labels: conlist(str, min_items=1) class ScoredLabelsOut(BaseModel): labels: List[str] scores: List[float] @app.post("/classification", response_model=ScoredLabelsOut) def read_classification(user_request_in: UserRequestIn): return classifier(user_request_in.text, user_request_in.labels) *: Production grade if used in combination with workers, A python quirk I felt is not relevant to the topic of decorators.
- Galanwe 4y agoDear god, so much antipatterns on a single page. Interleaving configuration management in the business logic code is a massive technical debt waiting to engulf future generations in a world of maintainance pain.
- Maro 4y agoIt's example code in a blog post.
- joelschw 4y agoI'm not sure it's framed like that, it's specifically talking about production patterns
- pvg 4y ago'production' for many kinds of data analysis is not the same kind of production as for, say, a web app or service. You can have a production environment for running ad hoc jobs.
- deleted 4y ago[deleted]
- bigDinosaur 4y agoUnironically that means it will end up in hundreds if not thousands of production codebases.
- roomey 4y agoI'm so glad I'm not the only one. I was reading the code thinking this seems like a different language to the one I use (albeit I'm not too experienced). That seems obtuse to me in the extreme. Maybe I'm just not used to that style however.
- deleted 4y ago[deleted]
- lpmiss 4y ago
- Synersoft 4y ago[dead]
- henrydark 4y agoAll of these are great examples of useful higher functions, meaning they take and output functions. I just wouldn't use them as decorators in proper code.
- sscarduzio 4y agoWhy not tho? Decorators are very explicit markers of higher functions. When I see one, I know what's for.
- LaMarineta 4y ago
- henrydark 4y agoIt's rare that all users of a function will want it to be "production" or "parallel" or the rest in the piece. Applying these functions as decorators, i.e. with @, means you can't run the non parallel version, or the test not in production, etc. In the end, decorators, though nice on the first day of usage, reduce composability by restricting usage to whatever you wanted on that same day. (this is not a general remark, it doesn't apply to DSLs that use decorators, e.g. flask)
- dragonwriter 4y agoTo the extent that this is a real issue, you could just use a decorator that exposed the wrapped function as an attribute of the returned function object. You could even write a higher order function, callable as a decorator, that would transform a decorator that didn't do this into one that was otherwise identical that did.
- LaMarineta 4y ago
- jimmytucson 4y agoYou’re saying write it this way instead? def my_func: ... my_decorated_func = my_decorator(my_func)
- gjulianm 4y agoThese decorators are an absolute nightmare, sorry. - @parallel: Doesn't let the caller configure the actual amount of parallelism, and no, the only overhead is not "having to define a merge function". Creating the processes has quite a bit of overhead and calls might end up being far slower than just a regular serial execution. Also, you might not want to eat all CPUs on a single function (not to mention that cpu_count tells you the amount of CPUs in the system, not the amount that are available to your process). - @production/@deployable: Functions that sometimes return values and sometimes don't depending on the hostname? That's just a bug waiting to happen. Also, at least use an environment flag and not a hardcoded list of servers. - @redirect: Don't reinvent the wheel, use the logging module. Log messages will be more useful, you'll be able to redirect them wherever you want, enable, disable, increase the level, even use different logging modules and enable/disable them at runtime. There are even handlers for pretty IPython output. - @stacktrace/@traceable Somewhat more useful but, again, the logging module usually does this. Also, debuggers exist. If anyone is using this, be very aware of where you're using them, how and possible side effects and maintenance problems. Personally, I wouldn't let anything like this run in a somewhat serious codebase.
- deleted 4y ago[deleted]
- s_Hogg 4y agoAnother classic of the genre I've seen is excessive memoization, done using precisely these sorts of decorators. By the time it has to be debugged in production, the culprit has left the building. It can even obscure the source of other bugs on top of the ones it introduces. Long story short, if you want to have a cache, have a cache. If you want to make something parallel, make it parallel. This stuff just gets in the way. The only real exceptions I make for decorators by and large are for pydantic things and similar. It's very noticeable that if you're doing stuff in pytorch lightning you don't tend to fiddle with decorators much at all but in tensorflow it's common. Cute code isn't good code, end of story
- misnome 4y ago- @redirect - logging This suggestion (logging) doesn't help if it's e.g. other code you don't have control over, but this is also available as a context manager in the standard library as "redirect_stdout": https://docs.python.org/3/library/contextlib.html#contextlib.redirect_stdout https://docs.python.org/3/library/contextlib.html#contextlib... At least the article doesn't advocate inserting them into __builtins__ (which I have seen people do with their own special custom functions before).
- TheAlchemist 4y agoA better title would be 'Useful Python Decorators by a Data Scientist'. Those decorators are exactly what data scientists would do, while software engineers would be terrified.
- rmbyrro 4y ago> while software engineers would be terrified At least surprised. There are solutions to these problems already. Author seemed to have learned decorators and is enthusiastic about abusing them, instead of learning the stdlib. The capturing of print statements. Why not use the Logging machinery, instead? Or the @stacktrace, it seems what they really want/need is a debugger. But anyway, if these solutions fit their programming style better, so be it.
- agumonkey 4y agoTo be fair, logging is badly made. There are at least two libs to make it human again.
- rmbyrro 4y agoIndeed fair. But even if I preferred the decorator style, I'd be cautious about dropping a stdlib or any other mature solution. When I create my own solutions, I tend to underestimate how hard it will be to get it working bug-free and to maintain it.
- anu7df 4y agoWhich ones do you suggest/recommend?
- imcoconut 4y agoim a big fan of loguru.
- mattkrause 4y ago@redirect could be handy when the print statements are embedded in someone else's code. We use a library that is very...chatty (some function calls send a screenful of info/progress to the screen), and I think I'm going to steal this to make it quieter.
- kwertyoowiyop 4y ago@production looks risky, compared to checking an environment variable. Hopefully that host_name function is guaranteed to return the same result every time it is called, and can never fail or raise an exception.
- rmbyrro 4y agoIt's also a bad idea, because it implies you have different implementations for each environment. Which means your dev and production environments are quite different, increasing risks of letting stupid mistakes slip into prod.
- kwertyoowiyop 4y agoOh so true. No matter how you express it, “if production” will likely land you in debugging hell sooner or later.
- giantrobot 4y agoYou can do the @production one with an envar or any other mechanism. I've been using it for years (with an envar) and it works really well. Besides switching between something like production and test you can catch a case where an envar isn't set or is an unexpected value.
- julvo 4y agoTo add to the list of very handy footguns: @reloading to hot-reload a function from source before every invocation (https://github.com/julvo/reloading https://github.com/julvo/reloading)
- gknoy 4y agoThat looks amazing, thank you! I wouldn't want it in my production code but it seems like it would be great for something that I'm working on in the REPL.
- neves 4y agoif you are using a REPL, first you should be using ipython. Second you should be running: %load_ext autoreload %autoreload 2 now all your imports will be automatically reloaded. So good that I have it in ipython startup scripts
- pizza 4y agoThere's also reloadium [0] for tight dev iteration loops https://reloadium.io https://reloadium.io
- TekMol 4y agoThe first example illustrates perfectly why I dislike explicit typing: primes: set[int] = set() candidate: int = randint(4, domain) It's just so ugly and redundant. My feeling is that redundant stuff like type hints make programmers create more bugs because they don't see the forest for the trees anymore. As if you wrapped the driver of a car in so many seat belts that they cannot see the road anymore.
- jsmeaton 4y ago> My feeling is that redundant stuff like type hints make programmers create more bugs because they don't see the forest for the trees anymore. Not ones that ship to production. Some developers have a terrible time trying to get through the mypy errors. But from my observations, I haven't seen any typing related errors in a very long time.
- iknownothow 4y agoI don't exactly know what Pycharm does but I only add type hints when Pycharm's default type checker raises a complaint or is unable to infer the type. To me that's just the perfect middle ground. Sometimes adding a type to a single object (e.g. `df: pd.DataFrame = ...`) only once at the top of a 20-50 line function, is usually sufficient for Pycharm to infer all other types downstream without ambiguity.
- gjulianm 4y ago> My feeling is that redundant stuff like type hints make programmers create more bugs because they don't see the forest for the trees anymore. I've been using type hints heavily and it's been such a great help in avoiding bugs and also faster completion in IDEs. Most of the time, with correctly typed functions you don't have to write explicit types for variables. It's a little bit more code, yes, but it's definitely worth it for any codebase that grows more than a handful of files. Even for smaller ones I have found it very useful.
- mrtranscendence 4y ago> I've been using type hints heavily and it's been such a great help in avoiding bugs and also faster completion in IDEs Agreed. I've been using type hints for the past few years now even though most of my projects are fairly small (< 10,000 LOC). I wouldn't say they've prevented a huge amount of bugs, but they've definitely prevented a few, and I think they've helped in library design and documentation.
- didip 4y agoI never liked decorator/annotation pattern. It sweeps complexity under the rug with a deceivingly simple facade.
- dymk 4y agoThat’s called abstraction. Decorators are sugar over simple composition, nothing more.
- lysecret 4y agoI am struggling a bit with decorators. I find myself in a lot of situations, where I could use a decorator on a function, but only some of the time. In this case its much more convenient to design it as a higher order function.
- lysecret 4y agoI am strugeling a bit with the @production decorator. I never had the issue of only wanting to run a function on prod. Often I want slightly different behaviour, but then id use env variables. (say using a Prod and a Dev API adr or DB adr). Wouldnt it also be better design to keep dev and prod as close as possible?
- giantrobot 4y agoThere's more environments than just dev and prod. Even if the environments differ having some extra telemetry in a test environment often makes sense. So a @telemetry or @production decorator that does something in a non-production environment is an easy way to add that capability to functions. I've found I like a decorator better than some envar testing custom logging/telemetry. If I have a custom_log function I use everywhere, I have to use it everywhere I might want it. With a decorator I can add it to only a few functions and get far less noise.
- anu7df 4y agoOne place I am using decorators is pytorch in combination with hydra. Decorating functions so that an instantiation returns a function , not a call of the function. This allows easy reuse of existing functions in ETL pipeline setting it up with config files and hydra. But I have to say, the native python logging module is pure evil.
- memco 4y agoI just posted another comment advocating that using logging would be better than redirect: I use it quite extensively in production and am mostly happy with its setup and use: what makes it evil for you and what is different about data science that makes it so?
- superbatfish 4y agoThe redirect() method shown in this post only works for output that is written via the Python sys.stdout handle. If your code links against C/C++ extensions that have internal printf calls, then you need something lower-level, which can intercept the data at the file handle. There's a great solution for that on StackOverflow: https://stackoverflow.com/a/22434262/162094 https://stackoverflow.com/a/22434262/162094
- memco 4y agoEven on the Python level it seems much better to use the logging module and attach a handler to sys.stdout. if all you need is to print some stuff to the console just attach a handler with logger.addHandler(logging.StreamHandler(sys.stdout)). You can attach other handlers for files or to send the logs via some network interface and you can silence or filter as desired. You can even attach metadata so that you can process the messages in different ways depending on what you want to do with that metadata. This can all be done separate from the calling code so the main logic just need to know log this message at this severity and all the processing filtering and transmission will be handled by the logger.
- bbkane 4y agoCould you give an example of "attach metadata"? Are you talking about attaching metadata to the logger object or handlers? How is that done? If I recall correctly, the only thing you can send with an individual log event is a string message, and an log level, no other metadata. I looked into this because I love structured logging, and I wasn't able to figure out how to make that work easily with the default logging module.
- memco 4y agoA solution I have used is to creat a logging recordfactory similar to what’s described here: https://stackoverflow.com/questions/59585861/using-logrecordfactory-in-python-to-add-custom-fields-for-logging https://stackoverflow.com/questions/59585861/using-logrecord.... There’s also logging adapters as described here: https://docs.python.org/3/howto/logging-cookbook.html https://docs.python.org/3/howto/logging-cookbook.html. The docs how-to also describes how to add contextual information via a filter. Not knowing your particular setup I can’t say how well any of these will work for you, but these gave me all the tools I needed to selectively attach IDs and other metadata to celery task logs so I can easily grep for the task ID or a few other unique pieces of data and easily see the whole log for a given task or for tasks that all used the same device, etc.
- diarrhea 4y agoThis is either a bug or could be handled using `enumerate(..., start=1)`: for i, line in enumerate(lines): i += 1
- spywaregorilla 4y agoI've only really ever needed one decorator. The lovely @fuckit from fuckit. Tells your code to ignore all errors and keep on truckin. Great for prod if an error is unnacceptable (or more realistically, web scraping where you just don't care about one off errors)
- staticautomatic 4y agoDoes it just wrap the code in a context manager where __exit__ returns True or something?
- dnadler 4y agoI'd imagine it's just a try/except wrapper with a funny name.
- spywaregorilla 4y agoIt's not just a try/except wrapper, it's a try/except wrapper for every single statement individually in a given class or function :) It has some ast stuff built in. @fuckit def this_is_fine(garbage_in): x = 1 / 0 return "garbage_out" this_is_fine("") ### Returns "garbage out"
- stainablesteel 4y ago> for data scientists > Let's assume I write a really inefficient way to find primes let me stop you right there
- blunova 4y agoI often use my custom timer decorator to time execution of functions/methods. This is not the only way I can do that, but I think it's a very convenient option.