8 ms·

A Guide to Python's Magic Methods (2012)

194 points by apsec112 8y ago

kqr 8y ago
This is a great explanation and a must-read for Python developers. Once you know this stuff, you'll be a much stronger debugger. But it's also worth throwing a warning out there: when you first learn about a new type of magic, you'll be tempted to use it everywhere. Be careful! I can recognise a clear specific period in my Python work which was just after I had read this, and a bunch of things suddenly became implicit.
- florabuzzword 8y ago
  My reaction to this, as someone who doesn’t use many of these on a regular basis (probably guilty of overusing __new__ though) is actually that I can imagine these causing some real problems. The operator ones seem particularly tempting and dangerous at the same time.
  - ubernostrum 8y ago
    Operator overloading, in any language that supports it, has always been controversial, so Python doesn't really add anything new there. The thing is, it's also incredibly useful at times. Language features that are "useful when you have the use case for it, confusing when you don't" tend to provoke that reaction.
    - int_19h 8y ago
      In Python, though, operator overloading is not merely "supported". It's how all operators are defined, including built-in ones. >>> dir(123) ['__abs__', '__add__', '__and__', '__bool__', '__ceil__', '__class__', '__delattr__', '__dir__', '__divmod__', '__doc__', '__eq__', '__float__', '__floor__', '__floordiv__', '__format__', '__ge__', '__getattribute__', '__getnewargs__', '__gt__', '__hash__', '__index__', '__init__', '__init_subclass__', '__int__', '__invert__', '__le__', '__lshift__', '__lt__', '__mod__', '__mul__', '__ne__', '__neg__', '__new__', '__or__', '__pos__', '__pow__', '__radd__', '__rand__', '__rdivmod__', '__reduce__', '__reduce_ex__', '__repr__', '__rfloordiv__', '__rlshift__', '__rmod__', '__rmul__', '__ror__', '__round__', '__rpow__', '__rrshift__', '__rshift__', '__rsub__', '__rtruediv__', '__rxor__', '__setattr__', '__sizeof__', '__str__', '__sub__', '__subclasshook__', '__truediv__', '__trunc__', '__xor__', 'bit_length', 'conjugate', 'denominator', 'from_bytes', 'imag', 'numerator', 'real', 'to_bytes'] >>> (1).__add__ <method-wrapper '__add__' of int object at 0x678C40A0> >>> (123).__add__.__doc__ 'Return self+value.' >>> (1).__add__(2) 3 Which is a good thing, because it makes the whole arrangement a lot more consistent that languages in which this sort of thing is just magic associated with a specific primitive type (sometimes it's even inconsistent - e.g. in C#, + for ints and strings is magic, but + for decimal is an overloaded operator). In Python, all that magic is confined to the magic methods. Learn them once, and they work the same everywhere.
    - florabuzzword 8y ago
      Oh yeah, don’t get me wrong. It’s cool as hell! I’m totally gonna use it for a GUI menu-building script interface I’m making right now :). I have just never done it before, and I think this is the first time I have seen the concept, so the first thing that popped in my mind we’re all the things that can go wrong. But, also all the things that can go right.
- Puer 8y ago
  Could you give an example of how you would use these methods to debug versus other methods/approaches? I'd love to hear if you're willing to share--I program mainly in Python these days but Python's magic methods are definitely something I need to understand at a deeper level.
  - travisjungroth 8y ago
    Fluent Python has a chapter on these methods and I highly recommend it. Also, these are much better described as dunder methods (short for double underscore) than magic methods. There's really no magic to them.
- nomel 8y ago
  > Once you know this stuff, you'll be a much stronger debugger. As you suggest, more accurately, you'll require the next person who debugs your code to read this, making them much better at debugging your code. I no longer use magic and completely avoid meta programming in python. It confuses everyone, is not needed (just use a boring factory function), and severely limits who can help you work on your code, for what? A few lines of less boilerplate here and there? "Succinctness"? Some sort of "elegance"? More likely, a DSL that nobody understands but you! The more I write, and mostly the more I interact with other people, the more I realize that code needs to be boring. No magic. I think this is why go is succeeding: no magic.
  - rootusrootus 8y ago
    100% agree. I figure this is a big part of what turns normal developers into senior developers, and lead developers. It's easy to code. What takes wisdom is 1) knowing what not to write, and 2) self control on how you write it. As a mentor I try to convey this to the fresh new supercoders joining our team.
montecarl 8y ago
This part is a little dated (it is a 2012 post after all): https://rszalski.github.io/magicmethods/#comparisons https://rszalski.github.io/magicmethods/#comparisons In Python 3 __cmp__ is ignored and you have to use the other methods. To make this easier, you can use https://docs.python.org/3/library/functools.html#functools.total_ordering https://docs.python.org/3/library/functools.html#functools.t...
- gvx 8y ago
  Yeah. In Python 3, __div__ is gone, and only __floordiv__ and __truediv__ exist.
- opencl 8y ago
  It's amazing that Python 3 was 4 years old when this post was made and yet saying "Python" was still implicitly assumed by basically everyone to mean 2.x.
  - kjeetgill 8y ago
    Is it that amazing? I don't think I've had use 3 until this year and I still have to flip-flop. AND I say this as someone who was reading peps going into 3's release. Do Ubuntu or Fedora default to python 3 yet? Arch? My RHEL experience isn't as representative.
    - bhrgunatha 8y ago
      Arch's default is Python 3, but there are still many official Python 2 packages, mostly in community i.e. adopted from the AUR: Core: 1 Extra: 96 Community: 822 AUR: Over 2300 depending on how you search
      - lathiat 8y ago
        The official python recommendation is not to "default" the 'python' command as python3, and they should just explicitly be 'python' and 'python3' This is what every distro except arch does (for whatever reason) https://www.python.org/dev/peps/pep-0394/ https://www.python.org/dev/peps/pep-0394/
florabuzzword 8y ago
Wonderful. I’ve never seen this resource before. A related article I got a lot out of was ‘Understanding Python Metaclasses’, an in-depth breakdown of Python class instantiation. Here’s the link to that: https://blog.ionelmc.ro/2015/02/09/understanding-python-metaclasses/ https://blog.ionelmc.ro/2015/02/09/understanding-python-meta...
olooney 8y ago
Be sure to read "Appendix 2: Changes in Python 3" at the very end of the document! I was preparing a long "Actually..." rant before I saw that.
devxpy 8y ago
It is when you get to this part of python, that you achieve true Nirvana. It's absolutely bonkers the extent to which you can be dynamic in this language. It's the very reason why Django is so DRY.
- misiti3780 8y ago
  can you explain what you mean about django ?
  - lalos 8y ago
    This would be an example: https://www.agiliq.com/blog/2018/01/understanding-python-magic-methods-with-djang-querysets/ https://www.agiliq.com/blog/2018/01/understanding-python-mag...
- sbjs 8y ago
  But this article is just explaining operator overloading, right? Doesn't everyone learn that halfway through semester in C++ 101? And operator overloading has nothing to do with DRY, in fact it's generally better to avoid it except for in very niche situations, like adding __add__ to a matrix type.
  - devxpy 8y ago
    https://docs.python.org/3/reference/datamodel.html#customizing-attribute-access https://docs.python.org/3/reference/datamodel.html#customizi...
- nikofeyn 8y ago
  why is this so bonkers? it seems like it's just basic patterns of method overriding and operator overloading. if people think this is so magic, it's almost as if lisps, schemes, and prologs never existed.
  - jlarocco 8y ago
    > if people think this is so magic, it's almost as if lisps, schemes, and prologs never existed. I write quite a bit of Lisp, and the conclusion I've come to is that for 95% of software developers, Lisp, Scheme, and Prolog might as well not exist. Even here on HN, where I'd expect the bar to be a little higher, the amount of incorrect info posted about Lisp is pretty crazy. I'm almost certain most people's only experience with Lisp(s) and Prolog was a week or two using it in their programming languages class at university.
saagarjha 8y ago
> Be careful, however, as there is no guarantee that __del__ will be executed if the object is still alive when the interpreter exits, so __del__ can't serve as a replacement for good coding practices (like always closing a connection when you're done with it. When the interpreter exits, your program's resources should be reclaimed by the operating system, right? So why not just use __del__?
- cjhanks 8y ago
  Say it's a socket and you want to send some sort of "END_OF_STREAM" message before closing.
- gvx 8y ago
  You almost always only want to implement __del__ if you're writing an extension in C. Otherwise, the GC generally takes care of cleanup perfectly fine. And in cases like the one cjhanks describes, __del__ doesn't get called reliably enough. In those cases, using a contextmanager is better.
- pfranz 8y ago
  That wouldn't work if you expected __del__ to cleanup lock files or temp files or to manage other external resources.
- lstyls 8y ago
  If this were the case, there would be no need for SIGTERM. The OS doesn't know the application logic and can't guarantee clean termination from the application's perspective. As others have said, there's rarely reason to override `__del__`, and when you do you know exactly why you need it.
- falsedan 8y ago
  The python interpreter can choose not to call it. I've had to register essential cleanup with `atexit` instead.
pypypypypypy 8y ago
It's awesome that someone has taken the time to write this all up. With reference to this section (https://rszalski.github.io/magicmethods/#descriptor https://rszalski.github.io/magicmethods/#descriptor) the author mentions that descriptors are "meant to be held by an owner class." which is true, however in their example, they are storing the value as `self.value` on the descriptor object itself meaning that if the user were to have two instances of `Distance` all their values would be the same for meter or foot on all instances of `Distance`. One way to solve this is to store the values in a dictionary-like structure on the descriptor, e.g. `self.data[instance] = your_stored_value`.
meowface 8y ago
Good guide, but missing Python 3's __next__.
jl2718 8y ago
This is a godsend. Thank you. Without this, I’d nearly given up on python already, but, wow, thank you.
- deathanatos 8y ago
  While there's no harm in having a second guide, I just want to make sure folks do know this is in the official documentation: https://docs.python.org/3/reference/datamodel.html https://docs.python.org/3/reference/datamodel.html (The article will perhaps be a better guide to those newer; the data model page is … pretty dense.) Also, I find is exceptionally handy to have a Chrome "search engine" for the Python docs while I work. That is, on chrome://settings/searchEngines I have added a custom search engine with this setup: Search Engine: Python Docs Keyword: py URL: https://google.com/search?q=site:https://docs.python.org/3+%s&btnI (The URL is an "I'm feeling lucky" search on Google, limited to the Python 3 docs. Note that it is set to search Python 3.) This allows me to type, e.g., py data model in the URL and get the above page. Want the asyncio module docs? py asyncio (I similarly have "mdn" for the developer.mozilla.org docs, which are also excellent.)

194 points by apsec112 8y ago

kqr 8y ago

This is a great explanation and a must-read for Python developers. Once you know this stuff, you'll be a much stronger debugger. But it's also worth throwing a warning out there: when you first learn about a new type of magic, you'll be tempted to use it everywhere. Be careful! I can recognise a clear specific period in my Python work which was just after I had read this, and a bunch of things suddenly became implicit.

florabuzzword 8y ago
My reaction to this, as someone who doesn’t use many of these on a regular basis (probably guilty of overusing __new__ though) is actually that I can imagine these causing some real problems. The operator ones seem particularly tempting and dangerous at the same time.
- ubernostrum 8y ago
  Operator overloading, in any language that supports it, has always been controversial, so Python doesn't really add anything new there. The thing is, it's also incredibly useful at times. Language features that are "useful when you have the use case for it, confusing when you don't" tend to provoke that reaction.
  - int_19h 8y ago
    In Python, though, operator overloading is not merely "supported". It's how all operators are defined, including built-in ones. >>> dir(123) ['__abs__', '__add__', '__and__', '__bool__', '__ceil__', '__class__', '__delattr__', '__dir__', '__divmod__', '__doc__', '__eq__', '__float__', '__floor__', '__floordiv__', '__format__', '__ge__', '__getattribute__', '__getnewargs__', '__gt__', '__hash__', '__index__', '__init__', '__init_subclass__', '__int__', '__invert__', '__le__', '__lshift__', '__lt__', '__mod__', '__mul__', '__ne__', '__neg__', '__new__', '__or__', '__pos__', '__pow__', '__radd__', '__rand__', '__rdivmod__', '__reduce__', '__reduce_ex__', '__repr__', '__rfloordiv__', '__rlshift__', '__rmod__', '__rmul__', '__ror__', '__round__', '__rpow__', '__rrshift__', '__rshift__', '__rsub__', '__rtruediv__', '__rxor__', '__setattr__', '__sizeof__', '__str__', '__sub__', '__subclasshook__', '__truediv__', '__trunc__', '__xor__', 'bit_length', 'conjugate', 'denominator', 'from_bytes', 'imag', 'numerator', 'real', 'to_bytes'] >>> (1).__add__ <method-wrapper '__add__' of int object at 0x678C40A0> >>> (123).__add__.__doc__ 'Return self+value.' >>> (1).__add__(2) 3 Which is a good thing, because it makes the whole arrangement a lot more consistent that languages in which this sort of thing is just magic associated with a specific primitive type (sometimes it's even inconsistent - e.g. in C#, + for ints and strings is magic, but + for decimal is an overloaded operator). In Python, all that magic is confined to the magic methods. Learn them once, and they work the same everywhere.
  - florabuzzword 8y ago
    Oh yeah, don’t get me wrong. It’s cool as hell! I’m totally gonna use it for a GUI menu-building script interface I’m making right now :). I have just never done it before, and I think this is the first time I have seen the concept, so the first thing that popped in my mind we’re all the things that can go wrong. But, also all the things that can go right.
Puer 8y ago
Could you give an example of how you would use these methods to debug versus other methods/approaches? I'd love to hear if you're willing to share--I program mainly in Python these days but Python's magic methods are definitely something I need to understand at a deeper level.
- travisjungroth 8y ago
  Fluent Python has a chapter on these methods and I highly recommend it. Also, these are much better described as dunder methods (short for double underscore) than magic methods. There's really no magic to them.
nomel 8y ago
> Once you know this stuff, you'll be a much stronger debugger. As you suggest, more accurately, you'll require the next person who debugs your code to read this, making them much better at debugging your code. I no longer use magic and completely avoid meta programming in python. It confuses everyone, is not needed (just use a boring factory function), and severely limits who can help you work on your code, for what? A few lines of less boilerplate here and there? "Succinctness"? Some sort of "elegance"? More likely, a DSL that nobody understands but you! The more I write, and mostly the more I interact with other people, the more I realize that code needs to be boring. No magic. I think this is why go is succeeding: no magic.
- rootusrootus 8y ago
  100% agree. I figure this is a big part of what turns normal developers into senior developers, and lead developers. It's easy to code. What takes wisdom is 1) knowing what not to write, and 2) self control on how you write it. As a mentor I try to convey this to the fresh new supercoders joining our team.

montecarl 8y ago

This part is a little dated (it is a 2012 post after all): https://rszalski.github.io/magicmethods/#comparisons https://rszalski.github.io/magicmethods/#comparisons In Python 3 __cmp__ is ignored and you have to use the other methods. To make this easier, you can use https://docs.python.org/3/library/functools.html#functools.total_ordering https://docs.python.org/3/library/functools.html#functools.t...

gvx 8y ago
Yeah. In Python 3, __div__ is gone, and only __floordiv__ and __truediv__ exist.
opencl 8y ago
It's amazing that Python 3 was 4 years old when this post was made and yet saying "Python" was still implicitly assumed by basically everyone to mean 2.x.
- kjeetgill 8y ago
  Is it that amazing? I don't think I've had use 3 until this year and I still have to flip-flop. AND I say this as someone who was reading peps going into 3's release. Do Ubuntu or Fedora default to python 3 yet? Arch? My RHEL experience isn't as representative.
  - bhrgunatha 8y ago
    Arch's default is Python 3, but there are still many official Python 2 packages, mostly in community i.e. adopted from the AUR: Core: 1 Extra: 96 Community: 822 AUR: Over 2300 depending on how you search
    - lathiat 8y ago
      The official python recommendation is not to "default" the 'python' command as python3, and they should just explicitly be 'python' and 'python3' This is what every distro except arch does (for whatever reason) https://www.python.org/dev/peps/pep-0394/ https://www.python.org/dev/peps/pep-0394/

florabuzzword 8y ago

Wonderful. I’ve never seen this resource before. A related article I got a lot out of was ‘Understanding Python Metaclasses’, an in-depth breakdown of Python class instantiation. Here’s the link to that: https://blog.ionelmc.ro/2015/02/09/understanding-python-metaclasses/ https://blog.ionelmc.ro/2015/02/09/understanding-python-meta...

olooney 8y ago

Be sure to read "Appendix 2: Changes in Python 3" at the very end of the document! I was preparing a long "Actually..." rant before I saw that.

devxpy 8y ago

It is when you get to this part of python, that you achieve true Nirvana. It's absolutely bonkers the extent to which you can be dynamic in this language. It's the very reason why Django is so DRY.

misiti3780 8y ago
can you explain what you mean about django ?
- lalos 8y ago
  This would be an example: https://www.agiliq.com/blog/2018/01/understanding-python-magic-methods-with-djang-querysets/ https://www.agiliq.com/blog/2018/01/understanding-python-mag...
sbjs 8y ago
But this article is just explaining operator overloading, right? Doesn't everyone learn that halfway through semester in C++ 101? And operator overloading has nothing to do with DRY, in fact it's generally better to avoid it except for in very niche situations, like adding __add__ to a matrix type.
- devxpy 8y ago
  https://docs.python.org/3/reference/datamodel.html#customizing-attribute-access https://docs.python.org/3/reference/datamodel.html#customizi...
nikofeyn 8y ago
why is this so bonkers? it seems like it's just basic patterns of method overriding and operator overloading. if people think this is so magic, it's almost as if lisps, schemes, and prologs never existed.
- jlarocco 8y ago
  > if people think this is so magic, it's almost as if lisps, schemes, and prologs never existed. I write quite a bit of Lisp, and the conclusion I've come to is that for 95% of software developers, Lisp, Scheme, and Prolog might as well not exist. Even here on HN, where I'd expect the bar to be a little higher, the amount of incorrect info posted about Lisp is pretty crazy. I'm almost certain most people's only experience with Lisp(s) and Prolog was a week or two using it in their programming languages class at university.

saagarjha 8y ago

> Be careful, however, as there is no guarantee that __del__ will be executed if the object is still alive when the interpreter exits, so __del__ can't serve as a replacement for good coding practices (like always closing a connection when you're done with it. When the interpreter exits, your program's resources should be reclaimed by the operating system, right? So why not just use __del__?

cjhanks 8y ago
Say it's a socket and you want to send some sort of "END_OF_STREAM" message before closing.
gvx 8y ago
You almost always only want to implement __del__ if you're writing an extension in C. Otherwise, the GC generally takes care of cleanup perfectly fine. And in cases like the one cjhanks describes, __del__ doesn't get called reliably enough. In those cases, using a contextmanager is better.
pfranz 8y ago
That wouldn't work if you expected __del__ to cleanup lock files or temp files or to manage other external resources.
lstyls 8y ago
If this were the case, there would be no need for SIGTERM. The OS doesn't know the application logic and can't guarantee clean termination from the application's perspective. As others have said, there's rarely reason to override `__del__`, and when you do you know exactly why you need it.
falsedan 8y ago
The python interpreter can choose not to call it. I've had to register essential cleanup with `atexit` instead.

pypypypypypy 8y ago

It's awesome that someone has taken the time to write this all up. With reference to this section (https://rszalski.github.io/magicmethods/#descriptor https://rszalski.github.io/magicmethods/#descriptor) the author mentions that descriptors are "meant to be held by an owner class." which is true, however in their example, they are storing the value as `self.value` on the descriptor object itself meaning that if the user were to have two instances of `Distance` all their values would be the same for meter or foot on all instances of `Distance`. One way to solve this is to store the values in a dictionary-like structure on the descriptor, e.g. `self.data[instance] = your_stored_value`.

meowface 8y ago

Good guide, but missing Python 3's __next__.

jl2718 8y ago

This is a godsend. Thank you. Without this, I’d nearly given up on python already, but, wow, thank you.

deathanatos 8y ago
While there's no harm in having a second guide, I just want to make sure folks do know this is in the official documentation: https://docs.python.org/3/reference/datamodel.html https://docs.python.org/3/reference/datamodel.html (The article will perhaps be a better guide to those newer; the data model page is … pretty dense.) Also, I find is exceptionally handy to have a Chrome "search engine" for the Python docs while I work. That is, on chrome://settings/searchEngines I have added a custom search engine with this setup: Search Engine: Python Docs Keyword: py URL: https://google.com/search?q=site:https://docs.python.org/3+%s&btnI (The URL is an "I'm feeling lucky" search on Google, limited to the Python 3 docs. Note that it is set to search Python 3.) This allows me to type, e.g., py data model in the URL and get the above page. Want the asyncio module docs? py asyncio (I similarly have "mdn" for the developer.mozilla.org docs, which are also excellent.)