19 ms·

A Guide to Python's Magic Methods

234 points by cskau 15y ago

MikeOnFire 15y ago
For each special method, you should show what syntax magically invokes it. Something like this: foo.__iter__() is invoked by for x in foo a.__iadd__(b) is invoked by a += b a.__rmult__(b) is invoked by b * a when b * a fails. foo.__len__() is invoked by if foo when foo.__bool__ or foo.__nonzero__ is not defined. foo.__repr__() is invoked by '%r' % foo, also by pprint.pprint(foo) and deprecated `foo`. It's also invoked by __str__ magic when foo.__str__ is not defined. etc. Putting the above info in a table would also be helpful.
- rafekett 15y ago
  This was my work (not sure why someone randomly posted it here, but such is the internet). I do exactly what you just described, see http://www.rafekettler.com/magicmethods.html#appendix http://www.rafekettler.com/magicmethods.html#appendix
  - cskau 15y ago
    Such is the Internet indeed. I stumbled upon this while researching some Python magic. I felt it was a pretty good write-up so I decided to share it with my fellow HN'ers. Thank you for posting this.
    - rafekett 15y ago
      And thank you for posting. I'll take all of the traffic I can get :)
  - ceol 15y ago
    I'm quite glad someone posted it here! I'm not sure if I would have found such a useful reference otherwise. Thanks for writing this up!
  - masklinn 15y ago
    You should not document __coerce__, it shouldn't be used anymore. Also why put __nonzero__, __str__ and __unicode__ in "Representing your Classes" instead of "Type conversion magic methods"? Why not talk about the collections ABCs for container operators? (using them as mixins greatly implifies implementing containers, and makes it clear what's required and what is not)? And the __concat__ note is incorrect, that does not exist in the userland datamodel it exists only in the `operator` module and in the C sequence protocol.
  - jerf 15y ago
    The word example is dangerous; two different words of the same length are neither less than each other, greater than each other, or equal to each other. I think sorting may not work as well as you'd like, and you should just be passing a custom cmp function to the sort method. The other comparison methods are for optimization; they should never disagree with __cmp__, and if __cmp__ is already optimal you should usually not define them.
- masklinn 15y ago
  > a.__rmult__(b) is invoked by b * a when b * a fails. And for boolean comparisons, mirrored is an other operator: if trying a >= b and a.__ge__(b) fails, then the interpreter will fall back on b.__le__(a) (and if that fails as well, then both CPython and Pypy will compare the value's type's name in Python 2 although that's considered an implementation detail — and the interpreters may behave differently as some types have different casings) Also, if I remember correctly "failure" is either the method not existing or the method returning NotImplemented. An other interesting fallback is `in` (`__contains__`) which falls back on iteration when there is no `__contains__` and iteration itself (`__iter__`) falls back on `__getitem__(int)` when `__iter__` is not defined. So you can have `a in b` ending up calling `a.__getitem__(0)`, which the container does not always expect (when it's not been coded correctly) It's rather sad that TFA does not explain these, or in which case reflected arithmetic operators get called (his example makes very little sense, especially for non-commutative operators). > foo.__bool__ or foo.__nonzero__ Important note: the former is the Python 3 version of the latter, defining __bool__ in Python 2 does not work.
RShackleford 15y ago
__ is fucking idiot syntax, fuck python
mitchellh 15y ago
This is a great post, but I also want to point out that this page is invaluable to any Python developer wanting to know the details about the magic methods: http://docs.python.org/reference/datamodel.html#special-method-names http://docs.python.org/reference/datamodel.html#special-meth... Whenever working on any reasonably complex class, I find that open in a tab to verify that I'm using the magic methods properly, and that they'll be called as I expect.
- rafekett 15y ago
  I agree. What I wrote here was based on that. I wanted my post to be more akin to a tutorial than a reference. I don't think the data model reference is useful to people who don't know the basics -- the idea was to bridge that gap so that it could be useful.
deleted 15y ago
[deleted]
steve-howard 15y ago
>__getattr__(self, name) > ... This can be useful for catching and redirecting common misspellings ... I can think of only one good reason to redirect spelling errors, and that's if you are a widely-used library and you want to support both "color" and "colour" (and similar Britishisms). Other than that, you're just asking for a headache.
- chernevik 15y ago
  I am but an egg . . . and while I agree this stuff is great, it is magic and I think it helps to write code so your usage is explicit. Sometimes all that's needed is a hint that something is Not Normal, so the code reader is alerted to look carefully. A test showing Foo(4) * Bar('grandma') == 'nice' can at least flag and often explain whatever mysteries you've created. If you don't put the tests in doc strings with doc tests, at least put a reference to them in the module so the reviewer knows to look for them. Stuff like __getattr__ is particularly tricky. I also try to end __getattr__ code with an AttributeException, which I find really helps debugging. I like using this to dispatch calls to objects better built to fill them, but I try to do so explicitly within the __getattr__ code to give reviewers a clue as to what's going on. For example, I'll check the attribute name against a list before passing it on to the target object. Even if I intent to pass _everything_ to a target, I'll first check the target for the passed attribute with getattr(target, attribute name, False), so that failed passthroughs stay with the object calling them. (You can generate messages indicating the problem -- "Can't find 'RIAA' in object 'Nice'" and pass those to an AttributeException.) Ideally any string validly usable as an instance attribute should be findable somewhere in the code by grep.
- jules 15y ago
  It would be perhaps be useful to throw an error saying "X object has no attribute Y, did you mean Z instead?".
  - steve-howard 15y ago
    That's probably a better idea. In either case being "liberal in what you accept" is a discredited software practice.
zanny 15y ago
This might be tangental, but the syntax of having magic methods be surrounded by double underscores always drives me insane. For a language designed around readability, it seems like the python dev team just kicked the can on things like __init_ and said "Sure, we have a few dozen english words already restricted usage in the language, but let's kick the whole "pretty" thing out and start wrapping every special statement in ugly underscores like its 1995 again!". They probably started this trend around 1995, now that I think about it, ya know. since the language is that old and all. It is still really annoying, I just cringe when I use __init__ constructors because they look so ugly in otherwise very English-document-like source. </rant>
- njharman 15y ago
  Special methods are special. It is a feature that they look different and stand out.
  - batista 15y ago
    Well, they could stand out with some better say. How about a "special" keyboard? def special init() How about @? def @init() or :: def special::init() or, all caps, like Go does initial-cap for exported members: def INIT()
    - lclarkmichalek 15y ago
      Well @ is taken by decorator syntax. I like the python way however (probably due to using python a lot). Special methods are special so it does make sense that they look different, and python all ready the convention that underscores in front of them function name indicate the function shouldn't be accessed directly, and that is true for special methods. When was the last time you called __init__ directly (discounting uses inside another __init__ method)? Most likely never. I guess it's personal preference, but that naming convention saves me a lot of time when calling die on an object; I know I can basically ignore anything after the first name beginning with an underscore.
    - njharman 15y ago
njharman 15y ago
Controlling Attribute Access section is really poor. Giving poor explanation of getattr vs getattribute, wrongly recommending to not use these methods, claiming there is little use, and providing poor examples of common use cases. The biggest of which is object composition.
- jeffjose 15y ago
  I'm not sure who downvoted you, but I felt the same. Its been a while since I've been through getattr vs getattribute pages, but I distinctly remember there's a big big difference in how you use them. I was kinda hoping the page had it, but unfortunately it didnt. Original Author, if you're reading this - kindly do that.
  - SoftwareMaven 15y ago
    __getattribute__ (new classes only!) hijacks the entire lookup machinery; every time you write instance., you will call it. __getattr__ allows "normal" lookup (is it a method, does it exist in __dict__, etc) to be done by the VM and isn't called until you would otherwise get an AttributeError.
sophacles 15y ago
One of the better treatments of __get__ and __set__ I've seen. Descriptors are deep and powerful magic, and a bit complex, but when they are appropriate, they are less complex than the alternatives, and allow for cleaner interfaces.

234 points by cskau 15y ago

For each special method, you should show what syntax magically invokes it. Something like this: foo.__iter__() is invoked by for x in foo a.__iadd__(b) is invoked by a += b a.__rmult__(b) is invoked by b * a when b * a fails. foo.__len__() is invoked by if foo when foo.__bool__ or foo.__nonzero__ is not defined. foo.__repr__() is invoked by '%r' % foo, also by pprint.pprint(foo) and deprecated `foo`. It's also invoked by __str__ magic when foo.__str__ is not defined. etc. Putting the above info in a table would also be helpful.

rafekett 15y ago
This was my work (not sure why someone randomly posted it here, but such is the internet). I do exactly what you just described, see http://www.rafekettler.com/magicmethods.html#appendix http://www.rafekettler.com/magicmethods.html#appendix
- cskau 15y ago
  Such is the Internet indeed. I stumbled upon this while researching some Python magic. I felt it was a pretty good write-up so I decided to share it with my fellow HN'ers. Thank you for posting this.
  - rafekett 15y ago
    And thank you for posting. I'll take all of the traffic I can get :)
- ceol 15y ago
  I'm quite glad someone posted it here! I'm not sure if I would have found such a useful reference otherwise. Thanks for writing this up!
- masklinn 15y ago
  You should not document __coerce__, it shouldn't be used anymore. Also why put __nonzero__, __str__ and __unicode__ in "Representing your Classes" instead of "Type conversion magic methods"? Why not talk about the collections ABCs for container operators? (using them as mixins greatly implifies implementing containers, and makes it clear what's required and what is not)? And the __concat__ note is incorrect, that does not exist in the userland datamodel it exists only in the `operator` module and in the C sequence protocol.
- jerf 15y ago
  The word example is dangerous; two different words of the same length are neither less than each other, greater than each other, or equal to each other. I think sorting may not work as well as you'd like, and you should just be passing a custom cmp function to the sort method. The other comparison methods are for optimization; they should never disagree with __cmp__, and if __cmp__ is already optimal you should usually not define them.
masklinn 15y ago
> a.__rmult__(b) is invoked by b * a when b * a fails. And for boolean comparisons, mirrored is an other operator: if trying a >= b and a.__ge__(b) fails, then the interpreter will fall back on b.__le__(a) (and if that fails as well, then both CPython and Pypy will compare the value's type's name in Python 2 although that's considered an implementation detail — and the interpreters may behave differently as some types have different casings) Also, if I remember correctly "failure" is either the method not existing or the method returning NotImplemented. An other interesting fallback is `in` (`__contains__`) which falls back on iteration when there is no `__contains__` and iteration itself (`__iter__`) falls back on `__getitem__(int)` when `__iter__` is not defined. So you can have `a in b` ending up calling `a.__getitem__(0)`, which the container does not always expect (when it's not been coded correctly) It's rather sad that TFA does not explain these, or in which case reflected arithmetic operators get called (his example makes very little sense, especially for non-commutative operators). > foo.__bool__ or foo.__nonzero__ Important note: the former is the Python 3 version of the latter, defining __bool__ in Python 2 does not work.

RShackleford 15y ago

__ is fucking idiot syntax, fuck python

mitchellh 15y ago

This is a great post, but I also want to point out that this page is invaluable to any Python developer wanting to know the details about the magic methods: http://docs.python.org/reference/datamodel.html#special-method-names http://docs.python.org/reference/datamodel.html#special-meth... Whenever working on any reasonably complex class, I find that open in a tab to verify that I'm using the magic methods properly, and that they'll be called as I expect.

rafekett 15y ago
I agree. What I wrote here was based on that. I wanted my post to be more akin to a tutorial than a reference. I don't think the data model reference is useful to people who don't know the basics -- the idea was to bridge that gap so that it could be useful.

deleted 15y ago

[deleted]

steve-howard 15y ago

>__getattr__(self, name) > ... This can be useful for catching and redirecting common misspellings ... I can think of only one good reason to redirect spelling errors, and that's if you are a widely-used library and you want to support both "color" and "colour" (and similar Britishisms). Other than that, you're just asking for a headache.

chernevik 15y ago
I am but an egg . . . and while I agree this stuff is great, it is magic and I think it helps to write code so your usage is explicit. Sometimes all that's needed is a hint that something is Not Normal, so the code reader is alerted to look carefully. A test showing Foo(4) * Bar('grandma') == 'nice' can at least flag and often explain whatever mysteries you've created. If you don't put the tests in doc strings with doc tests, at least put a reference to them in the module so the reviewer knows to look for them. Stuff like __getattr__ is particularly tricky. I also try to end __getattr__ code with an AttributeException, which I find really helps debugging. I like using this to dispatch calls to objects better built to fill them, but I try to do so explicitly within the __getattr__ code to give reviewers a clue as to what's going on. For example, I'll check the attribute name against a list before passing it on to the target object. Even if I intent to pass _everything_ to a target, I'll first check the target for the passed attribute with getattr(target, attribute name, False), so that failed passthroughs stay with the object calling them. (You can generate messages indicating the problem -- "Can't find 'RIAA' in object 'Nice'" and pass those to an AttributeException.) Ideally any string validly usable as an instance attribute should be findable somewhere in the code by grep.
jules 15y ago
It would be perhaps be useful to throw an error saying "X object has no attribute Y, did you mean Z instead?".
- steve-howard 15y ago
  That's probably a better idea. In either case being "liberal in what you accept" is a discredited software practice.

zanny 15y ago

This might be tangental, but the syntax of having magic methods be surrounded by double underscores always drives me insane. For a language designed around readability, it seems like the python dev team just kicked the can on things like __init_ and said "Sure, we have a few dozen english words already restricted usage in the language, but let's kick the whole "pretty" thing out and start wrapping every special statement in ugly underscores like its 1995 again!". They probably started this trend around 1995, now that I think about it, ya know. since the language is that old and all. It is still really annoying, I just cringe when I use __init__ constructors because they look so ugly in otherwise very English-document-like source. </rant>

njharman 15y ago
Special methods are special. It is a feature that they look different and stand out.
- batista 15y ago
  Well, they could stand out with some better say. How about a "special" keyboard? def special init() How about @? def @init() or :: def special::init() or, all caps, like Go does initial-cap for exported members: def INIT()
  - lclarkmichalek 15y ago
    Well @ is taken by decorator syntax. I like the python way however (probably due to using python a lot). Special methods are special so it does make sense that they look different, and python all ready the convention that underscores in front of them function name indicate the function shouldn't be accessed directly, and that is true for special methods. When was the last time you called __init__ directly (discounting uses inside another __init__ method)? Most likely never. I guess it's personal preference, but that naming convention saves me a lot of time when calling die on an object; I know I can basically ignore anything after the first name beginning with an underscore.
  - njharman 15y ago

njharman 15y ago

Controlling Attribute Access section is really poor. Giving poor explanation of getattr vs getattribute, wrongly recommending to not use these methods, claiming there is little use, and providing poor examples of common use cases. The biggest of which is object composition.

jeffjose 15y ago
I'm not sure who downvoted you, but I felt the same. Its been a while since I've been through getattr vs getattribute pages, but I distinctly remember there's a big big difference in how you use them. I was kinda hoping the page had it, but unfortunately it didnt. Original Author, if you're reading this - kindly do that.
- SoftwareMaven 15y ago
  __getattribute__ (new classes only!) hijacks the entire lookup machinery; every time you write instance., you will call it. __getattr__ allows "normal" lookup (is it a method, does it exist in __dict__, etc) to be done by the VM and isn't called until you would otherwise get an AttributeError.

sophacles 15y ago

One of the better treatments of __get__ and __set__ I've seen. Descriptors are deep and powerful magic, and a bit complex, but when they are appropriate, they are less complex than the alternatives, and allow for cleaner interfaces.