9 ms·

Understanding Python's augmented assignment (a += b)

72 points by ceronman 12y ago

icebraining 12y ago
Nice article, though I already understood the matter. Thanks for the link to PEP 463[1], though; I can see it cleaning up some of our code. Do you know if there have been any developments into implementing it? [1] http://legacy.python.org/dev/peps/pep-0463/ http://legacy.python.org/dev/peps/pep-0463/
- maxerickson 12y ago
  It died: https://mail.python.org/pipermail/python-dev/2014-March/133118.html https://mail.python.org/pipermail/python-dev/2014-March/1331... (the PEP author posts a couple messages in the thread too)
zrail 12y ago
This is a wonderful explanation of a deeply confusing situation. Thank you for posting.
aftbit 12y ago
Why couldn't Python do something like: >>> _tmp = getitem(t, 0) >>> setitem(t, 0, _tmp) >>> try: >>> _tmp2 = _tmp.__iadd__([2]) >>> except AttributeError: >>> _tmp2 = _tmp.__add__([2]) >>> setitem(t, 0, _tmp2) Then if the first setitem raises, the rest of the code will be skipped. Obviously this isn't perfect: 1. If setitem has strange side-effects, this will cause them to happen twice. 2. If setitem raises based on the value (rather than the key), this won't notice that. But otherwise, it seems like a pretty straightforward improvement, no?
peterderivaz 12y ago
I ran into this when computing PSNR comparisons using a Numpy function that returned a numpy integer type: >>> import numpy >>> x=0 >>> y=numpy.int32(2*10**9) >>> x+=y >>> x+=y >>> x -294967296 In later versions of Python this gives a RuntimeWarning so this is less likely to cause problems now.
- PhantomGremlin 12y ago
  Python 2.7.2 on OS X 10.8.5 behaves even more strangely. I get: >>> import numpy >>> x=0 >>> y=numpy.int32(2*10**9) >>> y 2000000000 >>> x+=y >>> x 2000000000 >>> x+=y >>> x 4000000000 >>> x*x -2446744073709551616 >>> y*y __main__:1: RuntimeWarning: overflow encountered in int_scalars -1651507200 I didn't think it was possible to get integer wraparound in Python. I guess it's because the "int32" from numpy "breaks" x, which subsequently no longer behaves like a python variable.
danbruc 12y ago
What this article says about C++ is not true.
- oldmanjay 12y ago
  "Not true" sort of undersells it. They seem to make up some hypothetical language entirely and call it c++ for rhetorical convenience.
- anon4 12y ago
  It seems to me it describes pretty much what happens in C++, but with pythonized syntax, because explaining the exact syntax of all the sigils would be too much for such a short article.
  - danbruc 12y ago
    Absolutely not. Take for example the part about t = [0, 1] followed by t[0] = 2. t[0] does return nothing at all, neither 0 nor any strange complex object. It just evaluates to the memory address of the first element of the array. The idea expressed later that C++ unlike Python treats every piece of code exactly the same no matter whether it appears on the left hand side or the right hand side of an assignment operator is fundamentally wrong.
    - Leszek 12y ago
      Well no, for std::vector<int> t = {0, 1}; t[0] calls operator[], which returns an int&. This is precisely an lvalue reference, which is kind of like a memory address, but int& is a type in its own right. Calling it a "reference object" is not entirely correct, but it's also not entirely incorrect. And let's not get started on vector<bool>...
    - anon4 12y ago
      At the semantic level it evaluates to an lvalue. Or an lvalue reference. It depends on what the type of t is. Given the syntax, the author obviously meant some vector-like type, so it should, if you're following good style, evaluate to a reference. And that's what he was getting at - t[0] evaluates to something that can be assigned to. In C++, the thing that t[0] evaluates to handles the assignment (whether t is a pointer and t[0] "handles" the assignment by simple copy to an address, or t is an object and t[0] returns a reference or some object overriding operator= for the type that t contains is irrelevant - it's the same semantically), while in Python, the t object itself handles the assignment. Python's behaviour is akin to C++ having an operator[]=(size_t, const T&). It all stems from the different behaviour of = in Python and C++ - in Python = binds the name on the left to the value on the right and you can have any number of names for a value; in C++ each name is exactly one value (even in the case of references - the reference itself is a pointer and there's no other name that signifies the memory in which that pointer value is stored) and = copies the value from the thing on the right to the thing on the left, in a way determined by the thing on the left (so for references the value is copied to the thing that the reference points to and other types can override operator= to do whatever they like), but the base concept is that = copies a value from one place to another (or moves it if it's an xvalue and you're using C++11 and what's on the left implements move-assignment). P.S. Since C++ lets you do as you like, you can for instance implement operator[](size_t, const T&) as assignment and then t[0, 2] will assign 2 to t[0]. Try putting template <class T> void operator[](T* p, size_t i, const T& v) { p[i] = v; } in a header file in your next project for fun. Then use t[i, v] for all your array assignments.
danbruc 12y ago
The example look ill-conceived to begin with. If you want to extend the list that is the first element of the tuple just use t[0].extend([1]). Using an augmented assignment operator makes no sense if the left hand side can not be assigned to. I don't know Python well enough to tell why it does not check if the left hand side can be assigned to before evaluating the right hand side, i.e. why it dos not check that __setitem__ is present. This would make the behavior more similar to a compiled statically typed language where this would be a compile time error and the right hand side would never get evaluated.
- aftbit 12y ago
  In fact, the second half of the article is about why it doesn't do that. Also, Python is not a compiled statically typed language, and doesn't have compile errors.
cagriaksay 12y ago
Official FAQ is easier to understand. I recommend reading that first before the article.

72 points by ceronman 12y ago

Nice article, though I already understood the matter. Thanks for the link to PEP 463[1], though; I can see it cleaning up some of our code. Do you know if there have been any developments into implementing it? [1] http://legacy.python.org/dev/peps/pep-0463/ http://legacy.python.org/dev/peps/pep-0463/

maxerickson 12y ago
It died: https://mail.python.org/pipermail/python-dev/2014-March/133118.html https://mail.python.org/pipermail/python-dev/2014-March/1331... (the PEP author posts a couple messages in the thread too)

zrail 12y ago

This is a wonderful explanation of a deeply confusing situation. Thank you for posting.

aftbit 12y ago

Why couldn't Python do something like: >>> _tmp = getitem(t, 0) >>> setitem(t, 0, _tmp) >>> try: >>> _tmp2 = _tmp.__iadd__([2]) >>> except AttributeError: >>> _tmp2 = _tmp.__add__([2]) >>> setitem(t, 0, _tmp2) Then if the first setitem raises, the rest of the code will be skipped. Obviously this isn't perfect: 1. If setitem has strange side-effects, this will cause them to happen twice. 2. If setitem raises based on the value (rather than the key), this won't notice that. But otherwise, it seems like a pretty straightforward improvement, no?

peterderivaz 12y ago

I ran into this when computing PSNR comparisons using a Numpy function that returned a numpy integer type: >>> import numpy >>> x=0 >>> y=numpy.int32(2*10**9) >>> x+=y >>> x+=y >>> x -294967296 In later versions of Python this gives a RuntimeWarning so this is less likely to cause problems now.

PhantomGremlin 12y ago
Python 2.7.2 on OS X 10.8.5 behaves even more strangely. I get: >>> import numpy >>> x=0 >>> y=numpy.int32(2*10**9) >>> y 2000000000 >>> x+=y >>> x 2000000000 >>> x+=y >>> x 4000000000 >>> x*x -2446744073709551616 >>> y*y __main__:1: RuntimeWarning: overflow encountered in int_scalars -1651507200 I didn't think it was possible to get integer wraparound in Python. I guess it's because the "int32" from numpy "breaks" x, which subsequently no longer behaves like a python variable.

danbruc 12y ago

What this article says about C++ is not true.

oldmanjay 12y ago
"Not true" sort of undersells it. They seem to make up some hypothetical language entirely and call it c++ for rhetorical convenience.
anon4 12y ago
It seems to me it describes pretty much what happens in C++, but with pythonized syntax, because explaining the exact syntax of all the sigils would be too much for such a short article.
- danbruc 12y ago
  Absolutely not. Take for example the part about t = [0, 1] followed by t[0] = 2. t[0] does return nothing at all, neither 0 nor any strange complex object. It just evaluates to the memory address of the first element of the array. The idea expressed later that C++ unlike Python treats every piece of code exactly the same no matter whether it appears on the left hand side or the right hand side of an assignment operator is fundamentally wrong.
  - Leszek 12y ago
    Well no, for std::vector<int> t = {0, 1}; t[0] calls operator[], which returns an int&. This is precisely an lvalue reference, which is kind of like a memory address, but int& is a type in its own right. Calling it a "reference object" is not entirely correct, but it's also not entirely incorrect. And let's not get started on vector<bool>...
  - anon4 12y ago
    At the semantic level it evaluates to an lvalue. Or an lvalue reference. It depends on what the type of t is. Given the syntax, the author obviously meant some vector-like type, so it should, if you're following good style, evaluate to a reference. And that's what he was getting at - t[0] evaluates to something that can be assigned to. In C++, the thing that t[0] evaluates to handles the assignment (whether t is a pointer and t[0] "handles" the assignment by simple copy to an address, or t is an object and t[0] returns a reference or some object overriding operator= for the type that t contains is irrelevant - it's the same semantically), while in Python, the t object itself handles the assignment. Python's behaviour is akin to C++ having an operator[]=(size_t, const T&). It all stems from the different behaviour of = in Python and C++ - in Python = binds the name on the left to the value on the right and you can have any number of names for a value; in C++ each name is exactly one value (even in the case of references - the reference itself is a pointer and there's no other name that signifies the memory in which that pointer value is stored) and = copies the value from the thing on the right to the thing on the left, in a way determined by the thing on the left (so for references the value is copied to the thing that the reference points to and other types can override operator= to do whatever they like), but the base concept is that = copies a value from one place to another (or moves it if it's an xvalue and you're using C++11 and what's on the left implements move-assignment). P.S. Since C++ lets you do as you like, you can for instance implement operator[](size_t, const T&) as assignment and then t[0, 2] will assign 2 to t[0]. Try putting template <class T> void operator[](T* p, size_t i, const T& v) { p[i] = v; } in a header file in your next project for fun. Then use t[i, v] for all your array assignments.

danbruc 12y ago

The example look ill-conceived to begin with. If you want to extend the list that is the first element of the tuple just use t[0].extend([1]). Using an augmented assignment operator makes no sense if the left hand side can not be assigned to. I don't know Python well enough to tell why it does not check if the left hand side can be assigned to before evaluating the right hand side, i.e. why it dos not check that __setitem__ is present. This would make the behavior more similar to a compiled statically typed language where this would be a compile time error and the right hand side would never get evaluated.

aftbit 12y ago
In fact, the second half of the article is about why it doesn't do that. Also, Python is not a compiled statically typed language, and doesn't have compile errors.

cagriaksay 12y ago

Official FAQ is easier to understand. I recommend reading that first before the article.