5 ms·

I have a mildly surprising snippet which I believe they don't have. It's about interaction of class and instance attributes with behavior of "X += Y". Python ex

I have a mildly surprising snippet which I believe they don't have. It's about interaction of class and instance attributes with behavior of "X += Y". Python expands it to "X = X + Y" but only if X implements "+" but not "+=" class A: a = (1,) # tuples are immutable and don't have "+=" b = [1,] # lists have "+=" obj = A() obj.a += (2,) # this creates obj.a obj.b += (2,) # this modifies A.b print(A.a, A.b) The snippet prints "(1,) [1,2]".

iknownothow 4y ago
Oh, that's a wtf for sure. More to be scared of and more paranoid I'm going to appear to junior devs while reviewing merge requests. If I'm not mistaken there's a chapter in Effective Python about this, where they recommend never using lists or dicts as Class attributes or default values for parameters and instead use factories or use None and then initialize them with default values when a new instance is created.
- roflyear 4y ago
  Yeah it is a little tricky, but using them as class attributes can actually be handy in some circumstances. Note to readers unfamiliar with Python: class attributes is a special thing, different than instance attributes. Definitely one of the trickier things with Python for sure.
- praptak 4y ago
  Thanks for bringing this up, I didn't make the connection to default args. I think this behavior is actually "default args" but reversed. In case of default args it is the mutability that causes the nasty surprise (ooops, all calls share the same object, which is mutable). In case of the "+=" it is the _immutability_ that causes the surprise - I would not expect "obj.a += ..." to create an instance attribute that shadows the class one. I think the practical conclusion here is "don't call += on objects that don't support it - because it may work and this is not what you want!)".
  - iknownothow 4y ago
    Maybe we are looking at it differently. To me, the behavior of "obj.a += (2, )" is the obvious and expected one i.e. I expect a tuple "(1, 2)" to be created and stored in "obj.a" and the class attribute "A.a" is untouched and remains "(1,)". The behavior of the "obj.b += [2,]" for the list is the real gotcha here where it correctly alters the value of "obj.b" but wtf'ingly also alters the value of the class attribute "A.b" which completely alters the behavior of A. I'm curious if you see it the same way?
    - praptak 4y ago
      My intuitions come from C and C++, so "+=" should always modify a single object (in C lingo it would be "lvalue") in place. Storing the result somewhere else than the left side of "+=" is counterintuitive to me. Side note: the object can be immutable, but then the (variable or attribute) left side should be replaced with a new immutable object.
      - iknownothow 4y ago
        Gotcha, I see where we differ. I've exclusively programmed in Python and since int, str and tuple are immutable in Python, I've always assumed "+=" operators create copies and don't update the object.
        kazinator 4y ago
        > int is immutable But what does that have to do with assignment? >>> a = 3 >>> a += 4 >>> print(a) 7 3 and 7 are immutable, yet a takes on those objects. Obviously, this is inspired by the C += operator, and behaves accordingly, when it's not being weird.
        iknownothow 4y ago
        If you go back to the original comment from the user praptak, you will notice in his example the assignments "obj.a += (2,)" and "obj.b += [2,]". These assignments work exactly as expected no matter which school of thought you come from. The difference is in how the Class attributes are affected i.e. the values of "A.a" and "A.b". # obj.a += (2,) has no impact on A.a since tuples # are immutable and copies are made. # This is unexpected from a C/C++ intuition. >>> print(A.a) (1,) # obj.b += [2,] has an indirect impact on A.b # since lists are mutable and updates are made in place. # This is unexpected for beginners in Python since it # deviates from how int, str and tuples behave as "+=" # can be used for int, str, tuples and lists. >>> print(A.b) [1, 2] Edit: formatting
- anthony_romeo 4y ago
  > If I'm not mistaken there's a chapter in Effective Python about this, where they recommend never using lists or dicts as Class attributes or default values for parameters and instead use factories or use None and then initialize them with default values when a new instance is created. Ya, this is because the initialized object is stored as the default parameter, and is not initialized every time the function runs. def functy(dicty={}): print('dicty=', dicty) return dicty output = functy() # dicty = {} output.update({'key': 'value'}) functy() # dicty = {'key': 'value'}
thrdbndndn 4y ago
Edit: I understand it now. See comment below. > # tuples are immutable and don't have "+=" What do you mean? Because it looks to me tuples have "+=": a = (1,) a += (3,) # a = (1, 3) I get that tuple is immutable so you're actually creating a new tuple and shove it back into variable a, but it does not conflict with "having += operation".
- DasIch 4y ago
  I would read that as not implementing `__iadd__` https://docs.python.org/3/reference/datamodel.html#object.__iadd__ https://docs.python.org/3/reference/datamodel.html#object.__...
  - thrdbndndn 4y ago
    Ah I got it. So it would fallback to __add__ if __iadd__ doesn't exist. Thanks!
- maweki 4y ago
  > shove it back into variable a Python is more of a lisp. The assignment semantics are different to C. Actually, you're letting the name "a" point to the newly created object. In C, a variable is a memory position you can fill and point to. In Python, a variable is a name that points to a memory position.
  - kazinator 4y ago
    Other than in regard to the way type works, and lexical capture, mainstream Lisps have variable semantics that work more or less like C. The names of lexical variables disappear at compile time; a given a variable may be just a slot in some environment frame (perhaps a stack frame), a machine register, or disappear entirely via constant folding: or any of these three in different areas of the code. In any case, symbols do not "point" at memory positions. Similarity to C is no accident here, since the lexical scoping concepts in Lisps like Scheme and CL, as well as in C, both trace back to Algol. Classic Lisp global/dynamic variables may store a global value a "value cell" which is closely tied to the symbol itself (perhaps stored in it).
dotancohen 4y ago
In Python 3.8 the tuple is in fact changed: >>> print(obj.a, obj.b) (1, 2) [1, 2]
- Diggsey 4y ago
  What happens to `obj.a`/`obj.b` is not the point of the example, you should be looking at `A.a`/`A.b`.
  - hoseja 4y ago
    Is that like a static member? Truly, dynamic languages are God's curse upon the Earth.
    - digisign 4y ago
      It refers to the class, not the instance.
edflsafoiewq 4y ago
obj.b += (2,) creates obj.b too, it's just that obj.b still points to the same object as A.b.
madsr 4y ago
I think this example is quite deceptive. When setting obj.a and obj.b, what is your intention? If you want to set the instance attributes, you should have created self.a and self.b in the class constructor. class A: a = (1,) b = [1,] def __init__(self): self.a = (1,) self.b = [1,] A.a # this is the class attribute (1,) A.b # class attribute [1] obj = A() obj.a # instance attribute (1,) obj.b [1] obj.a += (2,) obj.b += [2,] A.a (1,) # still the same class attribute A.b [1] obj.a (1, 2) # instance attribute appended obj.b [1, 2]
- supercheetah 4y ago
  I think that's the point is that it does seem rather ambiguous as to the intent. If someone is new to programming entirely, what are they thinking is happening with an operation *on obj.a or obj.b? If they know other languages and are just starting out learning Python, what expectations are they bringing from those languages? If I had had magic wand, I'd make operations on class attributes from an instance a syntax error, and only allow it from the class name, and perhaps with special syntax like in C++, e.g.: A::a = (1,) A::b = [1,] *edit: operation on obj.a or obj.b...
ghusbands 4y ago
There's a similar surprising += 'problem' in JS. If you have the following code: async function totalSize(fol) { const files = await fol.getFiles(); let totalSize = 0; await Promise.all(files.map(async file => { totalSize += await file.getSize(); })); // totalSize is now way too small return totalSize; } You get an overly low totalSize. It's caused by 'a += b' expanding to 'a = a + b', and the double-mention of 'a' creating a concurrency issue. If '+=' were a single operation with the right-hand-side being calculated first, it wouldn't be an issue.
- tln 4y ago
  Wow, that is surprising. That must have been a fun one to track down!
- aastronaut 4y ago
  stupid question incomming: isn't that a '+= await' problem?
- DrammBA 4y ago
  The problem here is the closure on totalSize and where you're awaiting, totalSize is going to be 0 for all files and then you await for the result of getSize, so in the end totalSize is going to be equal to the size of the last file that completes the getSize call. To fix this, simply await the result of getSize first, and then add the result to totalSize using +=
  - masklinn 4y ago
    > and where you're awaiting That is the sole source of the issue, and exactly what the comment you're replying to talks about. The problem is that `a += b` desugars to `a = a + b`, so if `b` is an await you get totalSize = totalSize + await file.getSize(); Since javascript evaluates left to right, it first evaluates `totalSize`, gets zero, then `file.getSize()`, then suspends waiting for the result... at which point the handler for the next file can run, doing the exact same thing, repeat for all files.
- sharno 4y ago
bobbylarrybobby 4y ago
I think this is more or less covered by "▶ Class attributes and instance attributes"
deleted 4y ago
[deleted]

iknownothow 4y ago

Oh, that's a wtf for sure. More to be scared of and more paranoid I'm going to appear to junior devs while reviewing merge requests. If I'm not mistaken there's a chapter in Effective Python about this, where they recommend never using lists or dicts as Class attributes or default values for parameters and instead use factories or use None and then initialize them with default values when a new instance is created.

roflyear 4y ago
Yeah it is a little tricky, but using them as class attributes can actually be handy in some circumstances. Note to readers unfamiliar with Python: class attributes is a special thing, different than instance attributes. Definitely one of the trickier things with Python for sure.
praptak 4y ago
Thanks for bringing this up, I didn't make the connection to default args. I think this behavior is actually "default args" but reversed. In case of default args it is the mutability that causes the nasty surprise (ooops, all calls share the same object, which is mutable). In case of the "+=" it is the _immutability_ that causes the surprise - I would not expect "obj.a += ..." to create an instance attribute that shadows the class one. I think the practical conclusion here is "don't call += on objects that don't support it - because it may work and this is not what you want!)".
- iknownothow 4y ago
  Maybe we are looking at it differently. To me, the behavior of "obj.a += (2, )" is the obvious and expected one i.e. I expect a tuple "(1, 2)" to be created and stored in "obj.a" and the class attribute "A.a" is untouched and remains "(1,)". The behavior of the "obj.b += [2,]" for the list is the real gotcha here where it correctly alters the value of "obj.b" but wtf'ingly also alters the value of the class attribute "A.b" which completely alters the behavior of A. I'm curious if you see it the same way?
  - praptak 4y ago
    My intuitions come from C and C++, so "+=" should always modify a single object (in C lingo it would be "lvalue") in place. Storing the result somewhere else than the left side of "+=" is counterintuitive to me. Side note: the object can be immutable, but then the (variable or attribute) left side should be replaced with a new immutable object.
    - iknownothow 4y ago
      Gotcha, I see where we differ. I've exclusively programmed in Python and since int, str and tuple are immutable in Python, I've always assumed "+=" operators create copies and don't update the object.
      - kazinator 4y ago
        > int is immutable But what does that have to do with assignment? >>> a = 3 >>> a += 4 >>> print(a) 7 3 and 7 are immutable, yet a takes on those objects. Obviously, this is inspired by the C += operator, and behaves accordingly, when it's not being weird.
        iknownothow 4y ago
        If you go back to the original comment from the user praptak, you will notice in his example the assignments "obj.a += (2,)" and "obj.b += [2,]". These assignments work exactly as expected no matter which school of thought you come from. The difference is in how the Class attributes are affected i.e. the values of "A.a" and "A.b". # obj.a += (2,) has no impact on A.a since tuples # are immutable and copies are made. # This is unexpected from a C/C++ intuition. >>> print(A.a) (1,) # obj.b += [2,] has an indirect impact on A.b # since lists are mutable and updates are made in place. # This is unexpected for beginners in Python since it # deviates from how int, str and tuples behave as "+=" # can be used for int, str, tuples and lists. >>> print(A.b) [1, 2] Edit: formatting
anthony_romeo 4y ago
> If I'm not mistaken there's a chapter in Effective Python about this, where they recommend never using lists or dicts as Class attributes or default values for parameters and instead use factories or use None and then initialize them with default values when a new instance is created. Ya, this is because the initialized object is stored as the default parameter, and is not initialized every time the function runs. def functy(dicty={}): print('dicty=', dicty) return dicty output = functy() # dicty = {} output.update({'key': 'value'}) functy() # dicty = {'key': 'value'}

thrdbndndn 4y ago

Edit: I understand it now. See comment below. > # tuples are immutable and don't have "+=" What do you mean? Because it looks to me tuples have "+=": a = (1,) a += (3,) # a = (1, 3) I get that tuple is immutable so you're actually creating a new tuple and shove it back into variable a, but it does not conflict with "having += operation".

DasIch 4y ago
I would read that as not implementing `__iadd__` https://docs.python.org/3/reference/datamodel.html#object.__iadd__ https://docs.python.org/3/reference/datamodel.html#object.__...
- thrdbndndn 4y ago
  Ah I got it. So it would fallback to __add__ if __iadd__ doesn't exist. Thanks!
maweki 4y ago
> shove it back into variable a Python is more of a lisp. The assignment semantics are different to C. Actually, you're letting the name "a" point to the newly created object. In C, a variable is a memory position you can fill and point to. In Python, a variable is a name that points to a memory position.
- kazinator 4y ago
  Other than in regard to the way type works, and lexical capture, mainstream Lisps have variable semantics that work more or less like C. The names of lexical variables disappear at compile time; a given a variable may be just a slot in some environment frame (perhaps a stack frame), a machine register, or disappear entirely via constant folding: or any of these three in different areas of the code. In any case, symbols do not "point" at memory positions. Similarity to C is no accident here, since the lexical scoping concepts in Lisps like Scheme and CL, as well as in C, both trace back to Algol. Classic Lisp global/dynamic variables may store a global value a "value cell" which is closely tied to the symbol itself (perhaps stored in it).

dotancohen 4y ago

In Python 3.8 the tuple is in fact changed: >>> print(obj.a, obj.b) (1, 2) [1, 2]

Diggsey 4y ago
What happens to `obj.a`/`obj.b` is not the point of the example, you should be looking at `A.a`/`A.b`.
- hoseja 4y ago
  Is that like a static member? Truly, dynamic languages are God's curse upon the Earth.
  - digisign 4y ago
    It refers to the class, not the instance.

edflsafoiewq 4y ago

obj.b += (2,) creates obj.b too, it's just that obj.b still points to the same object as A.b.

madsr 4y ago

I think this example is quite deceptive. When setting obj.a and obj.b, what is your intention? If you want to set the instance attributes, you should have created self.a and self.b in the class constructor. class A: a = (1,) b = [1,] def __init__(self): self.a = (1,) self.b = [1,] A.a # this is the class attribute (1,) A.b # class attribute [1] obj = A() obj.a # instance attribute (1,) obj.b [1] obj.a += (2,) obj.b += [2,] A.a (1,) # still the same class attribute A.b [1] obj.a (1, 2) # instance attribute appended obj.b [1, 2]

supercheetah 4y ago
I think that's the point is that it does seem rather ambiguous as to the intent. If someone is new to programming entirely, what are they thinking is happening with an operation *on obj.a or obj.b? If they know other languages and are just starting out learning Python, what expectations are they bringing from those languages? If I had had magic wand, I'd make operations on class attributes from an instance a syntax error, and only allow it from the class name, and perhaps with special syntax like in C++, e.g.: A::a = (1,) A::b = [1,] *edit: operation on obj.a or obj.b...

ghusbands 4y ago

There's a similar surprising += 'problem' in JS. If you have the following code: async function totalSize(fol) { const files = await fol.getFiles(); let totalSize = 0; await Promise.all(files.map(async file => { totalSize += await file.getSize(); })); // totalSize is now way too small return totalSize; } You get an overly low totalSize. It's caused by 'a += b' expanding to 'a = a + b', and the double-mention of 'a' creating a concurrency issue. If '+=' were a single operation with the right-hand-side being calculated first, it wouldn't be an issue.

tln 4y ago
Wow, that is surprising. That must have been a fun one to track down!
aastronaut 4y ago
stupid question incomming: isn't that a '+= await' problem?
DrammBA 4y ago
The problem here is the closure on totalSize and where you're awaiting, totalSize is going to be 0 for all files and then you await for the result of getSize, so in the end totalSize is going to be equal to the size of the last file that completes the getSize call. To fix this, simply await the result of getSize first, and then add the result to totalSize using +=
- masklinn 4y ago
  > and where you're awaiting That is the sole source of the issue, and exactly what the comment you're replying to talks about. The problem is that `a += b` desugars to `a = a + b`, so if `b` is an await you get totalSize = totalSize + await file.getSize(); Since javascript evaluates left to right, it first evaluates `totalSize`, gets zero, then `file.getSize()`, then suspends waiting for the result... at which point the handler for the next file can run, doing the exact same thing, repeat for all files.
sharno 4y ago

bobbylarrybobby 4y ago

I think this is more or less covered by "▶ Class attributes and instance attributes"

deleted 4y ago

[deleted]