6 ms·
Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code
- pbrowne011 3y agoThe author has a number of other good articles going into the depths of CPython here: https://codeconfessions.substack.com/t/cpython-internals https://codeconfessions.substack.com/t/cpython-internals. One of them goes into how many lines of C it takes to execute "a + b" in Python (great explanation, but you have to go to the comments for the answer!). I remember one of my professors opening up gdb and printing every C line to execute "print("hello, world")", and the author's writings give a better understanding of what some of those lines are. I'd be very interested in a follow-up post going in-depth on the ins and outs of list comprehension.
- abhi9u 3y agoThank you (author here). I have not looked into list comprehensions. I think that and how generators work internally might be interesting to write about. I will try to cover them.
- hildenae 3y agoIt is a good explanation of what is happening. However, I think that many of not most Python linters will warn you about this. It is perfectly valid, and something you might actually want, so it should not throw an error.
- EdwardDiego 3y agoUsing * on a sequence wouldn't get past code review with me, it's an obscure feature with surprising side-effects and it doesn't explain your thinking. Only context I've ever found it useful in, is if you're unit testing something that accepts a string of max length M, then bad_str = "A" * (M + 1) Is quite nice and also, easy to interpret. But if you're calling multiply on a list[list[Any]] lol nah mate, what the hell are you playing at, make a for loop and add comments.
- thomasahle 3y agoThere's always the `zip(*[iter(x)]*2)` to split an iterable into even/odds.
- maweki 3y ago> Using * on a sequence wouldn't get past code review On a mutable sequence. Or anything mutable. [] being a sequence is not the issue here. If you work with immutable stuff, * is really not an issue.
- boxed 3y agoIt's not great though. I do use it to repeat strings but it should have been a method and not weirdly overloading *. This type of misuse of operator overloading is why people from a C++ background think operator overloading is evil. In python it's mostly ok, due to a more restrained culture, but this is a good example of when it's bad.
- maweki 3y agoIt's not weirdly overloaded. 'a'*3 == 'a'+'a'+'a'
- ponco 3y agoI feel like far too much attention is given to `[[]] * n`. In the grand scheme of things, no serious python programmer is using multiply on a sequence outside of the string construction convenience. It's also remarkebly easy to diagnose once you see the unexpected behaviour, so anyone asking for help is going to be instantly told not to do it and use a comprehension instead. I'm sure there are plenty of other pitfalls in other languages, though I concede this one is especially unintuitive to a new person.
- 10000truths 3y agoTL;DR is that multiplying a list performs a shallow copy of the list's elements. Great article. The only thing it's missing is a snippet that shows what you should do instead: [[] for _ in range(N)] Since generator-like expressions evaluates the inner expression (i.e. the []) on each iteration, the elements of the outer list are guaranteed to be unique references.
- blt 3y agoThis is pretty subtle. I wish more languages explicitly distinguished references and values like C++ and Rust.
- scbrg 3y agoEverything falls in place when you accept that in python all names (including list indexes) are just pointers to an object, and objects aren't copied unless you explicitly copy them. Everything else follows from that. If you had done [1] * 4 you'd ended up with a list of four references to the same object (1) as well. It's just that 1 isn't mutable, so nobody cares.
- deleted 3y ago[deleted]
- bmitc 3y agoF# does this nicely with the mutable keyword and the <- assignment operator.
- Skeime 3y agoIn Python, everything is a reference, so you don't need special syntax to mark them. There just happen to be some types where you cannot change the referenced value in any way. These behave like value types in other languages. (If you believe that they should be marked explicitly, do you also believe the same thing about Rust? I.e. should the Rust compiler throw an error if you never actually change a reference type, as you "should" be using a value type in that case?)
- 3y ago
- kerkeslager 3y agoI like the deep dive into the source, but big picture, I think if you understand the general order of evaluation which applies to most languages, this isn't all that unintuitive. The arguments going into the outer [ ] get evaluated before the outer [ ], meaning the inner [] gets evaluated first. Once that happens the inner list is created, and there's no code left that even visually could create more inner lists. The more unintuitive example of this to me is in function defaults, because those feel more like part of the function signature--i.e. I'd expect them to be assigned at function call time. But apparently, function defaults get called once when the function is defined. >>> def grow(items=[0]): ... items.append(items[-1] + 1) ... return items ... >>> grow() [0, 1] >>> grow() [0, 1, 2] >>> grow() [0, 1, 2, 3]
- Joker_vD 3y agoWell, those things are called "default argument values", not "default argument expressions", you know. It's a value that's reused for every function call, not an expression that re-evaluated at every function call.
- throwaway0665 3y agoSure there is a reason it is that way. But it doesn't mean it's not incredibly unintuitive and error-prone.
- Joker_vD 3y agoYes, the reason is that it's a tradition to have it this way. Python IMHO has decent enough assignment semantics that using "default argument expressions" model would've been fine, unlike e.g. in C++.
- chii 3y agoif you compare > a = [0] and > b = [0] these two expressions produces different 'value' objects for `[0]`, and modifying `a.append(a[-1] + 1)` does not change b. so it's unintuitive that using `[0]` as a default value in a function argument can cause that argument to retain the same value across multiple function calls, when programmers would assume the argument to a function is not the same across multiple invocations. The issue really here is the usage of an array object, which is not a value object (immutable), but a reference object. It would not have been a problem if the default argument was a string, or a number. This is a reason not to use arrays in default arguments.
- mamonster 3y agoI think anybody that ever did LeetCode/HackerRank in Python had that bug at least once when making an empty 2-D DP table or a return matrix. My bigger question is how does a professor of machine learning does not know about one of the oldest Python footguns to exist: https://stackoverflow.com/questions/12791501/why-does-this-code-for-initializing-a-list-of-lists-apparently-link-the-lists-to https://stackoverflow.com/questions/12791501/why-does-this-c... Very nice article otherwise, I never actually bothered looking into why it behaved this way(at this point most people initialize using list comprehension, so this bug is IMO very rare in production), very well explained.
- bmitc 3y ago> My bigger question is how does a professor of machine learning does not know about one of the oldest Python footguns to exist Because there's a lot of them. And as soon as you touch a sane language that fits your intuitive model of how things should work, going back to Python means that you'll start doing intuitive things that Python will happily and silently transform into unintuitive operations.
- Affric 3y agoYep, common enough that everyone learns it. I started using loops and it motivated me to start writing list comprehensions. One of the foot guns that will really motivate you to understand Python as a tool more than anything.
- wruza 3y agoHow is that counter-intuitive? Or at least what’s your general intuition then? If it was [x] with x being a complex object, would you expect it to be deep copied? Shallow copied? Copied via IRepeatable? What’s your intuition on a process of repeating a value that isn’t reference-copy? What should a[3] = a[2] = a[1] = a[0] or an equivalent chunk do, assuming trivial __setitem__()? Why that should be different from a * 4? I’m not even a python guy, if you think I’m biased. These questions are natural for a programmer and a programmer has to ask themselves about what they expect, why, is it consistent, is it safe to assume. Making [[]]*4 to create three new lists implies that you add a non-obvious ephemeral re-evaluation point in your grammar. Should [[f()]]*4 call f() three more times? These may be added for convenience. But you have to make sure that syntactic context makes it dead clear. E.g. in a for-i iterator part, loop conditions, function argument init, etc. Locations which are known to be executed more than once. (Python specifically doesn’t even do that for arguments. Every def f(x=[]) shares the same list as a default for x.)
- twixfel 3y agoIndeed at first glance it looks odd, but the alternative is that multiplying a sequence always does a deep copy. Actually what is "strange" is the behaviour in combination with the syntax of a nested list. The behaviour itself is fine. Replace it with class X: pass y = [X()] * 5 and suddenly I think the behaviour looks fairly reasonable.
- banish-m4 3y agoIn other realms, SSA and immutable data makes things faster (cache friendly) and more predictable without impure side-effects cascading in unanticipated ways. In general, references are half pointers and quarter gotos in terms of papercut yourself factor. I think Erlang got this one right. Perhaps folks replace their containers and fundamental types like strs, lists, and dicts with immutable ones using an appropriate library.
- mmaniac 3y agoThe root problem here is how references and mutability combine to cause spooky action at a distance, and the non-obviousness of how they combine. The higher-level a programming language is, the more likely it is to hide the use of references, and also the more likely it is to hide when a new object is created, which would break shared mutable state. In C or Go any reference is taken explicitly and objects are also created explicitly, so it's very easy to see at a glance what can be mutated from outside the immediate scope or when a defensive copy is made. Java and C# use lots of implicit references but still require keyword new for nearly all object instantiations. It's easy to look at the expression Collections.nCopies(4, new ArrayList<>()) and understand that the same list will be referenced 4 times. At the highest extreme you have languages like Haskell where immutability makes details about when objects are copied or referenced irrelevant. Surprising behaviour is outright impossible now. Python appears to be in a tarpit where object instantiation and referencing is obscured just enough that it can take you by surprise when it silently results in shared mutable state.