11 ms·
Python: Common Newbie Mistakes, Part 1
- akoumjian 13y agoI've been coding in Python for 8 years now and only this year came up against this behavior. I had a good long laugh after realizing what had just eaten up my previous couple hours. I actually figured it out by printing out the id of the object being modified in my code, and when seeing it was the same had to really scratch my head. Default args are initialized when the function (or similarly, class) is defined, not when executed.
- pyoung 13y agoHa, that's funny. Just ran into this issue a few days ago. Spent an hour or two trying to figure what was going on, thought I was going crazy. Finally caved and asked an experienced python programmer, and they immediately pointed out that lists are mutable. Goes to show the value of working with experienced programmers.
- RyanZAG 13y agoOuch, that is one extremely non-user friendly feature going on there. The only reason it works at all is because most people use default for strings/numbers only. That functionality really should be changed for the next major python release, and have default evaluate each time the function is called - which is how 99.9% of people using it expect it to work.
- mickeyp 13y agoYou don't change behaviour like that once it's been introduced. That will just break existing code in very subtle ways. There's plenty of code out there that rely on default values being assigned a mutable value and changing it -- even in a major point release -- would be far worse off. Python's been around for 20 years; it's too late to make a change like that. It IS a "wart" in that you need to be aware of it to know that it's an issue, but once you are, it is an obvious thing to spot.
- Camillo 13y agoThey made other compatibility-breaking changes in Python 3, though. It'd be interesting to know why this particular feature was kept. I guess from another point of view it's simpler to have everything in the `def` line executed when that line is reached (which is when the function is defined).
- mixmastamyk 13y agoI don't know. Scripts have to be somewhat rewritten in Py 3 anyway, I think it would have been a good time to fix it.
- rmc 13y agoPython 3 is deliberately not backwards compatible. They made several big breaking changes to remove warts.
- spamizbad 13y ago> That functionality really should be changed for the next major python release Make lists immutable you mean? Like tuple()? Edit: Or force re-evaluation on every function call (could be pretty slow)? The best practice is to just use a tuple (immutable) instead of a list to begin with. You can always do mylist = list(defaulttuple) inside your function first thing if you really want to work with it as a list. Also, both pylint and pep8, two python static analyzers, will all warn you about any mutable default arguments you provide. Many Python developers have these hooked into their editor or precommit hooks to catch things like this (among others).
- wahnfrieden 13y agoTuples shouldn't be differentiated from lists primarily in terms of mutable vs immutable. They represent different usages semantically, even though their primary functional difference is in their mutability. Lists are largely for homogeneous collections, whereas tuples are heterogeneous. When you mix them, you abuse what they signal to the reader. Python doesn't do much for type safety. Either accept the pitfalls of mutability as a tradeoff, or subclass list to make a safer type.
- spamizbad 13y agoThat's actually super insightful (homogeneous vs heterogeneous). I never made that connection but looking back I see it everywhere in Python code, and even did it unknowingly myself.
- klibertp 13y agoWhy didn't you use an array (from array module in stdlib) for homogeneous collection then? Python is dynamically typed, which means that, by default, we assume different types everywhere (heterogeneity). If you want to express homogeneity just do so explicitly.
- Myrmornis 13y agoNo, spamizbad is correct that wahnfrieden's comment was insightful. List and tuple have totally different semantics; they also happen to differ in mutability which just confuses the issue unfortunately.
- bluecalm 13y agoThere are actual reasons to make it and keep it that way. Easy caching function's results is one. You may want to read this SO thread: http://stackoverflow.com/questions/1132941/least-astonishment-in-python-the-mutable-default-argument http://stackoverflow.com/questions/1132941/least-astonishmen...
- Tloewald 13y agoThe accepted explanation seems wrong to me. Whether or not functions are first class objects or not, the behavior of def is a matter of design choice and in this case the wrong choice has been made. All the rationales are broken -- they're uncommon use cases and perfectly easy to implement without the behavior, i like Python but this is a wart.
- njharman 13y agoNo.
- the1 13y agono, default parameter should not be evaluated every time the function gets called. current behavior is totally fine. it is well defined that somelist.append(x) appends x to somelist inplace.
- mjhea0 13y agoyou could also use a generator function - def foo(num=[0]): num[0] = num[0] + 1 return num[0]
- tome 13y agoHuh? How's that a generator function?
- wahnfrieden 13y agoIt's not a generator, but his intention is clear - mutable keyword arguments allow for some clever tricks.
- deleted 13y ago[deleted]
- jzwinck 13y agoThey sure do--that's why we shouldn't have them. Instead: class Foo: def __init__(self): self.n = 0 def __call__(self): self.n += 1 return self.n foo = Foo()
- tome 13y agoCool, it looks like the Rachums are Israeli Pythonista brothers: https://news.ycombinator.com/item?id=5998675 https://news.ycombinator.com/item?id=5998675
- Nurdok 13y agoYeah, we are!
- smegel 13y agoWow I have been using Python for years and didn't realize default values were only created when defined. I don't think I have ever been bitten by it, but it would have been one helluva bug to track down.
- gizmo686 13y agoI've been bitten by that exactly once. In order to track it down, I had to tear out pieces of my program to issolate where it is, to the point where I was working with a single function that did nothing other than mutate a dict that was a default value and print said value; and code that called it twice.
- Siecje 13y agoWhy not use parenthesis with print so that it works for python 2 and python 3?
- lightcatcher 13y agoWhen I saw the title for this post, my first thought was "hmm, mutable default arguments are pretty tricky, hopefully that's somewhere on the list". I wasn't disappointed.
- arms 13y agoI went into this thinking "This'll be an obvious example - one I'm sure I wouldn't make." I was wrong, and now I know better. Thank you.
- rhizome31 13y agoAlso worth mentioning is the closure gotcha: when you create functions in a loop, the functions' context points to the context of the latest iteration of the loop. Other languages might have this issue as well and provide alternatives based on map-style constructs. In Python, you can take advantage of the behaviour described by the OP to make your inner functions have their own context, which is initialized when the function gets created.
- wahnfrieden 13y agoIn other words, function definitions create scope.
- stormbrew 13y agoThe most frustrating thing that bit me when I was first learning python was the error when you pass the wrong number of arguments to a method: >>> class A(object): ... def method(self, a): pass ... >>> A().method() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: method() takes exactly 2 arguments (1 given) Once you understand more about how method dispatch works it makes sense, but it's really confusing to be told you were doing something you don't think you are.
- _ZeD_ 13y agopass to python 3 :D >>> class C: ... def m(self,a): ... pass ... >>> C().m() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: m() missing 1 required positional argument: 'a' >>>
- stormbrew 13y agoNice. I haven't committed this particular error in python 3 yet, so it's good to know they made it more sane.
- nknighthb 13y agoA simple fix to the time/now example is also useful for demonstrating first-class functions: def print_now(now=time.time): print now()
- DrJ 13y agoif I recall, this is how SQLAlchemy does default values that are functions.
- ajanuary 13y agoYou do then have to wrap values in lambdas if you want to pass them in.
- herge 13y agoThe classic way is: def print_now(now=None): now = now or time.time() print now
- mnordhoff 13y agoI hope "now" is never 1970-01-01 00:00:00. Edit: if now is None: now = time.time()
- herge 13y agoOr even more paranoid: import mock def print_now(now=mock.sentinel.now): now = now if now != mock.sentinel.now else time.time()
- mnordhoff 13y agoI don't have experience with mock, but I've used object() -- literally -- as a sentinel. _sentinel = object() def print_now(now=_sentinel): if now is _sentinel: now = time.time() ...
- NiceOneBrah 13y agoAs a newbie to Python, my biggest issue when coming up to speed on a new project is chasing down the types of arguments to and return values from functions. Previously I was a Java programmer and while I don't miss its verbosity, I do miss how I always knew what types I was working with. Does anybody have any tips that make this easier?
- Bitwse 13y agoYou can make use of the type() and isinstance() functions when needed.
- nknighthb 13y agoThe biggest tip: The Python interactive shell is your friend. Use it to play with things you're not sure of. IPython[1] is a more advanced Python shell you may like. Lots of people also swear by pdb, the Python debugger. Personally, I get less use out of it than others, but YMMV. But overall, this is a problem involving documentation, design and naming, and your own mindset. Ideally you would never really be put in the position of not knowing, because it would be clearly documented. Of course, we all know how well that tends to work. Second, well-designed systems with good naming practices will make it fairly obvious most of the time what sort of arguments are expected, and what you can expect to get back. (This is related to the next point.) Finally, coming from a "static" language, your brain just isn't trained to perform its own type inference and deal with duck typing. This is mostly a matter of practice, you get better at it. But if you aren't already familiar with various kinds of typing, be sure to do some reading. A better understanding of the differences will help make sense of what you're seeing. [1] http://ipython.org/ http://ipython.org/
- sethammons 13y agoAnother alternative is bpython
- cpeterso 13y agoI think this is why Rust's type inference only applies within a function.
- thrownaway2424 13y agoI agree. Python is a common newbie mistake.
- TheSOB88 13y agoSeriously? Not even gonna back up your claim, just gonna troll?
- lostlogin 13y agoNewbie here with limited Python experience. What would you suggest? I have no projects in mind, just want to learn and have been mashing away at Python for months. What ever it is I make needs to run on a Mac or iDevice - via browser is fine.
- randomchars 13y agoPython is fine, parent is just a troll. If you're interested in OSX or iOS programming Objective-C would be the obvious choice.
- lostlogin 13y agoIt was suggested to me that I get a grasp of basic concepts before going there. I found it fairly easy to make a very very basic application with series of buttons that worked etc via objective c and the interface builder, but had no idea what I was doing when I hit code (which was the whole point of playing around). I have a better understanding of some coding basics now that I have done a small amount of python (Zed Shaw). Maybe it's time I went back to objective c - you've got me thinking... Thanks.
- randomchars 13y agoYou're going to miss the interpreter. I started programming with python too, and now I'm moving onto Java (for Android), and while static typing is nice, I really miss being able to just try things out in a few seconds. Working with a GUI framework is very different than web development and CLI.
- ak217 13y agoI would put the unfortunate scoping design above mutable defaults on the list of things that confused me. Also, I really don't like the decision to make Python 2 disobey the system locale and use ascii as the default encoding. Lots of Python programs out there have broken i18n for no good reason because of that (nobody starts teaching python by telling you to use .decode('utf-8')/.encode('utf-8') when you pipe data, and by the time you realize what's wrong, you often already have code out there.)
- sgdnogb2n 13y agoI expected far more from the article. It's not like this hasn't been blogged about before.
- bryanh 13y agoA safe pattern for this without the None check is keyword expansion which will define defaults when a function is called. def foo(**kwargs): numbers = kwargs.pop('numbers', []) numbers.append(9) print numbers
- kevinwuhoo 13y agoI've encountered a similar bug before. >>> arrays = [[]] * 5 >>> print arrays ... [[], [], [], [], []] >>> arrays[0].append(1) >>> print arrays ... [[1], [1], [1], [1], [1]]
- icebraining 13y agoThat's not a bug, it's just the result of Python variables being references to values, instead of actual values. When you write "[[]]", what you're doing is creating a list that holds a reference to another list. If you use a variable it's easier to understand: a = [] arrays = [a] * 5 What you have now is [a, a, a, a, a], so if you modify one of the lists, it's no wonder they all get modified: they're references to the same list.
- socillion 13y agoThat's because of the implicit copy() only copying by value 1 level deep? On the topic of the bug in OP, would a Python newbie expect the similarly confusing behavior caused by interactions between (im)mutability and function arguments being passed as a "reference"? Using OP's foo(): >>> a = [1,2,3] >>> foo(a) [1, 2, 3, 9] >>> a [1, 2, 3, 9] if 'a' were an immutable object like an int, it wouldn't be modified. >>> def bar(n): n += 1 print(n) >>> b = 3 >>> bar(b) 4 >>> b 3
- t1m 13y agoNote that tuples are immutable in python. I usually use tuples as default values for collections: def f(x=()): x = list(x) ... def g(y=(())) y = dict(y) ... f() is the pattern for lists, g() is for dicts. Note that you can pass in regular lists to f() and regular dicts to g().
- jlas 13y agoCool. Related: https://news.ycombinator.com/item?id=5925467 https://news.ycombinator.com/item?id=5925467