16 ms·

Understanding Python through its builtins

544 points by tusharsadhwani 5y ago

xojoc 5y ago
In a similar vein you may like "WTF Python: Exploring and understanding Python through surprising snippets": https://github.com/satwikkansal/wtfpython https://github.com/satwikkansal/wtfpython HN thread: https://news.ycombinator.com/item?id=26097732 https://news.ycombinator.com/item?id=26097732 (163 comments) PS: found with a site I'm building: https://discussions.xojoc.pw/?q=Understanding+Python+through+its+builtins https://discussions.xojoc.pw/?q=Understanding+Python+through...
submeta 5y ago
Nicely written article! - Slightly off topic: I love seeing and reading Python code. Used to see many flaws in the language (things like `.append` changing the object, returning `None` instead of creating a copy and returning that), but after ten years of working with Python I really appreciate its versatility, it‘s ubiquitous availability, the large number of libraries and the community. There‘s nothing I can‘t solve with it. It‘s the swiss army knife in my pocket, available whenever I need to solve something by coding.
- tusharsadhwani 5y ago
  Thank you! Things like `list.append` modifying in-place might feel like a flaw to some, but I think Python is really consistent when it comes to its behaviour. If you ask a person who comes from an object-oriented world, they'll say it only makes sense for a method on an object to modify that object's data directly. There's always ways to do things the other way, for example you can use x = [*x, item] to append and create a new copy, while being quite a bit more explicit that a new list is being created.
  - pokepim 5y ago
    Huh I never even thought we would need to create copy of an object when adding new item to it (like a new item to list for example). Is there any drawback on doing that in standard pythonic way? I actually learned to program using Python and it was my first language. Since then I only used JS. In both I like using functions a lot and rarely dabble in OOP since it is more conveniet to me.
    - tyingq 5y ago
      You can control the behavior manually, like: first = [0,1,2] second = [*a,3] # first is unchanged, second = [0,1,2,3] Or second=itertools.chain(first, [3]), which avoids the copy. Though, to me, it's asking for trouble later.
      - deleted 5y ago
        [deleted]
    - sgeisenh 5y ago
      You often lose performance in traditional imperative languages when aiming for persistence. When you have immutability guarantees (like in many functional programming languages like ML or Haskell) you can avoid making copies by sharing the parts of the data structure that don't change. If this kind of thing interests you, you should check out Chris Okasaki's book "Purely Functional Data Structures".
      - eximius 5y ago
        "avoid making copies" dors not always equal "performance". Depending on your access patterns, having the data colocated can be more important. But immutability sure is nice when you can have it.
    - tusharsadhwani 5y ago
      whether mutating data is better than creating a new copy for everything is a really long debate about immutability and functional programming, with good points on either sides, but that's really beyond the point here. In my opinion, you should use whichever method makes your code easy to read and understand for your usecase.
  - tyingq 5y ago
    >Python is really consistent when it comes to its behaviour True, though you end up with things like: ' '.join(thelist) Instead of thelist.join(' ') Because of the somewhat aggressive mantra to be consistent.
    - Marazan 5y ago
      But that's good. Because a string just needs to know aboit interable to perform that operation whereas every iterable would need to implement it's own join if you had it the other way around.
      - goatlover 5y ago
        Yet Ruby and JS manage to do it somehow. To me it seems natural that join should be a method on the iterable, and I always have to pause to remember Python is different.
        aidos 5y ago
        How does that work? Don’t you have to effectively convert your general iterable to an array and then join on that? Array.from(iterable).join(…)?
        deleted 5y ago
        [deleted]
        globular-toast 5y ago
        I don't think it should be a method at all. It's just a function: join(iterable, separator). It can also be implemented with reduce naturally: `reduce(lambda x, y: x + separator + y, iterable)`.
        Redoubts 5y ago
        Reduce sounds like a really slow way to do string building
  - int_19h 5y ago
    One related area where Python is not consistent is operators like +=. In pretty much all other languages that have them, the expected behavior of A+=B is exactly the same as A=A+B, except that A is only evaluated once. Now lets look at lists in Python: xs = [1, 2] ys = xs ys = ys + [3] print(xs, ys) This prints [1, 2] [1, 2, 3], because the third line created a new list, and made ys reference that. On the other hand, this: xs = [1, 2] ys = xs ys += [3] print(xs, ys) prints [1, 2, 3] [1, 2, 3], because += changes the list itself, and both xs and ys refer to that same list. (Note that this is not the same as C++, because in the latter, the variables store values directly, while in Python, all variables are references to values.) The worst part of it is that Python isn't even self-consistent here. If you only define __add__ in your custom class, you can use both + and += with its instances, with the latter behaving normally. But if you define __iadd__, as list does, then you can do whatever you want - and the idiomatic behavior is to modify the instance! For comparison, C# lets you overload + but not +=, and automatically synthesizes the latter from the former to enforce the correct behavior.
    - milliams 5y ago
      The rule in Python is that `=` creates a name binding, where as `+=` does not. That's pretty consistent as far as I can tell.
      - roelschroeven 5y ago
        The inconsistency is that for immutable objects (or more generally, objects that have __add__ but not __iadd__), '+=' does create a name binding. After 'a = 300; b = 50000', a += b is exactly the same as a = a + b In this case, a new object is created with the value of a + b which then gets bound to the name a.
      - int_19h 5y ago
        Nope, += also creates a binding! Try this: xs = [1, 2] def foo(): xs += [3] foo() You'll get an exception saying that local variable xs was used before it was assigned - precisely because += created a new local binding for xs inside foo.
- radarsat1 5y ago
  > (things like `.append` changing the object, returning `None` instead of creating a copy and returning that) This would be horrendously inefficient without immutable data structures like Clojure's. Very few languages have that, so it's a strange assumption to make, especially for a language as old as Python. Although it is a very nice feature of Clojure.
  - l33tc0der 5y ago
    One of the many reasons why I love Clojure so much! Rich implement his own brand of persistent data structures which makes Clojure's immutability a lot more efficient.
    - radarsat1 5y ago
      Thanks yeah, persistent is the term I was looking for, did not come to mind as I haven't used Clojure in a few years.
  - Blikkentrekker 5y ago
    It should also be worth nothing that Clojure sacrificed quite a bit to make this as efficient as possible. “persistent vectors” are certainly an interesting data structure that strike a compromise between fast indexing and being able to relatively quickly create a copy where only one element changes, but it's a compromise and indexing is made slower to allow for the latter. — They also take up more memory on their own but are allowed to share memory with their copies. I will say that my ideal language contains them in the standard library alongside standard vectors that index in constant time. Further, it should be noted that much of the performance talk is on the assumption that accessing from memory is truly random access; — with the existence of c.p.u. caches that assumption is not entirely accurate and accessing from contiguous rather than scattered memory in practice is considerably cheaper so one also pays the price for their being scattered more in memory.
    - adrusi 5y ago
      Random access into a clojure vector is going to need more memory lookups than conventional sequential buffer array (I don't recall the constants used in the implementation, I think it's either 4 or 8 lookups). But when you're indexing into the vector sequentially, the memory layout plays rather well with memory caching behavior, and most lookups are going to be in L1 cache, just like they would be in a conventional array. So lookups are a bit more expensive, but not as much more expensive as one might imagine.
- Alex3917 5y ago
  > things like `.append` changing the object, returning `None` instead of creating a copy and returning that The obvious question is why it can't return a reference to the list instead of returning None. I feel like if I've been using the language on an almost daily basis for ten years now and I still get burned by that all the time, then it's just a poorly designed feature.
  - canjobear 5y ago
    The advantage of mutating operations always returning None is that you can easily tell whether a mutation is happening by looking at the code. If you see y = f(x) that means x is unchanged, whereas if you see just f(x) on a line that means something stateful is happening.
    - macintux 5y ago
      There are counter-examples: functions that return values while also having side effects, cases where the developer simply made a mistake. Coming from a language with baked-in immutability, Python’s behavior in this regard was very difficult to get used to.
    - brundolf 5y ago
      Agreed. JavaScript's Array.sort is an example of this. Most of JavaScript's other array methods return a new array and people get used to chaining them, but sort mutates the array and also returns a reference to it. You can actually get pretty far before being bitten by this so long as you're sorting already-copied arrays. But then one day you hit a bizarre bug caused by behavior that's been sneaking past your radar the whole time.
    - Xavdidtheshadow 5y ago
      I really like ruby's method naming convention for this: * `sort(arr)` returns a sorted copy of the input * `sort!(arr)` returns the sorted original methods that return booleans end in `?`, like `arr.sorted?` It's just a convention, but it's a nice way to let the writer know what will happen.
- brundolf 5y ago
  > Used to see many flaws in the language (things like `.append` changing the object, returning `None` instead of creating a copy and returning that) I think it's pretty off-base to call this a "flaw". Immutable structures have their place and can be very helpful where appropriate, but making its core primitives work this way is far outside the scope or the philosophy of Python. If you want otherwise, you're really wanting an entirely different language. And there's nothing wrong with that! But I think it would be a "flaw" for Python to make these operations immutable, even though I love immutability personally.
- listenallyall 5y ago
  And also like a Swiss army knife, it's not particularly great at anything, can be awkward even when functional, and there's always a better tool for any specific job.
cinntaile 5y ago
Are there any good books that deal with writing pythonic code? As well as being focused on more intermediate or advanced features like this? If the book is project focused that's a bonus. Performance trade-offs another bonus.
- asdfgeoff 5y ago
  Not a book, but I have found Trey Hunner's https://www.pythonmorsels.com/ https://www.pythonmorsels.com/ exercises very useful on this front. The solutions presented typically include both a "basic" approach, a "as pythonic as possible" approach, and a brief discussion of the trade-offs between elegance and readability, etc.
- tusharsadhwani 5y ago
  I can personally recommend Fluent Python (its 2nd edition is about to come out in a couple months) for learning these intermediate/advanced concepts, and Python Cookbook for code examples using many of these features. I don't know any books for projects per-se, maybe HN will know!
  - deleted 5y ago
    [deleted]
  - cinntaile 5y ago
    To me it looks like a lot of this knowledge is spread out over many different excellent technical blogs like yours. While the content is good, it's hard to get something that resembles a more complete picture compared to just another piece of a big puzzle.
- tracyhenry 5y ago
  Searching python on HackerNews readings (a site I built): https://hacker-recommended-books.vercel.app/category/0/all-time/page/0/0/search?q=python https://hacker-recommended-books.vercel.app/category/0/all-t... The results include Fluent Python recommended by tusharsadhwani
  - 5y ago
tyingq 5y ago
In a somewhat similar way, this post about hacking the import system to load modules from strings finally helped me understand how imports work: https://cprohm.de/blog/python-packages-in-a-single-file/ https://cprohm.de/blog/python-packages-in-a-single-file/
d_burfoot 5y ago
Great article. This is what I come to HN to find.
ps173 5y ago
I am not even half way through it and I now understand how python actually works under hood. This is great for understanding how a lot of interpreted languages work
Pearse 5y ago
This is such a good write up for someone like myself still trying to get their head around Python. I can appreciate this must have taken considerable effort, It reads really well. Thank you!
- tusharsadhwani 5y ago
  I was concerned if my writing style will click with people, this is good to know :)
eximius 5y ago
In addition to this, I highly recommend just reading the codebase. I haven't written C since college and it's remarkably readable. I once tried to catalogue all the stdlib operations which release the GIL, meaning if you use only those (well, only those "heavy" bits, you can still use other small blocking glue bits), you can do "real" multithreading. It was a fun exercise!
- dharmab 5y ago
  Agreed. CPython makes readability and maintainability a priority.
- riazrizvi 5y ago
  And it helps much more in the actual job of writing software than in say, practicing coding puzzles
- matheusmoreira 5y ago
  > I highly recommend just reading the codebase Me too. When I used to write Ruby I read a lot of CRuby source code. I achieved a much deeper understanding of the language that way. Even answered some really fun stackoverflow questions. Now the first thing I do when I see a new language is read its source code.
  - jamesfinlayson 5y ago
    Yep, when I was writing PHP I found myself digging into php-src on a regular basis to see what exactly was going on. PHP's documentation is good but the code is much more explicit.
- hultner 5y ago
  There's a really nice (although old now) walk through of the cpython code base on YouTube. I watched it on a long 24 hour flight between Canada and Sweden a couple of years back. Edit: Found it! You're in for about 9 hours of quality watching. https://youtube.com/playlist?list=PLwyG5wA5gIzgTFj5KgJJ15lxq5Cv6lo_0 https://youtube.com/playlist?list=PLwyG5wA5gIzgTFj5KgJJ15lxq...
collsni 5y ago
You know what helped me with python? Breakpoints inside vscode's module. It all just kinda clicked.
- yuy910616 5y ago
  python actually has a build-in `breakpoint()` function. I think it brings up a repl at the line. I've been using that instead of print debug, it's been great.
  - int_19h 5y ago
    It's actually customizable via the PYTHONBREAKPOINT environment variable and sys.breakpointhook(). The default does pdb.set_trace(), which gives you a built-in debugger prompt (not a REPL) at that location. But it can be set to execute arbitrary code, and most Python IDEs make it behave like "normal" breakpoints.
sireat 5y ago
I am a pretty average Python programmer(5 years teaching, 15 years writing). I still wonder what was the reasoning for allowing creation of local objects with the same name as builtins. Okay it can be nice to redefine pprint as print I suppose. Still how many sum, list, min, max, dict(!) have been erroneously redefined in beginner tutorials and beginner code. From my experience sum and list suffer the most. Sure there are linters that will warn you but there should be a setting for the interpreter (as in -Werror in GCC) to disallow this silliness.
- tusharsadhwani 5y ago
  I think linters are really effective to figure out such issues in professional code. For students for example, I'll have to agree. Maybe having a flag or environment variable that teachers can set up for it would be a nice idea. You should start a thread on the python-ideas mailing list about this, and it might get somewhere :)
- nneonneo 5y ago
  Python itself doesn’t disallow this because there are quite a lot of builtins with useful names - for example, `file`, `id`, and `hash` to name a few. Disallowing setting these would be tantamount to adding a bunch of new keywords to the language, which they’ve been quite loathe to do in general. A good linter will catch these, so in production environments you usually don’t run into issues. I agree that it can be a beginner trap though!
- wpietri 5y ago
  No idea what their actual reasoning is, but here's how I think about it: This is better for novices, because otherwise you create a whole bunch of land mines for people who are desperately trying to get something done. If they aren't aware of the built-in then they aren't trying to use it. Insisting that they become aware of something they don't want right then will be frustrating. It's also better for experts, in that they're generally aware they're overriding a built-in and are doing it on purpose, and if not they'll have an IDE or linter reminding them. To me, I see tooling as a spectrum from supportive to controlling. Python is very much on the supportive end. It feels controlling when I get interrupted because some programmer who has never met me programmed a tool to insist I do things their way. That would very much include insisting I respect a bunch of names they decided long ago to put in the global namespace.
nneonneo 5y ago
Cute fact about __debug__: it is one of the only ways to get compile-time conditionals in Python. Performing a comparison with `if __debug__:` will output byte code for the ensuing statement if and only if the interpreter is in debug mode - notably, in `-O` mode, it will not even generate a load of __debug__ and a conditional jump, and acts as if the statement didn’t exist at all.
- globular-toast 5y ago
  I knew that `assert` behaved in a similar way. Turns out it's actually equivalent to an `if __debug__:` https://docs.python.org/3/reference/simple_stmts.html#grammar-token-python-grammar-assert_stmt https://docs.python.org/3/reference/simple_stmts.html#gramma...
- tusharsadhwani 5y ago
  that is indeed interesting. Mind if I add this in the article?
  - nneonneo 5y ago
    Sure, go ahead :)
- Fordec 5y ago
  Not going to lie, if I could enable stricter compile time conditions without debug mode that would be a very welcome RFC
- BiteCode_dev 5y ago
  Unfortunatly you often can't use -o because of the 3rd party libs that didnt' get the memo and use assert for error checking. We still have -X dev and sys.flags but it's runtime only.
  - alshel 5y ago
    Care to name any 3rd party libs in particular?
DangitBobby 5y ago
Very well written. Fun little tidbit, Django abuses the fact that bools are ints in it's partition util: https://github.com/django/django/blob/01bf679e59850bb7b3e6395f1683bd1463ed9969/django/utils/functional.py#L414 https://github.com/django/django/blob/01bf679e59850bb7b3e639...
- hultner 5y ago
  Where? I don't see anything in that code relying on that unless I'm Sunday blind.
  - iezepov 5y ago
    `results[predicate(item)]` here they get the first and the second elements of a tuple. Essentially it’s `results[False]` and `results[True]`
    - hultner 5y ago
      Oh, that's obvious now. Must be that Sunday night blindness.
  - NegativeLatency 5y ago
    Results has 2 elements indexed by 0 and 1 (it’s a tuple not a dict)
- int_19h 5y ago
  It does make for some amusing code golf techniques, e.g.: print([ f"{(not x % 3) * 'Fizz'}{(not x % 5) * 'Buzz'}" or x for x in range(1, 20) ])
  - DangitBobby 5y ago
    Fantastic
- np_tedious 5y ago
  cool! >>> l = ['a','b'] >>> l[False] 'a' >>> l[True] 'b' >>> d = {0: 'a', 1: 'b'} >>> d[False] 'a' >>> d[True] 'b' >>> # TIL!
matsemann 5y ago
> List comprehensions are basically a more Pythonic, more readable way to write these exact same things More pythonic maybe, but you can't have more than a single expression in a list comprehension without it becoming completely unintelligible. I also often miss other standard list features. Reduce, flatmap, indexed versions, utils like first of predicate, split, filternonnull etc
- BerislavLopac 5y ago
  Are you saying that: l = [] for a in range(10): for b in range(10): for c in range (10): l.append(a + b + c) is more intelligible than: l = [ a + b + c for a in range(10) for b in range(10) for c in range(10) ] ???
- weatherlight 5y ago
  Anything remotely interesting like that is dumped in itertools. Python's creator, Guido van Rossum, doesn't like functional/functional-ish programming a lot. That's well-known. Guido: "I value readability and usefulness for real code. There are some places where map() and filter() make sense, and for other places Python has list comprehensions. I ended up hating reduce() because it was almost exclusively used (a) to implement sum(), or (b) to write unreadable code. So we added built-in sum() at the same time we demoted reduce() from a built-in to something in functools (which is a dumping ground for stuff I don't really care about :-)."
  - matsemann 5y ago
    Yes, lots of them are available. But I also would like to be able to call .sum() on my iterable at the end of a chain, instead of having to mentally unwrap sum(map(filter(filter(map(...))))
    - weatherlight 5y ago
      I strongly dislike python, I often wonder if it would have been as popular if Guido didn't work at Google early on.
PostThisTooFast 5y ago
"Builtins?"
escanor 5y ago
> for index, item in enumerate(menu): should be > for index, item in enumerate(menu, start=1): for the example to be correct :)
- tusharsadhwani 5y ago
  you're right, thanks!
kgm 5y ago
This is a neat article, but it does have some errors. One subtle point that the post gets wrong: > So where does that come from? The answer is that Python stores everything inside dictionaries associated with each local scope. Which means that every piece of code has its own defined “local scope” which is accessed using locals() inside that code, that contains the values corresponding to each variable name. The dictionary returned by `locals()` is not literally a function's local namespace, it's a copy of that namespace. The actual local namespace is an array that is part of the frame object; in this way, references to local variables may happen much more quickly than would be the case if it had to look each variable up in a dictionary every time. One consequence of this is that you can't mutate the dict returned by `locals()` in order to change the value of a function-local variable. Another, less-subtle error in the post is this: > int is another widely-used, fundamental primitive data type. It’s also the lowest common denominator of 2 other data types: , float and complex. complex is a supertype of float, which, in turn, is a supertype of int. > What this means is that all ints are valid as a float as well as a complex, but not the other way around. Similarly, all floats are also valid as a complex. Oh, no no no. Python integers are arbitrary-precision integers. Floats are IEEE 754 double-precision binary floating-point values, and as such only support full integer precision up to 2^53. The int type can represent values beyond that range which the float type cannot. And while it is true that the complex type is just two floats stuck together, I would very much not call it a supertype. It performs distinct operations. > Accessing an attribute with obj.x calls the __getattr__ method underneath. Similarly setting a new attribute and deleting an attribute calls __setattr__ and __detattr__ respectively. Attribute lookup in Python is way more complex than this. It's an enormous tar pit, too much so to detail in this comment, but __getattr__ is most often not involved, and the `object` type doesn't even have a __getattr__ method.
- tusharsadhwani 5y ago
  Oh, I didn't know that about locals! Yeah, calling float and complex "supertypes" probably wasn't the best idea, but I couldn't think of a better explanation that wouldn't take too long to explain. I'll ponder about that one. the getattr thing seems like a huge rabbit hole, I'm totally going to look into this. Thank you :)
- 5y ago
howolduis 5y ago
ever heard of line separator?
tored 5y ago
Thanks, got a better understanding of the Python "philosophy" because of this article, easy to follow even if you haven’t written a single line of Python like me.
- tusharsadhwani 5y ago
  That's great to hear :D hopefully you'll try out Python sometime.
thrdbndndn 5y ago
>As a bonus, this also adds support for adding two MyNumber classes together: Merely having `__add__` (without `__radd__`) is enough to add two MyNumber classes together in your case. >It mostly exists to support type annotations, The link for "type annotations" is broken.
- tusharsadhwani 5y ago
  Type annotations link is fixed, thank you! Just `__add__` doesn't work for me. I get: TypeError: unsupported operand type(s) for +: 'int' and 'Number' Here's the code: class Number: def __add__(self, x): return 42 + x num = Number() print(num + num) Edit: Okay. replacing `42 + x` with `x + 42` actually makes it work. But I'll be honest I have no idea what happened there.
  - thrdbndndn 5y ago
    It took me quite a long time to understand what happened there too! (Pseudo code, let's call them num1 and num2 for better readability) num1 + num2 = num1.__add__(num2) = num2 + 42 (that's why the order is important) = num2.__add__(42) = 42 + 42 = 84
  - thrdbndndn 5y ago
    >Type annotations link is fixed, thank you! I'm not sure about that - it's still `<a href="mypy-guide">type annotations</a>` which jumps to https://sadh.life/post/builtins/mypy-guide https://sadh.life/post/builtins/mypy-guide and then jumps to your homepage.
tyilo 5y ago
> Python has exactly 6 primitive data types (well, actually just 5, but we’ll get to that). 4 of these are numerical in nature, and the other 2 are text-based. Let’s talk about the text-based first, because that’s going to be much simpler. What is your definition of a primitive data types? All of these have object as a superclass, so I wouldn't call them primitive data types in python. Maybe there is just 1 primitive type: type? Or none at all?
- tusharsadhwani 5y ago
  In my case I meant it as things that only extend from object. Kinda like how prime numbers only have two factors including themselves.
  - tyilo 5y ago
    What about list, dict, set, tuple, range and map then?
ripe 5y ago
Thank you for writing the post. Newbie question about “nonlocal” from your example: def outer_function(): x = 11 def inner_function(): nonlocal x x = 22 print('Inner x:', x) inner_funcion() print('Outer x:', x) I get how the example works, but don’t see the point of the declaration? If I just left out the “nonlocal x” line, wouldn’t the example still work the same?
- ekimekim 5y ago
  Python assumes that all assignments assign to the current scope. So by default when you reach "x = 22", it would create a new variable called "x" in the inner_function() scope which overrides the variable "x" in the outer_function() scope. So when you print "Inner x" you would only be printing the inner_function() version of x, not the outer_function() version, which would remain at 11.
  - faho 5y ago
    This is a consequence of python not having explicit variable definition. Here it'll decide to define a new x instead of seeing the old one. And that's also usually what you want because otherwise a function would start altering variables in the enclosing scope if they happen to exist! E.g. foo = 42 def myfunc(bar): foo = bar + 1 print(foo) myfunc(6) print(foo) # would print "7" if the "foo =" above took the nonlocal foo automatically! So the trade-off is to require "nonlocal" if you ever need a variable from the enclosing scope.
faho 5y ago
The list comparison here is also true when the first list is a prefix of the other: class list: def __eq__(self, other): return all(x == y for x, y in zip(self, other)) # Can also be written as: return all(self[i] == other[i] for i in range(len(self))) run that with `[1,2,3]` and `[1,2,3,4]` and it'll be true because it only checks up to the 3. It's probably simplest to compare `len(self) == len(other)` before. Similarly, the set comparison will also be true if the first is a subset of the other.
- tusharsadhwani 5y ago
  Yup, I'll fix this, thanks for pointing it out.
tallguytyo 5y ago
Your article is really easy to get through and digest. I'd like to keep it as a reference going forward. Please consider adding a floating TOC to the page.

In a similar vein you may like "WTF Python: Exploring and understanding Python through surprising snippets": https://github.com/satwikkansal/wtfpython https://github.com/satwikkansal/wtfpython HN thread: https://news.ycombinator.com/item?id=26097732 https://news.ycombinator.com/item?id=26097732 (163 comments) PS: found with a site I'm building: https://discussions.xojoc.pw/?q=Understanding+Python+through+its+builtins https://discussions.xojoc.pw/?q=Understanding+Python+through...

submeta 5y ago

Nicely written article! - Slightly off topic: I love seeing and reading Python code. Used to see many flaws in the language (things like `.append` changing the object, returning `None` instead of creating a copy and returning that), but after ten years of working with Python I really appreciate its versatility, it‘s ubiquitous availability, the large number of libraries and the community. There‘s nothing I can‘t solve with it. It‘s the swiss army knife in my pocket, available whenever I need to solve something by coding.

tusharsadhwani 5y ago
Thank you! Things like `list.append` modifying in-place might feel like a flaw to some, but I think Python is really consistent when it comes to its behaviour. If you ask a person who comes from an object-oriented world, they'll say it only makes sense for a method on an object to modify that object's data directly. There's always ways to do things the other way, for example you can use x = [*x, item] to append and create a new copy, while being quite a bit more explicit that a new list is being created.
- pokepim 5y ago
  Huh I never even thought we would need to create copy of an object when adding new item to it (like a new item to list for example). Is there any drawback on doing that in standard pythonic way? I actually learned to program using Python and it was my first language. Since then I only used JS. In both I like using functions a lot and rarely dabble in OOP since it is more conveniet to me.
  - tyingq 5y ago
    You can control the behavior manually, like: first = [0,1,2] second = [*a,3] # first is unchanged, second = [0,1,2,3] Or second=itertools.chain(first, [3]), which avoids the copy. Though, to me, it's asking for trouble later.
    - deleted 5y ago
      [deleted]
  - sgeisenh 5y ago
    You often lose performance in traditional imperative languages when aiming for persistence. When you have immutability guarantees (like in many functional programming languages like ML or Haskell) you can avoid making copies by sharing the parts of the data structure that don't change. If this kind of thing interests you, you should check out Chris Okasaki's book "Purely Functional Data Structures".
    - eximius 5y ago
      "avoid making copies" dors not always equal "performance". Depending on your access patterns, having the data colocated can be more important. But immutability sure is nice when you can have it.
  - tusharsadhwani 5y ago
    whether mutating data is better than creating a new copy for everything is a really long debate about immutability and functional programming, with good points on either sides, but that's really beyond the point here. In my opinion, you should use whichever method makes your code easy to read and understand for your usecase.
- tyingq 5y ago
  >Python is really consistent when it comes to its behaviour True, though you end up with things like: ' '.join(thelist) Instead of thelist.join(' ') Because of the somewhat aggressive mantra to be consistent.
  - Marazan 5y ago
    But that's good. Because a string just needs to know aboit interable to perform that operation whereas every iterable would need to implement it's own join if you had it the other way around.
    - goatlover 5y ago
      Yet Ruby and JS manage to do it somehow. To me it seems natural that join should be a method on the iterable, and I always have to pause to remember Python is different.
      - aidos 5y ago
        How does that work? Don’t you have to effectively convert your general iterable to an array and then join on that? Array.from(iterable).join(…)?
      - deleted 5y ago
        [deleted]
      - globular-toast 5y ago
        I don't think it should be a method at all. It's just a function: join(iterable, separator). It can also be implemented with reduce naturally: `reduce(lambda x, y: x + separator + y, iterable)`.
        Redoubts 5y ago
        Reduce sounds like a really slow way to do string building
- int_19h 5y ago
  One related area where Python is not consistent is operators like +=. In pretty much all other languages that have them, the expected behavior of A+=B is exactly the same as A=A+B, except that A is only evaluated once. Now lets look at lists in Python: xs = [1, 2] ys = xs ys = ys + [3] print(xs, ys) This prints [1, 2] [1, 2, 3], because the third line created a new list, and made ys reference that. On the other hand, this: xs = [1, 2] ys = xs ys += [3] print(xs, ys) prints [1, 2, 3] [1, 2, 3], because += changes the list itself, and both xs and ys refer to that same list. (Note that this is not the same as C++, because in the latter, the variables store values directly, while in Python, all variables are references to values.) The worst part of it is that Python isn't even self-consistent here. If you only define __add__ in your custom class, you can use both + and += with its instances, with the latter behaving normally. But if you define __iadd__, as list does, then you can do whatever you want - and the idiomatic behavior is to modify the instance! For comparison, C# lets you overload + but not +=, and automatically synthesizes the latter from the former to enforce the correct behavior.
  - milliams 5y ago
    The rule in Python is that `=` creates a name binding, where as `+=` does not. That's pretty consistent as far as I can tell.
    - roelschroeven 5y ago
      The inconsistency is that for immutable objects (or more generally, objects that have __add__ but not __iadd__), '+=' does create a name binding. After 'a = 300; b = 50000', a += b is exactly the same as a = a + b In this case, a new object is created with the value of a + b which then gets bound to the name a.
    - int_19h 5y ago
      Nope, += also creates a binding! Try this: xs = [1, 2] def foo(): xs += [3] foo() You'll get an exception saying that local variable xs was used before it was assigned - precisely because += created a new local binding for xs inside foo.
radarsat1 5y ago
> (things like `.append` changing the object, returning `None` instead of creating a copy and returning that) This would be horrendously inefficient without immutable data structures like Clojure's. Very few languages have that, so it's a strange assumption to make, especially for a language as old as Python. Although it is a very nice feature of Clojure.
- l33tc0der 5y ago
  One of the many reasons why I love Clojure so much! Rich implement his own brand of persistent data structures which makes Clojure's immutability a lot more efficient.
  - radarsat1 5y ago
    Thanks yeah, persistent is the term I was looking for, did not come to mind as I haven't used Clojure in a few years.
- Blikkentrekker 5y ago
  It should also be worth nothing that Clojure sacrificed quite a bit to make this as efficient as possible. “persistent vectors” are certainly an interesting data structure that strike a compromise between fast indexing and being able to relatively quickly create a copy where only one element changes, but it's a compromise and indexing is made slower to allow for the latter. — They also take up more memory on their own but are allowed to share memory with their copies. I will say that my ideal language contains them in the standard library alongside standard vectors that index in constant time. Further, it should be noted that much of the performance talk is on the assumption that accessing from memory is truly random access; — with the existence of c.p.u. caches that assumption is not entirely accurate and accessing from contiguous rather than scattered memory in practice is considerably cheaper so one also pays the price for their being scattered more in memory.
  - adrusi 5y ago
    Random access into a clojure vector is going to need more memory lookups than conventional sequential buffer array (I don't recall the constants used in the implementation, I think it's either 4 or 8 lookups). But when you're indexing into the vector sequentially, the memory layout plays rather well with memory caching behavior, and most lookups are going to be in L1 cache, just like they would be in a conventional array. So lookups are a bit more expensive, but not as much more expensive as one might imagine.
Alex3917 5y ago
> things like `.append` changing the object, returning `None` instead of creating a copy and returning that The obvious question is why it can't return a reference to the list instead of returning None. I feel like if I've been using the language on an almost daily basis for ten years now and I still get burned by that all the time, then it's just a poorly designed feature.
- canjobear 5y ago
  The advantage of mutating operations always returning None is that you can easily tell whether a mutation is happening by looking at the code. If you see y = f(x) that means x is unchanged, whereas if you see just f(x) on a line that means something stateful is happening.
  - macintux 5y ago
    There are counter-examples: functions that return values while also having side effects, cases where the developer simply made a mistake. Coming from a language with baked-in immutability, Python’s behavior in this regard was very difficult to get used to.
  - brundolf 5y ago
    Agreed. JavaScript's Array.sort is an example of this. Most of JavaScript's other array methods return a new array and people get used to chaining them, but sort mutates the array and also returns a reference to it. You can actually get pretty far before being bitten by this so long as you're sorting already-copied arrays. But then one day you hit a bizarre bug caused by behavior that's been sneaking past your radar the whole time.
  - Xavdidtheshadow 5y ago
    I really like ruby's method naming convention for this: * `sort(arr)` returns a sorted copy of the input * `sort!(arr)` returns the sorted original methods that return booleans end in `?`, like `arr.sorted?` It's just a convention, but it's a nice way to let the writer know what will happen.
brundolf 5y ago
> Used to see many flaws in the language (things like `.append` changing the object, returning `None` instead of creating a copy and returning that) I think it's pretty off-base to call this a "flaw". Immutable structures have their place and can be very helpful where appropriate, but making its core primitives work this way is far outside the scope or the philosophy of Python. If you want otherwise, you're really wanting an entirely different language. And there's nothing wrong with that! But I think it would be a "flaw" for Python to make these operations immutable, even though I love immutability personally.
listenallyall 5y ago
And also like a Swiss army knife, it's not particularly great at anything, can be awkward even when functional, and there's always a better tool for any specific job.

cinntaile 5y ago

Are there any good books that deal with writing pythonic code? As well as being focused on more intermediate or advanced features like this? If the book is project focused that's a bonus. Performance trade-offs another bonus.

asdfgeoff 5y ago
Not a book, but I have found Trey Hunner's https://www.pythonmorsels.com/ https://www.pythonmorsels.com/ exercises very useful on this front. The solutions presented typically include both a "basic" approach, a "as pythonic as possible" approach, and a brief discussion of the trade-offs between elegance and readability, etc.
tusharsadhwani 5y ago
I can personally recommend Fluent Python (its 2nd edition is about to come out in a couple months) for learning these intermediate/advanced concepts, and Python Cookbook for code examples using many of these features. I don't know any books for projects per-se, maybe HN will know!
- deleted 5y ago
  [deleted]
- cinntaile 5y ago
  To me it looks like a lot of this knowledge is spread out over many different excellent technical blogs like yours. While the content is good, it's hard to get something that resembles a more complete picture compared to just another piece of a big puzzle.
tracyhenry 5y ago
Searching python on HackerNews readings (a site I built): https://hacker-recommended-books.vercel.app/category/0/all-time/page/0/0/search?q=python https://hacker-recommended-books.vercel.app/category/0/all-t... The results include Fluent Python recommended by tusharsadhwani
- 5y ago

tyingq 5y ago

In a somewhat similar way, this post about hacking the import system to load modules from strings finally helped me understand how imports work: https://cprohm.de/blog/python-packages-in-a-single-file/ https://cprohm.de/blog/python-packages-in-a-single-file/

d_burfoot 5y ago

Great article. This is what I come to HN to find.

ps173 5y ago

I am not even half way through it and I now understand how python actually works under hood. This is great for understanding how a lot of interpreted languages work

Pearse 5y ago

This is such a good write up for someone like myself still trying to get their head around Python. I can appreciate this must have taken considerable effort, It reads really well. Thank you!

tusharsadhwani 5y ago
I was concerned if my writing style will click with people, this is good to know :)

eximius 5y ago

In addition to this, I highly recommend just reading the codebase. I haven't written C since college and it's remarkably readable. I once tried to catalogue all the stdlib operations which release the GIL, meaning if you use only those (well, only those "heavy" bits, you can still use other small blocking glue bits), you can do "real" multithreading. It was a fun exercise!

dharmab 5y ago
Agreed. CPython makes readability and maintainability a priority.
riazrizvi 5y ago
And it helps much more in the actual job of writing software than in say, practicing coding puzzles
matheusmoreira 5y ago
> I highly recommend just reading the codebase Me too. When I used to write Ruby I read a lot of CRuby source code. I achieved a much deeper understanding of the language that way. Even answered some really fun stackoverflow questions. Now the first thing I do when I see a new language is read its source code.
- jamesfinlayson 5y ago
  Yep, when I was writing PHP I found myself digging into php-src on a regular basis to see what exactly was going on. PHP's documentation is good but the code is much more explicit.
hultner 5y ago
There's a really nice (although old now) walk through of the cpython code base on YouTube. I watched it on a long 24 hour flight between Canada and Sweden a couple of years back. Edit: Found it! You're in for about 9 hours of quality watching. https://youtube.com/playlist?list=PLwyG5wA5gIzgTFj5KgJJ15lxq5Cv6lo_0 https://youtube.com/playlist?list=PLwyG5wA5gIzgTFj5KgJJ15lxq...

collsni 5y ago

You know what helped me with python? Breakpoints inside vscode's module. It all just kinda clicked.

yuy910616 5y ago
python actually has a build-in `breakpoint()` function. I think it brings up a repl at the line. I've been using that instead of print debug, it's been great.
- int_19h 5y ago
  It's actually customizable via the PYTHONBREAKPOINT environment variable and sys.breakpointhook(). The default does pdb.set_trace(), which gives you a built-in debugger prompt (not a REPL) at that location. But it can be set to execute arbitrary code, and most Python IDEs make it behave like "normal" breakpoints.

sireat 5y ago

I am a pretty average Python programmer(5 years teaching, 15 years writing). I still wonder what was the reasoning for allowing creation of local objects with the same name as builtins. Okay it can be nice to redefine pprint as print I suppose. Still how many sum, list, min, max, dict(!) have been erroneously redefined in beginner tutorials and beginner code. From my experience sum and list suffer the most. Sure there are linters that will warn you but there should be a setting for the interpreter (as in -Werror in GCC) to disallow this silliness.

tusharsadhwani 5y ago
I think linters are really effective to figure out such issues in professional code. For students for example, I'll have to agree. Maybe having a flag or environment variable that teachers can set up for it would be a nice idea. You should start a thread on the python-ideas mailing list about this, and it might get somewhere :)
nneonneo 5y ago
Python itself doesn’t disallow this because there are quite a lot of builtins with useful names - for example, `file`, `id`, and `hash` to name a few. Disallowing setting these would be tantamount to adding a bunch of new keywords to the language, which they’ve been quite loathe to do in general. A good linter will catch these, so in production environments you usually don’t run into issues. I agree that it can be a beginner trap though!
wpietri 5y ago
No idea what their actual reasoning is, but here's how I think about it: This is better for novices, because otherwise you create a whole bunch of land mines for people who are desperately trying to get something done. If they aren't aware of the built-in then they aren't trying to use it. Insisting that they become aware of something they don't want right then will be frustrating. It's also better for experts, in that they're generally aware they're overriding a built-in and are doing it on purpose, and if not they'll have an IDE or linter reminding them. To me, I see tooling as a spectrum from supportive to controlling. Python is very much on the supportive end. It feels controlling when I get interrupted because some programmer who has never met me programmed a tool to insist I do things their way. That would very much include insisting I respect a bunch of names they decided long ago to put in the global namespace.

nneonneo 5y ago

Cute fact about __debug__: it is one of the only ways to get compile-time conditionals in Python. Performing a comparison with `if __debug__:` will output byte code for the ensuing statement if and only if the interpreter is in debug mode - notably, in `-O` mode, it will not even generate a load of __debug__ and a conditional jump, and acts as if the statement didn’t exist at all.

globular-toast 5y ago
I knew that `assert` behaved in a similar way. Turns out it's actually equivalent to an `if __debug__:` https://docs.python.org/3/reference/simple_stmts.html#grammar-token-python-grammar-assert_stmt https://docs.python.org/3/reference/simple_stmts.html#gramma...
tusharsadhwani 5y ago
that is indeed interesting. Mind if I add this in the article?
- nneonneo 5y ago
  Sure, go ahead :)
Fordec 5y ago
Not going to lie, if I could enable stricter compile time conditions without debug mode that would be a very welcome RFC
BiteCode_dev 5y ago
Unfortunatly you often can't use -o because of the 3rd party libs that didnt' get the memo and use assert for error checking. We still have -X dev and sys.flags but it's runtime only.
- alshel 5y ago
  Care to name any 3rd party libs in particular?

DangitBobby 5y ago

Very well written. Fun little tidbit, Django abuses the fact that bools are ints in it's partition util: https://github.com/django/django/blob/01bf679e59850bb7b3e6395f1683bd1463ed9969/django/utils/functional.py#L414 https://github.com/django/django/blob/01bf679e59850bb7b3e639...

hultner 5y ago
Where? I don't see anything in that code relying on that unless I'm Sunday blind.
- iezepov 5y ago
  `results[predicate(item)]` here they get the first and the second elements of a tuple. Essentially it’s `results[False]` and `results[True]`
  - hultner 5y ago
    Oh, that's obvious now. Must be that Sunday night blindness.
- NegativeLatency 5y ago
  Results has 2 elements indexed by 0 and 1 (it’s a tuple not a dict)
int_19h 5y ago
It does make for some amusing code golf techniques, e.g.: print([ f"{(not x % 3) * 'Fizz'}{(not x % 5) * 'Buzz'}" or x for x in range(1, 20) ])
- DangitBobby 5y ago
  Fantastic
np_tedious 5y ago
cool! >>> l = ['a','b'] >>> l[False] 'a' >>> l[True] 'b' >>> d = {0: 'a', 1: 'b'} >>> d[False] 'a' >>> d[True] 'b' >>> # TIL!

matsemann 5y ago

> List comprehensions are basically a more Pythonic, more readable way to write these exact same things More pythonic maybe, but you can't have more than a single expression in a list comprehension without it becoming completely unintelligible. I also often miss other standard list features. Reduce, flatmap, indexed versions, utils like first of predicate, split, filternonnull etc

BerislavLopac 5y ago
Are you saying that: l = [] for a in range(10): for b in range(10): for c in range (10): l.append(a + b + c) is more intelligible than: l = [ a + b + c for a in range(10) for b in range(10) for c in range(10) ] ???
weatherlight 5y ago
Anything remotely interesting like that is dumped in itertools. Python's creator, Guido van Rossum, doesn't like functional/functional-ish programming a lot. That's well-known. Guido: "I value readability and usefulness for real code. There are some places where map() and filter() make sense, and for other places Python has list comprehensions. I ended up hating reduce() because it was almost exclusively used (a) to implement sum(), or (b) to write unreadable code. So we added built-in sum() at the same time we demoted reduce() from a built-in to something in functools (which is a dumping ground for stuff I don't really care about :-)."
- matsemann 5y ago
  Yes, lots of them are available. But I also would like to be able to call .sum() on my iterable at the end of a chain, instead of having to mentally unwrap sum(map(filter(filter(map(...))))
  - weatherlight 5y ago
    I strongly dislike python, I often wonder if it would have been as popular if Guido didn't work at Google early on.

PostThisTooFast 5y ago

"Builtins?"

escanor 5y ago

> for index, item in enumerate(menu): should be > for index, item in enumerate(menu, start=1): for the example to be correct :)

tusharsadhwani 5y ago
you're right, thanks!

kgm 5y ago

This is a neat article, but it does have some errors. One subtle point that the post gets wrong: > So where does that come from? The answer is that Python stores everything inside dictionaries associated with each local scope. Which means that every piece of code has its own defined “local scope” which is accessed using locals() inside that code, that contains the values corresponding to each variable name. The dictionary returned by `locals()` is not literally a function's local namespace, it's a copy of that namespace. The actual local namespace is an array that is part of the frame object; in this way, references to local variables may happen much more quickly than would be the case if it had to look each variable up in a dictionary every time. One consequence of this is that you can't mutate the dict returned by `locals()` in order to change the value of a function-local variable. Another, less-subtle error in the post is this: > int is another widely-used, fundamental primitive data type. It’s also the lowest common denominator of 2 other data types: , float and complex. complex is a supertype of float, which, in turn, is a supertype of int. > What this means is that all ints are valid as a float as well as a complex, but not the other way around. Similarly, all floats are also valid as a complex. Oh, no no no. Python integers are arbitrary-precision integers. Floats are IEEE 754 double-precision binary floating-point values, and as such only support full integer precision up to 2^53. The int type can represent values beyond that range which the float type cannot. And while it is true that the complex type is just two floats stuck together, I would very much not call it a supertype. It performs distinct operations. > Accessing an attribute with obj.x calls the __getattr__ method underneath. Similarly setting a new attribute and deleting an attribute calls __setattr__ and __detattr__ respectively. Attribute lookup in Python is way more complex than this. It's an enormous tar pit, too much so to detail in this comment, but __getattr__ is most often not involved, and the `object` type doesn't even have a __getattr__ method.

tusharsadhwani 5y ago
Oh, I didn't know that about locals! Yeah, calling float and complex "supertypes" probably wasn't the best idea, but I couldn't think of a better explanation that wouldn't take too long to explain. I'll ponder about that one. the getattr thing seems like a huge rabbit hole, I'm totally going to look into this. Thank you :)
5y ago

howolduis 5y ago

ever heard of line separator?

tored 5y ago

Thanks, got a better understanding of the Python "philosophy" because of this article, easy to follow even if you haven’t written a single line of Python like me.

tusharsadhwani 5y ago
That's great to hear :D hopefully you'll try out Python sometime.

thrdbndndn 5y ago

>As a bonus, this also adds support for adding two MyNumber classes together: Merely having `__add__` (without `__radd__`) is enough to add two MyNumber classes together in your case. >It mostly exists to support type annotations, The link for "type annotations" is broken.

tusharsadhwani 5y ago
Type annotations link is fixed, thank you! Just `__add__` doesn't work for me. I get: TypeError: unsupported operand type(s) for +: 'int' and 'Number' Here's the code: class Number: def __add__(self, x): return 42 + x num = Number() print(num + num) Edit: Okay. replacing `42 + x` with `x + 42` actually makes it work. But I'll be honest I have no idea what happened there.
- thrdbndndn 5y ago
  It took me quite a long time to understand what happened there too! (Pseudo code, let's call them num1 and num2 for better readability) num1 + num2 = num1.__add__(num2) = num2 + 42 (that's why the order is important) = num2.__add__(42) = 42 + 42 = 84
- thrdbndndn 5y ago
  >Type annotations link is fixed, thank you! I'm not sure about that - it's still `<a href="mypy-guide">type annotations</a>` which jumps to https://sadh.life/post/builtins/mypy-guide https://sadh.life/post/builtins/mypy-guide and then jumps to your homepage.

tyilo 5y ago

> Python has exactly 6 primitive data types (well, actually just 5, but we’ll get to that). 4 of these are numerical in nature, and the other 2 are text-based. Let’s talk about the text-based first, because that’s going to be much simpler. What is your definition of a primitive data types? All of these have object as a superclass, so I wouldn't call them primitive data types in python. Maybe there is just 1 primitive type: type? Or none at all?

tusharsadhwani 5y ago
In my case I meant it as things that only extend from object. Kinda like how prime numbers only have two factors including themselves.
- tyilo 5y ago
  What about list, dict, set, tuple, range and map then?

ripe 5y ago

Thank you for writing the post. Newbie question about “nonlocal” from your example: def outer_function(): x = 11 def inner_function(): nonlocal x x = 22 print('Inner x:', x) inner_funcion() print('Outer x:', x) I get how the example works, but don’t see the point of the declaration? If I just left out the “nonlocal x” line, wouldn’t the example still work the same?

ekimekim 5y ago
Python assumes that all assignments assign to the current scope. So by default when you reach "x = 22", it would create a new variable called "x" in the inner_function() scope which overrides the variable "x" in the outer_function() scope. So when you print "Inner x" you would only be printing the inner_function() version of x, not the outer_function() version, which would remain at 11.
- faho 5y ago
  This is a consequence of python not having explicit variable definition. Here it'll decide to define a new x instead of seeing the old one. And that's also usually what you want because otherwise a function would start altering variables in the enclosing scope if they happen to exist! E.g. foo = 42 def myfunc(bar): foo = bar + 1 print(foo) myfunc(6) print(foo) # would print "7" if the "foo =" above took the nonlocal foo automatically! So the trade-off is to require "nonlocal" if you ever need a variable from the enclosing scope.

faho 5y ago

The list comparison here is also true when the first list is a prefix of the other: class list: def __eq__(self, other): return all(x == y for x, y in zip(self, other)) # Can also be written as: return all(self[i] == other[i] for i in range(len(self))) run that with `[1,2,3]` and `[1,2,3,4]` and it'll be true because it only checks up to the 3. It's probably simplest to compare `len(self) == len(other)` before. Similarly, the set comparison will also be true if the first is a subset of the other.

tusharsadhwani 5y ago
Yup, I'll fix this, thanks for pointing it out.

tallguytyo 5y ago

Your article is really easy to get through and digest. I'd like to keep it as a reference going forward. Please consider adding a floating TOC to the page.