10 ms·
How the Python object system works
- r4victor 6y agoHi! This is part 6 of my Python behind the scenes series. Each part of the series covers how some aspect of the language is implemented in the interpreter, CPython. This time we'll focus on the Python object system. You'll learn: - What Python objects and types are and how they are implemented. - What slots are and how they determine the behavior of objects. - How slots are related to special methods. I'd be glad to get your feedback and answer your questions. Thanks!
- ryan-duve 6y agoI have a C background from graduate school and a Python background from my professional work, but I've never explored how Python is implemented in C until I came across one of your previous parts of this series. I don't have any questions. I just wanted to let you know I think you do a fantastic job writing these up. The thinking process is very clear, and the resource links are well placed for followup research. High quality stuff, for sure. Thanks for making these. I look forward to future installations.
- r4victor 6y agoThanks, Ryan! The series is going on. Stay tuned!
- azhenley 6y agoAwesome, thanks for this! In the spring I’ll be teaching a graduate-level university course on how the Python compiler and tooling works, so I’ll be showing them your articles.
- stonecharioteer 6y agoYou should give a go at reading or recommending Anthony Shaw's excellent book[1] (or at the very least the free article on Real Python[2]). 1. https://realpython.com/products/cpython-internals-book/ https://realpython.com/products/cpython-internals-book/ 2. https://realpython.com/cpython-source-code-guide/ https://realpython.com/cpython-source-code-guide/
- lsferreira42 6y agoCan i translate this series in Portuguese? I think that would help a lot of coders that i know, but they can't read english.
- tcbawo 6y agoOff topic, but do the programmers you know that can't read English struggle using programming languages and libraries where the keywords, class, variable, and function names are primarily in English? I'm curious how beginning or ESL programmers manage. Are there ecospheres where they can manage using only their first languages?
- dontTango 6y agoVariables and nouns are easy to translate. Sentence meaning is significantly harder.
- 411111111111111 6y agoThere is a vast difference between people that don't fluidly read/speak English and people that can't be arsed to learn what the 20 English keywords mean a programming language might use.
- swirepe 6y ago20 keywords, but what about EscapeString, or SpringContextIntegrationTest? A program can use a lot of English words.
- 411111111111111 6y agoHow does a cord for fleeing tell you anything about what it actually is? Both of your examples bring very little meaning from their English definition. You just know a second definition in the context of programming, which is why you can deduce what it does. The same applies to the people that don't fluidly speak the language.
- brodouevencode 6y agoThis is a great series. Thank you for this.
- fantod 6y agoThis is great! There aren't enough references on Python internals, I find. Not sure you're at all interested in going in this direction, but I (and likely many people) would be really interested in a similar explanation of NumPy internals.
- callamdelaney 6y agoThere are some great videos on youtube, a little outdated but gold if you can find them. It's this guy who literally sits at his desk for an hour and a half straight and walks a class line by line through various parts of cpython2.7 - little outdated but was awesome. There are multiple parts.
- mattip 6y agoPlease open a documentation issue at https://github.com/numpy/numpy/issues https://github.com/numpy/numpy/issues describing exactly which internals you would like to see explained: indexing, ufuncs, linkage, random, dtypes, ... and what format you would like to see the documentation in: text, video, lessons with questions/answers, ... As for python internals, https://snarky.ca/ https://snarky.ca/ has some great short high level “how does this construct work” text explainations.
- edgyquant 6y agoMore people are looking for passive articles to consume than filing a bug report to request information on specific systems. This reply is unnecessary.
- miduil 6y agoThank you so much for this article, I remember seeing some talk on this topic many years ago but could never find a comprehensive summary so far (or even a video of the talk). I'd be really curious about this in other languages as well, such as Ruby and JavaScript.
- scruple 6y agoI'm actually tempted to start a series of blog posts in this vein for Ruby. The book Ruby Under a Microscope [0] covers these topics (and more), at least from a cursory glance of the blog series, but it would be nice to refresh that material in my own head for Ruby 3. [0]: http://patshaughnessy.net/ruby-under-a-microscope http://patshaughnessy.net/ruby-under-a-microscope
- choeger 6y agoVery interesting and quite ... instructive. The lack of optimizations is interesting. One would naively imagine that treating, e.g., ints, strings, dicts, and lists special would gain some performance.
- kevin_thibedeau 6y agoSmall integers do get special treatment as the same values share the same object.
- Pxtl 6y agoOkay, that's dodging the object instantiation/destruction thing but it's still doing all the reference counting on them isn't it?
- saagarjha 6y agoWhy would reference counting be necessary an immutable, immortal global?
- Pxtl 6y agoIt isn't, but IIRC refcounting is cooked into the python interpreter in so many places that everything that can be referenced must also be refcounted. If you get a reference to something, you increment its refcount. If you lose the reference, you decrement it. No ifs, no branches.
- masklinn 6y agoAFAIK the implementation of cpython is purposefully kept relatively simple such that it remains approachable.
- ketralnis 6y agoThere are fast-paths for these cases, like this one: https://github.com/python/cpython/blob/4b8cdfcb22fbeaab9d954cb693a7fb3362a382b6/Python/ceval.c#L1831-L1833 https://github.com/python/cpython/blob/4b8cdfcb22fbeaab9d954... (and note the comment above it about integer addition!) But the cpython folk make understandability an explicit goal of the codebase which tends to preclude things like fancy optimisations or JITs
- 83457 6y agoI did a quick search and didn't see it. Does this article cover why python has mutable default arguments?
- __mharrison__ 6y ago(Default) arguments are bound to the function when it is created not executed...
- blackrock 6y agoWhen do the objects get destroyed or garbage collected? This part was always magical. It seems to free up the local memory usage of the program, but it doesn’t seem to release the memory back to the operating system, until some type of triggered event.
- rng_civ 6y agoI quickly skimmed over OP's previous posts and I don't think they mention it. According to [0], CPython's GC is ran every X instructions (not sure how up to date the source is). From this, I guess CPython opted for simplicity instead of implementing something like a memory-usage monitor. [0] https://realpython.com/cpython-source-code-guide/#garbage-collection https://realpython.com/cpython-source-code-guide/#garbage-co...
- callamdelaney 6y agoI haven't read the article but as I understand python will hold on to acquired memory and pre-allocates 'blocks' of memory ready for items of various sizes. When you use more memory and it's freed iirc it holds on to most of these blocks.
- duskwuff 6y ago> This part was always magical. It seems to free up the local memory usage of the program, but it doesn’t seem to release the memory back to the operating system, until some type of triggered event. That's up to the C runtime's memory allocator. Modern memory allocators don't typically request a new chunk of memory for every malloc() call -- instead, they allocate a single large region of memory at a time and carve that up as needed. This is massively more efficient (system calls are expensive), but also means that those regions can't be released to the OS until all allocations in them are gone.
- remexre 6y agoThere's an interesting talk [0] from Bobby Powers on how to make a malloc() and free() that perform compaction, to be able to release memory back to the operating system more frequently (and improve cache hit rates, etc.). But this isn't standard at all yet. [1]: https://www.youtube.com/watch?v=c1UBJbfR-H0 https://www.youtube.com/watch?v=c1UBJbfR-H0
- deleted 6y ago[deleted]
- thrower 6y agoI want to start contributing to open source projects like these, but the process seems somewhat intimidating. Are there any good guides or maybe an introductory tutorial somewhere on how to get involved?
- pfalcon 6y agoStart your own open-source project, and make process of contributing to it not "intimidating".
- deleted 6y ago[deleted]
- pansa2 6y agoPython's descriptor protocol is very elegant. Given: class C: def __init__(self): self.a = 1 def b(self): print(f'b({self})') @property def c(self): print(f'c({self})') o = C() The descriptor protocol is a generic way to enable `o.a` to access an attribute of `o`, `o.b` to access an attribute of `C` and bind `self` to `o`, and `o.c` to access an attribute of `C`, bind `self` to `o`, and call it. Do any other dynamic languages use a similar protocol for attribute access? Ruby, for example, doesn't need to - it can use a simpler approach because `o.x` never refers to an attribute of `o`. JavaScript's solution is less elegant - the way it sets the value of `this` is infamous, and AFAIK properties are treated as a special case rather than built on top of a generic protocol.
- lifthrasiir 6y agoI don't think Python's approach is particularly more elegant than JavaScript, as they both distinguish property-as-value and property-as-function. It's just a matter of where to put property-as-function (and unlike Python, JavaScript can have non-configurable properties which can be optimized). As property-as-function can subsume property-as-value, no built-in property-as-value would be truly elegant (Ruby is close, but not exact).
- sillysaurusx 6y agoI agree. Python's descriptor pattern was an endless source of confusion for me. But, it's much easier to do mixins in Python than in JS, even if the mro algorithm is a bit complicated...
- pansa2 6y agoSorry, could you clarify what you mean by “distinguish property-as-value and property-as-function“? How does Python distinguish between these two?
- lifthrasiir 6y agoThe very descriptor protocol exists to distinguish property-as-function (descriptor) from property-as-value (slot).
- rurban 6y agoThis is exactly the reason why python is so slow. The everything is an object approach has no real benefit, as the code is still full of special cases. Faster VMs of course special-case int and float. Best at compile-time.
- pfalcon 6y agoSpecial-casing (only) ints and floats in VM is too trivial and a bloat for quite minor benefit. To start with, faster VMs are not VMs, but JITs. And they are able to specialize for arbitrary types, not just 2 random ones. I thought there were some "lessons learned" from ParrotVM... ;-).
- rurban 6y agoThis has nothing to do with parrot. Every self respecting VM treats value types differently than reference types, and esp. objects. Ruby, Perl, Javascript, every lisp, lua... That's why they are all so much faster than python. Python made the weird/ insane decision to overload + for strings, so treating numbers as classes goes from there. It just cannot be fast then. But even most Common Lisp's treat every value as class, with an atom being a struct of class and value. So it can be made fast if you do it properly. Lessons learned: Don't ever look at the python vm, if you want to look for a well designed VM. Even Ruby is miles better there.
- pfalcon 6y agoAs a shameless plug, here's a proposal on how to optimize some things in Python object system: https://github.com/pycopy/PycoEPs/blob/master/StrictMode.md https://github.com/pycopy/PycoEPs/blob/master/StrictMode.md . Implemented in my lightweight Python dialect, https://github.com/pfalcon/pycopy https://github.com/pfalcon/pycopy