7 ms·
CPython Internals Explained
- westurner 8mo agovstinner's Python docs; "Unofficial Python Development (Victor's notes) documentation" > Garbage Collector > "Implement the GC protocol in a type": https://pythondev.readthedocs.io/garbage_collector.html#implement-the-gc-protocol-in-a-type https://pythondev.readthedocs.io/garbage_collector.html#impl... Python Developer's Guide > "CPython's internals": https://devguide.python.org/internals/index.html https://devguide.python.org/internals/index.html Python/cpython//InternalDocs/README.md > "CPython Internals Documentation": https://github.com/python/cpython/blob/main/InternalDocs/README.md https://github.com/python/cpython/blob/main/InternalDocs/REA...
- westurner 8mo agoIDK why /InternalDocs/ instead of /Doc/internals/ ? ( `ln -s` works with Mac/Lin/WSL. ) Ideally what's in InternalDocs/ would be built into the docs.python.org docs . Is it just that markdown support in sphinx is not understood to exist? Sphinx has native markdown support. Sphinx does not have native MyST Markdown support. To support MyST Markdown in a sphinx-doc project, you must e.g. `pip install myst_parser` and add "myst_parser" to the extensions list in conf.py. MyST Markdown supports docutils and sphinx RestructuredText roles and directives: https://myst-parser.readthedocs.io/en/latest/syntax/roles-and-directives.html https://myst-parser.readthedocs.io/en/latest/syntax/roles-an... Directive in ReStructuredText .rst: .. directivename:: arguments :key1: val1 :key2: val2 This is directive content Directive in MyST Markdown .md: ```{directivename} arguments :key1: val1 :key2: val2 This is directive content ``` RestructuredText Role, MyST Markdown Role: :role-name:`role content` {role-name}`role content` Sphinx resolves reference labels at docs build time, so that references will be replaced with the full relative URL to the path#fragment of the document where they occur; in ReStructuredText and then MyST Markdown: .. _label-name: (label-name)= :ref:`Link title <label-name>` {ref}`Link title <label-name>`
- shakna 8mo ago> Ideally what's in InternalDocs/ would be built into the docs.python.org docs . If you expose internals in documentation, then people depend on internals. And when you break it, because it isn't meant to be tracked by any kind of API, there are wonderful groups who will sue you (usually under "devaluation").
- westurner 8mo agoThat's why three different procedures for docs? Python docs procedures: (0) Devguide, (1) PEPs w/ front matter in RST, (2) RST in /Doc with Sphinx, (3) MD and TXT in /InternalDocs without a toctree The .. warning: or even admonition directives could be used for indicating that docs under /internals are not public API and can change with or without a PEP; though that should also or at least be indicated in the source unless that's a given expectation that not marked public APIs are not to be considered stable
- shakna 8mo agoHow many, many times has a project said, "Don't use, internal only", only for it to become an industry-wide common "trick"? Saying "here be dragons" is not enough to discourage people whose job it is to be creative.
- westurner 8mo agoThat's the bad kind of lazy. It is advantageous to format, interlink, and create a table of contents for documentation. I doubt anyone would advocate for removing the Makefile and conf.py from /Doc. That a document describes something internal does not mean that it should be excluded from the docs. Is there a consistent standard for whether something is internal or external to the CPython project? Aren't there already internal things documented in /Doc? Should they be removed from the docs build then? Or moved to an internal folder with a DRY additional toctree?
- squeedles 8mo agoHad to write a fairly substantial native extension to Python a couple years ago and one of the things I enjoyed was that the details were not easily "Googleable" because implementation results were swamped by language level results. It took me back to the old days of source diving and accumulated knowledge that you carried around in your head. https://www.dave.org/posts/20220806_python/ https://www.dave.org/posts/20220806_python/
- davepeck 8mo agoI made some small contributions to cpython during the 3.14 cycle. The codebase is an interesting mix of modern and “90s style” C code. I found that agentic coding tools were quite good at answering my architectural questions; even when their answers were only half correct, they usually pointed me in the right direction. (I didn’t use AI to write code and I wonder if agentic tools would struggle with certain aspects of the codebase like, for instance, the Cambrian explosion of utility macros used throughout.)
- squeedles 8mo agoThis was around 2021 so AI code tools had not yet eaten everyone. One of the most interesting challenges was finding the right value judgements when blending multiple type systems. I doubt any agentic coding tool could do it today. I blended the python type system with a large low-level type system (STEP AIM low level types) and a smaller set of higher-level types (STEP ARM, similar to a database view). I already was familiar with STEP, so I needed to really grok what Python was doing under the covers because I needed to virtualize the STEP ARM and AIM access while making it look like "normal" Python.
- davepeck 8mo agoOh, that's very interesting work. And, yes, I'd also be surprised if (today's) agentic tools were at all helpful for that: it's way outside of distribution, and conceptual correctness truly matters.
- EdwardDiego 8mo ago
- elcapitan 8mo agoThis looks quite nice. I always wished there was something like "Ruby Under a Microscope" for Python (and other languages). It was quite instrumental for my deeper understanding of the language.
- stonecharioteer 8mo agoThere is. https://realpython.com/products/cpython-internals-book/ https://realpython.com/products/cpython-internals-book/ But it's for 3.9 and doesn't cover the massive changes regarding delayed annotations and the GIL updates The Ruby under a Microscope guy is updating it.
- elcapitan 8mo agoThat's nice too, but it seems to be more a tour of the code base, and doesn't have the detailled diagrams of memory layout that the Ruby book and the one posted here have.
- mvATM99 8mo agoVery interesting! Gonna look through this for sure in the next weeks
- deleted 8mo ago[deleted]
- damjon 8mo agoI've been comparing various platforms and discussing them with ChatGPT—for instance, why Python's execution is slower than JavaScript's V8. It claimed this is due to mtechnical debt and the inability to change because libraries like NumPy bypass public interfaces and access data directly. I'm wondering how much of that is true and what is just a hallucination." Btw: JavaScript seems to have similar complexity issues. Edit: Python has no JIT
- johndough 8mo ago> Edit: Python has no JIT There are quite a few JITs: JIT-compiler for Python https://pypy.org/ https://pypy.org/ Python enhancement proposal for JIT in CPython https://peps.python.org/pep-0744/ https://peps.python.org/pep-0744/ And there are several JIT-compilers for various subsets of Python, usually with focus on numerical code and often with GPU support, for example Numba https://numba.pydata.org/numba-doc/dev/user/jit.html https://numba.pydata.org/numba-doc/dev/user/jit.html Taichi Lang https://github.com/taichi-dev/taichi https://github.com/taichi-dev/taichi
- davepeck 8mo agoPer PEP 744, cpython shipped with an experimental JIT (default disabled) in 3.13. It remains experimental in 3.14. See https://docs.python.org/3/whatsnew/3.13.html#an-experimental-just-in-time-jit-compiler https://docs.python.org/3/whatsnew/3.13.html#an-experimental...
- gf000 8mo agoIf we are being very pedantic, languages don't have "speed", only implementations do. Of course in the real life there are de facto implementations and language features give way to better/worse tradeoffs. With that out of the way, Python is basically the de facto glue language. It is very often used to provide a scripting API over lower level C libraries. To be ergonomic in this function, CPython (the major implementation) exposed some internal details of its execution model, which C libraries can reach into. This makes it very hard to make more aggressive optimizations, as one example a C library can just increase/decrease the reference count of an object. Another design decision (that got some discussion recently) is the GIL (global interpreter lock) that makes python much less competitive than something like Java. (JS also does a single thread of execution, though there are ways around it). JS has a different use case, so access to the C world doesn't impose such restrictions on it.
- tonymet 8mo agoI wish they would just go back to calling it Python, since it’s the Python that everyone knows and uses. No one gets confused over Python the spec and Python the implementation. Every time I see “CPython” i have to double check we’re just talking about Python. I guess they “CPython’ed” back when people thought Jython would take off , and it never did because Java sucks.
- mkoubaa 8mo agoPrecision in language is important for software engineering.
- paulddraper 8mo agoA lot to unpack there, but the language and the implementation are different. JavaScript and Node.js are different too.
- palata 8mo agoI feel like when the goal is to talk about the internals of it, then it makes sense to call it CPython. In general, I never, ever see anyone saying "I will write a CPython script". Everybody says "Python" in my experience... do you see it differently? EDIT: I don't think that your opinion deserves to be downvoted, though...
- EdwardDiego 8mo agoI find it's usually referred to as CPython when discussing something specific to the implementation or internals of Python that don't apply to Pypy, which seems to be the alternative Python implementation with the most traction. No harm in being explicit right? Tis part of the zen of Python after all.
- vkazanov 8mo agoJust to name alternatives: Cpython, Pypy, jython, ironpython. Then, there quite a few python-likes out there. I wish they would stay precise.
- foresto 8mo ago