3 ms·
Excellent post! Python Bytecode is a little more naive / high level than this though, so stuff like exceptions and their nested handlers are actually implemente
by formerly_proven 5y ago
Excellent post! Python Bytecode is a little more naive / high level than this though, so stuff like exceptions and their nested handlers are actually implemented in the VM itself. The VM has essentially a second stack for exception and context manager blocks. The compiled bytecode essentially looks like this:
SETUP_TRY 10 # address where exceptions will be handled
some stuff that might explode
POP_BLOCK
JUMP 20 # jump over exception handlers
LOAD exception type # address 10
COMPARE # check if exception type matches
... handler for the type
stuff after the try-except # address 20
The corresponding source would look like
try:
some stuff that might explode
except exception type:
... handler for the type
stuff after the try-except
Python's bytecode compiler is generally a 1:1 translation of the AST; it never optimizes, e.g:
[value for value in list]
Translates to something like
LIST_NEW
FOR_EACH
STORE
LOAD
LIST_APPEND
JUMP BACK
Note how "value" generated a store-load pair.
The VM checks whether PyErr (pointer to exception) is set after basically everything. Similarly, extension modules check PyErr after every call into the interpreter, e.g.
PyObject *attr = PyObject_GetAttr(someattr, somepystr);
if(PyErr_Occurred()) { // or !attr
// handle exception
}
This gets old pretty fast.
- mixedmath 5y agoI'm growing more interested in actually understanding some of the internals that you mention here. I know this is a bit of a tangent, but is there a better way to approach understanding python's internals than reading the source (which feels a bit monolithic to me right now)?
- formerly_proven 5y agoThe central dispatch of the VM is a good place to start: https://github.com/python/cpython/blob/main/Python/ceval.c#L1678 https://github.com/python/cpython/blob/main/Python/ceval.c#L...
- boothby 5y agoI thought I had a pretty good handle on Python internals, until some time early this year when I took an interest in the generated bytecode. I'd read plenty of the cpython source, written lots of cython extensions, etc., but somehow missed the middle piece. Fortunately, it's really easy to get at the bytecode, and quite instructive. Random inquiry: how do generator functions work? In [1]: def foo(): ...: yield from range(10) ...: In [2]: import dis In [3]: dis.dis(foo) 2 0 LOAD_GLOBAL 0 (range) 2 LOAD_CONST 1 (10) 4 CALL_FUNCTION 1 6 GET_YIELD_FROM_ITER 8 LOAD_CONST 0 (None) 10 YIELD_FROM 12 POP_TOP 14 LOAD_CONST 16 RETURN_VALUE From there, you can read how each of those bytecode instructions is implemented in ceval.c, which formerly_proven links to. edit: probably nice to have the actual disassembly of a list comprehension, too: In [4]: def bar(): ...: return [x for x in range(10)] ...: In [5]: dis.dis(bar) 2 0 LOAD_CONST 1 (<code object <listcomp> at 0x7f814059f450, file "<ipython-input-4-a7a3cc7e7d7f>", line 2>) 2 LOAD_CONST 2 ('bar.<locals>.<listcomp>') 4 MAKE_FUNCTION 0 6 LOAD_GLOBAL 0 (range) 8 LOAD_CONST 3 (10) 10 CALL_FUNCTION 1 12 GET_ITER 14 CALL_FUNCTION 1 16 RETURN_VALUE Disassembly of <code object <listcomp> at 0x7f814059f450, file "<ipython-input-4-a7a3cc7e7d7f>", line 2>: 2 0 BUILD_LIST 0 2 LOAD_FAST 0 (.0) >> 4 FOR_ITER 8 (to 14) 6 STORE_FAST 1 (x) 8 LOAD_FAST 1 (x) 10 LIST_APPEND 2 12 JUMP_ABSOLUTE 4 >> 14 RETURN_VALUE
- Chris2048 5y ago> Python's bytecode compiler is generally a 1:1 translation of the AST; it never optimizes I thought python had a "peephole optimizer" that optimises?