6 ms·
One obvious improvement here is the use of `HPyContext`. This allows the Python interpreter state to be encapsulated rather than globally-accessible, which woul
by pansa2 6y ago
One obvious improvement here is the use of `HPyContext`. This allows the Python interpreter state to be encapsulated rather than globally-accessible, which would enable multiple interpreters in a single process.
Lua’s C API is generally considered to be well-designed, and does something similar with `lua_State`. How does the HPy API compare to the Lua API more generally?
- leecb 6y ago> enable multiple interpreters in a single process. CPython (the default interpreter) has had support for multiple interpreters in the same process since 1997, but it has only been exposed to the C API, and not in the language itself. Python 3.10, coming out later this year, will expose multiple interpreters in one process (subinterpreters, see PEP 554 [1]). I'm excited about what could eventually come out of this. If there is one GIL per interpreter, we could have something like the `multiprocessing` library for parallel execution, but all within one process. 1 https://www.python.org/dev/peps/pep-0554/ https://www.python.org/dev/peps/pep-0554/
- rkeene2 6y agoThis is basically how the Threads extension [0] works for Tcl. Tcl has long supported creating many interpreters per thread. The Thread extension exposes some threading commands to the interpreter, which lets you create a new thread+interp. You can then send messages to that interpreter, or pass it file descriptors, but otherwise they are fairly isolated (apartment threading model). [0] https://www.tcl-lang.org/man/tcl8.6/ThreadCmd/thread.htm https://www.tcl-lang.org/man/tcl8.6/ThreadCmd/thread.htm
- pansa2 6y agoCan multiple Tcl interpreters run truly parallel on different threads in the same process? That’s possible with Lua but AFAIK not with Python, because one of the pieces of global Python interpreter state is the Global Interpreter Lock. Encapsulated interpreter state via HPyContext would allow replacing the per-process lock with a per-interpreter lock, enabling such parallelism.
- leecb 6y agoRelated but separate from the subinterpreters PEP, there is interest in moving from a global interpreter lock to one lock per subinterpreter. This would need to be in place before running Python interpreters truly in parallel within one process. There’s a great summary here: https://lwn.net/Articles/820424/ https://lwn.net/Articles/820424/
- rkeene2 6y agoYes, threads from the Thread extension are real OS threads, which have a Tcl interp in them. Since they are separate interpreters then there's no impact from any kind of "per-interpreter lock". Passing messages between Threads can be done asynchronously which does not block either thread. Example Tcl program with threads package require Thread set tid [thread::create] thread::send -async $tid { while true { after 200; puts Jazz } } while true { after 100; puts Party } This creates one thread which is in an infinite loop printing "Jazz" to stdout every 200ms, and the "main" thread which is in an infinite loop printing "Party" to stdout every 100ms.
- arigo 6y agoNo. See A Disclaimer about the GIL right after the Abstract.
- int_19h 6y agoThe big difference between Lua and Python C APIs is that Python values are exposed directly as first-class values (as pointers in CPython, and as handles in HPy). If one native function wants to call another native function and pass it a Python object, it can do so directly - the VM is none the wiser for it, except for reference count updates. In Lua API, all Lua values are bound to a stack, and C functions have to exchange values via that stack - there's no free-standing "Lua value" type in the API. Thus, the runtime is fully aware of all the objects that flow back and forth, even if one native Lua function is calling another native Lua function.
- shakna 6y ago> In Lua API, all Lua values are bound to a stack, and C functions have to exchange values via that stack - there's no free-standing "Lua value" type in the API. That's not 100% correct. You can have values unbound from the stack, they're just typed, rather than having one "value" type. (lua_Number, lua_CFunction, etc). Though functions bound from do still use the stack for taking/returning values. (However your C functions can directly work on other value types).
- int_19h 6y agoCorrect me if I'm wrong, but doesn't this only work for value types? i.e. there's no way you can hold a reference to a Lua (not C) function, or a table, without the stack.
- shakna 6y agoWhat you're looking for in that case is lua_topointer, which can turn any Lua userdata, table, thread, or function into a void pointer.
- int_19h 6y agoNot really. Looking at the doc for that function: "There is no way to convert the pointer back to its original value. Typically this function is used only for debug information." (In CPython, you can e.g. stash a PyObject* away in C globals.)