6 ms·

I think the support for isolated sub-interpreters with separate Global Interpreter Locks is the most interesting new feature in python. It is doubtful not the b

I think the support for isolated sub-interpreters with separate Global Interpreter Locks is the most interesting new feature in python. It is doubtful not the best way to offer some sort of concurrency but still a step closer to maybe one day get rid of GILs.

yxhuvud 3y ago
It may not be a step towards that. Ruby has guilds which is a very similar idea and they are explicitly not working towards removing them altogether at this point. Matz did a full keynote on the latest Euruku defending not working towards removing the whole of it. See https://www.youtube.com/watch?v=5WmhTMcnO7U&t=1244s https://www.youtube.com/watch?v=5WmhTMcnO7U&t=1244s for the full talk if you are interested. I find the convergent evolution of features in these two languages pretty funny, as it is very clear that they don't really look at implementation details of the other language even if they quite often land on ideas that are pretty close in practice.
zozbot234 3y ago
Well, that's where the whole "for Workgroups" part comes in I suppose. See my sibling comment.
qsort 3y ago
Too bad there's no Python API available yet :( It's scheduled to be delivered next year if I'm not mistaken. The new syntax for generic types is also a very nice QOL improvement, you can now just do: class MyList[T]: ...
- remram 3y ago
  I think you are talking about the "stable Python ABI"?
- rmbyrro 3y ago
  Yes, they say it's scheduled for 3.13. But it seems not confirmed yet (probably still not ready for prime time).
ptx 3y ago
Since it currently lacks any way to transfer objects between interpreters other than pickling, does it offer any advantage over the multiprocessing module?
- ynik 3y ago
  Not for pure python code; but there's massive advantages for mixed C(++) and Python: I can now have multiple sub interpreters running concurrently and accessing the same shared state in a thread-safe C++ library. Previously this required rewriting the whole C++ library to support either pickling (multiplying the total memory consumption by the number of cores), or support allocating everything in shared memory (which means normal C++ types like `std::string` are unusable, need to switch e.g. to boost::interprocess). Now is sufficient to pickle a pointer to a C++ object as an integer, and it'll still be a valid pointer in the other subinterpreter.
  - ledauphin 3y ago
    this is the _first_ place I've seen anyone clearly articulate an actual use case for subinterpreters. Thank you!
  - veber-alex 3y ago
    But why? There is no GIL in C/C++/Rust/Zig/Whatever, just use threads.
    - nutate 3y ago
      Let's say you wanted to run some python code that was written and test it against some C/C++/Rust for accuracy of some sort (numerical, lexicographical, etc). In the old way you would have to fire up multiple processes to do that (like OS level processes) but now you can have your multithreaded compiled code running in threads and your multi-GIL'd interpreted code running all in one process and comparing their results in the `main` of your C/C++/Rust. That's a contrived example, but the issue was that a single GIL isn't threadsafe in and of itself. So if you're using these compiled languages as sort of python runners you couldn't multithread python interpreter execution and guarantee the code working. Also as the above comment stated, you could do hacks, but you'd double your memory allocation by needing a python and C/C++/Rust representation for everything that went back and forth.

yxhuvud 3y ago

It may not be a step towards that. Ruby has guilds which is a very similar idea and they are explicitly not working towards removing them altogether at this point. Matz did a full keynote on the latest Euruku defending not working towards removing the whole of it. See https://www.youtube.com/watch?v=5WmhTMcnO7U&t=1244s https://www.youtube.com/watch?v=5WmhTMcnO7U&t=1244s for the full talk if you are interested. I find the convergent evolution of features in these two languages pretty funny, as it is very clear that they don't really look at implementation details of the other language even if they quite often land on ideas that are pretty close in practice.

zozbot234 3y ago

Well, that's where the whole "for Workgroups" part comes in I suppose. See my sibling comment.

qsort 3y ago

Too bad there's no Python API available yet :( It's scheduled to be delivered next year if I'm not mistaken. The new syntax for generic types is also a very nice QOL improvement, you can now just do: class MyList[T]: ...

remram 3y ago
I think you are talking about the "stable Python ABI"?
rmbyrro 3y ago
Yes, they say it's scheduled for 3.13. But it seems not confirmed yet (probably still not ready for prime time).

ptx 3y ago

Since it currently lacks any way to transfer objects between interpreters other than pickling, does it offer any advantage over the multiprocessing module?

ynik 3y ago
Not for pure python code; but there's massive advantages for mixed C(++) and Python: I can now have multiple sub interpreters running concurrently and accessing the same shared state in a thread-safe C++ library. Previously this required rewriting the whole C++ library to support either pickling (multiplying the total memory consumption by the number of cores), or support allocating everything in shared memory (which means normal C++ types like `std::string` are unusable, need to switch e.g. to boost::interprocess). Now is sufficient to pickle a pointer to a C++ object as an integer, and it'll still be a valid pointer in the other subinterpreter.
- ledauphin 3y ago
  this is the _first_ place I've seen anyone clearly articulate an actual use case for subinterpreters. Thank you!
- veber-alex 3y ago
  But why? There is no GIL in C/C++/Rust/Zig/Whatever, just use threads.
  - nutate 3y ago
    Let's say you wanted to run some python code that was written and test it against some C/C++/Rust for accuracy of some sort (numerical, lexicographical, etc). In the old way you would have to fire up multiple processes to do that (like OS level processes) but now you can have your multithreaded compiled code running in threads and your multi-GIL'd interpreted code running all in one process and comparing their results in the `main` of your C/C++/Rust. That's a contrived example, but the issue was that a single GIL isn't threadsafe in and of itself. So if you're using these compiled languages as sort of python runners you couldn't multithread python interpreter execution and guarantee the code working. Also as the above comment stated, you could do hacks, but you'd double your memory allocation by needing a python and C/C++/Rust representation for everything that went back and forth.