4 ms·

The article says that embedding (calling Python from C/C++) is currently buggy. It is not clear whether the Python interpreter was compiled with --disable-gil.

by wrlanv 2y ago

The article says that embedding (calling Python from C/C++) is currently buggy. It is not clear whether the Python interpreter was compiled with --disable-gil. I did not check the claim, since, as the article rightly points out, there are general issues in the Python space and other languages like Lua are far better for embedding.

elnatro 2y ago
Thanks, I’m not an embedded developer and was losing the train of thought of the author.
- deleted 2y ago
  [deleted]
- cclac109 2y ago
  These threading issues may also be present when not using embedding. Embedding the interpreter frequently shows general threading bugs much quicker. In other words, when "torturing" the interpreter with embedding a threading bug might show up after 30 min. The same threading bug might show up once every 6 months in a web server (which is a nasty security risk, since it is hard to detect and to reproduce).
  - elnatro 2y ago
    Ouch. So we (app developers) are doomed with this feature?
    - SAHChandler 2y ago
      I did open an issue on GitHub with the CPython implementation and it's being looked into. The crash happens for sure in the debug version of Python. However we're seeing valgrind and address sanitizer fire off for the upcoming 3.13 release in release mode when using the C API as I showed in my example. However, this behavior only appears to occur when embedding python directly, not via extending it, into an application. Because of how the (hidden because it is experimental) `_xxsubinterpreters` module works internally, it's extra internal bookkeeping seems to give enough time for the race condition to disappear.
SAHChandler 2y ago
It might not have been obvious, but I used the currently available Python 3.12.5 release. However upon filing a ticket, and being asked for more testing, we found this issue persists into 3.13 and 3.14 alpha with and without the GIL disabled :(
- elnatro 2y ago
  Thanks for your work and for bringing up this issue to them!

by wrlanv 2y ago

elnatro 2y ago

Thanks, I’m not an embedded developer and was losing the train of thought of the author.

deleted 2y ago
[deleted]
cclac109 2y ago
These threading issues may also be present when not using embedding. Embedding the interpreter frequently shows general threading bugs much quicker. In other words, when "torturing" the interpreter with embedding a threading bug might show up after 30 min. The same threading bug might show up once every 6 months in a web server (which is a nasty security risk, since it is hard to detect and to reproduce).
- elnatro 2y ago
  Ouch. So we (app developers) are doomed with this feature?
  - SAHChandler 2y ago
    I did open an issue on GitHub with the CPython implementation and it's being looked into. The crash happens for sure in the debug version of Python. However we're seeing valgrind and address sanitizer fire off for the upcoming 3.13 release in release mode when using the C API as I showed in my example. However, this behavior only appears to occur when embedding python directly, not via extending it, into an application. Because of how the (hidden because it is experimental) `_xxsubinterpreters` module works internally, it's extra internal bookkeeping seems to give enough time for the race condition to disappear.

SAHChandler 2y ago

It might not have been obvious, but I used the currently available Python 3.12.5 release. However upon filing a ticket, and being asked for more testing, we found this issue persists into 3.13 and 3.14 alpha with and without the GIL disabled :(

elnatro 2y ago
Thanks for your work and for bringing up this issue to them!