3 ms·

> It has unsafe escape hatches (via ffi, at the very least). Yep, ctypes is part of the stdlib and lets you corrupt the VM on the fly. Fun stuff like changing

> It has unsafe escape hatches (via ffi, at the very least). Yep, ctypes is part of the stdlib and lets you corrupt the VM on the fly. Fun stuff like changing the value of cached integers and everything. But ctypes being a terrifying pain in the ass, people tread very carefully around it. Cffi’s a lot better though it requires an external package. At the end of the day I think I’d be more enclined to bind through pyo3 or cython than write C in python (which is what ctypes has you do without even what little type system C has, to say nothing of -Wall -Weverything).

rraval 4y ago
> But ctypes being a terrifying pain in the ass, people tread very carefully around it. I'm not sure how much people treading carefully actually translates into safety in practice. CPython in particular has ad-hoc refcounting semantics where references can either be borrowed or stolen and you have to carefully verify both the documentation and implementation of functions you call because it's the wild west and nothing can be trusted: https://docs.python.org/3.9/c-api/intro.html#reference-count-details https://docs.python.org/3.9/c-api/intro.html#reference-count... This ad-hoc borrowed vs stolen references convention bleeds into cffi as well. If you annotate an FFI function as returning `py_object`, cffi assumes that the reference is stolen and thus won't increment the ref count. However, if that same function instead returns a `struct` containing a `py_object`, cffi assumes the reference is borrowed and will increment the ref count instead. So a harmless looking refactoring that changes a directly returned `py_object` into a composite `struct` containing a `py_object` is now a memory leak. Memory leaks aren't so bad (even Rust treats them as safe after the leakpocalypse [1] [2]). It's when you go the other way and treat what should have been a borrowed reference as stolen that real bad things happen. Here's a quick demo that deallocates the `None` singleton: Python 3.9.13 (main, May 17 2022, 14:19:07) [GCC 11.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.getrefcount(None) 4584 >>> import ctypes >>> ctypes.pythonapi.Py_DecRef.argtypes = [ctypes.py_object] >>> for i in range(5000): ... ctypes.pythonapi.Py_DecRef(None) ... 0 0 0 0 0 [snip] Fatal Python error: none_dealloc: deallocating None Python runtime state: initialized Current thread 0x00007f28b22b7740 (most recent call first): File "<stdin>", line 2 in <module> fish: Job 1, 'python3' terminated by signal SIGABRT (Abort) [1]: https://rust-lang.github.io/rfcs/1066-safe-mem-forget.html https://rust-lang.github.io/rfcs/1066-safe-mem-forget.html [2] https://cglab.ca/~abeinges/blah/everyone-poops/ https://cglab.ca/~abeinges/blah/everyone-poops/
- masklinn 4y ago
  > Here's a quick demo that deallocates the `None` singleton: As I said, you can trivially corrupt the VM through ctypes. However I don't think I've ever seen anyone wilfully interact with the VM for reasons other than shit and giggles. The few uses of ctypes I've seen were actual FFI (interacting with native libraries), and IME it's rare enough and alien enough that people tread quite carefully around that. I've actually seen a lot less care with the native library on the other side of the FFI call than on the FFI call itself (I've had to point issues with that just this morning during a code review, if anything the ctypes call was over-protected, otoh the update to the so's source had multiple major issues).

rraval 4y ago

> But ctypes being a terrifying pain in the ass, people tread very carefully around it. I'm not sure how much people treading carefully actually translates into safety in practice. CPython in particular has ad-hoc refcounting semantics where references can either be borrowed or stolen and you have to carefully verify both the documentation and implementation of functions you call because it's the wild west and nothing can be trusted: https://docs.python.org/3.9/c-api/intro.html#reference-count-details https://docs.python.org/3.9/c-api/intro.html#reference-count... This ad-hoc borrowed vs stolen references convention bleeds into cffi as well. If you annotate an FFI function as returning `py_object`, cffi assumes that the reference is stolen and thus won't increment the ref count. However, if that same function instead returns a `struct` containing a `py_object`, cffi assumes the reference is borrowed and will increment the ref count instead. So a harmless looking refactoring that changes a directly returned `py_object` into a composite `struct` containing a `py_object` is now a memory leak. Memory leaks aren't so bad (even Rust treats them as safe after the leakpocalypse [1] [2]). It's when you go the other way and treat what should have been a borrowed reference as stolen that real bad things happen. Here's a quick demo that deallocates the `None` singleton: Python 3.9.13 (main, May 17 2022, 14:19:07) [GCC 11.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.getrefcount(None) 4584 >>> import ctypes >>> ctypes.pythonapi.Py_DecRef.argtypes = [ctypes.py_object] >>> for i in range(5000): ... ctypes.pythonapi.Py_DecRef(None) ... 0 0 0 0 0 [snip] Fatal Python error: none_dealloc: deallocating None Python runtime state: initialized Current thread 0x00007f28b22b7740 (most recent call first): File "<stdin>", line 2 in <module> fish: Job 1, 'python3' terminated by signal SIGABRT (Abort) [1]: https://rust-lang.github.io/rfcs/1066-safe-mem-forget.html https://rust-lang.github.io/rfcs/1066-safe-mem-forget.html [2] https://cglab.ca/~abeinges/blah/everyone-poops/ https://cglab.ca/~abeinges/blah/everyone-poops/

masklinn 4y ago
> Here's a quick demo that deallocates the `None` singleton: As I said, you can trivially corrupt the VM through ctypes. However I don't think I've ever seen anyone wilfully interact with the VM for reasons other than shit and giggles. The few uses of ctypes I've seen were actual FFI (interacting with native libraries), and IME it's rare enough and alien enough that people tread quite carefully around that. I've actually seen a lot less care with the native library on the other side of the FFI call than on the FFI call itself (I've had to point issues with that just this morning during a code review, if anything the ctypes call was over-protected, otoh the update to the so's source had multiple major issues).