3 ms·

In my experience, having python as the root node for multithreaded C/python programs generally doesn't present a problem. When calling into external C function

by hagy 16y ago

In my experience, having python as the root node for multithreaded C/python programs generally doesn't present a problem. When calling into external C functions through ctypes, the GIL is released such that multithreading at the C-level is unaffected. I personally use this feature heavily to perform numerically intensive work with multiple threads as directed through python with C doing the grinding. Additionally, all IO code in the standard library and most C-extensions that I've seen release the GIL around blocking IO calls. This allows multi-threading to be used for the standard blocking IO use cases. Further, the computationally intensive routines in numpy and scipy that I’m familiar with also release the GIL to facilitate multithreading at the C-level The GIL only becomes an issue if you want multiple threads to be executing Python bytecode concurrently. I don’t believe this is a common problem as one rarely runs computationally intensive code at the python level. Such computationally intensive work is generally performed through C extensions or through external C functions (as accessed through ctypes). As long as such C extensions properly release the GIL, multithreading at the C-level is unaffected.

tmsh 16y ago
These are really good points. I suppose the only problem I see is when the C modules develop their own hierarchy of types and then they have to interface with Python's hierarchy of types. That's really the only issue. If what you're doing in C is numerical or computational in a functional way that doesn't need a hierarchy of types -- then keeping all your types in Python and calling into C for IO, computation, etc., makes a lot of sense. Where I've run into problems is if the C modules themselves get sophisticated and start having scene graphs or hierarchies of shapes, etc. Because then it makes sense to mirror the hierarchy in Python -- but then you get real performance issues. So I guess if you can ensure that your C modules have a clear functional interface (or at least one whose side effects are clearly defined) and doesn't involve anything but a shallow type hierarchy, then Python + C is all good for that. I.e., the C modules have to be very well-defined or have just about no 'code smell' (at least near their interface with Python). Arguably you can always do this if you make an effort to refactor your C modules. But yeah, because Python must always be at the root node, one is sort of constrained to serve the 'top', so to speak. And that may encourage a sort of top-down design which is less amenable to bottom-up programming (as mentioned in On Lisp), etc.

by hagy 16y ago

tmsh 16y ago

These are really good points. I suppose the only problem I see is when the C modules develop their own hierarchy of types and then they have to interface with Python's hierarchy of types. That's really the only issue. If what you're doing in C is numerical or computational in a functional way that doesn't need a hierarchy of types -- then keeping all your types in Python and calling into C for IO, computation, etc., makes a lot of sense. Where I've run into problems is if the C modules themselves get sophisticated and start having scene graphs or hierarchies of shapes, etc. Because then it makes sense to mirror the hierarchy in Python -- but then you get real performance issues. So I guess if you can ensure that your C modules have a clear functional interface (or at least one whose side effects are clearly defined) and doesn't involve anything but a shallow type hierarchy, then Python + C is all good for that. I.e., the C modules have to be very well-defined or have just about no 'code smell' (at least near their interface with Python). Arguably you can always do this if you make an effort to refactor your C modules. But yeah, because Python must always be at the root node, one is sort of constrained to serve the 'top', so to speak. And that may encourage a sort of top-down design which is less amenable to bottom-up programming (as mentioned in On Lisp), etc.