4 ms·

I wonder why so many library developers even chose to build native libraries on the shaky and poorly-architected foundation that Python is. Even writing a JVM

I wonder why so many library developers even chose to build native libraries on the shaky and poorly-architected foundation that Python is. Even writing a JVM native JNI library would have allowed to avoid a lot of that pain (and the library would have been useable from Clojure, Kotlin , Scala, JRuby[1], Jython[2], Java, etc) without any painful threading issues. [1] which I’m aware of having been used in production by companies in the past [2] which I’m aware has been quite a bit under maintained for the last several years

lanstin 3y ago
Because with the GIL it was dead simple to glue a C library into Python, and also was the canonical way to address hot inner loops in Python: rewrite into C. Nothing fancy but a little trial and error with module loading and you get 30% speed ups without being a great C programmer. I don’t think it’s false to say that the ease of moving hot spots into C is part of the reason Python has been so successful for thirty years. I hope this works, but I am very sceptical about being able to port code that worked with a locking solution provided for the enthusiast trying out C working without running into concurrency bugs.
- Cacti 3y ago
  My understanding is the thing about the binding system that drove adoption more than anything was due to how easy it was to generate bindings for an existing C library. This drove stdlib expansion and allowed rapid repurpose of huge amounts of long-standing and popular C libraries, and it also added instant credibility for those that already trusted those libraries. This helped avoid the chicken and egg problem of not enough devs for a serious stdlib, but not serious enough stdlib to draw dev (which is where most languages, even good ones, die). I distinctly remember that during the time period where Python grew from “minor” to “dominate” (roughly 1995-2005), doing Python dev was often a process of answering “are there bindings for that?” And usually _there were_, because of that tooling.
- d0mine 3y ago
  btw, many C extensions release GIL already e.g., one Python thread can do regex work, another some number crunching, yet another waiting IO--all in parallel.
- samus 3y ago
  C extensions are a different use case than accessing a C library using `ctypes`. The former allows intimate interactions with Python objects and therefore requires the GIL. The latter should only cause trouble if the C library executes callbacks on another thread. In that case, a Python thread is created for every invocation, which could cause race conditions with Python code. However, this problem also exists in GIL mode. In most cases, foreign libraries usually have Pythonic wrappers that should prevent some of these issues.
deleted 3y ago
[deleted]
gjulianm 3y ago
I think there are several reasons: - The history is backwards: it isn’t that devs wanted to make a native library and then chose Python, it’s that they chose Python and then they needed a native library. - Python works very well for scripts and small programs, and decently well for medium-size programs. This makes libraries with concrete purposes very useful and productive. If your library, say, helps devs do some basic calculation with time series, the ability to be used in quick scripts is a big plus. - The C API is fairly good, and libraries such as pybind11 make it even better. You don’t need a lot of code or boilerplate for an extension.
imtringued 3y ago
Actually, Java virtual threads have a similar problem with JNI. Your virtual thread gets pinned to a core if you call native code. This isn't as bad as the python GIL but it still limits the usefulness of virtual threads if they can't migrate to other cores.

lanstin 3y ago

Because with the GIL it was dead simple to glue a C library into Python, and also was the canonical way to address hot inner loops in Python: rewrite into C. Nothing fancy but a little trial and error with module loading and you get 30% speed ups without being a great C programmer. I don’t think it’s false to say that the ease of moving hot spots into C is part of the reason Python has been so successful for thirty years. I hope this works, but I am very sceptical about being able to port code that worked with a locking solution provided for the enthusiast trying out C working without running into concurrency bugs.

Cacti 3y ago
My understanding is the thing about the binding system that drove adoption more than anything was due to how easy it was to generate bindings for an existing C library. This drove stdlib expansion and allowed rapid repurpose of huge amounts of long-standing and popular C libraries, and it also added instant credibility for those that already trusted those libraries. This helped avoid the chicken and egg problem of not enough devs for a serious stdlib, but not serious enough stdlib to draw dev (which is where most languages, even good ones, die). I distinctly remember that during the time period where Python grew from “minor” to “dominate” (roughly 1995-2005), doing Python dev was often a process of answering “are there bindings for that?” And usually _there were_, because of that tooling.
d0mine 3y ago
btw, many C extensions release GIL already e.g., one Python thread can do regex work, another some number crunching, yet another waiting IO--all in parallel.
samus 3y ago
C extensions are a different use case than accessing a C library using `ctypes`. The former allows intimate interactions with Python objects and therefore requires the GIL. The latter should only cause trouble if the C library executes callbacks on another thread. In that case, a Python thread is created for every invocation, which could cause race conditions with Python code. However, this problem also exists in GIL mode. In most cases, foreign libraries usually have Pythonic wrappers that should prevent some of these issues.

deleted 3y ago

[deleted]

gjulianm 3y ago

I think there are several reasons: - The history is backwards: it isn’t that devs wanted to make a native library and then chose Python, it’s that they chose Python and then they needed a native library. - Python works very well for scripts and small programs, and decently well for medium-size programs. This makes libraries with concrete purposes very useful and productive. If your library, say, helps devs do some basic calculation with time series, the ability to be used in quick scripts is a big plus. - The C API is fairly good, and libraries such as pybind11 make it even better. You don’t need a lot of code or boilerplate for an extension.

imtringued 3y ago

Actually, Java virtual threads have a similar problem with JNI. Your virtual thread gets pinned to a core if you call native code. This isn't as bad as the python GIL but it still limits the usefulness of virtual threads if they can't migrate to other cores.