3 ms·

Are there really that many Python libraries implemented in C? Genuine question - I thought there were a few very high profile ones, but I always figured there w

by smcl 4y ago

Are there really that many Python libraries implemented in C? Genuine question - I thought there were a few very high profile ones, but I always figured there weren't many (like 10% or less, total finger-in-the-air guess)

randlet 4y ago
"there were a few very high profile ones" The absolute number (or fraction) implemented in C is less important than the high profile ones. numpy/scipy alone are used in a huge number of projects so if you can't support those two libraries alone it's pretty much a non starter.
- smcl 4y ago
  Oh I’m not downplaying them, if they were small in number and minor then I’m sure CPython would’ve happily dumped the GIL already.
bloblaw 4y ago
The number of them isn't as relevant as their usage. Lots of data science related modules are effectively wrappers around C code. Here's a few: NumPy pandas Matplotlib TensorFlow PyTorch These modules are critical to many of the workloads using Python (ML / AI / Data Science)
- smcl 4y ago
  Right this is what I thought - a small number of popular ones, rather than an overall majority across pypi packages
  - plonk 4y ago
    In the data science space, which is probably one of the biggest users of Python, an interpreter that doesn't support these is dead in the water.
    - ameliaquining 4y ago
      I think the question is whether it would suffice to port the few most popular native modules to directly use this project's bespoke Rust API instead of the standard CPython API. This'd probably be less work than building a real emulation layer for the CPython API, especially if they don't want to include the GIL. But if Python users need a lot of different native modules and not just a few popular ones, then that won't suffice.
dragonwriter 4y ago
> Are there really that many Python libraries implemented in C? There are a lot that use the C API, even if some of those aren't implemented in C (some are actually implemented in Rust.)
- smcl 4y ago
  This is what I meant, “implemented in C” and “use the C API” are basically the same thing at some level - if they depend on the C/Python interface at all
jeroenhd 4y ago
The ones that are fast are implemented in something other than Python. Most of them C, some Rust, but all native code. Python itself is just too slow to be useful in many scenarios.

by smcl 4y ago

randlet 4y ago

"there were a few very high profile ones" The absolute number (or fraction) implemented in C is less important than the high profile ones. numpy/scipy alone are used in a huge number of projects so if you can't support those two libraries alone it's pretty much a non starter.

smcl 4y ago
Oh I’m not downplaying them, if they were small in number and minor then I’m sure CPython would’ve happily dumped the GIL already.

bloblaw 4y ago

The number of them isn't as relevant as their usage. Lots of data science related modules are effectively wrappers around C code. Here's a few: NumPy pandas Matplotlib TensorFlow PyTorch These modules are critical to many of the workloads using Python (ML / AI / Data Science)

smcl 4y ago
Right this is what I thought - a small number of popular ones, rather than an overall majority across pypi packages
- plonk 4y ago
  In the data science space, which is probably one of the biggest users of Python, an interpreter that doesn't support these is dead in the water.
  - ameliaquining 4y ago
    I think the question is whether it would suffice to port the few most popular native modules to directly use this project's bespoke Rust API instead of the standard CPython API. This'd probably be less work than building a real emulation layer for the CPython API, especially if they don't want to include the GIL. But if Python users need a lot of different native modules and not just a few popular ones, then that won't suffice.

dragonwriter 4y ago

> Are there really that many Python libraries implemented in C? There are a lot that use the C API, even if some of those aren't implemented in C (some are actually implemented in Rust.)

smcl 4y ago
This is what I meant, “implemented in C” and “use the C API” are basically the same thing at some level - if they depend on the C/Python interface at all

jeroenhd 4y ago

The ones that are fast are implemented in something other than Python. Most of them C, some Rust, but all native code. Python itself is just too slow to be useful in many scenarios.