3 ms·

15 years ago I remember reading Guido van Rossum saying that Python is a connector language and if you need performance, just drop into C and write/use a C modu

15 years ago I remember reading Guido van Rossum saying that Python is a connector language and if you need performance, just drop into C and write/use a C module. I thought it was crazy at the time, but now I see that he was absolutely right. It took a while, but now Python has a high-performing C module for pretty much every task.

chrisseaton 4y ago
But these don’t compose right? Each is a black-box to each other? A black-box add and a black-box multiply don’t fuse.
- dekhn 4y ago
  C and Python are not black boxes to each other. The entire python interpreter is literally a C API. You can create pyobjects, add heterogenous PyObjects to PyLists, etc. So evewrything in Python can be introspected from C. Turned around, Python has arbitrary access into the C programming space (really, the UNIX or Windows process it's running inside), so long as it has access to headers or other type info it can see C with more than black box info. Most python numerics is implement in numpy; the low levels of numpy are actually (or were) effectively a C API implementing multiple dimension arrays, with a python wrapper.
  - klyrs 4y ago
    You're talking past chrisseaton's point here. If you want two C extensions to interoperate with bare-metal performance, you can't just do from lib1 import makedata from lib2 import processdata data = makedata() print(processdata(data)) Because makedata needs to provide a c->py bridge and processdata needs a py->c bridge, so your process inherently has python in the middle unless lib2 has intimate knowledge of lib1. It can absolutely be done (I've written plenty of c extensions that handle numpy arrays, for example) but if somebody hasn't done the work, you don't get it for free. If your c extension expects a list of lists of floats, the numpy array totally supports that interface... but (last I checked) the overhead there is way slower than calling list(map(list, data)) and throwing that into your numpy-naive c extension.
  - chrisseaton 4y ago
    > C and Python are not black boxes to each other Yes they are - the C interpreter knows nothing about what your C extension does. It can’t optimise it because all it has is your machine code - no higher level logic.
- kylebarron 4y ago
  They can! Numpy exposes a C API to other Python programs [0]. It's not hard to write a Cython library that uses the Numpy C API directly and does not cross into Python [1]. [0]: https://numpy.org/doc/stable/reference/c-api/index.html https://numpy.org/doc/stable/reference/c-api/index.html [1]: https://github.com/kylebarron/pymartini/blob/4774549ffa2051c4ddaa254aca191bedcc4d30c5/pymartini/martini.pyx#L2 https://github.com/kylebarron/pymartini/blob/4774549ffa2051c...
  - chrisseaton 4y ago
    So they can if you use their specific API? It doesn’t naturally compose in conventional Python code?

chrisseaton 4y ago

But these don’t compose right? Each is a black-box to each other? A black-box add and a black-box multiply don’t fuse.

dekhn 4y ago
C and Python are not black boxes to each other. The entire python interpreter is literally a C API. You can create pyobjects, add heterogenous PyObjects to PyLists, etc. So evewrything in Python can be introspected from C. Turned around, Python has arbitrary access into the C programming space (really, the UNIX or Windows process it's running inside), so long as it has access to headers or other type info it can see C with more than black box info. Most python numerics is implement in numpy; the low levels of numpy are actually (or were) effectively a C API implementing multiple dimension arrays, with a python wrapper.
- klyrs 4y ago
  You're talking past chrisseaton's point here. If you want two C extensions to interoperate with bare-metal performance, you can't just do from lib1 import makedata from lib2 import processdata data = makedata() print(processdata(data)) Because makedata needs to provide a c->py bridge and processdata needs a py->c bridge, so your process inherently has python in the middle unless lib2 has intimate knowledge of lib1. It can absolutely be done (I've written plenty of c extensions that handle numpy arrays, for example) but if somebody hasn't done the work, you don't get it for free. If your c extension expects a list of lists of floats, the numpy array totally supports that interface... but (last I checked) the overhead there is way slower than calling list(map(list, data)) and throwing that into your numpy-naive c extension.
- chrisseaton 4y ago
  > C and Python are not black boxes to each other Yes they are - the C interpreter knows nothing about what your C extension does. It can’t optimise it because all it has is your machine code - no higher level logic.
kylebarron 4y ago
They can! Numpy exposes a C API to other Python programs [0]. It's not hard to write a Cython library that uses the Numpy C API directly and does not cross into Python [1]. [0]: https://numpy.org/doc/stable/reference/c-api/index.html https://numpy.org/doc/stable/reference/c-api/index.html [1]: https://github.com/kylebarron/pymartini/blob/4774549ffa2051c4ddaa254aca191bedcc4d30c5/pymartini/martini.pyx#L2 https://github.com/kylebarron/pymartini/blob/4774549ffa2051c...
- chrisseaton 4y ago
  So they can if you use their specific API? It doesn’t naturally compose in conventional Python code?