3 ms·

> You loose CPU-bound parallelism with threading, but you gain easy to write C extensions I'm extremely interested in your thoughts on this subject as a NumPy

by _dps 13y ago

> You loose CPU-bound parallelism with threading, but you gain easy to write C extensions I'm extremely interested in your thoughts on this subject as a NumPy developer. To what extent do you think C interop designs like Python CFFI (which mimics LuaJIT's FFI) will make this an unimpressive accomplishment? As a former heavy user of Python for scientific programming (for easy access to C libraries) I have moved everything I do to LuaJIT and C. The ease of writing a LuaJIT FFI (or Python CFFI binding) is at least an order of magnitude greater than that of writing PyObject* style bindings. Do you think it's possible (or likely) that a mature CFFI will make this tradeoff you describe seem no-longer-positive? Added in Edit: I should point out that multicore patterns in Lua are already very different from those in Python, e.g. running a Lua VM per core and communicating through shared memory. So I realize that the existence of CFFI won't map one-to-one onto changes in Python multicore design, at least in the short term.

cdavid 13y ago
For interfacing with C, I think cython is better than cffi. Well, I don't have much experience with cffi, but I used to use ctypes, and cython is much better. Cffi may be nicer than ctypes, but not enough to make me switch (I may be missing something). Unfortunately, I think that for python, it is too late to have a better C API: there is so much legacy that depends on python C API details that moving away from that would take almost as much manpower as rewriting it to a different language. Look at pypy: even though it is 5x times faster on average than python, I don't see people flying from cpython to pypy. 5x speedup is more than what you can hope on todays' CPU with near perfect scaling with # cores. This tells me that no-GIL python, to be successful, would need to have a much lower barrier to entry than pypy. Maybe the future is closer to having hooks to 'escape' python for the numerical stuff: numba and the numerous similar projects jitting a subset of python, or interoperating with some other language more amenable to optimization (e.g. Julia: http://www.youtube.com/watch?v=Eb8CMuNKdJ0 http://www.youtube.com/watch?v=Eb8CMuNKdJ0) I am quite interested in Lua-like design, though: one of my pet project around numpy would be to refactor it internally to use something closer to how lua works, exposing numpy to cpython only at the outer layer.
- _dps 13y ago
  > For interfacing with C, I think cython is better than cffi. I think we'll have to agree to disagree on this. To quote the CFFI project description [0]: the goal is to be able to seamlessly call C from Python without having to learn a third language/API. Cython, Ctypes, and CPython's C API all fail to meet this criterion, and in my opinion do so by a large margin. Let's also not forget the fact that Cython requires a compilation step not only for the bound C code but also for the portion of the "Python" code that interfaces with it. Compare that (or writing CPython C API manipulation code) with the following: >>> from cffi import FFI >>> ffi = FFI() >>> ffi.cdef("int printf(const char *format, ...);") >>> C = ffi.dlopen(None) # loads the entire C namespace >>> arg = ffi.new("char[]", "world") # char arg[] = "world"; >>> C.printf("hi there, %s!\n", arg) # call printf hi there, world! For me CFFI wins without question. Of course, it's even simpler in LuaJIT (but the CFFI people are getting closer every day): local ffi = require "ffi" ffi.cdef [[ int printf(const char*, ...); ]] ffi.C.printf("hi there, %s!\n", "world") Now, it would be misleading for me not to point out the limitations of an approach like CFFI/LJ-FFI vs CPython API: you may have to contort yourself to make C calls that safely manipulate the scripting language VM (e.g. instantiate new Python objects from within a C FFI call). In my experience this has not been a serious problem because it's easy to write struct-to-object mappers in idiomatic Python or Lua rather than having to muck about with PyObject* and friends. Of course, in LuaJIT it's even easier with FFI metatypes (which let you attach dispatch tables to FFI native types, so your structs can behave just like Lua tables with method dispatch, inheritance, etc.). Anyway, leaving personal taste aside for a moment, let me rephrase the original question that sparked my interest in your opinion: in a world with a mature CFFI option providing the above binding capabilities do you think that easy C extensions are still a worthwhile tradeoff for the GIL? (FWIW I find the point to be quite compelling for why Python had a GIL to begin with in the late 90s, but I'm suspecting it's less of an intrinsic design tradeoff and more historical cruft as the PyPy and CFFI people roll out their work). [0] http://cffi.readthedocs.org/en/release-0.6/ http://cffi.readthedocs.org/en/release-0.6/
  - cdavid 13y ago
    I don't think the points you highlighted above matter fundamentally: they don't handle memory management for you, and that's the difficulty, especially at the language boundary. What is interesting for me in Lua is the stack-based argument passing and the GC: it avoids leaking the reference counting and the binary representation of objects (I think, I am rather clueless when it comes to Lua). Wrapping things like crypto, file format and co is relatively trivial, because not much crosses the language barrier. Now, when you need to handle non native object life cycle, that's another matter, and I don't see how cffi makes it any easier than cython

by _dps 13y ago

cdavid 13y ago

For interfacing with C, I think cython is better than cffi. Well, I don't have much experience with cffi, but I used to use ctypes, and cython is much better. Cffi may be nicer than ctypes, but not enough to make me switch (I may be missing something). Unfortunately, I think that for python, it is too late to have a better C API: there is so much legacy that depends on python C API details that moving away from that would take almost as much manpower as rewriting it to a different language. Look at pypy: even though it is 5x times faster on average than python, I don't see people flying from cpython to pypy. 5x speedup is more than what you can hope on todays' CPU with near perfect scaling with # cores. This tells me that no-GIL python, to be successful, would need to have a much lower barrier to entry than pypy. Maybe the future is closer to having hooks to 'escape' python for the numerical stuff: numba and the numerous similar projects jitting a subset of python, or interoperating with some other language more amenable to optimization (e.g. Julia: http://www.youtube.com/watch?v=Eb8CMuNKdJ0 http://www.youtube.com/watch?v=Eb8CMuNKdJ0) I am quite interested in Lua-like design, though: one of my pet project around numpy would be to refactor it internally to use something closer to how lua works, exposing numpy to cpython only at the outer layer.

_dps 13y ago
> For interfacing with C, I think cython is better than cffi. I think we'll have to agree to disagree on this. To quote the CFFI project description [0]: the goal is to be able to seamlessly call C from Python without having to learn a third language/API. Cython, Ctypes, and CPython's C API all fail to meet this criterion, and in my opinion do so by a large margin. Let's also not forget the fact that Cython requires a compilation step not only for the bound C code but also for the portion of the "Python" code that interfaces with it. Compare that (or writing CPython C API manipulation code) with the following: >>> from cffi import FFI >>> ffi = FFI() >>> ffi.cdef("int printf(const char *format, ...);") >>> C = ffi.dlopen(None) # loads the entire C namespace >>> arg = ffi.new("char[]", "world") # char arg[] = "world"; >>> C.printf("hi there, %s!\n", arg) # call printf hi there, world! For me CFFI wins without question. Of course, it's even simpler in LuaJIT (but the CFFI people are getting closer every day): local ffi = require "ffi" ffi.cdef [[ int printf(const char*, ...); ]] ffi.C.printf("hi there, %s!\n", "world") Now, it would be misleading for me not to point out the limitations of an approach like CFFI/LJ-FFI vs CPython API: you may have to contort yourself to make C calls that safely manipulate the scripting language VM (e.g. instantiate new Python objects from within a C FFI call). In my experience this has not been a serious problem because it's easy to write struct-to-object mappers in idiomatic Python or Lua rather than having to muck about with PyObject* and friends. Of course, in LuaJIT it's even easier with FFI metatypes (which let you attach dispatch tables to FFI native types, so your structs can behave just like Lua tables with method dispatch, inheritance, etc.). Anyway, leaving personal taste aside for a moment, let me rephrase the original question that sparked my interest in your opinion: in a world with a mature CFFI option providing the above binding capabilities do you think that easy C extensions are still a worthwhile tradeoff for the GIL? (FWIW I find the point to be quite compelling for why Python had a GIL to begin with in the late 90s, but I'm suspecting it's less of an intrinsic design tradeoff and more historical cruft as the PyPy and CFFI people roll out their work). [0] http://cffi.readthedocs.org/en/release-0.6/ http://cffi.readthedocs.org/en/release-0.6/
- cdavid 13y ago
  I don't think the points you highlighted above matter fundamentally: they don't handle memory management for you, and that's the difficulty, especially at the language boundary. What is interesting for me in Lua is the stack-based argument passing and the GC: it avoids leaking the reference counting and the binary representation of objects (I think, I am rather clueless when it comes to Lua). Wrapping things like crypto, file format and co is relatively trivial, because not much crosses the language barrier. Now, when you need to handle non native object life cycle, that's another matter, and I don't see how cffi makes it any easier than cython