4 ms·

Can you comment on the difference between Pymod and Cython? Cython's pretty mature now and it's straight up amazing how easy it is to generate cross-platform ex

by p4wnc6 11y ago

Can you comment on the difference between Pymod and Cython? Cython's pretty mature now and it's straight up amazing how easy it is to generate cross-platform extension modules. It even exposes a lot of C++ too. Cython is a separate programming language that allows you to write C code with a special Python-like syntax, or write Python with Python syntax (including NumPy), and has a few extra type annotation bits here and there (like array syntax or pointer syntax). At the end, it creates the equivalent C code (with all needed CPython API boilerplate) and compiles it into an importable shared object file. Just from the description above, it sounds superficially the same as Pymod; is it fair to say Pymod is to Nim what Cython is to C/C++? I'd be very interested to know more about how the two tools compare and contrast.

jboy 11y ago
Sure, this is a question that people ask quite frequently. I most recently answered it here: https://news.ycombinator.com/item?id=10569768 https://news.ycombinator.com/item?id=10569768 """Actually, Pymod was designed to be almost an anti-Cython. :) My issue with Cython is that it's a limited sub-language within Python, where you add Cython elements incrementally & iteratively (diverging from Python in the process) until the code runs "fast enough". I'd rather work directly in a full-featured, internally-self-consistent language from the start. Nim has a clean Pythonic syntax, with all the best parts of C++ (including its runtime speed). Hence, Pymod takes the form of an `exportpy` annotation (a user-defined Nim pragma) onto existing Nim functions, which are then auto-compiled into a Python extension module. So there's no gradual divergence of my Python code (as it becomes more "Cythonized"); rather, the high-performance code is written directly in pure Nim. :) """ There are a few more details in that thread comparing the wrapping of existing C libraries in Cython vs Pymod. (It doesn't seem right to copy-paste an entire thread...)
- p4wnc6 11y ago
  I'm not sure I appreciate that criticism of Cython. A general strategy in Cython is to use the annotation feature (cython -a) and visually inspect the degree of CPython API necessary for the lines of Cython source code. I've generally found this process to be really enlightening. Of course you can use that information to select portions of the code that don't need to be involving the CPython API, add typed contructs to those parts of the Cython source, and iterate. But you can also learn a lot about how CPython works ... for example how using the 'and' keyword can invoke a long chain of Python special functions with lots of type checking overhead. What this lets me do is to be extremely fine-grained about my optimizations, or conversely, to also see when some optimizations are not worth it because they don't help much but they do hurt the readability, or cause too much Python-to-Cython divergence as you put it. In a lot of cases, I prefer that this is left up to me to do, rather than if Cython had already made hand-mapped choices about which Python things compile to which C or machine code things. If taken to the limit, a Cython that did that would just become what PyPy is, except it would be ahead of time compiled instead of JIT compiled. But I do see the benefit of both approaches. Sometimes you don't want the burden of choosing your annotations to induce the desired compilation effect, and you don't want to allow for similar but not identical Cython source files to result in dramatically different C code, as can often happen currently.
  - jboy 11y ago
    The preferred workflow that you've described seems to be a (more considered) form of the standard Cython workflow that I see described: (1) Write Python. (2) Compile with Cython. (3) Run compiled Cython, profile & review. (4) Consider what Cython annotations to make to code; make code changes. (5) Goto 2. The emphasis is always on iteration & incremental additions. Of course I practice iterative & incremental development, and of course I'll prototype a quick proof-of-concept implementation first (often in Python+Numpy) before profiling & algorithmic optimization. But the Cython workflow seems to me to add more iteration (of incremental Cython syntax additions) than is really necessary. When I'm working to implement some algorithm, I don't really want to iteratively learn how Cython or CPython implement various Python functions; I'd rather write my "proper version" code just once, properly the first time. So why not take it to the logical extreme and write it all in Cython-lang from the start? If we're writing for-keeps code in a language with Pythonic syntax & static types, I find Nim-lang a more expressive, more full-featured language (with features such as generics & type-safe Lisp-like macros, in particular; I note that Cython does support pointers & operator overloading) than Cython-lang in general-purpose uses, without being very different at all in simple uses. For example, there is an example `primes` function in the Cython tutorial: http://docs.cython.org/src/tutorial/cython_tutorial.html#primes http://docs.cython.org/src/tutorial/cython_tutorial.html#pri... Here is an equivalent implementation in Nim. As you can see, it's really almost identical in syntax: proc primes(kmax: int): seq[int] = var kmax = kmax var n, k, i: int var p: array[1000, int] result = @[] if kmax > 1000: kmax = 1000 k = 0 n = 2 while k < kmax: i = 0 while i < k and n mod p[i] != 0: i = i + 1 if i == k: p[k] = n k = k + 1 result.add(n) n = n + 1 return result All of this said, I understand that a great deal of this decision comes down to personal preference: Would you rather start with Python & then iteratively diverge? Would you rather start & stay in Nim? And I can also see the benefit of both approaches in different circumstances. :)
    - Nrpf 11y ago
      Or you can use numba and with pure python syntax and get really fast.
    - 11y ago
- andreasvc 11y ago
  It is strange that you characterize Cython as a limited sub-language. Cython supports almost the full Python language, plus adding a superset for producing native C code. In addition, Cython can be used to wrap C code, so what you describe as "exportpy" is possible as well. Cython is a mature project, being in use in various parts of the scientific python ecosystems such as in Pandas and scikit-learn. Nim sure looks interesting but from what I hear, the language and the compiler still have rough edges. In terms of stability, maturity, and ecosystem, I'd say Cython has a clear advantage.
  - jboy 11y ago
    I elaborated on my thoughts about the Cython language in this sibling comment: https://news.ycombinator.com/edit?id=10578834 https://news.ycombinator.com/edit?id=10578834 I agree with you that Cython is a mature, very respectable project, and it evidently has more widespread adoption at this time. I'm betting on Nim because I think it has a larger positive gradient, even though its y-coordinate is lower right now.
pathsjs 11y ago
I will try to give my 2 cents. As you correctly state, Cython is in effect another language. Now, a core principle in the design of Cython is to be as much as possible source compatible with Python, which is a language without static types. This is not a very good principle for me - I would rather work in a language that is not literally identical to Python, but has its own design. Adding static types often requires to do things slightly differently. In Nim a find a very principled and well designed language, and PyMod seems a great to tool to make interoperability with Python easy.
- andreasvc 11y ago
  You have a point that adding static types can mean having to rewrite. But in practice I often start out with Cython and static types, making this a non-issue. If you use Cython in this way it can approach C with Python syntax. See also http://spacy.io/blog/writing-c-in-cython/ http://spacy.io/blog/writing-c-in-cython/
  - pathsjs 11y ago
    I am not only saying that one needs to write things differently in Cython. I am arguing that the whole language needs to be designed differently, to take into consideration the presence of types. This includes adding features to recover the lost dynamism, such as interface or macros, and modifying the standard library to a style that best suits a typed language. Just grafting types onto Python is not enough
    - andreasvc 11y ago
      > I am arguing that the whole language needs to be designed differently No I think you are arguing that you want it to be designed differently. It's like the contrast between having a high and low level language (Python, C/Cython) versus having a unified middle-level language such as Julia (see http://graydon2.dreamwidth.org/189377.html http://graydon2.dreamwidth.org/189377.html ). It's a matter of preference which is better, and an empirical matter which approach will be more successful in the end. > Just grafting types onto Python is not enough Clearly it is already successful though...

by p4wnc6 11y ago

jboy 11y ago

Sure, this is a question that people ask quite frequently. I most recently answered it here: https://news.ycombinator.com/item?id=10569768 https://news.ycombinator.com/item?id=10569768 """Actually, Pymod was designed to be almost an anti-Cython. :) My issue with Cython is that it's a limited sub-language within Python, where you add Cython elements incrementally & iteratively (diverging from Python in the process) until the code runs "fast enough". I'd rather work directly in a full-featured, internally-self-consistent language from the start. Nim has a clean Pythonic syntax, with all the best parts of C++ (including its runtime speed). Hence, Pymod takes the form of an `exportpy` annotation (a user-defined Nim pragma) onto existing Nim functions, which are then auto-compiled into a Python extension module. So there's no gradual divergence of my Python code (as it becomes more "Cythonized"); rather, the high-performance code is written directly in pure Nim. :) """ There are a few more details in that thread comparing the wrapping of existing C libraries in Cython vs Pymod. (It doesn't seem right to copy-paste an entire thread...)

p4wnc6 11y ago
I'm not sure I appreciate that criticism of Cython. A general strategy in Cython is to use the annotation feature (cython -a) and visually inspect the degree of CPython API necessary for the lines of Cython source code. I've generally found this process to be really enlightening. Of course you can use that information to select portions of the code that don't need to be involving the CPython API, add typed contructs to those parts of the Cython source, and iterate. But you can also learn a lot about how CPython works ... for example how using the 'and' keyword can invoke a long chain of Python special functions with lots of type checking overhead. What this lets me do is to be extremely fine-grained about my optimizations, or conversely, to also see when some optimizations are not worth it because they don't help much but they do hurt the readability, or cause too much Python-to-Cython divergence as you put it. In a lot of cases, I prefer that this is left up to me to do, rather than if Cython had already made hand-mapped choices about which Python things compile to which C or machine code things. If taken to the limit, a Cython that did that would just become what PyPy is, except it would be ahead of time compiled instead of JIT compiled. But I do see the benefit of both approaches. Sometimes you don't want the burden of choosing your annotations to induce the desired compilation effect, and you don't want to allow for similar but not identical Cython source files to result in dramatically different C code, as can often happen currently.
- jboy 11y ago
  The preferred workflow that you've described seems to be a (more considered) form of the standard Cython workflow that I see described: (1) Write Python. (2) Compile with Cython. (3) Run compiled Cython, profile & review. (4) Consider what Cython annotations to make to code; make code changes. (5) Goto 2. The emphasis is always on iteration & incremental additions. Of course I practice iterative & incremental development, and of course I'll prototype a quick proof-of-concept implementation first (often in Python+Numpy) before profiling & algorithmic optimization. But the Cython workflow seems to me to add more iteration (of incremental Cython syntax additions) than is really necessary. When I'm working to implement some algorithm, I don't really want to iteratively learn how Cython or CPython implement various Python functions; I'd rather write my "proper version" code just once, properly the first time. So why not take it to the logical extreme and write it all in Cython-lang from the start? If we're writing for-keeps code in a language with Pythonic syntax & static types, I find Nim-lang a more expressive, more full-featured language (with features such as generics & type-safe Lisp-like macros, in particular; I note that Cython does support pointers & operator overloading) than Cython-lang in general-purpose uses, without being very different at all in simple uses. For example, there is an example `primes` function in the Cython tutorial: http://docs.cython.org/src/tutorial/cython_tutorial.html#primes http://docs.cython.org/src/tutorial/cython_tutorial.html#pri... Here is an equivalent implementation in Nim. As you can see, it's really almost identical in syntax: proc primes(kmax: int): seq[int] = var kmax = kmax var n, k, i: int var p: array[1000, int] result = @[] if kmax > 1000: kmax = 1000 k = 0 n = 2 while k < kmax: i = 0 while i < k and n mod p[i] != 0: i = i + 1 if i == k: p[k] = n k = k + 1 result.add(n) n = n + 1 return result All of this said, I understand that a great deal of this decision comes down to personal preference: Would you rather start with Python & then iteratively diverge? Would you rather start & stay in Nim? And I can also see the benefit of both approaches in different circumstances. :)
  - Nrpf 11y ago
    Or you can use numba and with pure python syntax and get really fast.
  - 11y ago
andreasvc 11y ago
It is strange that you characterize Cython as a limited sub-language. Cython supports almost the full Python language, plus adding a superset for producing native C code. In addition, Cython can be used to wrap C code, so what you describe as "exportpy" is possible as well. Cython is a mature project, being in use in various parts of the scientific python ecosystems such as in Pandas and scikit-learn. Nim sure looks interesting but from what I hear, the language and the compiler still have rough edges. In terms of stability, maturity, and ecosystem, I'd say Cython has a clear advantage.
- jboy 11y ago
  I elaborated on my thoughts about the Cython language in this sibling comment: https://news.ycombinator.com/edit?id=10578834 https://news.ycombinator.com/edit?id=10578834 I agree with you that Cython is a mature, very respectable project, and it evidently has more widespread adoption at this time. I'm betting on Nim because I think it has a larger positive gradient, even though its y-coordinate is lower right now.

pathsjs 11y ago

I will try to give my 2 cents. As you correctly state, Cython is in effect another language. Now, a core principle in the design of Cython is to be as much as possible source compatible with Python, which is a language without static types. This is not a very good principle for me - I would rather work in a language that is not literally identical to Python, but has its own design. Adding static types often requires to do things slightly differently. In Nim a find a very principled and well designed language, and PyMod seems a great to tool to make interoperability with Python easy.

andreasvc 11y ago
You have a point that adding static types can mean having to rewrite. But in practice I often start out with Cython and static types, making this a non-issue. If you use Cython in this way it can approach C with Python syntax. See also http://spacy.io/blog/writing-c-in-cython/ http://spacy.io/blog/writing-c-in-cython/
- pathsjs 11y ago
  I am not only saying that one needs to write things differently in Cython. I am arguing that the whole language needs to be designed differently, to take into consideration the presence of types. This includes adding features to recover the lost dynamism, such as interface or macros, and modifying the standard library to a style that best suits a typed language. Just grafting types onto Python is not enough
  - andreasvc 11y ago
    > I am arguing that the whole language needs to be designed differently No I think you are arguing that you want it to be designed differently. It's like the contrast between having a high and low level language (Python, C/Cython) versus having a unified middle-level language such as Julia (see http://graydon2.dreamwidth.org/189377.html http://graydon2.dreamwidth.org/189377.html ). It's a matter of preference which is better, and an empirical matter which approach will be more successful in the end. > Just grafting types onto Python is not enough Clearly it is already successful though...