31 ms·

Pythran: A Python subset to C++ compiler that takes advantage of SIMD

51 points by maxmouchet 11y ago

gamesbrainiac 11y ago
There are quite a few attempts to make faster python implementation these days from pypy (A JIT built on top of rpython), pyston (yet another JIT that which uses llvm as a backend), pyparallel, nukita and I'm pretty sure there are a lot more. Either the python community is being very prolific, or has no clue what will actually solve the problem, hence its trying everything.
- StavrosK 11y ago
  Or there are multiple ways to solve the speed problem, with different approaches having different benefits.
- _yosefk 11y ago
  Well, these things are fun to hack on and then each might solve a part of the generally near-unsolvable problem, and for each such part there are people caring about that part the most so the solution has its uses. (Incidentally, my own preference to date has been to only use CPython and fall back to C extension modules when speed starts to matter... But I could have a different preference in a different environment working on different things.) Kind of like making C or C++ safe is a near-unsolvable problem for most values of "safe" and there's an ever-growing set of static and dynamic analysis tools detecting things like uninitialized memory access, out of bounds access, data races, integer overflow, etc. etc. etc. And each of these tools gives you a different mix of false positives, false negatives, build time overhead (including getting the program to build with the tool at all) and runtime overhead (including getting the program to run), not to mention availability on a given platform and pricing. And different people have a different preference with respect to what this mix should be for their tool of choice. But the problem is unsolvable which is why it's worth to be "trying everything."
  - pjmlp 11y ago
    > But the problem is unsolvable which is why it's worth to be "trying everything." The problem is solvable by switching languages, but what these people all try to do, is to solve the problem while keep using the language, which is by definition unsolvable. As the original causes of the problem won't go away. But it is good, if in the end at least helps to improve the overall situation.
    - themartorana 11y ago
      Switching languages may solve one problem while introducing another. If it were so simple, no one would use Python.
  - jboy 11y ago
    As a longtime programmer of Python, C & C++ (and a longtime appreciator of your FQA Lite, by the way), my preference until recently was also to use CPython & fall back on extension modules for speed. But I grew weary of writing all the Python C-API boilerplate -- especially checking parameter types and managing reference counts properly. So I created (and recently open-sourced) Pymod, a Nim+Python project that auto-generates all the Python C-API boilerplate & auto-compiles a Python C extension module, to wrap the functions in a Nim module: https://github.com/jboy/nim-pymod https://github.com/jboy/nim-pymod As a pragmatic Python programmer, Pymod may be of interest to you too... (As someone who learned C++ in 1999 and spent most of 200x working in C++, I also grew weary of staying on top of C++0x's new features & new gotchas. So about a year ago, I switched from C++ to Nim-lang and haven't looked back. To me, Nim combines all the best features of Python & C++, while avoiding most of the worst: C++ speed, C++ static types, C++ generics, C++ operator overloading; combined with a clean Pythonic syntax & Lispy macros.)
    - maxerickson 11y ago
      Cython (and before it Pyrex) occupies a similar space between C and Python. Was any of your design inspired by them? I also wonder how practical it would be to reuse parts of it when making a bridge for another language.
- detaro 11y ago
  Each of these projects has different goals if you look a bit more closely than "make python faster" and has made design decisions and trade-offs accordingly, and as a result each has use-cases it is better suited for than others.
- maxerickson 11y ago
  Nuitka for anybody searching. There is also Shedskin and Numba. It'd be nice if this project mentioned why they decided not to continue/enhance Shedskin. I think another possible explanation is that compiling a subset of Python isn't such a huge project. Only a couple of the projects mentioned are anywhere close to even just supporting the standard library that ships with CPython.
toolslive 11y ago
But does it retain the python semantics? for example: def fac(n): if n == 0 : return 1 else: return n * fac(n-1) print fac(100) will yield a big string, while any naive C++ translation will print the string "0" due to overflow.
- e12e 11y ago
  Good question (and I don't know the answer, as I'm not familiar with this project). Note that a naive c++ translation will also error out on simply i=2 * * 524. Automatic extension/promotion of numbers is pretty fundamental to python.
nickcw 11y ago
I just wondered if anyone had any insight in how this differs from shedskin - https://code.google.com/p/shedskin/ https://code.google.com/p/shedskin/ From the shedskin website: Shed Skin is an experimental compiler, that can translate pure, but implicitly statically typed Python (2.4-2.6) programs into optimized C++. It can generate stand-alone programs or extension modules that can be imported and used in larger Python programs.
- drdaeman 11y ago
  Don't know about the differences, but I think shedskin had died some years ago.
  - stuaxo 11y ago
    Nope, they are still responsive on the bug list, I think it lives - just very slowly.

51 points by maxmouchet 11y ago

There are quite a few attempts to make faster python implementation these days from pypy (A JIT built on top of rpython), pyston (yet another JIT that which uses llvm as a backend), pyparallel, nukita and I'm pretty sure there are a lot more. Either the python community is being very prolific, or has no clue what will actually solve the problem, hence its trying everything.

StavrosK 11y ago
Or there are multiple ways to solve the speed problem, with different approaches having different benefits.
_yosefk 11y ago
Well, these things are fun to hack on and then each might solve a part of the generally near-unsolvable problem, and for each such part there are people caring about that part the most so the solution has its uses. (Incidentally, my own preference to date has been to only use CPython and fall back to C extension modules when speed starts to matter... But I could have a different preference in a different environment working on different things.) Kind of like making C or C++ safe is a near-unsolvable problem for most values of "safe" and there's an ever-growing set of static and dynamic analysis tools detecting things like uninitialized memory access, out of bounds access, data races, integer overflow, etc. etc. etc. And each of these tools gives you a different mix of false positives, false negatives, build time overhead (including getting the program to build with the tool at all) and runtime overhead (including getting the program to run), not to mention availability on a given platform and pricing. And different people have a different preference with respect to what this mix should be for their tool of choice. But the problem is unsolvable which is why it's worth to be "trying everything."
- pjmlp 11y ago
  > But the problem is unsolvable which is why it's worth to be "trying everything." The problem is solvable by switching languages, but what these people all try to do, is to solve the problem while keep using the language, which is by definition unsolvable. As the original causes of the problem won't go away. But it is good, if in the end at least helps to improve the overall situation.
  - themartorana 11y ago
    Switching languages may solve one problem while introducing another. If it were so simple, no one would use Python.
- jboy 11y ago
  As a longtime programmer of Python, C & C++ (and a longtime appreciator of your FQA Lite, by the way), my preference until recently was also to use CPython & fall back on extension modules for speed. But I grew weary of writing all the Python C-API boilerplate -- especially checking parameter types and managing reference counts properly. So I created (and recently open-sourced) Pymod, a Nim+Python project that auto-generates all the Python C-API boilerplate & auto-compiles a Python C extension module, to wrap the functions in a Nim module: https://github.com/jboy/nim-pymod https://github.com/jboy/nim-pymod As a pragmatic Python programmer, Pymod may be of interest to you too... (As someone who learned C++ in 1999 and spent most of 200x working in C++, I also grew weary of staying on top of C++0x's new features & new gotchas. So about a year ago, I switched from C++ to Nim-lang and haven't looked back. To me, Nim combines all the best features of Python & C++, while avoiding most of the worst: C++ speed, C++ static types, C++ generics, C++ operator overloading; combined with a clean Pythonic syntax & Lispy macros.)
  - maxerickson 11y ago
    Cython (and before it Pyrex) occupies a similar space between C and Python. Was any of your design inspired by them? I also wonder how practical it would be to reuse parts of it when making a bridge for another language.
detaro 11y ago
Each of these projects has different goals if you look a bit more closely than "make python faster" and has made design decisions and trade-offs accordingly, and as a result each has use-cases it is better suited for than others.
maxerickson 11y ago
Nuitka for anybody searching. There is also Shedskin and Numba. It'd be nice if this project mentioned why they decided not to continue/enhance Shedskin. I think another possible explanation is that compiling a subset of Python isn't such a huge project. Only a couple of the projects mentioned are anywhere close to even just supporting the standard library that ships with CPython.

toolslive 11y ago

But does it retain the python semantics? for example: def fac(n): if n == 0 : return 1 else: return n * fac(n-1) print fac(100) will yield a big string, while any naive C++ translation will print the string "0" due to overflow.

e12e 11y ago
Good question (and I don't know the answer, as I'm not familiar with this project). Note that a naive c++ translation will also error out on simply i=2 * * 524. Automatic extension/promotion of numbers is pretty fundamental to python.

nickcw 11y ago

I just wondered if anyone had any insight in how this differs from shedskin - https://code.google.com/p/shedskin/ https://code.google.com/p/shedskin/ From the shedskin website: Shed Skin is an experimental compiler, that can translate pure, but implicitly statically typed Python (2.4-2.6) programs into optimized C++. It can generate stand-alone programs or extension modules that can be imported and used in larger Python programs.

drdaeman 11y ago
Don't know about the differences, but I think shedskin had died some years ago.
- stuaxo 11y ago
  Nope, they are still responsive on the bug list, I think it lives - just very slowly.