8 ms·
Pythran: a Python to C++ compiler with a focus on scientific computing
- akandiah 10y agoThe problem with a lot of these optimizing tools for Python is that they only support Python 2.7 not Python 3.
- njbooher 10y agoNot really a problem here as most of science still seems to be using Python 2. edit: Cython was always great for Python 2 and apparently supports 3. http://cython.org http://cython.org
- ericfrederich 10y agoYeah. I came across C library that used Cython to create Python bindings. It was funny, they could easily support both Python 2 and Python 3 at the same time since all their binding code was in Cython, it was only their installation script witch used an incompatibility (print statement)
- brian_herman 10y agoWhat was the library?
- shoyer 10y agoNumba also supports Python 3.
- Demiurge 10y agoAs a happy Python user for 10 years, I didn't even wonder whether it supports python3. Maybe the problem is somewhere else, and not with the library I'm looking forward to trying?
- nerdponx 10y agoAre you insinuating that the problem is with Python 3? What is actually so much more amazing about Python 2 that you would rather freeze the language than upgrade?
- kolokolo 10y agoIt ain't broken. Why fix it?
- witty_username 10y agoPython 3 has many new features and fixes. Maybe 5 years ago this was true, but not today.
- Nullabillity 10y agoAnd how many of them can't be backported to 2? You don't need to freeze the language, but if the alternative is breaking backwards compatibility after all that time...
- snovv_crash 10y agoWhat is there in 3 that would benefit scientific computing which 2.7 doesn't have? TBH I don't see the value proposition. For web stuff you have asyncio, and for hackers there is the venv improvements, but what is there for scientific users who have work to get done?
- bmer 10y agoI use Python for my research work...and this has never been a problem for me, I suppose because I use Python 2.7, and I guess I am "part of the problem"?
- frozenport 10y agoYes you are.
- labrador 10y agoYou have the problem, not bmer.
- goatlover 10y agoThe problem is not upgrading after years of 3 being out.
- snovv_crash 10y agoWhat if you don't care about the version of python, but you have a bunch of code that runs on 2.7. You don't get anything from moving to 3, you have to learn the new syntax features (friction), you need to upgrade all you code (friction) and some of the libraries you used before don't work and need fixing (friction). If you don't use asyncio, and you already had venv working, what do you gain? For someone to whom python is just a fancy pipe-wrench, why keep up with the new shiny?
- jjawssd 10y agoYou are right. Just like we should all just be using Word 97 on Windows 95 because, you know, word processing is just word processing in the end, isn't it?
- dagw 10y agoI take it you don't remember the Office 2000 -> 2003 upgrade debacle and peoples unwillingness to just get on board. Most people I know simply skipped 2003 and only reluctantly hopped to 2007 after much gnashing of teeth. I still know a handful of people still using Word 2000.
- mrweasel 10y agoI never looked into, but I wonder if there's something in Python 3 that makes it hard to do these kind of things to, compared to Python 2.7. Python 3 is the platform I use for almost everything, going back to 2.7 is always a major pain. 10 minutes into some 2.7 work and you have to deal with encoding and you start wondering if it would be quicker to just upgrade to 3.5. Of cause if you mainly do numbers, there's perhaps little gained from Python 3.
- Arkanosis 10y agoCan't talk for Pythran devs and others, but in the case of shedskin, one of the issues is the removal of the “compiler” module in Python 3, on which shedskin relies to, well, parse the Python code. This, plus some syntax changes, that are not really bigger that some that were introduced between, say, 2.6 and 2.7, but that are mandatory and breaking backward compatibility. So, nothing “hard”, really, you just have to do the work again (and prepare to support two versions for a long time, instead of virtually only one).
- m_mueller 10y agoWhat is the advantage of doing this compared to writing Fortran (90+) kernels, use F2Py bindings and combining them with NumPy? For HPC you generally need to be able to have full control over compiler flags. Plus, Fortran already has way better multi-dim array handling built-in - NumPy is basically just a wrapper around that.
- greglindahl 10y agoI'm a huge Fortran fan, but even I can see that reducing the number of languages required to get high performance can be a good thing. Not all of my science colleagues are interested in learning a ton of languages.
- m_mueller 10y agoWell, most of the scientific computing in Python is happening on NumPy. Which uses Fortran compatible data structures. I doubt that a C++ backed python extension will be compatible with that. For that reason, if anything, I'd advocate to transpile to Fortran rather than C++ if you want more comfort for scientific users.
- shoyer 10y agoNumPy natively supports both row-major (C/C++) and column-major (Fortran) arrays, so from the perspective of NumPy inter-operability there would be no advantage either way.
- m_mueller 10y agoI didn't know that. How does NumPy use its Fortran kernels with row-major arrays? Reordering would have quite significant performance implications.
- shoyer 10y agoNumPy has an abstract iterator interface so you don't need to write a kernel with a specific memory layout in mind: http://docs.scipy.org/doc/numpy-1.11.0/reference/c-api.iterator.html#simple-iteration-example http://docs.scipy.org/doc/numpy-1.11.0/reference/c-api.itera... For operations like adding two arrays or summing an array along an axis, this means that NumPy automatically iterates in the fastest order.
- vonnik 10y agoReminds me of JavaCPP: https://github.com/bytedeco/javacpp https://github.com/bytedeco/javacpp
- nerdponx 10y agoWhat does this do that I can't already do with some combination of Numpy, Cython, and Numba?
- bmer 10y agoI use numba quite a bit, and with every new version, the developers have notes like: PR #1970: Optimize the power operator with a static exponent. PR #1961: Support printing constant strings. PR #1927: Support np.linalg.lstsq. My point being that the developers have to go about re-implementing (in LLVM IR?) features that are very mature in C libraries (e.g. numpy) and similarly C++ libraries. Could it be that one gain of going from Python to C++ will be that one could take advantage of all the mature libraries that C++ has access to, without having to do this re-inventing of the wheel? I am not sure, and regardless, I'll be using numba for a long time coming, because...I like the devs, they work hard on it, and numba works like a charm for what I need to do, so I'm not looking for an overkill.
- nerdponx 10y agoThat makes sense. Numba aside, though, it's strange to me that I see no mention of Cython on their project page. They ought to at least acknowledge that other fairly mature products exist in this space. Also, maybe it's more of a curiosity for you than a practical tool, but Stan (which is becoming the de-facto standard DSL for Bayesian statistics) compiles to C++, where it takes advantage of libraries like Eigen and Boost.
- bmer 10y agoThat's a really good point---Cython doesn't have the same issues, it's basically Pythran except it compiles to C. Yeah, no clue.
- _ZeD_ 10y agoWhat's the difference between this and nuitka?
- dagw 10y agoThe big one seems to be that pythran is targeting only a subset of python with the goal of getting faster performance, while Nuitka is aims at being able to compile all valid python programs (currently only up to python 3.4), but in doing so it has fewer opportunities for optimization and thus gives you slower performance than pythran. Pythran also introduces its own type annotation system you have to use if you want to get good performance. So If you write a python program that is compatible with pythran, then pythran will almost certainly be much faster than nuitka. If however you take a random python program it will almost certainly compile with Nuitka and almost certainly not compile with pythran.
- jsjsjsjsjsjs 10y agoNo idea why they could not contribute some patches to nuitka to make it do exactly that. Because further down the road it is the goal of nuitka - to natively compile as much as possible.
- guyzmo 10y agoWel, too bad they did not implement this with python 3, and actually gave some meaning to the type annotations… I mean def dprod(l0: list[int], l1: list[int]): return sum(x * y for x, y in zip(l0, l1)) Wouldn't it be much nicer?
- srean 10y agoTo be brutally honest, they probably couldn't give a flying fuck, I couldn't when I tried. (sorry) To the number crunchers, Python 3.* offers no compelling reason to move (except for threats, "you are on your own now, 2.7.* will not get updates"). I see two compelling advantages that 3.* offers (i) better abstractions for asynchrony (ii) UTF strings (that exacts a cost even if you don't need them). If you don't need these two there I see no technical reason to move (political/social reasons yes, technical not really).
- module0000 10y agoFor what it's worth, Python 3 does offer at least 1 compelling reason to move. Multiprocessing lib in python 3 is significantly faster than python 2. My use case is spawning 30-35 worker processes managed by a pool, and recycling them after 5 workloads pass through. The switch from 2.x to 3.x without any code changes, increased my throughput ~150%. Your mileage may vary. edit: typo
- srean 10y agoOh! thanks for this, I did not know.
- leaningtower 10y agoAs I once discussed with a friend of mine, once you remove semicolons, and give spacing a syntactic and semantic value, you almost have FORTRAN... this is just the icing on the cake :¬)
- nhamausi 10y agoWondering how does it compare to Cython?
- hairy_man674 10y agoSo we have CPython which has an ffi and a native implementation of the source interpreter. We have PyPy which is JIT for Python that emits assembly at runtime though not as successful Cpython for its support. The complexity class of this problem, without type annotations, namely completely transforming an arbitary Dynamic Python program to a static, partially inferred typed language (C++) is O(N): https://en.m.wikipedia.org/wiki/Hindley–Milner_type_system https://en.m.wikipedia.org/wiki/Hindley–Milner_type_system The transformations are also not guaranteed to improve runtime performance and might even do the opposite or have side effects for precision and a program that is slower due to generalizations that an algorithm makes. With type annotations... why, just why? Use C++ or C if you want types and performance gains thereof..nThe potential for bugs and failures is less than relying on an algorithm if you realise the fear and laziness of compiled languages is umfounded.
- toolslive 10y ago(updated, see comment below) There used to be another python to c++ compiler called "shed skin" it did quite a literal translation, fe python classes would turn into c++ classes and it was quite readable..... but utterly useless: a python integer would be translated into a c++ int. Where python automatically starts using a wider type this one would just wrap.
- sitkack 10y agoCompare to https://github.com/shedskin/shedskin https://github.com/shedskin/shedskin which does whole program type flow analysis. Shedskin offers excellent performance on numerical workloads.
- madsohm 10y agoWe do something similar with Bohrium[1]. Bohrium intercepts your NumPy calls and compile a kernel for either CPU, Multicore CPU, or GPU and run that instead. It will also fuse together all NumPy calls until one has a side-effect, that the run-time needs. We are currently building smart filters into it as well, so that we can do byte-code optimizations. [1] https://github.com/bh107/bohrium https://github.com/bh107/bohrium