7 ms·
GoPy – Extend CPython by writing Go
- Goranek 14y agoI would love to see some benchmarks here! Great idea!
- danieldk 14y agoNice! I assume this only works with gccgo (to create a shared library)?
- levesque 14y agoNice! It looks pretty simple, but it makes me wonder. Why bother interfacing your Python code with Go? Why not go straight to C for more performance?
- bvdbijl 14y agoBecause Go is way more pleasant to write and is still very performant
- pavanky 14y ago1. Because it can be done. 2. The native way of doing this in C (without using SWIG or cython or any other such)is more verbose. 3. It may be easier (not necessarily faster) to write certain code in Go as opposed to both python or C.
- rattray 14y agoWould Go's static binaries be an added advantage?
- tiredofcareer 14y agoNo, because in Python's interop model, importing a compiled module involves a shared object (so you don't have to relink the interpreter to add one).
- cdavid 14y agostrictly speaking, it is not required: you can build a giant python binary with all the extensions you need, though it can be quite a bit of work to do so. Granted, the vast majority of users use the loadable module approach.
- Laremere 14y agoPython doesn't do any parallel computation. Go does it very well. I honestly don't know how to do parallel computation in C or C++, but I doubt it's very easy. Slower code can run faster overall if you have multiple processors, so pure processing speed has the potential to matter less.
- danieldk 14y agoI honestly don't know how to do parallel computation in C or C++, but I doubt it's very easy. Actually, with OpenMP parellelization is often easy in C and C++. In embarrassingly parellel problems, it is often a matter of adding an OpenMP pragma before the for-loop. To give one example, in some machine learning code I wrote, it was simply a matter of replacing for (int i = 0; i < static_cast<int>(contexts.size()); ++i) by: #pragma omp parallel for for (int i = 0; i < static_cast<int>(contexts.size()); ++i) Obviously, you have to ensure that shared mutable variables are properly protected. But, in many cases it's fairly simple. I have tried to emulate some of OpenMP's functionality in the par package: http://github.com/danieldk/par http://github.com/danieldk/par I think the more convincing argument is that many Python programmers will feel far more comfortable writing Go than C or C++. It offers good performance and some control over memory layout, while being substantially higher level than C and not the enormous language C++ has become.
- burntsushi 14y ago> In embarrassingly parellel problems To be fair, it is trivial to parallelize embarrassingly parallel problems in most languages. Including Python (with `multiprocessing`). Go separates itself from C/C++ by making concurrent programming much easier. Namely, channels are the predominant form of both communication and synchronization between concurrent processes. This removes much of the need for locks to protect state, which is a source of common bugs. Go goes an extra step with M:N scheduling, but that is orthogonal to concurrent programming. Also, you might be interested in my type parametric version of `ParMap` [1]. It loses compile time type safety, but performance still seems to remain roughly the same. [1] - http://godoc.org/github.com/BurntSushi/ty/fun#ParMap http://godoc.org/github.com/BurntSushi/ty/fun#ParMap
- dvirsky 14y agooutsourcing parallel work to goroutines? yes please!
- jff 14y agoNice! Does this let you get around the GIL?
- mmariani 14y agoI believe so, at least you can in C. You have to release the GIL in your extension, do your business, and cleanup before giving the control back to Pyhton. http://docs.python.org/3.3/c-api/init.html http://docs.python.org/3.3/c-api/init.html Edit: Even though you can do it in C, Go is a no go. Under item four in the caveats [1] is stated: | When using goroutines py.Lock must be used. No other | GIL or threading interface functions have been | tested - and they most likely will not work. [1] http://gopy.qur.me/extensions/ http://gopy.qur.me/extensions/
- tiredofcareer 14y ago> Calling Go code from a second thread will currently cause the code to raise an exception (i.e. all calls into Go code must be made from the same thread) That's ... interesting.
- georgemcbay 14y agoI haven't looked at this much (not much of a Python user these days) but presumably you can still multithread your Go code using Go semantics regardless of this restriction. The Go code you call from Python should be able to create goroutines as needed which then report back to the main Go thread as needed, as long as they follow the Go conventions about pointer ownership (channel receiver owns the memory sent to it). Of course it would be nicer if the main Go thread didn't lock out the Python caller from running (which it seems to do), but there should still be lots of concurrency available on the Go side.
- d0mine 14y agoyes. See lock.UnblockThreads(); in the parallel example from the article (the sleep is executed without holding GIL).
- rattray 14y agoThis looks fantastic. Having only worked extensively with Python and Javascript, I haven't been looking forward to learning C or C++ to write performant bits of an application, but Go really excites me. What are the limitations? Are there situations where GoPy won't work that (eg) ctypes would?
- toyg 14y agoThe main limitation atm is that it looks like a very experimental project, nowhere near production-quality.
- pjmlp 14y agoFor new code, I would rather use Julia.
- nnq 14y ago...was thinking of the same thing. Julia seems much nicer for the use case of writing scicomp code, but Go overall will be a much more popular language. How would you choose one of them for writing performance tuned Python extensions?
- pjmlp 14y agoIf it is for writing extensions, then PyPy would be a better option if it is to stay in Python land. For writing new code from scratch, Julia is a much better option. The language is not yet 1.0 and already achieves parity to C compilers in many algorithms. Go compilers are yet to match JVM and .NET speeds, let alone C.
- codygman 14y agoThanks, I hadn't previously heard of Julia.
- pjmlp 14y agoYou will probably find comments from me here dismissing the language when it was initially announced, but after watching the MIT presentations on its current state I think the language has a future in technical computing. http://julialang.org/blog/2013/03/julia-tutorial-MIT/ http://julialang.org/blog/2013/03/julia-tutorial-MIT/
- codygman 14y agoI respectfully disagree with some of your other opinions, but I think you might be on to something with Julia :D
- seunosewa 14y agoWhen can we expect a Windows port?
- scrambam 14y agololz internet
- apendleton 14y agoThis looks really slick, but it's a bit unfortunate that this was implemented in a way that's tied to the CPython API specifically, given that there are Python-VM-neutral methods of low-level interop available (ctypes, CFFI, etc.). My general sense (and certainly my own experience) has been that many people doing speed-sensitive Python work are already running on PyPy now, and that writing lower-level extensions has become a way to squeeze out an extra level of performance beyond that, so having to go back to CPython to take advantage of this infrastructure would be kind of a bummer, and almost definitely a non-starter in my projects, which leaves me stuck with C.
- masklinn 14y ago> My general sense (and certainly my own experience) has been that many people doing speed-sensitive Python work are already running on PyPy now Depends, an other category drops down to Cython (which in many cases of low type dynamism consists of just running Python code through Cython).
- apendleton 14y agoRight, I definitely don't think PyPy+[ctypes|cffi] is universal ("many," not "all"). Lots of people are doing Cython. Also, anyone who has a performance-sensitive workload that depends on SciPy is out of luck on PyPy at least for the time being. All I'm saying is that PyPy as a performant-Python strategy is common enough now that it's worth supporting in new performant-Python projects.
- fixxer 14y agoMy thoughts exactly. Great concept and I will definitely follow it as it develops, but I'd love to see an abstraction layer that takes the pain out of the a$$ that is the Python C-api. I left that years ago and I'm not going back.
- dman 14y agoIve been using the Python C api recently and I find it the most straight forward option of getting performance out of Python. If you write a few helper classes in C++ using RAII - the python c-api becomes actually pretty pleasant to use.
- SatyajitSarangi 14y agoThis looks brilliant. My only concern being, consdering an extra layer comes in, how much is the performance affected by this?