5 ms·

Nice! It looks pretty simple, but it makes me wonder. Why bother interfacing your Python code with Go? Why not go straight to C for more performance?

bvdbijl 14y ago
Because Go is way more pleasant to write and is still very performant
pavanky 14y ago
1. Because it can be done. 2. The native way of doing this in C (without using SWIG or cython or any other such)is more verbose. 3. It may be easier (not necessarily faster) to write certain code in Go as opposed to both python or C.
- rattray 14y ago
  Would Go's static binaries be an added advantage?
  - tiredofcareer 14y ago
    No, because in Python's interop model, importing a compiled module involves a shared object (so you don't have to relink the interpreter to add one).
    - cdavid 14y ago
      strictly speaking, it is not required: you can build a giant python binary with all the extensions you need, though it can be quite a bit of work to do so. Granted, the vast majority of users use the loadable module approach.
Laremere 14y ago
Python doesn't do any parallel computation. Go does it very well. I honestly don't know how to do parallel computation in C or C++, but I doubt it's very easy. Slower code can run faster overall if you have multiple processors, so pure processing speed has the potential to matter less.
- danieldk 14y ago
  I honestly don't know how to do parallel computation in C or C++, but I doubt it's very easy. Actually, with OpenMP parellelization is often easy in C and C++. In embarrassingly parellel problems, it is often a matter of adding an OpenMP pragma before the for-loop. To give one example, in some machine learning code I wrote, it was simply a matter of replacing for (int i = 0; i < static_cast<int>(contexts.size()); ++i) by: #pragma omp parallel for for (int i = 0; i < static_cast<int>(contexts.size()); ++i) Obviously, you have to ensure that shared mutable variables are properly protected. But, in many cases it's fairly simple. I have tried to emulate some of OpenMP's functionality in the par package: http://github.com/danieldk/par http://github.com/danieldk/par I think the more convincing argument is that many Python programmers will feel far more comfortable writing Go than C or C++. It offers good performance and some control over memory layout, while being substantially higher level than C and not the enormous language C++ has become.
  - burntsushi 14y ago
    > In embarrassingly parellel problems To be fair, it is trivial to parallelize embarrassingly parallel problems in most languages. Including Python (with `multiprocessing`). Go separates itself from C/C++ by making concurrent programming much easier. Namely, channels are the predominant form of both communication and synchronization between concurrent processes. This removes much of the need for locks to protect state, which is a source of common bugs. Go goes an extra step with M:N scheduling, but that is orthogonal to concurrent programming. Also, you might be interested in my type parametric version of `ParMap` [1]. It loses compile time type safety, but performance still seems to remain roughly the same. [1] - http://godoc.org/github.com/BurntSushi/ty/fun#ParMap http://godoc.org/github.com/BurntSushi/ty/fun#ParMap
    - danieldk 14y ago
      > To be fair, it is trivial to parallelize embarrassingly parallel problems in most languages. Including Python (with `multiprocessing`). Of course, but usually you are switching to C because the code that is being parallelized needs to be faster. Go separates itself from C/C++ by making concurrent programming much easier. Right. I was reacting to the grandparent, who mentioned parallel computation, not concurrency. Also, you might be interested in my type parametric version of `ParMap` [1]. Nice :). Though my Haskell heart cries when people use interface{} plus reflection as a replacement for parametric polymorphism. Before we know it the library stack will effectively turn Go into a dynamically typed language.
dvirsky 14y ago
outsourcing parallel work to goroutines? yes please!

bvdbijl 14y ago

Because Go is way more pleasant to write and is still very performant

pavanky 14y ago

1. Because it can be done. 2. The native way of doing this in C (without using SWIG or cython or any other such)is more verbose. 3. It may be easier (not necessarily faster) to write certain code in Go as opposed to both python or C.

rattray 14y ago
Would Go's static binaries be an added advantage?
- tiredofcareer 14y ago
  No, because in Python's interop model, importing a compiled module involves a shared object (so you don't have to relink the interpreter to add one).
  - cdavid 14y ago
    strictly speaking, it is not required: you can build a giant python binary with all the extensions you need, though it can be quite a bit of work to do so. Granted, the vast majority of users use the loadable module approach.

Laremere 14y ago

Python doesn't do any parallel computation. Go does it very well. I honestly don't know how to do parallel computation in C or C++, but I doubt it's very easy. Slower code can run faster overall if you have multiple processors, so pure processing speed has the potential to matter less.

danieldk 14y ago
I honestly don't know how to do parallel computation in C or C++, but I doubt it's very easy. Actually, with OpenMP parellelization is often easy in C and C++. In embarrassingly parellel problems, it is often a matter of adding an OpenMP pragma before the for-loop. To give one example, in some machine learning code I wrote, it was simply a matter of replacing for (int i = 0; i < static_cast<int>(contexts.size()); ++i) by: #pragma omp parallel for for (int i = 0; i < static_cast<int>(contexts.size()); ++i) Obviously, you have to ensure that shared mutable variables are properly protected. But, in many cases it's fairly simple. I have tried to emulate some of OpenMP's functionality in the par package: http://github.com/danieldk/par http://github.com/danieldk/par I think the more convincing argument is that many Python programmers will feel far more comfortable writing Go than C or C++. It offers good performance and some control over memory layout, while being substantially higher level than C and not the enormous language C++ has become.
- burntsushi 14y ago
  > In embarrassingly parellel problems To be fair, it is trivial to parallelize embarrassingly parallel problems in most languages. Including Python (with `multiprocessing`). Go separates itself from C/C++ by making concurrent programming much easier. Namely, channels are the predominant form of both communication and synchronization between concurrent processes. This removes much of the need for locks to protect state, which is a source of common bugs. Go goes an extra step with M:N scheduling, but that is orthogonal to concurrent programming. Also, you might be interested in my type parametric version of `ParMap` [1]. It loses compile time type safety, but performance still seems to remain roughly the same. [1] - http://godoc.org/github.com/BurntSushi/ty/fun#ParMap http://godoc.org/github.com/BurntSushi/ty/fun#ParMap
  - danieldk 14y ago
    > To be fair, it is trivial to parallelize embarrassingly parallel problems in most languages. Including Python (with `multiprocessing`). Of course, but usually you are switching to C because the code that is being parallelized needs to be faster. Go separates itself from C/C++ by making concurrent programming much easier. Right. I was reacting to the grandparent, who mentioned parallel computation, not concurrency. Also, you might be interested in my type parametric version of `ParMap` [1]. Nice :). Though my Haskell heart cries when people use interface{} plus reflection as a replacement for parametric polymorphism. Before we know it the library stack will effectively turn Go into a dynamically typed language.

dvirsky 14y ago

outsourcing parallel work to goroutines? yes please!