3 ms·

And that's what we did during our first pass: identify the slow parts and port them to C. This gave us about 30% better performance. Without getting too much in

And that's what we did during our first pass: identify the slow parts and port them to C. This gave us about 30% better performance. Without getting too much into the nitty gritty of our specific case, this wasn't enough. We needed some massive speed improvements, I'm talking in the order of 100X faster. The nature of our algorithms was such that they could be done in parallel (i.e map/reduce) - an ideal candidate for Go's goroutines. We actually tried to make it parallelizable at first in Python, using gevent (and even just multiprocessing) and the results were not great. One other aspect that really guided us towards Go was memory usage. Python was just sucking up so much memory whereas our Go implementation thus far is so much thinner.

danieldk 14y ago
One other aspect that really guided us towards Go was memory usage. Python was just sucking up so much memory whereas our Go implementation thus far is so much thinner. It's funny that you mention that, because in computationally intensive work memory management is an issue with Go. E.g. I wrote a maximum entropy parameter estimator in Go, which was terribly slow until I circumvented the garbage collector by preallocating a huge block of memory and doing my own management. In C malloc() and free()-ing had practically no overhead. After putting the Go garbage collector out of the game, the Go version was approximately within 2x of the C version. I am interested how Go gave you one or two orders of magnitude speedup, while rewriting hot spots in C didn't...
- Ziomislaw 14y ago
  > I am interested how Go gave you one or two orders of magnitude speedup, while rewriting hot spots in C didn't... paralellization
  - danieldk 14y ago
    One can easily do that in C with OpenMP.
    - Symmetry 14y ago
      If you can get away with only parallelizing the C code that'll work, but if you're replacing individual Python functions calls it won't so much. You could use a lot of Python processes, but that might be awkward depending on how much they need to coordinate and might lead to too much memory usage.

danieldk 14y ago

One other aspect that really guided us towards Go was memory usage. Python was just sucking up so much memory whereas our Go implementation thus far is so much thinner. It's funny that you mention that, because in computationally intensive work memory management is an issue with Go. E.g. I wrote a maximum entropy parameter estimator in Go, which was terribly slow until I circumvented the garbage collector by preallocating a huge block of memory and doing my own management. In C malloc() and free()-ing had practically no overhead. After putting the Go garbage collector out of the game, the Go version was approximately within 2x of the C version. I am interested how Go gave you one or two orders of magnitude speedup, while rewriting hot spots in C didn't...

Ziomislaw 14y ago
> I am interested how Go gave you one or two orders of magnitude speedup, while rewriting hot spots in C didn't... paralellization
- danieldk 14y ago
  One can easily do that in C with OpenMP.
  - Symmetry 14y ago
    If you can get away with only parallelizing the C code that'll work, but if you're replacing individual Python functions calls it won't so much. You could use a lot of Python processes, but that might be awkward depending on how much they need to coordinate and might lead to too much memory usage.