5 ms·

I found an interesting subtlety to the message here. They had a Python wrapper around a C library, and effectively never updated the library. I wonder if this w

I found an interesting subtlety to the message here. They had a Python wrapper around a C library, and effectively never updated the library. I wonder if this was due to engineering decisions, hiring/expertise of their staff (or future hires), or something else (risk? etc). I imagine it is easier these days to find Go devs than C devs who want to work at a company like this? Also, for a sense of scale, this rewrite took about 4 years to complete from first concepts.

sho 7y ago
For 9 out of 10 articles like this - "We rewrote our old code in tired old language FOO with hot new language BAR and it's SO MUCH BETTER!" - I suspect they could have simply rewritten with the same language, engineered it better and applied all the lessons learned, and achieved basically the same result, probably faster. But that doesn't scratch the "new hotness" itch, does it? That said, considering the requirements this thing sounds like it had to meet, perhaps golang was the right choice after all. Unlike certain other projects I know about where they HAD to write everything in golang/microservices/k8s/etc because it HAD TO SCALE, took 18 months instead of the 3 months it would have taken with rails but credit where credit is due - those 2 or 3 requests a minute (peak) are handled very, very quickly.
- Barrin92 7y ago
  > That said, considering the requirements this thing sounds like it had to meet, perhaps golang was the right choice after all. Yes, in this particular case I don't think 'the new hotness syndrome' applies. The performance limits of python given that it's dynamic and the multithreading problems are fundamental to the language.
  - pjmlp 7y ago
    The same old theme. Common Lisp, Dylan, Smalltalk, JavaScript are just as dynamic as Python and yet run circles around its performance. The only thing lacking is more community love for PyPy and similar endeavours.
    - Barrin92 7y ago
      Sure it's true that if you go with one of the compiled versions of dynamic languages you're going to achieve speedups. And in particular in CL some of those can be achieved by making your programs less dynamic (inlining, early binding etc..), but it is in the nature of dynamic programs to be less optimizable than static programs. While interpreted python code is probably as slow as you can get, Go is still ahead of javascript, and Java beats Smalltalk, and something like C/C++ will take you even further. Implementing compilers for dynamic languages is no trivial feat. The amount of labour that went into something like V8 to get it to its current performance is astonishing.
- sidlls 7y ago
  That may be true in many cases, but this article really resonated with me. I spent more years than I want to admit working on a large machine learning and analytics platform that was originally built with Python. It was, frankly, a nightmare. The dependency problems, the performance issues, the problems with typing and use of data structures were all huge headaches to manage. I have a blog in the engineering organization I work in where I'm basically publishing a series of articles that reflect a similar path the Salesforce article describes, for basically the same reasons. (My group is using Java, not Go, though.)
  - blub 7y ago
    You're doing ML in Java? :) What are the equivalents of some of the popular Python libs for that?
    - roseway4 7y ago
      DL4J and the ND4J library are a good start and at a high-level offer equivalent functionality to tensorflow, pytorch, and pandas: https://github.com/eclipse/deeplearning4j https://github.com/eclipse/deeplearning4j
- fiedzia 7y ago
  >I suspect they could have simply rewritten with the same language, engineered it better and applied all the lessons learned No, you can't do that. There are many benefits that come from underlying technology choices that you can't backport to old stack, and if you try, you'd create a monster nobody would like to touch. There might be fewer developers willing to work with Elm than something else, there are none willing to work with "we rewrote javascript standard libraries so that it kind of resembles Elm, but poorly".
bsg75 7y ago
It is also easier for Python devs to pick up Go and become productive than with C. I suspect SF had more people that could move quickly from Python to Go than those who could add features to the C lib. This is from personal experience where Go is used when performance is needed, and the end result is a single program, vs a C lib and a Python wrapper, gaining some level of simplicity along with the needed performance. We don't get quite the performance of C, but we also don't need to include memory management. Sort of a happy medium between Python and C.
robbrit 7y ago
> it is easier these days to find Go devs than C devs Not really actually, there are still loads of C devs (or C++ at least) and a lot more jobs in C and C++ than for Go. Having made engineering decisions between C++ and Go, my key reasons for picking Go over C++ are: * Simplicity when multi-threading * It's much easier for someone who knows neither language to become productive in a professional environment in Go than in C/C++ * Fast compile times - no more typing "make" and then going off to get a coffee * Lots of modern niceties that are more fragmented in the C world: third-party vendoring, unit testing, style guides, etc.
- blub 7y ago
  Most C++ devs won't want to touch C and being the performance person in a Python project sounds like a sideshow job. Mixing C and C++ like you do makes little sense, they're very different.
  - robbrit 7y ago
    Yep they are two very different languages. However with the possible exception of the build times argument, the points for choosing Go over C++ also apply to Go over C.
scythe 7y ago
So I worked on this project. As with many stories this one has been simplified to be less boring and emphasize key points. I generally agree with Guillaume’s characterization—he’s a brilliant engineer and it was great to work with him. The most important advantage of Go was that it is statically typed and there was a broad agreement that we should use a typed language to prevent errors. The other advantage was that Go is faster than Python, allowing us to move logic out of the C codebase and unify the high-level operations in a single codebase. The other other advantage of Go was that it is not C++, for which we were all grateful. As with most real projects, the rewrite occurred concurrently with adding features and tightening coding standards — `go fmt` was a big motivator and the Go codebase had much more comprehensive test coverage than the Python one. >Also, for a sense of scale, this rewrite took about 4 years to complete from first concepts. That’s not quite accurate. In fact the Python production code was largely ported by mid-2015, and the Go library was feature-complete by 2016, albeit still linking to C. However, there was a lot of non-production prototype code, implementing more complex optimizations and written in a few additional languages, which took another two years to port, and also to be made production-ready. Perhaps what’s important is that we were able to keep adding new improvements while still “porting” this code. (Real life is a poor laboratory.) >effectively never updated the library. It is, or was, still maintained. Most of the very low level file operations had been optimized to death already. Higher-level functionality was slowly moved out of the C library. I left Salesforce in 2017 so I’m not sure what happened next. (Incidentally I left not only Salesforce but tech entirely - I’m now at grad school for medical physics.) >I imagine it is easier these days to find Go devs than C devs Maybe? None of us knew Go when the porting began. I think if you know at least 2 programming languages and one of those is C-like, Go will be a piece of cake. I was hired with no Go experience and ramped up quickly.
- derivagral 7y ago
  Thanks for the detailed reply! Getting a real perspective instead of an article is always great.

sho 7y ago

For 9 out of 10 articles like this - "We rewrote our old code in tired old language FOO with hot new language BAR and it's SO MUCH BETTER!" - I suspect they could have simply rewritten with the same language, engineered it better and applied all the lessons learned, and achieved basically the same result, probably faster. But that doesn't scratch the "new hotness" itch, does it? That said, considering the requirements this thing sounds like it had to meet, perhaps golang was the right choice after all. Unlike certain other projects I know about where they HAD to write everything in golang/microservices/k8s/etc because it HAD TO SCALE, took 18 months instead of the 3 months it would have taken with rails but credit where credit is due - those 2 or 3 requests a minute (peak) are handled very, very quickly.

Barrin92 7y ago
> That said, considering the requirements this thing sounds like it had to meet, perhaps golang was the right choice after all. Yes, in this particular case I don't think 'the new hotness syndrome' applies. The performance limits of python given that it's dynamic and the multithreading problems are fundamental to the language.
- pjmlp 7y ago
  The same old theme. Common Lisp, Dylan, Smalltalk, JavaScript are just as dynamic as Python and yet run circles around its performance. The only thing lacking is more community love for PyPy and similar endeavours.
  - Barrin92 7y ago
    Sure it's true that if you go with one of the compiled versions of dynamic languages you're going to achieve speedups. And in particular in CL some of those can be achieved by making your programs less dynamic (inlining, early binding etc..), but it is in the nature of dynamic programs to be less optimizable than static programs. While interpreted python code is probably as slow as you can get, Go is still ahead of javascript, and Java beats Smalltalk, and something like C/C++ will take you even further. Implementing compilers for dynamic languages is no trivial feat. The amount of labour that went into something like V8 to get it to its current performance is astonishing.
sidlls 7y ago
That may be true in many cases, but this article really resonated with me. I spent more years than I want to admit working on a large machine learning and analytics platform that was originally built with Python. It was, frankly, a nightmare. The dependency problems, the performance issues, the problems with typing and use of data structures were all huge headaches to manage. I have a blog in the engineering organization I work in where I'm basically publishing a series of articles that reflect a similar path the Salesforce article describes, for basically the same reasons. (My group is using Java, not Go, though.)
- blub 7y ago
  You're doing ML in Java? :) What are the equivalents of some of the popular Python libs for that?
  - roseway4 7y ago
    DL4J and the ND4J library are a good start and at a high-level offer equivalent functionality to tensorflow, pytorch, and pandas: https://github.com/eclipse/deeplearning4j https://github.com/eclipse/deeplearning4j
fiedzia 7y ago
>I suspect they could have simply rewritten with the same language, engineered it better and applied all the lessons learned No, you can't do that. There are many benefits that come from underlying technology choices that you can't backport to old stack, and if you try, you'd create a monster nobody would like to touch. There might be fewer developers willing to work with Elm than something else, there are none willing to work with "we rewrote javascript standard libraries so that it kind of resembles Elm, but poorly".

bsg75 7y ago

It is also easier for Python devs to pick up Go and become productive than with C. I suspect SF had more people that could move quickly from Python to Go than those who could add features to the C lib. This is from personal experience where Go is used when performance is needed, and the end result is a single program, vs a C lib and a Python wrapper, gaining some level of simplicity along with the needed performance. We don't get quite the performance of C, but we also don't need to include memory management. Sort of a happy medium between Python and C.

robbrit 7y ago

> it is easier these days to find Go devs than C devs Not really actually, there are still loads of C devs (or C++ at least) and a lot more jobs in C and C++ than for Go. Having made engineering decisions between C++ and Go, my key reasons for picking Go over C++ are: * Simplicity when multi-threading * It's much easier for someone who knows neither language to become productive in a professional environment in Go than in C/C++ * Fast compile times - no more typing "make" and then going off to get a coffee * Lots of modern niceties that are more fragmented in the C world: third-party vendoring, unit testing, style guides, etc.

blub 7y ago
Most C++ devs won't want to touch C and being the performance person in a Python project sounds like a sideshow job. Mixing C and C++ like you do makes little sense, they're very different.
- robbrit 7y ago
  Yep they are two very different languages. However with the possible exception of the build times argument, the points for choosing Go over C++ also apply to Go over C.

scythe 7y ago

So I worked on this project. As with many stories this one has been simplified to be less boring and emphasize key points. I generally agree with Guillaume’s characterization—he’s a brilliant engineer and it was great to work with him. The most important advantage of Go was that it is statically typed and there was a broad agreement that we should use a typed language to prevent errors. The other advantage was that Go is faster than Python, allowing us to move logic out of the C codebase and unify the high-level operations in a single codebase. The other other advantage of Go was that it is not C++, for which we were all grateful. As with most real projects, the rewrite occurred concurrently with adding features and tightening coding standards — `go fmt` was a big motivator and the Go codebase had much more comprehensive test coverage than the Python one. >Also, for a sense of scale, this rewrite took about 4 years to complete from first concepts. That’s not quite accurate. In fact the Python production code was largely ported by mid-2015, and the Go library was feature-complete by 2016, albeit still linking to C. However, there was a lot of non-production prototype code, implementing more complex optimizations and written in a few additional languages, which took another two years to port, and also to be made production-ready. Perhaps what’s important is that we were able to keep adding new improvements while still “porting” this code. (Real life is a poor laboratory.) >effectively never updated the library. It is, or was, still maintained. Most of the very low level file operations had been optimized to death already. Higher-level functionality was slowly moved out of the C library. I left Salesforce in 2017 so I’m not sure what happened next. (Incidentally I left not only Salesforce but tech entirely - I’m now at grad school for medical physics.) >I imagine it is easier these days to find Go devs than C devs Maybe? None of us knew Go when the porting began. I think if you know at least 2 programming languages and one of those is C-like, Go will be a piece of cake. I was hired with no Go experience and ramped up quickly.

derivagral 7y ago
Thanks for the detailed reply! Getting a real perspective instead of an article is always great.