21 ms·

Learning Go by porting a medium-sized web back end from Python

242 points by benhoyt 9y ago

crdoconnor 9y ago
>However, it was only 1900 lines of Go (about 50% more than the 1300 lines of Python). >However, I’d seriously consider using Go for larger projects (static typing makes refactoring easier) Assuming a fully tested codebase and a relatively sane runtime type system, I'd much, much rather maintain the shorter codebase than the statically typed one. Having less code makes refactoring easier.
- boomlinde 9y ago
  Assuming a fully tested codebase, there is an overhead in writing and maintaining unit tests that has to be considered as well, and using a static type system obviates a whole category of them. You move the type specification out of tests and manual documentation directly into the code where you'll find out at compile time what you did wrong. Not at run-time, or during testing because you diligently wrote a test that considered the case. I agree that brevity sometimes aids refactoring, but there are many parameters that have to be weighed in, and often I find that the benefit of static typing outweigh whatever ignoring the type or checking it at run-time might offer.
  - crdoconnor 9y ago
    >Assuming a fully tested codebase, there is an overhead in writing and maintaining unit tests that has to be considered as well, and using a static type system obviates a whole category of them. I've not found this to be true, and tests that do type checking are a waste of time in any language. Bugs that are caught by static type systems are almost always caught by a combination of behavior tests and a sprinkling of sanity/type checks on border code (e.g. a module's publicly facing API). >you'll find out at compile time what you did wrong In a fully tested code base that is regularly tested the distinction between compile time and runtime does not matter a lot. By contrast n code bases in dire need of tests (which is more common than generally assumed), the distinction is absolutely vital.
reacharavindh 9y ago
I'm at the same spot myself, trying to learn Go by building a simple web app. Your article was a refreshing read to me. One extra thing that is the major motivation for me to go to Go is the ability to deploy as a single God damn binary and not deal with dependancies and pip. pip and counting on OS repositories may be the right thing to do, but it is just too cumbersome. I have an air gapped setup at work, and there is no easy way for me to deploy Python applications - because all projects just say pip instal this and pip install that. It would make my life so much easier building something on my laptop, scp the binary to servers and expect them to "just work".
- pletnes 9y ago
  I don’t disagree, but FYI pip has a —cache flag to use packages from file, and the files could be on a usb stick of course. Same approach can be used to speed up CI setups.
- proto-n 9y ago
  Coming from a data science background, don't anaconda and environment description files [1] solve this problem with python? Not even sudo is needed to install anaconda and setup / clone environments, also it can manage python versions independently from the system's python. [1] https://conda.io/docs/user-guide/tasks/manage-environments.html#building-identical-conda-environments https://conda.io/docs/user-guide/tasks/manage-environments.h...
  - reacharavindh 9y ago
    Thanks for your comment. I have just started using Conda locally to learn before taking it to work. I'm a new sysadmin and my predecessor maintained a restricted list of python packages on all cluster systems like a dictator. I don't want to be one. Does conda cloning the environment mean I can just copy the project directory to another server and expect my program to run without it looking for the packages on the internet?
    - StavrosK 9y ago
trumpeta 9y ago
Recently I have had the opportunity to write a microservice in Go. It was a very refreshing experience switching from Scala. Go is a lot faster to compile and runs with a far smaller footprint. The channels are nice. On the other hand the testing feels wrong because its so annoying to do mocks. And don't even get me started on the dependency management. Overall Go feels to me like some version compiled PHP with better concurrency support.
- egeozcan 9y ago
  > compiled PHP with better concurrency support ...and simpler syntax and without the quirks. But that's one of the best descriptions of Go I've seen so far.
  - cies 9y ago
    Indeed. Go and PHP are at the complete opposites when it comes to quirkiness. And syntactically PHP is at a very noticeable disadvantage.
- oblio 9y ago
  I really wish that there was a language which had these features: - simple (so not Scala, Haskell, Perl 6, etc.; no Rust either, unfortunately) - clean (nice syntax, preferably Python inspired, but consistent) - modern (generics, some functional features, string interpolation, etc.) - decent concurrency/parallelism story - good IDE, preferably supported by the core dev team - compilation to a (possibly static) native binary - decently big ecosystem - and a big enough community of contributors or commercial backing from at least 1 stable entity - also preferably didn't have a lot of historical baggage From what I gather C# might actually be close to this, 2-3 years from now, if the .Net Core transition happens smoothly and the OSS community adopts it... Who knows, maybe even Kotlin 2019 or so?
  - jhh 9y ago
    Kotlin comes to mind.
  - 9y ago
pjmlp 9y ago
> compiles very fast (for a statically-typed language). Like Turbo Pascal 4.0 in 1987 or Modula-2 in 1978, as two possible examples. :)
4c2383f5c88e911 9y ago
I can agree with most of the article, but for some it looks like we have not been using the same language ecosystem at all. Go really feels opinionated around the wrong things, in order to claim "simplicity" as a feature: - No generics just mean you're going to be handing interface{} all the way in your stack, it makes things more complex and less readable for no reason (and less safe) - error handling which is essentially string-based, in 2017? Give me a type system, please - gofmt is a good idea (just like clang-format or yapf), and having it is great, but the maintainers specifically refuse to add simple features, saying "running it in an automated process is not supported", despite all github projects already having it in their automated CI suite - go test is good, but in the end it is found lacking - dep, well, if you don't mind dumping the code for all your deps in the source tree, it's probably ok, but it solves a deficiency that the language should not even have (see next point) - stability: the language itself is fine on that, but the ecosystem really isn't. The fact that they don't even have the package management quality of python (which I find awful already) is baffling. I don't even know how they could think "yeah, github-importing is a good idea, let's do it". Go is eight years old tomorrow and you still have to rely on third-party tooling to do proper imports that don't burn your house.
- zaarn 9y ago
  >- No generics just mean you're going to be handing interface{} all the way in your stack, it makes things more complex and less readable for no reason (and less safe) Depends on what you do. I've written a good chunk of go code and interface{} is the rare exception rather than the rule, usually employed where a user might supply arbitrary types (ie a unmarshaljson like function) But if you do a website or webapp, 99% of your code is not using interface{} in it's methods. >- error handling which is essentially string-based, in 2017? Give me a type system, please You can put anything behind an error: type IntErr uint64 func (i IntErr) Error() string { return strvonc.Atoi(i) } The string is only present when you either use the stdlib errors or you output an error. Additionally, nothing in Go forces you to actually use the error interface (sans stdlib). You can invent your own Error interface that uses ints. (Won't work with most external libraries but nothing is stopping you) Panic and Recover can to my knowledge both handle things that aren't errors; recover returns a interface{} that you check for value and type. You can put anything in there. >stability: the language itself is fine on that, but the ecosystem really isn't. A lot of projects focus on stability. If the API is breaking a lot they usually use Gopherpit or gopkg.in to ensure compatibility in the future. In my experience, breaking the API is not something well used libraries do lightly (libraries with no or little use do break it sometimes since they lack the usage to refine it) If you need more, you can use vendoring.
Dowwie 9y ago
What I've noticed, and personally experienced, about these porting projects is that they're generally a premature optimization but are a great way to learn a new language. The gains from a port are largely intrinsic in nature, residing with the programmers. However, systems and operations largely carry on just fine with the original language chosen. Only one story comes to mind where a programmer had a legitimate problem using a high-level language and solved it with a lower-level language, and that was Armin Roacher, who solved a significant performance issue using Rust: https://blog.sentry.io/2016/10/19/fixing-python-performance-with-rust.html https://blog.sentry.io/2016/10/19/fixing-python-performance-... Can you think of other stories? Please, share. I encourage all to be more skeptical about potential gains and realistic about who is actually gaining.
- _Codemonkeyism 9y ago
  Developers have the gain, company suffers, developers move on for higher salary with more skills, CTO is in it, doesn't fullfil his duty towards the company or investors, CEO has no clue and doesn't manage the CTO. The story I see over and over again as a startup CEO consultant.
  - nspattak 9y ago
    So if I get your argument correctly: Developers learn a new language that is faster to develop, safer and faster to run than existing solutions (which work but are of increased technical debt) and successfully & quickly port the existing solution to the new tool. The company doesn't really benefit from it apparently because they never scale enough or because they lacked a target so they had their developers working on sth small instead on working on the company's products. This skill is highly valued and the developers leave for an other company while the original one struggles to continue. Do you think that it may be the case that the problem in all these is not within the developers or tool? Furthermore, shouldn't the CEOs and the CTOs know better and to a better job?
    - _Codemonkeyism 9y ago
      "Developers learn a new language that is faster to develop, safer and faster to run than existing solutions" Most often only in the eye of the developers with no proper evaluation. "Furthermore, shouldn't the CEOs and the CTOs know better and to a better job?" Yes.
oblio 9y ago
I'm not sure about Javascript, but why do you have to have two APIs when you have async? At least in C# you write the async version of the API and that's it. Then you go asyncStuff.Request on it, and it blocks. There are some quirks but I remember that it worked all right.
bischofs 9y ago
I am doing some python to go conversions for some performance gains in parts of my stack - I really like it so far because of how they brought some of the perks of languages like python into a compiled language. One thing I really miss though is ipdb. The Delve debugger still doesn't let you run functions and I miss all the autocomplete fun with ipdb. I hope they can make go a little more friendly to debugging in general.
jpcarmona 9y ago
I really liked to read the article. You gave your opinion and didn't try to sound that this was the way, it was clear that it was YOUR opinion. Thanks for it.
deleted 9y ago
[deleted]

242 points by benhoyt 9y ago

>However, it was only 1900 lines of Go (about 50% more than the 1300 lines of Python). >However, I’d seriously consider using Go for larger projects (static typing makes refactoring easier) Assuming a fully tested codebase and a relatively sane runtime type system, I'd much, much rather maintain the shorter codebase than the statically typed one. Having less code makes refactoring easier.

boomlinde 9y ago
Assuming a fully tested codebase, there is an overhead in writing and maintaining unit tests that has to be considered as well, and using a static type system obviates a whole category of them. You move the type specification out of tests and manual documentation directly into the code where you'll find out at compile time what you did wrong. Not at run-time, or during testing because you diligently wrote a test that considered the case. I agree that brevity sometimes aids refactoring, but there are many parameters that have to be weighed in, and often I find that the benefit of static typing outweigh whatever ignoring the type or checking it at run-time might offer.
- crdoconnor 9y ago
  >Assuming a fully tested codebase, there is an overhead in writing and maintaining unit tests that has to be considered as well, and using a static type system obviates a whole category of them. I've not found this to be true, and tests that do type checking are a waste of time in any language. Bugs that are caught by static type systems are almost always caught by a combination of behavior tests and a sprinkling of sanity/type checks on border code (e.g. a module's publicly facing API). >you'll find out at compile time what you did wrong In a fully tested code base that is regularly tested the distinction between compile time and runtime does not matter a lot. By contrast n code bases in dire need of tests (which is more common than generally assumed), the distinction is absolutely vital.

reacharavindh 9y ago

I'm at the same spot myself, trying to learn Go by building a simple web app. Your article was a refreshing read to me. One extra thing that is the major motivation for me to go to Go is the ability to deploy as a single God damn binary and not deal with dependancies and pip. pip and counting on OS repositories may be the right thing to do, but it is just too cumbersome. I have an air gapped setup at work, and there is no easy way for me to deploy Python applications - because all projects just say pip instal this and pip install that. It would make my life so much easier building something on my laptop, scp the binary to servers and expect them to "just work".

pletnes 9y ago
I don’t disagree, but FYI pip has a —cache flag to use packages from file, and the files could be on a usb stick of course. Same approach can be used to speed up CI setups.
proto-n 9y ago
Coming from a data science background, don't anaconda and environment description files [1] solve this problem with python? Not even sudo is needed to install anaconda and setup / clone environments, also it can manage python versions independently from the system's python. [1] https://conda.io/docs/user-guide/tasks/manage-environments.html#building-identical-conda-environments https://conda.io/docs/user-guide/tasks/manage-environments.h...
- reacharavindh 9y ago
  Thanks for your comment. I have just started using Conda locally to learn before taking it to work. I'm a new sysadmin and my predecessor maintained a restricted list of python packages on all cluster systems like a dictator. I don't want to be one. Does conda cloning the environment mean I can just copy the project directory to another server and expect my program to run without it looking for the packages on the internet?
  - StavrosK 9y ago

trumpeta 9y ago

Recently I have had the opportunity to write a microservice in Go. It was a very refreshing experience switching from Scala. Go is a lot faster to compile and runs with a far smaller footprint. The channels are nice. On the other hand the testing feels wrong because its so annoying to do mocks. And don't even get me started on the dependency management. Overall Go feels to me like some version compiled PHP with better concurrency support.

egeozcan 9y ago
> compiled PHP with better concurrency support ...and simpler syntax and without the quirks. But that's one of the best descriptions of Go I've seen so far.
- cies 9y ago
  Indeed. Go and PHP are at the complete opposites when it comes to quirkiness. And syntactically PHP is at a very noticeable disadvantage.
oblio 9y ago
I really wish that there was a language which had these features: - simple (so not Scala, Haskell, Perl 6, etc.; no Rust either, unfortunately) - clean (nice syntax, preferably Python inspired, but consistent) - modern (generics, some functional features, string interpolation, etc.) - decent concurrency/parallelism story - good IDE, preferably supported by the core dev team - compilation to a (possibly static) native binary - decently big ecosystem - and a big enough community of contributors or commercial backing from at least 1 stable entity - also preferably didn't have a lot of historical baggage From what I gather C# might actually be close to this, 2-3 years from now, if the .Net Core transition happens smoothly and the OSS community adopts it... Who knows, maybe even Kotlin 2019 or so?
- jhh 9y ago
  Kotlin comes to mind.
- 9y ago

pjmlp 9y ago

> compiles very fast (for a statically-typed language). Like Turbo Pascal 4.0 in 1987 or Modula-2 in 1978, as two possible examples. :)

4c2383f5c88e911 9y ago

I can agree with most of the article, but for some it looks like we have not been using the same language ecosystem at all. Go really feels opinionated around the wrong things, in order to claim "simplicity" as a feature: - No generics just mean you're going to be handing interface{} all the way in your stack, it makes things more complex and less readable for no reason (and less safe) - error handling which is essentially string-based, in 2017? Give me a type system, please - gofmt is a good idea (just like clang-format or yapf), and having it is great, but the maintainers specifically refuse to add simple features, saying "running it in an automated process is not supported", despite all github projects already having it in their automated CI suite - go test is good, but in the end it is found lacking - dep, well, if you don't mind dumping the code for all your deps in the source tree, it's probably ok, but it solves a deficiency that the language should not even have (see next point) - stability: the language itself is fine on that, but the ecosystem really isn't. The fact that they don't even have the package management quality of python (which I find awful already) is baffling. I don't even know how they could think "yeah, github-importing is a good idea, let's do it". Go is eight years old tomorrow and you still have to rely on third-party tooling to do proper imports that don't burn your house.

zaarn 9y ago
>- No generics just mean you're going to be handing interface{} all the way in your stack, it makes things more complex and less readable for no reason (and less safe) Depends on what you do. I've written a good chunk of go code and interface{} is the rare exception rather than the rule, usually employed where a user might supply arbitrary types (ie a unmarshaljson like function) But if you do a website or webapp, 99% of your code is not using interface{} in it's methods. >- error handling which is essentially string-based, in 2017? Give me a type system, please You can put anything behind an error: type IntErr uint64 func (i IntErr) Error() string { return strvonc.Atoi(i) } The string is only present when you either use the stdlib errors or you output an error. Additionally, nothing in Go forces you to actually use the error interface (sans stdlib). You can invent your own Error interface that uses ints. (Won't work with most external libraries but nothing is stopping you) Panic and Recover can to my knowledge both handle things that aren't errors; recover returns a interface{} that you check for value and type. You can put anything in there. >stability: the language itself is fine on that, but the ecosystem really isn't. A lot of projects focus on stability. If the API is breaking a lot they usually use Gopherpit or gopkg.in to ensure compatibility in the future. In my experience, breaking the API is not something well used libraries do lightly (libraries with no or little use do break it sometimes since they lack the usage to refine it) If you need more, you can use vendoring.

Dowwie 9y ago

What I've noticed, and personally experienced, about these porting projects is that they're generally a premature optimization but are a great way to learn a new language. The gains from a port are largely intrinsic in nature, residing with the programmers. However, systems and operations largely carry on just fine with the original language chosen. Only one story comes to mind where a programmer had a legitimate problem using a high-level language and solved it with a lower-level language, and that was Armin Roacher, who solved a significant performance issue using Rust: https://blog.sentry.io/2016/10/19/fixing-python-performance-with-rust.html https://blog.sentry.io/2016/10/19/fixing-python-performance-... Can you think of other stories? Please, share. I encourage all to be more skeptical about potential gains and realistic about who is actually gaining.

_Codemonkeyism 9y ago
Developers have the gain, company suffers, developers move on for higher salary with more skills, CTO is in it, doesn't fullfil his duty towards the company or investors, CEO has no clue and doesn't manage the CTO. The story I see over and over again as a startup CEO consultant.
- nspattak 9y ago
  So if I get your argument correctly: Developers learn a new language that is faster to develop, safer and faster to run than existing solutions (which work but are of increased technical debt) and successfully & quickly port the existing solution to the new tool. The company doesn't really benefit from it apparently because they never scale enough or because they lacked a target so they had their developers working on sth small instead on working on the company's products. This skill is highly valued and the developers leave for an other company while the original one struggles to continue. Do you think that it may be the case that the problem in all these is not within the developers or tool? Furthermore, shouldn't the CEOs and the CTOs know better and to a better job?
  - _Codemonkeyism 9y ago
    "Developers learn a new language that is faster to develop, safer and faster to run than existing solutions" Most often only in the eye of the developers with no proper evaluation. "Furthermore, shouldn't the CEOs and the CTOs know better and to a better job?" Yes.

oblio 9y ago

I'm not sure about Javascript, but why do you have to have two APIs when you have async? At least in C# you write the async version of the API and that's it. Then you go asyncStuff.Request on it, and it blocks. There are some quirks but I remember that it worked all right.

bischofs 9y ago

I am doing some python to go conversions for some performance gains in parts of my stack - I really like it so far because of how they brought some of the perks of languages like python into a compiled language. One thing I really miss though is ipdb. The Delve debugger still doesn't let you run functions and I miss all the autocomplete fun with ipdb. I hope they can make go a little more friendly to debugging in general.

jpcarmona 9y ago

I really liked to read the article. You gave your opinion and didn't try to sound that this was the way, it was clear that it was YOUR opinion. Thanks for it.

deleted 9y ago

[deleted]