4 ms·

While I'm pretty sure that Go is much faster than python in nearly all tasks, I'm still skeptical that a) they built a full replacement of the system in only a

While I'm pretty sure that Go is much faster than python in nearly all tasks, I'm still skeptical that a) they built a full replacement of the system in only a week and b) that they didn't improve the algorithm in the go code. I work at a python/django shop and I recently updated a process that could only handle 5000 transactions in 5 minutes to be able to handle over 100,000 in just under 3 minutes. This was all in python would go been a lot faster? Sure but how much is the new code path and how much is the advantage of golang?

jaegerpicker 12y ago
Also what to point out that I fully realize that Go is a lot faster than python at preforming the same task but this kind of analysis is dangerous because it leads inexperienced engineers to believe that changing languages is a good performance optimization which should usually be the last step/idea considered. It's expensive and dangerous to switch languages mid stream and while it's sometimes the exact cure it needs to be weighed very heavily to make sure it's the right situation. This article, at least the headline misses that piece of the puzzle pretty fully IMO.
- blt 12y ago
  Not sure I agree with you there. IMO, porting between languages with similar abstraction levels is not that hard. I ported some Python code, whose logic took weeks of fine-tuning, to C# in less than a day. Go still has a garbage collector and its type system is similar in philosophy to Python's duck typing. Unless you are porting extra dynamic code (eval(), monkey-patching) or deeply intertwined with an unusual library, it should be a fast process. On the other hand, I was leaving Python get real OS threads, not to increase performance.
  - wting 12y ago
    This is nitpicky, but Python threads are kernel threads. It's the GIL that restricts CPython to a single core. OTOH goroutines are green threads multiplexed onto kernel threads (aka N:M thread model).
  - nkozyra 12y ago
    This obviously also depends heavily on the complexity of the application and the available resources. Even so, a production-ready transition in a few weeks time doesn't leave a ton of time for benchmarking and testing, but as someone who operates on a fly-by-the-seat ethos more often than I should I'm in no position to be tossing stones around.
- merrua 12y ago
  There has been articles talking about rewritting being less of a terrible idea if your writing a SaaS. Not to mention the throw one out, you will anyway idea. I'd be interested though, in hearing more about what they rewrote into Go. Its most probable its not all of their codebase.
  - dev360 12y ago
    Yes, it sounds like its a websocket server. A lot of that stuff is relatively low complexity if you glue everything together with a message queue since they can be made language agnostic very easily.
  - mattrobenolt 12y ago
    Just our realtime system, and we run some asynchronous tasks with Go now, specifying to replace a few things that were done by Celery previously. See this shitty thing I wrote to bridge the gap: https://github.com/mattrobenolt/go-celery https://github.com/mattrobenolt/go-celery :)
- nkozyra 12y ago
  Actually, I think "Go is faster" has two primary built-in points: 1. It's faster in terms of real performance - given its proximity to C, optimization at the compiler level for multiple platforms, that its highly concurrent and run compiled (usually), this is no shocker. But, 2. It's faster than transitioning from a dynamic scripting language than it would be to transition to, say, C or C++ or Java, etc. Not that it's syntactically similar (obviously), but that language, testing and deployment design won't present any egregious bottlenecks and that there's so much built into the core library that complements an http-based service such as this. In other words, the prospect of going from Python (or Rails or PHP or whatever) to "faster language X" has historically imposed an inherent cost in terms of development that I think a language like Go mitigates to some degree by design.
  - yebyen 12y ago
    > and run compiled (usually) I think you are talking about 'go run', which one might easily mistake for an interpreter because of the speed of the compiler, but it's actually compiling your code and running it. Were you talking about something else?
    - nkozyra 12y ago
      I wasn't talking about go run, no - there have been a few examples of running go interpreted I've seen posted here. They may well just be experiments but I left the "usually" as my caveat. Edit: I swear there was an interpreter demo'd, but I may well have been thinking of something like agora, which interprets Go-like code within Go.
      - yebyen 12y ago
        I was actually fishing for cool stuff I hadn't heard of, like agora. I don't get to dive into very much Go news! Thanks
crucialfelix 12y ago
It sounds like this is a very contained server process that does one thing: pull things from rabbitmq and pushes that to nginx which does the push stream. So this could be written in a week easily.
mclarke 12y ago
This specific use case is multiplexing messages to lots of connections, which is almost the ideal use case of Go anyways. It wasn't a matter of re-implementing complex business logic. 1 week is reasonable given the specific problem.
jedberg 12y ago
> I'm still skeptical that a) they built a full replacement of the system in only a week Keep in mind that Highscalability bases it's articles on other 3rd party research. They don't talk to the people they are writing about, even when those people (like me) offer to help them get their facts right. In all likelihood it's just completely wrong.
- mattrobenolt 12y ago
  I assure you, I built this in a week. :)
  - jedberg 12y ago
    See, a first person source! :) Nonetheless, my statement otherwise still stands.
- toddh 12y ago
  While I do certainly make mistakes I take great pains to be as accurate as I can. And I do have a lot of original interviews on the site, though some problems have kept me from doing them lately. I give the sources directly in the article and they are directly from the people involved in the project. And if there are any inaccuracies in any article I would be happy to make corrections. I don't recall when you offered to correct mistakes, but if you did so and I missed it then I apologize.
- emperorcezar 12y ago
  It seems as if they replaced one component of the system in a week, not the whole thing.
mattrobenolt 12y ago
To be fair, there were some optimizations that were made as well. Everyone does that, right? :) But overall, specifically our realtime service is a hybrid of CPU intensive tasks + lots of network IO. gevent was handling the network io without an issue, but at higher contention, the CPU was choking everything. Switching over to Go removed that contention for us, which was the primary issue that we were seeing.
- jaegerpicker 12y ago
  Right and that's awesome and ALOT more useful than the article because that seems an ideal use case to switch to go over python.
mamcx 12y ago
> b) that they didn't improve the algorithm in the go code The port to GO will do it (I think), because it have built-in facilities to do concurrent programing across all the library..

jaegerpicker 12y ago

Also what to point out that I fully realize that Go is a lot faster than python at preforming the same task but this kind of analysis is dangerous because it leads inexperienced engineers to believe that changing languages is a good performance optimization which should usually be the last step/idea considered. It's expensive and dangerous to switch languages mid stream and while it's sometimes the exact cure it needs to be weighed very heavily to make sure it's the right situation. This article, at least the headline misses that piece of the puzzle pretty fully IMO.

blt 12y ago
Not sure I agree with you there. IMO, porting between languages with similar abstraction levels is not that hard. I ported some Python code, whose logic took weeks of fine-tuning, to C# in less than a day. Go still has a garbage collector and its type system is similar in philosophy to Python's duck typing. Unless you are porting extra dynamic code (eval(), monkey-patching) or deeply intertwined with an unusual library, it should be a fast process. On the other hand, I was leaving Python get real OS threads, not to increase performance.
- wting 12y ago
  This is nitpicky, but Python threads are kernel threads. It's the GIL that restricts CPython to a single core. OTOH goroutines are green threads multiplexed onto kernel threads (aka N:M thread model).
- nkozyra 12y ago
  This obviously also depends heavily on the complexity of the application and the available resources. Even so, a production-ready transition in a few weeks time doesn't leave a ton of time for benchmarking and testing, but as someone who operates on a fly-by-the-seat ethos more often than I should I'm in no position to be tossing stones around.
merrua 12y ago
There has been articles talking about rewritting being less of a terrible idea if your writing a SaaS. Not to mention the throw one out, you will anyway idea. I'd be interested though, in hearing more about what they rewrote into Go. Its most probable its not all of their codebase.
- dev360 12y ago
  Yes, it sounds like its a websocket server. A lot of that stuff is relatively low complexity if you glue everything together with a message queue since they can be made language agnostic very easily.
- mattrobenolt 12y ago
  Just our realtime system, and we run some asynchronous tasks with Go now, specifying to replace a few things that were done by Celery previously. See this shitty thing I wrote to bridge the gap: https://github.com/mattrobenolt/go-celery https://github.com/mattrobenolt/go-celery :)
nkozyra 12y ago
Actually, I think "Go is faster" has two primary built-in points: 1. It's faster in terms of real performance - given its proximity to C, optimization at the compiler level for multiple platforms, that its highly concurrent and run compiled (usually), this is no shocker. But, 2. It's faster than transitioning from a dynamic scripting language than it would be to transition to, say, C or C++ or Java, etc. Not that it's syntactically similar (obviously), but that language, testing and deployment design won't present any egregious bottlenecks and that there's so much built into the core library that complements an http-based service such as this. In other words, the prospect of going from Python (or Rails or PHP or whatever) to "faster language X" has historically imposed an inherent cost in terms of development that I think a language like Go mitigates to some degree by design.
- yebyen 12y ago
  > and run compiled (usually) I think you are talking about 'go run', which one might easily mistake for an interpreter because of the speed of the compiler, but it's actually compiling your code and running it. Were you talking about something else?
  - nkozyra 12y ago
    I wasn't talking about go run, no - there have been a few examples of running go interpreted I've seen posted here. They may well just be experiments but I left the "usually" as my caveat. Edit: I swear there was an interpreter demo'd, but I may well have been thinking of something like agora, which interprets Go-like code within Go.
    - yebyen 12y ago
      I was actually fishing for cool stuff I hadn't heard of, like agora. I don't get to dive into very much Go news! Thanks

crucialfelix 12y ago

It sounds like this is a very contained server process that does one thing: pull things from rabbitmq and pushes that to nginx which does the push stream. So this could be written in a week easily.

mclarke 12y ago

This specific use case is multiplexing messages to lots of connections, which is almost the ideal use case of Go anyways. It wasn't a matter of re-implementing complex business logic. 1 week is reasonable given the specific problem.

jedberg 12y ago

> I'm still skeptical that a) they built a full replacement of the system in only a week Keep in mind that Highscalability bases it's articles on other 3rd party research. They don't talk to the people they are writing about, even when those people (like me) offer to help them get their facts right. In all likelihood it's just completely wrong.

mattrobenolt 12y ago
I assure you, I built this in a week. :)
- jedberg 12y ago
  See, a first person source! :) Nonetheless, my statement otherwise still stands.
toddh 12y ago
While I do certainly make mistakes I take great pains to be as accurate as I can. And I do have a lot of original interviews on the site, though some problems have kept me from doing them lately. I give the sources directly in the article and they are directly from the people involved in the project. And if there are any inaccuracies in any article I would be happy to make corrections. I don't recall when you offered to correct mistakes, but if you did so and I missed it then I apologize.
emperorcezar 12y ago
It seems as if they replaced one component of the system in a week, not the whole thing.

mattrobenolt 12y ago

To be fair, there were some optimizations that were made as well. Everyone does that, right? :) But overall, specifically our realtime service is a hybrid of CPU intensive tasks + lots of network IO. gevent was handling the network io without an issue, but at higher contention, the CPU was choking everything. Switching over to Go removed that contention for us, which was the primary issue that we were seeing.

jaegerpicker 12y ago
Right and that's awesome and ALOT more useful than the article because that seems an ideal use case to switch to go over python.

mamcx 12y ago

> b) that they didn't improve the algorithm in the go code The port to GO will do it (I think), because it have built-in facilities to do concurrent programing across all the library..