10 ms·

Disqus: It's Still About Realtime, But Go Demolishes Python

94 points by geerlingguy 13y ago

jaegerpicker 13y ago
While I'm pretty sure that Go is much faster than python in nearly all tasks, I'm still skeptical that a) they built a full replacement of the system in only a week and b) that they didn't improve the algorithm in the go code. I work at a python/django shop and I recently updated a process that could only handle 5000 transactions in 5 minutes to be able to handle over 100,000 in just under 3 minutes. This was all in python would go been a lot faster? Sure but how much is the new code path and how much is the advantage of golang?
- jaegerpicker 13y ago
  Also what to point out that I fully realize that Go is a lot faster than python at preforming the same task but this kind of analysis is dangerous because it leads inexperienced engineers to believe that changing languages is a good performance optimization which should usually be the last step/idea considered. It's expensive and dangerous to switch languages mid stream and while it's sometimes the exact cure it needs to be weighed very heavily to make sure it's the right situation. This article, at least the headline misses that piece of the puzzle pretty fully IMO.
  - blt 13y ago
    Not sure I agree with you there. IMO, porting between languages with similar abstraction levels is not that hard. I ported some Python code, whose logic took weeks of fine-tuning, to C# in less than a day. Go still has a garbage collector and its type system is similar in philosophy to Python's duck typing. Unless you are porting extra dynamic code (eval(), monkey-patching) or deeply intertwined with an unusual library, it should be a fast process. On the other hand, I was leaving Python get real OS threads, not to increase performance.
    - wting 13y ago
      This is nitpicky, but Python threads are kernel threads. It's the GIL that restricts CPython to a single core. OTOH goroutines are green threads multiplexed onto kernel threads (aka N:M thread model).
    - nkozyra 13y ago
      This obviously also depends heavily on the complexity of the application and the available resources. Even so, a production-ready transition in a few weeks time doesn't leave a ton of time for benchmarking and testing, but as someone who operates on a fly-by-the-seat ethos more often than I should I'm in no position to be tossing stones around.
  - merrua 13y ago
    There has been articles talking about rewritting being less of a terrible idea if your writing a SaaS. Not to mention the throw one out, you will anyway idea. I'd be interested though, in hearing more about what they rewrote into Go. Its most probable its not all of their codebase.
    - dev360 13y ago
      Yes, it sounds like its a websocket server. A lot of that stuff is relatively low complexity if you glue everything together with a message queue since they can be made language agnostic very easily.
    - mattrobenolt 13y ago
      Just our realtime system, and we run some asynchronous tasks with Go now, specifying to replace a few things that were done by Celery previously. See this shitty thing I wrote to bridge the gap: https://github.com/mattrobenolt/go-celery https://github.com/mattrobenolt/go-celery :)
  - nkozyra 13y ago
    Actually, I think "Go is faster" has two primary built-in points: 1. It's faster in terms of real performance - given its proximity to C, optimization at the compiler level for multiple platforms, that its highly concurrent and run compiled (usually), this is no shocker. But, 2. It's faster than transitioning from a dynamic scripting language than it would be to transition to, say, C or C++ or Java, etc. Not that it's syntactically similar (obviously), but that language, testing and deployment design won't present any egregious bottlenecks and that there's so much built into the core library that complements an http-based service such as this. In other words, the prospect of going from Python (or Rails or PHP or whatever) to "faster language X" has historically imposed an inherent cost in terms of development that I think a language like Go mitigates to some degree by design.
    - yebyen 13y ago
      > and run compiled (usually) I think you are talking about 'go run', which one might easily mistake for an interpreter because of the speed of the compiler, but it's actually compiling your code and running it. Were you talking about something else?
      - nkozyra 13y ago
        I wasn't talking about go run, no - there have been a few examples of running go interpreted I've seen posted here. They may well just be experiments but I left the "usually" as my caveat. Edit: I swear there was an interpreter demo'd, but I may well have been thinking of something like agora, which interprets Go-like code within Go.
        yebyen 13y ago
        I was actually fishing for cool stuff I hadn't heard of, like agora. I don't get to dive into very much Go news! Thanks
- crucialfelix 13y ago
  It sounds like this is a very contained server process that does one thing: pull things from rabbitmq and pushes that to nginx which does the push stream. So this could be written in a week easily.
- mclarke 13y ago
  This specific use case is multiplexing messages to lots of connections, which is almost the ideal use case of Go anyways. It wasn't a matter of re-implementing complex business logic. 1 week is reasonable given the specific problem.
- jedberg 13y ago
  > I'm still skeptical that a) they built a full replacement of the system in only a week Keep in mind that Highscalability bases it's articles on other 3rd party research. They don't talk to the people they are writing about, even when those people (like me) offer to help them get their facts right. In all likelihood it's just completely wrong.
  - mattrobenolt 13y ago
    I assure you, I built this in a week. :)
    - jedberg 13y ago
      See, a first person source! :) Nonetheless, my statement otherwise still stands.
  - toddh 13y ago
    While I do certainly make mistakes I take great pains to be as accurate as I can. And I do have a lot of original interviews on the site, though some problems have kept me from doing them lately. I give the sources directly in the article and they are directly from the people involved in the project. And if there are any inaccuracies in any article I would be happy to make corrections. I don't recall when you offered to correct mistakes, but if you did so and I missed it then I apologize.
  - emperorcezar 13y ago
    It seems as if they replaced one component of the system in a week, not the whole thing.
- mattrobenolt 13y ago
  To be fair, there were some optimizations that were made as well. Everyone does that, right? :) But overall, specifically our realtime service is a hybrid of CPU intensive tasks + lots of network IO. gevent was handling the network io without an issue, but at higher contention, the CPU was choking everything. Switching over to Go removed that contention for us, which was the primary issue that we were seeing.
  - jaegerpicker 13y ago
    Right and that's awesome and ALOT more useful than the article because that seems an ideal use case to switch to go over python.
- mamcx 13y ago
  > b) that they didn't improve the algorithm in the go code The port to GO will do it (I think), because it have built-in facilities to do concurrent programing across all the library..
a3n 13y ago
Go is looking more attractive, but the language is not just the language. When Go has available a significant fraction (subjectively) of the "batteries included" that Python has, then I'll start investing time in Go. Is it there yet (subjectively)?
- wtbob 13y ago
  I think Go is pretty competitive with Python's standard library. It doesn't have an FTP client (we use and are happy with github.com/jlaffaye/ftp) or an equivalent to imaplib (although github.com/mxk/go-imap/imap looks interesting), but for the rest of our workload Go has everything we need.
  - daave 13y ago
    Aside: for SFTP we use (and like) https://github.com/minusnine/gosftp/ https://github.com/minusnine/gosftp/
- awda 13y ago
  For a start, Go's SSL/TLS doesn't disable certificate checking by default ;-). Does that mean it's more "batteries included" than Python for TLS and every TLS-using protocol?
- ominous_prime 13y ago
  It depends on what batteries you want. The Go standard library is pretty amazing as far as standard libraries go (I'd say it's more useful than python's, and definitely contains less cruft). Community provided libraries are expanding rapidly, but there is a strong focus on web services all around.
- jnbiche 13y ago
  It's definitely getting there. I'd actually say that Go's network "batteries" are just about as comprehensive and possibly easier to use than Python's (ie, socket, urllib, urllib2, httplib, smtplib, pop3lib, etc.). It's also pretty cool that Go has a full crypto library in written in Go. I was seriously looking at Go for a desktop app using the promising new go-qml, but there seems to be no elegant way to use Qt's system tray feature with go-qml. I'd say Go still has a way to go before it's got as many possibilities as Python for desktop GUI development. But for command line utilities and network apps, I'll definitely reach for Go over Python now (although I'm increasingly playing around with Rust and hoping I can eventually just use Rust for just about everything). Edit: Removed imaplib since Go has no equivalent in its stdlib.
axelf 13y ago
why were they using python for performance critical code in the first place? Go seems closer to java's niche to me.
- rch 13y ago
  They had a fairly popular post on the subject a while back: http://blog.disqus.com/post/62187806135/scaling-django-to-8-billion-page-views http://blog.disqus.com/post/62187806135/scaling-django-to-8-... In short, since they were not really pushing volume with Python, just feeding data to Varnish, throughput wasn't such a big deal.
- disputin 13y ago
  Yes, why is Go constantly compared to Python?
  - mattrobenolt 13y ago
    For me, it felt comfortable writing Go, considering I've been mostly writing Python for the past 8 years. So the transition was a lot more natural, compared to using Scala or Erlang or anything else. Again, this is my subjective opinion, and this is why we chose to use Go for our new stuff instead of something different.
- sologoub 13y ago
  Disqus was originally built on Django: http://blog.disqus.com/post/62187806135/scaling-django-to-8-billion-page-views http://blog.disqus.com/post/62187806135/scaling-django-to-8-... It seems that this is what the found team was most comfortable with, so it makes sense that they proceeded to solve problems using tools they already knew well. At some point, they exhausted how far they could take their existing tools and started investing into new tools.
  - melling 13y ago
    What Go web framework did they move to?
awda 13y ago
> yields it's own benefits as well. yields its own benefits as well.
peterwwillis 13y ago
> In only a week a replacement system was built This part is telling to me. It means their codebase isn't so complex, and optimizing their application for their hardware would take as much time (or less) than adding hardware. For people with large, complex codebases, scaling horizontally increases capacity much faster.
Oculus 13y ago
Node was not selected because it does not handle CPU intensive tasks well I'm curious to hear what sort of CPU intensive tasks Disqus does.
infogulch 13y ago
I was going to ask how pypy performed on their original python code but it looks like gevent doesn't support it yet (or didn't at the time).
wting 13y ago
In the past Go has had trouble with garbage collection lag due to a global mark and sweep implementation[0]. Was this not an issue with Disqus's new implementation? [0] https://groups.google.com/forum/m/#!msg/golang-nuts/S9goEGuoMRM/FZyd2M6uiVMJ https://groups.google.com/forum/m/#!msg/golang-nuts/S9goEGuo... For the record Python uses a generational garbage collector.
- devanti 13y ago
  You can run multiple instances of the same Go program to reduce the effect of GC lag

94 points by geerlingguy 13y ago

While I'm pretty sure that Go is much faster than python in nearly all tasks, I'm still skeptical that a) they built a full replacement of the system in only a week and b) that they didn't improve the algorithm in the go code. I work at a python/django shop and I recently updated a process that could only handle 5000 transactions in 5 minutes to be able to handle over 100,000 in just under 3 minutes. This was all in python would go been a lot faster? Sure but how much is the new code path and how much is the advantage of golang?

jaegerpicker 13y ago
Also what to point out that I fully realize that Go is a lot faster than python at preforming the same task but this kind of analysis is dangerous because it leads inexperienced engineers to believe that changing languages is a good performance optimization which should usually be the last step/idea considered. It's expensive and dangerous to switch languages mid stream and while it's sometimes the exact cure it needs to be weighed very heavily to make sure it's the right situation. This article, at least the headline misses that piece of the puzzle pretty fully IMO.
- blt 13y ago
  Not sure I agree with you there. IMO, porting between languages with similar abstraction levels is not that hard. I ported some Python code, whose logic took weeks of fine-tuning, to C# in less than a day. Go still has a garbage collector and its type system is similar in philosophy to Python's duck typing. Unless you are porting extra dynamic code (eval(), monkey-patching) or deeply intertwined with an unusual library, it should be a fast process. On the other hand, I was leaving Python get real OS threads, not to increase performance.
  - wting 13y ago
    This is nitpicky, but Python threads are kernel threads. It's the GIL that restricts CPython to a single core. OTOH goroutines are green threads multiplexed onto kernel threads (aka N:M thread model).
  - nkozyra 13y ago
    This obviously also depends heavily on the complexity of the application and the available resources. Even so, a production-ready transition in a few weeks time doesn't leave a ton of time for benchmarking and testing, but as someone who operates on a fly-by-the-seat ethos more often than I should I'm in no position to be tossing stones around.
- merrua 13y ago
  There has been articles talking about rewritting being less of a terrible idea if your writing a SaaS. Not to mention the throw one out, you will anyway idea. I'd be interested though, in hearing more about what they rewrote into Go. Its most probable its not all of their codebase.
  - dev360 13y ago
    Yes, it sounds like its a websocket server. A lot of that stuff is relatively low complexity if you glue everything together with a message queue since they can be made language agnostic very easily.
  - mattrobenolt 13y ago
    Just our realtime system, and we run some asynchronous tasks with Go now, specifying to replace a few things that were done by Celery previously. See this shitty thing I wrote to bridge the gap: https://github.com/mattrobenolt/go-celery https://github.com/mattrobenolt/go-celery :)
- nkozyra 13y ago
  Actually, I think "Go is faster" has two primary built-in points: 1. It's faster in terms of real performance - given its proximity to C, optimization at the compiler level for multiple platforms, that its highly concurrent and run compiled (usually), this is no shocker. But, 2. It's faster than transitioning from a dynamic scripting language than it would be to transition to, say, C or C++ or Java, etc. Not that it's syntactically similar (obviously), but that language, testing and deployment design won't present any egregious bottlenecks and that there's so much built into the core library that complements an http-based service such as this. In other words, the prospect of going from Python (or Rails or PHP or whatever) to "faster language X" has historically imposed an inherent cost in terms of development that I think a language like Go mitigates to some degree by design.
  - yebyen 13y ago
    > and run compiled (usually) I think you are talking about 'go run', which one might easily mistake for an interpreter because of the speed of the compiler, but it's actually compiling your code and running it. Were you talking about something else?
    - nkozyra 13y ago
      I wasn't talking about go run, no - there have been a few examples of running go interpreted I've seen posted here. They may well just be experiments but I left the "usually" as my caveat. Edit: I swear there was an interpreter demo'd, but I may well have been thinking of something like agora, which interprets Go-like code within Go.
      - yebyen 13y ago
        I was actually fishing for cool stuff I hadn't heard of, like agora. I don't get to dive into very much Go news! Thanks
crucialfelix 13y ago
It sounds like this is a very contained server process that does one thing: pull things from rabbitmq and pushes that to nginx which does the push stream. So this could be written in a week easily.
mclarke 13y ago
This specific use case is multiplexing messages to lots of connections, which is almost the ideal use case of Go anyways. It wasn't a matter of re-implementing complex business logic. 1 week is reasonable given the specific problem.
jedberg 13y ago
> I'm still skeptical that a) they built a full replacement of the system in only a week Keep in mind that Highscalability bases it's articles on other 3rd party research. They don't talk to the people they are writing about, even when those people (like me) offer to help them get their facts right. In all likelihood it's just completely wrong.
- mattrobenolt 13y ago
  I assure you, I built this in a week. :)
  - jedberg 13y ago
    See, a first person source! :) Nonetheless, my statement otherwise still stands.
- toddh 13y ago
  While I do certainly make mistakes I take great pains to be as accurate as I can. And I do have a lot of original interviews on the site, though some problems have kept me from doing them lately. I give the sources directly in the article and they are directly from the people involved in the project. And if there are any inaccuracies in any article I would be happy to make corrections. I don't recall when you offered to correct mistakes, but if you did so and I missed it then I apologize.
- emperorcezar 13y ago
  It seems as if they replaced one component of the system in a week, not the whole thing.
mattrobenolt 13y ago
To be fair, there were some optimizations that were made as well. Everyone does that, right? :) But overall, specifically our realtime service is a hybrid of CPU intensive tasks + lots of network IO. gevent was handling the network io without an issue, but at higher contention, the CPU was choking everything. Switching over to Go removed that contention for us, which was the primary issue that we were seeing.
- jaegerpicker 13y ago
  Right and that's awesome and ALOT more useful than the article because that seems an ideal use case to switch to go over python.
mamcx 13y ago
> b) that they didn't improve the algorithm in the go code The port to GO will do it (I think), because it have built-in facilities to do concurrent programing across all the library..

a3n 13y ago

Go is looking more attractive, but the language is not just the language. When Go has available a significant fraction (subjectively) of the "batteries included" that Python has, then I'll start investing time in Go. Is it there yet (subjectively)?

wtbob 13y ago
I think Go is pretty competitive with Python's standard library. It doesn't have an FTP client (we use and are happy with github.com/jlaffaye/ftp) or an equivalent to imaplib (although github.com/mxk/go-imap/imap looks interesting), but for the rest of our workload Go has everything we need.
- daave 13y ago
  Aside: for SFTP we use (and like) https://github.com/minusnine/gosftp/ https://github.com/minusnine/gosftp/
awda 13y ago
For a start, Go's SSL/TLS doesn't disable certificate checking by default ;-). Does that mean it's more "batteries included" than Python for TLS and every TLS-using protocol?
ominous_prime 13y ago
It depends on what batteries you want. The Go standard library is pretty amazing as far as standard libraries go (I'd say it's more useful than python's, and definitely contains less cruft). Community provided libraries are expanding rapidly, but there is a strong focus on web services all around.
jnbiche 13y ago
It's definitely getting there. I'd actually say that Go's network "batteries" are just about as comprehensive and possibly easier to use than Python's (ie, socket, urllib, urllib2, httplib, smtplib, pop3lib, etc.). It's also pretty cool that Go has a full crypto library in written in Go. I was seriously looking at Go for a desktop app using the promising new go-qml, but there seems to be no elegant way to use Qt's system tray feature with go-qml. I'd say Go still has a way to go before it's got as many possibilities as Python for desktop GUI development. But for command line utilities and network apps, I'll definitely reach for Go over Python now (although I'm increasingly playing around with Rust and hoping I can eventually just use Rust for just about everything). Edit: Removed imaplib since Go has no equivalent in its stdlib.

axelf 13y ago

why were they using python for performance critical code in the first place? Go seems closer to java's niche to me.

rch 13y ago
They had a fairly popular post on the subject a while back: http://blog.disqus.com/post/62187806135/scaling-django-to-8-billion-page-views http://blog.disqus.com/post/62187806135/scaling-django-to-8-... In short, since they were not really pushing volume with Python, just feeding data to Varnish, throughput wasn't such a big deal.
disputin 13y ago
Yes, why is Go constantly compared to Python?
- mattrobenolt 13y ago
  For me, it felt comfortable writing Go, considering I've been mostly writing Python for the past 8 years. So the transition was a lot more natural, compared to using Scala or Erlang or anything else. Again, this is my subjective opinion, and this is why we chose to use Go for our new stuff instead of something different.
sologoub 13y ago
Disqus was originally built on Django: http://blog.disqus.com/post/62187806135/scaling-django-to-8-billion-page-views http://blog.disqus.com/post/62187806135/scaling-django-to-8-... It seems that this is what the found team was most comfortable with, so it makes sense that they proceeded to solve problems using tools they already knew well. At some point, they exhausted how far they could take their existing tools and started investing into new tools.
- melling 13y ago
  What Go web framework did they move to?

awda 13y ago

> yields it's own benefits as well. yields its own benefits as well.

peterwwillis 13y ago

> In only a week a replacement system was built This part is telling to me. It means their codebase isn't so complex, and optimizing their application for their hardware would take as much time (or less) than adding hardware. For people with large, complex codebases, scaling horizontally increases capacity much faster.

Oculus 13y ago

Node was not selected because it does not handle CPU intensive tasks well I'm curious to hear what sort of CPU intensive tasks Disqus does.

infogulch 13y ago

I was going to ask how pypy performed on their original python code but it looks like gevent doesn't support it yet (or didn't at the time).

wting 13y ago

In the past Go has had trouble with garbage collection lag due to a global mark and sweep implementation[0]. Was this not an issue with Disqus's new implementation? [0] https://groups.google.com/forum/m/#!msg/golang-nuts/S9goEGuoMRM/FZyd2M6uiVMJ https://groups.google.com/forum/m/#!msg/golang-nuts/S9goEGuo... For the record Python uses a generational garbage collector.

devanti 13y ago
You can run multiple instances of the same Go program to reduce the effect of GC lag