12 ms·
Trying out this Go thing at Disqus
- kasey_junk 13y agoThis article uses the term "realtime" and then discusses latencies in seconds and tens of milliseconds. It further says that Go approaches C in terms of speed. Can Go actually achieve C latency which is an order of magnitude faster than this, or is that hyperbole? How does GC impact that? What about C to Go inter-op. Is there a latency penalty?
- mattrobenolt 13y agoMost of the latency involves other components in our system. This isn't just computational time. Most of the actual computation with our transformers and whatnot are done in less than 1ms. Are you referring to the cgo stuff?
- kasey_junk 13y agoI was very curious about the claim that Go approached C speeds. I've not seen those claims before and the latencies cited certainly don't back that up. C inter-op speed would be a big part of any such claims as any systems language would need to support that.
- mattrobenolt 13y agoWell, the latency referred to is not factor of the Go code. It's more of a factor of our nginx front end which are handling a few million concurrent connections. The time spent doing computation is being measured in nanoseconds.
- btilly 13y agoIf you need hard realtime, go is not an appropriate solution. For a start you should expect occasional GC pauses on the order of tens of milliseconds, with no upper bound if you're using a lot of memory. That said, go actually can come close to C on sustained throughput. (On some benchmarks, naive go programs beat naive C, but usually go is somewhat slower.) You can interoperate with C using cgo, and there is no penalty. (It may not work with all go compilers.) If you want to interoperate with C++ then see http://www.swig.org/Doc2.0/Go.html http://www.swig.org/Doc2.0/Go.html for the best solution. I believe that there is a performance penalty there.
- cdoxsey 13y agoYes the GC has a large impact on this and yes C to Go inter-op has a heavy penalty. Memcachier did a talk at a Go meetup where they discussed a custom slab allocator they built to workaround some of these problems. Like C, Go is compiled to a native binary. Certainly something like GCC will tend to produce faster code, but there's nothing fundamental here preventing you from writing code that can run fast in Go. The real question isn't "can go achieve C latency" but rather "will every-day code written in Go achieve the same latency as every-day code written in C". And the answer is probably: it'll be slower, though usually in the same ballpark, but it will be easier to write, easier to maintain and have fewer bugs.
- kasey_junk 13y agoFor me it is a very real question of whether "can Go achieve C latency" because if I'm going to deploy a Go application it is going to be to replace very performance critical components that are currently written in C. If they weren't latency sensitive neither C nor Go would be my choice (I'd go higher up the abstraction hierarchy to make it easier to write, easier to maintain etc). If Go can't do that (or can't do that yet) it's fine by me, I'll just let it bake a few more years and reassess, but there have been tons of articles lately talking about how great Go's performance is. I'm just curious if that is an artifact of the original languages being compared to, or if it has broken through to C levels of latency. If it has, it becomes a lot more interesting to me.
- _ak 13y agoIn raw performance, Go is relatively close to C. But since you keep mentioning latency: the garbage collector makes Go unsuitable for hard realtime applications.
- Symmetry 13y agoWell, according to the benchmark game Go tends to be about 3 times slower than C on tight, highly optomized benchmarks[1]. Python tends to be 40 times slower than C[2]. Those, of course, might not be representative of your tasks, or the optimization effort you're putting in. [1]http://benchmarksgame.alioth.debian.org/u32/go.php http://benchmarksgame.alioth.debian.org/u32/go.php
- karaziox 13y agoThese benchmarks seems to rely heavily on math operations and the Go bench seems compiled with gc. I suspect that gccgo with the latest runtime (There was no release of gcc with the 1.1 runtime yet) can show prettier results with the use of the more mature gcc optimisations.
- mseepgood 13y ago> This article uses the term "realtime" http://en.wikipedia.org/wiki/Real-time_web http://en.wikipedia.org/wiki/Real-time_web "The real-time web is fundamentally different from real-time computing"
- mattrobenolt 13y agoThis is true. :) Our definition of "realtime" is slightly different than a CS definition. Sub-1s end to end time usually falls under "realtime" when it comes to the web, but it depends on what the subject is of course. In our case, that's perfectly acceptable. We're obviously shooting for lower, but at this point, we can't do much to improve network latency and other factors involved. So for now... this is good. :)
- sergiotapia 13y agoI _wish_ I had these traffic load problems to use Go. :(
- eridius 13y agoYou don't need to wait until you have traffic load problems before you start investigating Go.
- chuhnk 13y agoI concur. I've been using go for close to a year now, discovered it inside Google and continued to play after leaving. It's really a fantastic language to work with. You can get away with using type inference most of the time which keeps the language feeling very dynamic. The inbuilt library support is very mature and provides almost all of what you need.
- bentlegen 13y agoI think part of the lesson here is that Disqus went very far with their realtime service using Python, a tool they were very familiar with and allowed them to iterate quickly. Migrating to Go took them very little time, once it became necessary.
- kmfrk 13y agoHow would you normally go about stress-testing a Django app like that to look into bottlenecks? All the articles and talks I see on scalability and optimization use production servers as examples, but as you way, you want to optimizing for traffic instead of responding to it.
- bayesianhorse 13y agoThere's no way to stress-test a complex web realtime system without actually running it on the real hardware. You'd expect very nonlinear behavior between number of users, response time and number of machines used. Not being a part of Disqus, the only way I see to "predict" performance beyond the load already seen is to try and use past data from the current real system.
- orangethirty 13y agoI have done work with Python + Gevent before, and used it to developed the second Nuuton prototype. Go blows it out of the water. It is way faster, and simpler to develop. I went from a bottleneck of 3K hits/second to around 5K without any optimization (and a lack of general knowledge about Go). These numbers generated during testing, and not in production.
- genwin 13y agoThanks to Go, my website Niflet can handle 2K hits/second on a cheap machine, each uniquely and usually within 5 ms. I'm finding Go to be well worth the effort to learn it.
- bratsche 13y agoFor someone who hasn't invested much time into looking beyond the basic syntax of Go, can you give us a little info on what your stack looks like? I haven't gotten to the point where I've looked into things like web frameworks, templating engines, Postgres adapters, anything like that. But I'm interested in it.
- genwin 13y agoNiflet is my first foray into modern web development (I'm a database developer by day), so I kept it simple, just Go and SQLite. I'm using Go's webserver, which is basic but wasn't hard to extend to do compression and other niceties. Templating is via Go's template package. SQLite gets embedded, making the entire app a single 5 MB file. A free Dropbox account is used for nightly backup, handled by a scheduled goroutine (Go's concurrency feature). For a heavyweight web app that needs to scale to multiple machines I would've done things differently. It turns out that Go is so fast that needing to scale would be a luxury problem for me. In theory I can serve 70M users spread over a 10-hour day for < $100 a month.
- sbov 13y agoThe standard library templating in Go is one of my favorite features. Between html.template and text.template, I probably use it in over half of the programs I've written.
- dignati 13y agoThis is not the first company switching from a a dynamic language to Go. Python and Ruby are great for getting started quick but when you scale up you need a workhorse like Go. Nothing too fancy but nice features that give the language a modern touch. I guess Github and more will pick it up soon.
- encoderer 13y agoI find blanket statements about performance like this to be pretty meaningless. I develop both Python and Go as well, and honestly performance is not the number one criteria I've used in deciding which to use for a project. I ask questions like "Are there existing Python packages that tip the scale", "If I expect to need real concurrency, would it be IO bound or CPU bound", "Do I want to make the tradeoff of the increased stability of static typing in exchange for more man-hours on this project?" Things like that.
- eliben 13y agoIt's perfectly fine to be using more than one tool. Starting with Python can be a good idea because of the rapid development and turnaround while experimenting. If Python ends up being fast enough, that's great. But if it doesn't, then it's nice to have this new language which is almost native speed and still fun to write, unlike Java or C++.
- encoderer 13y agoSo in your opinion, languages are ranked in order of "performance"? See, in my experience, it just doesn't work that way. Sometimes it does, sure, but it's not a rule that I can use in my life. Seems like it would be nice if I could just say "I need this done fast, I'll use Python" or "I need 'performance' so I'll use Golang" And to be fair, many benchmarks in Go don't perform nearly as fast as C or even the JVM. Go has gotten faster with each point release but again, these sorts of raw performance benchmarks aren't really good for anything except providing an apples-to-apples comparison. Because it's not usually a question of building the same exact app in the same exact way in one language or another.
- danieldk 13y agotl;dr We rewrote some internal Python software in Go, it is now faster. Unfortunately, there is no technical content (which seems endemic to the recent storm of Go articles). Is it faster because is a static language producing machine code? Was their choice of Python packages wrong? How much are the performance improvements caused by knowing better where the bottlenecks are? Were other languages with larger ecosystems considered?
- mattrobenolt 13y agoThis was partially intention on my part. A lot of the problems that we had with our Python backends were a bit unknown. We had some guesses, and not the right instrumentation to dig into it. There are memory leaks in gevent, and the CPU that was being consumed was ridiculous. Internally, we had each played with Go for a little bit, and felt very natural to us. There weren't any other considerations really. For comparison, we're still dark writing to the old realtime just to make sure that nothing is broken, and old realtime is consuming easily 16x resources and technically doing a bit less since it's not really publishing anymore.
- vanderZwan 13y agoMy first compsci teacher was Harm Bakker aka Harm the Almighty Recursion Master, and anyone who followed his classes has the following quote imprinted in their brains: "... and then it's tempting to start kludging. I have a suggestion: get it right the first time." Whenever I'm programming I can still feel his disappointed stare of disapproval when applying a quick and dirty hack.
- mattrobenolt 13y agoFortunately, it's not as black and white in the real world. Quick and dirty hacks are a real thing that will never go away. If anything, having the ability to do those under stressful scenarios is a good quality to have.
- vanderZwan 13y ago
- Mr_T_ 13y agoBut but but no generics! How dare they write production software without generics?!
- prodigal_erik 13y agoGo code which avoids collections is typesafe, unlike the Python version they started with.
- jackielii 13y agoThey dare
- cs648 13y agoSeems to be an ever increasing trend to replace Python components experiencing high load with ones written in Go. Is this Go's niche? Definitely a language I will be learning in the near future, it doesn't seem like it will be going away any time soon.
- mattrobenolt 13y agoIf you're CPU bound, I think it's a good option to at least check out. It can work with real threads, and doesn't compete with the GIL. In our case, our actual CPU was low, but due to the high concurrency, any little bit of CPU work requires a context switch and contends with everything, for a little work. We alleviated that by just using a bunch of processes. Will it replace all of our Python? Highly doubt it. Some very specific pieces? Probably.
- james4k 13y agoDo you know about GOMAXPROCS? Sounds like tuning it could help.
- mattrobenolt 13y agoI was referring to our Python processes being CPU bound. Not Go. :) Yes, I'm familiar with GOMAXPROCS.
- TylerE 13y agoI'd say Go's niche is pretty much any sort of network server. Right combination of high performance, easy multithreading, and making it hard to shoot yourself in the foot (no buffer overflows, for instance)
- bayesianhorse 13y agoI've heard this several times. But the success stories are mostly concerned with smaller parts under heavy load. Go still is a lot harder to work with than Python if you know Python and its tools well... With Go you don't have anything comparable to Django, Numpy, Pandas...
- nobodysfool 13y agoThat's cool, but I am wondering where are the trials of say Cython or PyPy? Why is there a need to switch to a different language, where are the benchmarks or profiling to show where the performance issues lie? I like Go, and I plan to learn it myself, but I just don't get why people jump ship so quick from Python. This post makes it seem like Python is slow. I say it's not the language but the interpreter. GIL is nonexistant in Jython, or IronPython. It can be avoided with the multiprocessing module. Gevent may be good for regular python, but if you have the GIL getting in the way, best to switch to something else.
- swah 13y agoTotally agreed. Also, its a great merit being the first language systems are written on so many places...
- mattrobenolt 13y agoOur assumptions were that the GIL was getting in the way with our excessive concurrency. gevent + PyPy is not really a safe thing yet. This entire project was also stood up in a week, so it's not like this service is some monster service with a lot of work invested into it. It was small enough that instrumenting and figuring out where the problems may lie and figuring them out would have probably taken just as much time.
- nobodysfool 13y agoPerfectly true, gevent + pypy requires a separate build of gevent, but pypy comes with eventlet, so it might have warranted an investigation to see if it makes sense to switch. However, I suppose without details of the service itself, it's hard to make a judgment call from the sidelines. Still, glad you were able to improve performance. Why was 'Go' chosen? Nevermind, I see that was answered in another thread.
- ineedtosleep 13y agoWith changes like these I always wonder if the users actually see the difference in performance. Was there positive feedback after these changes were rolled out onto production, or rather; was there a _decrease_ in people complaining about server performance issues?
- thetylerhayes 13y agoYes.
- up_and_up 13y ago> Everything was Python, except for the queue, which was something implemented on top of Redis. Could that be... Resque? If so whats the problem mentioning you are using Ruby as well?
- mattrobenolt 13y agoWe were using Thoonk. Jobs were originally pushed from our Django app, into a Celery job, which consumed the message, and published into Thoonk. Thoonk was then consumed by the old Python backend. With this rewrite, we've also removed Thoonk and are using a fanout exchange in Rabbit directly. Go consumes the Celery job from the Rabbit queue.
- lysium 13y agoSo, Thoonk could have been the bottleneck and it had nothing to do with Go? Just kidding.
- mattrobenolt 13y agoIt definitely wasn't helping. But it was not our bottleneck, I assure you.
- programminggeek 13y agoGo is the new Scala.
- _ak 13y agoIt's more like the anti-Scala, because can read other people's code within a few hours instead of shrugging for 6 months because they just stumbled upon yet another language feature.
- mseepgood 13y agoGo has an almost diametrical design philosophy. The only commonalities are message passing and curly braces.
- programminggeek 13y agoPerhaps I wasn't clear enough. Go is the new hot language on HN. It's what startups are switching to and blogging about en masse. Before Go a bunch of shops switched from ruby and rails to scala for various reasons. Before scala companies switched from either php or something enterprisey like Java/C# to rails. Before that people switched from C++ to Java/C#. Before that C to C++. Before that ASM to C. Before that punchcards to ASM. Thus, in terms of the hype cycle, Go is the new Scala.
- DiabloD3 13y agoSo why did they go with Go, a largely unproven language instead of Erlang? Seems like the right choice given what Disqus does.
- zemo 13y agoThe type system in Go is as important as the concurrency system. Sure, Erlang can do concurrency as well as (or better than) Go, but it doesn't have the type safety that Go offers.
- tptacek 13y agoCan it do concurrency better than Go? How so?
- pcwalton 13y agoFor one, a shared-nothing design makes migration to different machines easier and doesn't require a stop-the-world garbage collector. Erlang is memory safe in the presence of many-core (or many-system) parallelism. Go is not (you can segfault, possibly in an exploitable way, if GOMAXPROCS > 1). Erlang unbounded channels reduce deadlocks because your sender can continue execution without waiting for a receiver.
- tptacek 13y agoHave we heard a lot of stories about deployed Golang apps having problems because of garbage collection? It's true that this is a significant designed-in advantage for Erlang, which can run it's collector on a process-by-process basis. What's also true and potentially compensatory is the C-like degree of control Golang gives you over how you allocate memory and lay it out. I'm not sure the unbounded channel thing is a real advantage for Erlang. I'm happy to be convinced I'm wrong. What's a real, correct design which would be hard to realize in Golang (without unbounded channels) that relies on unbounded channels?
- 13y ago
- eliben 13y agoIt is a shame there aren't more technical details in that post though.
- mattrobenolt 13y agoThere will be more in the future. :)
- eliben 13y agoThat's good to hear :-) As you see, there's quite a bit of interest in such articles. I presume you can give a lot of additional details without exposing any IP.
- camus 13y agowell,they went pretty far with python, at the same time,and refactored their app when they needed it,that's the way to go.
- joelthelion 13y agoDid you try PyPy? I'm genuinely curious.
- brettjurgens 13y agoaccording to the comments on the page, they did
- Kiro 13y agoWhat does realtime refer to in this case? Live updating of Disqus threads? Does that mean there's an open connection for every post I visit that has Disqus comments?
- mattrobenolt 13y agoYes. Our realtime updates vote counts and new comments over websockets if your browser supports it. If not, we fall back to some more typical HTTP streaming.
- knodi 13y agoI been using Go for few months now and love it.
- da_n 13y agoI love Go and continue to play with it, glad to see another story. I have to say I always thought Disqus were a cool team, I have been doing websites for over a decade and whenever I needed a comment system they were the first choice. It was a real shame when I came up against their recent decision to push out the 'Discover' feature. It resulted in my having to remove Disqus from most of my clients websites as I received several angry emails complaining about porn links (it wasn't porn, but sleazy celeb stuff like bikini photos etc). Sad to say I have written them off as a commenting platform; what they did, although probably a brilliant business decision and making them loads more ad money, felt like a complete betrayal of trust.
- drivebyacct2 13y agoAnother HN thread, another half a dozen troll comments spouting inaccuracies about the language, then covering up for gross misunderstandings of the language and then saying "but I'm still right". Gotta love Go threads.
- EugeneOZ 13y agoNow I can't authorize with social networks buttons. Some stupid password is requested. Guys, whole idea of authorization buttons is to not ask user for email and password.