22 ms·

How We Went from 30 Servers to 2: Go

688 points by treeder 14y ago

jianshen 14y ago
Hey that's my photo of the GOpher! :) [1] http://www.flickr.com/photos/jianshen/8080852738/in/photostream http://www.flickr.com/photos/jianshen/8080852738/in/photostr...
- drivebyacct2 14y ago
  :) I don't know if the small plastic or larger plushy ones are cuter. Love having them on my desk.
  - treeder 14y ago
    We got a bunch of the little plastic ones sitting behind my desk: https://plus.google.com/u/0/101022900381697718949/posts/1DbeLzTVzvu https://plus.google.com/u/0/101022900381697718949/posts/1Dbe...
- treeder 14y ago
  Hah, nice photo! Are you ok with that? Found it on google images.
  - jianshen 14y ago
    It's cool. :)
- carimura 14y ago
  Monitor Stands!! [1] https://plus.google.com/photos/100264837158995116368/albums/5854596778257472465 https://plus.google.com/photos/100264837158995116368/albums/...
- martinced 14y ago
  +1 for the cool monitor stands but geez... Invest some $$$ in a real keyboard: scissor switches are the most painful type of keyboard switches ever : (
  - redbad 14y ago
    The Apple wireless keyboard is the best keyboard on the market right now IMO.
phasevar 14y ago
Excellent success story with Go. Love this quote... "We also weren't sure if we would be able hire top talent if we chose Go, but we soon found out that we could get top talent because we chose Go."
- GhotiFish 14y ago
  That's really obvious but overlooked. Developers just looking for their paycheck will only have learnt the established languages. Developers who enjoy their profession will look to the future, excitement gleaming in their eyes. I've heard of business developing in Haskell a good amount of success. I suspect, for the same reason.
  - jmilkbal 14y ago
    It could also be said that developers who use established languages are excited to get things done rather than reinvent wheels continuously.
    - thirsteh 14y ago
      No, it can't. People don't use only Java because they "just want to get things done." They use only Java because that's all they learned, and all they need for their day job. If not, then they will invariably step over languages that are very interesting but obscure.
      - jmilkbal 14y ago
        I'll give an anecdote. I've spent the last 7 years using Ada, a language that is /both/ interesting and obscure, almost exclusively. It's really a fantastic language that is frequently neglected much like poor Go, but with more merit. Anyway, in preparing the launch of my company's first iteration of its podcast network (http://76streetnetwork.com http://76streetnetwork.com), I absolutely wanted to use Ada, but the allure of the utterly fantastic tool support and breadth of libraries available for Java had me up and running in a day or two despite having never written anymore than basic hello world-like crap with it in the past. There's really nothing crazy about Java, you're right, but I live for coding and I have a deep interest in quality software development, but I guess I'm just looking for a paycheck for this side project that will never make me any money! Ironically, Ada is what pays my bills.
jabagonuts 14y ago
Out of curiousity, I would be interested to know which version/patches of ruby you were using. With ruby I'm used to running out of memory on servers long before running out of CPU. Of course, most of our workloads are IO bound, so that may be the diffent.
- duaneb 14y ago
  while true; end
- treeder 14y ago
  This was on Ruby 1.9, 2010 era. No patches. Memory was definitely a concern and probably contributed to some of it. Just to note though, after a machine was taken offline by the load balancer, it would generally come back to life.
  - TresAmiga 14y ago
    Memory "probably contributed to it"? You don't even know why the old version was slow but you want to credit golang with making it fast? Hype much? The lack of critical thinking here on Hacker News is astounding.
- FooBarWidget 14y ago
  When running I/O bound workloads you really should be using a multithreading-capable app server. For example, if you're using Unicorn then that is extremely bad for I/O bound workloads (the Unicorn website's Philosophy page documents this under section "Just Worse in Some Cases": http://unicorn.bogomips.org/PHILOSOPHY.html http://unicorn.bogomips.org/PHILOSOPHY.html). On the other hand, something like Phusion Passenger Enterprise 4 with multithreading turned on is excellent at handling I/O bound workloads and reduce your memory by a significant factor.
bokglobule 14y ago
Why was Go chosen over NodeJS ?
- treeder 14y ago
  Mostly because I can't stand Javascript. It makes me cringe just thinking about it. Go is much nicer to work with.
  - thibaut_barrere 14y ago
    Although I personally like CoffeeScript, I must say that Go fills me with joy for some reason. It feels "fresh" and lightweight to the beginner, and reminds me of TurboPascal in some ways.
  - methehack 14y ago
    Hey OP! I appreciate your sharing. Since you came from ruby and we're on the topic of the language itself, I'd appreciate your impression of how well Go supports collections. Is there or could one write something like http://underscorejs.org/ http://underscorejs.org/? Can you do this kind of thing? [1, 2, 3, 4, 5].reject {|i| i < 3}.map {|i| i + 9} I did the go tutorial the other day and I became a little worried that one would not be able to do this kind of thing without a bunch of unwieldy declarations. I think that would be a showstopper for me. Thanks for any insight you might have!
    - jbooth 14y ago
      Go has first-class functions, so you could write a library do the same thing with, yes, a few more declarations around it. In your example, .map { |i| i + 9 } is .map( func(i int) int { return i + 9 }), or .map(myFuncDefinedElsewhere) in most real-world scenarios. As a commenter upthread pointed out, Go and serverside Javascript are aimed at very different use cases and audiences.
      - deleted 14y ago
        [deleted]
jewel 14y ago
For large projects, it'd probably make more sense to rewrite the critical code sections in C. rubyinline makes this really easy, and ruby C extensions aren't that bad either. One time on a contracting gig I had some code that was interacting with cairo drawing and imagemagick. They didn't have a compatible raw image format at the time (one was RGBA and the other was ABGR, IIRC.) It's trivial to convert between the two in ruby, but it was taking 60 seconds per image. Once I switched that one loop to C, it took 0.02 seconds. I've got nothing against Go, just suggesting an alternative that might make more sense in some situations.
- bad_user 14y ago
  For web servers there are other things more important than the raw processing speed of a single routine. What you want is to get the CPU usage to 100%, CPU resources used preferably in actual processing and less on things like garbage collecting or blocking for I/O. For web servers in particular it is tricky since processing requests also involves a fair bit of I/O interwoven with CPU processing (in the case of apps that aren't just CRUD over a DB).
- dyselon 14y ago
  This definitely does make a lot more sense than throwing out your whole app and rewriting it in many situations, but I'm always a little hinky about moving stuff into C just because it has so many ways to shoot yourself in the foot. You can obviously create bugs in any language (I know I certainly have!), but Go really does hit a nice spot between safety, productivity, and speed, so I can definitely see the attraction.
siavosh 14y ago
I've read this domino-effect on server clusters on several HN postmortems and I've seen various flavors of this on our own web servers. I'd like to think there's a simple configuration in most servers that prevent 100% cpu utilization from taking place and preventing the server from telling its cluster that it's still alive. Anyone have any experience with this?
- laumars 14y ago
  It's not really possible without adding resources (ie a new server). The problem is, once your servers get saturated, requests queue up, processing slows down and users start to hit refresh which artificially and exponentially increases traffic. Then to compound things, servers buckle, which means you lose a resource right when you needed it the most. A domino effect is one analogy, another might be how a small hole in a damn can buckle and blow open from the force of the water pushing through.
  - ef4 14y ago
    > It's not really possible without adding resources (ie a new server). That's not quite true. With the right architecture, performance will degrade gracefully. You'll get slower and slower responses as the load goes up, and eventually start dropping requests, but the servers will not die and there will be no cascading failure. One way to achieve this is to make sure the queuing happens at the load balancer, and no large queues are allowed to build up in the individual application servers.
    - laumars 14y ago
      Well yeah. You can do all sorts of tricks from TCP/IP hacks to streamline HTTP requests through to disabling queuing entirely. But my point is you cannot entirely prevent your site from saturation without adding extra servers to your web farm (and more so, that siavosh's method of taking servers out of service has the inverse effect of what he was trying to achieve). Thus all you can do is slow the escalation in the hope that the traffic peaks before your resources buckle. The only method I'd found that is "guaranteed" to prevent such outages is the use of sorry pages (ie a static page stating "We're experiencing high volumes" which users are directed too if the dynamic page connections are maxed out). However even that is just essentially a prettier version of a page time out - and I mean this in terms of usability rather than technicality. ie the site is still unavailable, but you're killing the connection in a user friendly way rather than allowing connections to stack or just flat out disallowing "> n" active TCP/IP connections.
grey-area 14y ago
Interesting to hear of Go being used in production. It'd be great to hear some more details on your setup when deploying the go processes - how are you managing failover, what's your load balancer, and how are you handling swapping out processes etc? Are you compiling on the server or local machines before deployment? Most other languages have lots of solutions on the deployment side now but Go is so new there isn't much info out there. Also, as you are running an API presumably your app in Rails was pretty simple, did you have the impression things would be more complex in Go if you were writing an app with an extensive front end and UI and using sql? I'd miss all the view helpers etc available at present I think. Going from 30 servers to 2 certainly sounds like a huge improvement, so it was definitely worth it for you, are you thinking of writing any front-end apps in Go? I've been playing around with Go recently and it is a fantastic language for someone coming from dynamic languages like Ruby. I particularly liked interfaces as a way to define a contract for implementations to follow, and the simple package system which encourages you to make your code modular.
- carbocation 14y ago
  For my (toy) apps, I've been compiling locally and pushing to the server. It's trivial to compile for a target platform and architecture, and you get a single compiled binary. Certainly my response isn't getting all the way to what you're looking for, but it's not difficult to write shell scripts that manage the deploy process from here.
  - grey-area 14y ago
    Thanks, I use rsync wrapped in a script usually (because I'm dealing with more files than just a binary), haven't tried cross compiling yet but was going to give that a go. I suppose I was more concerned about what happens when things are on the server and how multiple processes can be managed. As soon as you have more than one and want to swap out server workers seamlessly and load balance etc it gets a little more complicated.
    - carbocation 14y ago
      Yes, I think you're right on the mark re: swapping out workers, etc. Those nuances I haven't had to deal with due to these being toy apps.
donebizkit 14y ago
Last month I picked NodeJS and build couple of sites with it. This month, I want to pick up something new. I was hesitating between Python and Go. Can you answer couple of questions about Go from someone coming from NodeJS: 1- What is the state of the external Go libraries, especially DB (MySql), caching libraries (memcached), protocol libraries (Oauth). Are they stable 100% 2- How easy is logging and tracing in go? Thanks.
- paddyforan 14y ago
  Memcached's author (Brad Fitzpatrick) is a Go user, and wrote a memcache client for Go. Go has a bunch of good stuff surrounding SQL. I haven't used MySQL, personally, so I can't comment on which MySQL driver is best, but I know YouTube uses Go for something related to MySQL, though I can't say for certain exactly what. http://code.google.com/p/vitess http://code.google.com/p/vitess The goauth2 (http://code.google.com/p/goauth2 http://code.google.com/p/goauth2) library is written by Brad Fitzpatrick and Andrew Gerrand (two members of the Go team), and is stable. Logging is great. Interfaces make it really, really flexible. I'm not sure what you mean by tracing. Stacktraces? Those are easy to retrieve: http://golang.org/pkg/runtime/debug/#Stack http://golang.org/pkg/runtime/debug/#Stack
  - tptacek 14y ago
    Well, Go has a bunch of libraries for different SQL servers. I wouldn't go so far as to say it has lots of good SQL stuff; it's in approximately the same place as C is w.r.t. databases. Having to interface with complicated SQL is a reason not to use Go (but not an insurmountable one).
    - paddyforan 14y ago
      I think it has lots of good SQL stuff just because I think it laid a nice foundation for developers to build SQL libraries on, and I think there's a lot of potential for using SQL in Go to get dramatically better in the next year or so. That said, I'm not an SQL expert or even very competent, so I'm probably not the best to comment on it--I just included it for the sake of addressing all the points. I would agree, though--working with SQL is the worst part of the software I'm currently writing in Go. I think that's more a praise of working with Go than a condemnation of its SQL support, though.
goatslacker 14y ago
It would be interesting to hear some of the reasons you went with Go vs Clojure (and also the other languages you mentioned)
shurcooL 14y ago
> "We also weren't sure if we would be able hire top talent if we chose Go, but we soon found out that we could get top talent because we chose Go." I feel[1] that a smart/talented C/C++/anything developer can go from someone who has never seen or heard of golang to a proficient and productive Go developer in a matter of a few weeks, maybe even _days_, if not less. That's how long it takes to go through the following materials (and fav some for later reference) and play with the language a bit. http://golang.org/ref/spec http://golang.org/ref/spec http://www.youtube.com/watch?v=ytEkHepK08c http://www.youtube.com/watch?v=ytEkHepK08c http://commandcenter.blogspot.com.au/2012/06/less-is-exponentially-more.html http://commandcenter.blogspot.com.au/2012/06/less-is-exponen... http://tour.golang.org/ http://tour.golang.org/ http://golang.org/doc/install http://golang.org/doc/install http://www.youtube.com/watch?v=XCsL89YtqCs http://www.youtube.com/watch?v=XCsL89YtqCs http://golang.org/doc/code.html http://golang.org/doc/code.html http://golang.org/doc/effective_go.html http://golang.org/doc/effective_go.html http://golang.org/pkg/ http://golang.org/pkg/ - use as reference http://www.youtube.com/watch?v=f6kdp27TYZs http://www.youtube.com/watch?v=f6kdp27TYZs http://talks.golang.org/2012/splash.article http://talks.golang.org/2012/splash.article https://gobyexample.com/ https://gobyexample.com/ http://golang-examples.tumblr.com/ http://golang-examples.tumblr.com/ And a some more similar things that you can mostly get to from golang.org site. The beauty of how concise the language and even its website are, is that you can literally just go through everything there one thing after another. [1] This is my personal opinion based on playing with go the last few weeks/months. I'd love to verify this theory. It's not yet the primary language in which I do things in (I use C++11 atm), but for all my side tasks[2] it proved to be indispensable. And I found it very easy to pick up. I can't wait until I start doing all my work in Go, that will be a true test of its productivity efficiency. [2] https://gist.github.com/shurcooL https://gist.github.com/shurcooL
- TillE 14y ago
  It's quite strange to me that people would identify as or look for a "[language] programmer". Sure, I happen to write more C++, Python, and C than anything else, but I've dabbled in just about everything and could reach comfortable proficiency in a matter of weeks. Most of programming and all of computer science is universal. Any serious programmer should be a polyglot by default.
petrohi 14y ago
Why Go over Erlang?
- Ixiaus 14y ago
  (Note: I'm a big Erlang supporter and 90% of my startup's code is written in Erlang) Erlang is not memory efficient (compared to Go? I don't have benchmarks to back this up) AND the learning curve for developers not exposed to functional programming (which is a lot) is quite a bit higher than it is in Go.
mrwnmonm 14y ago
man, you just made me worry very much now - we are making taxi dispatching api using ROR - and we will lunch it soon
- thibaut_barrere 14y ago
  Why worry? If you can handle the initial load, you can still optimize later on, rewriting some functions or parts of the system with something faster.
  - mrwnmonm 14y ago
    great
- carimura 14y ago
  don't worry... our Ruby servers kept us churning for a long time.
- programminggeek 14y ago
  If perf is that much of a worry, you can look at JRuby, that will get you a bit (or a lot) farther from what I've seen.
laumars 14y ago
Has anyone had an experience of running websites from Go; or more specifically, how you handle none-HTML content? I've been considering porting my CMS from mod_perl to Go, but I'm not sure how you work with the other files (CSS et al). I did read somewhere that you run Go from Apache but it's not recommended.
- carbocation 14y ago
  I've just started creating toy projects in Go. I put these Go projects (and Apache, which runs older PHP projects) behind nginx. For hosting CSS, you either generate it programmatically and send the right Header, or you can use http.FileServe from the standard library [1]. (Surely other approaches are possible, but those are the two I've played with so far.) [1] https://code.google.com/p/go-wiki/wiki/HttpStaticFiles https://code.google.com/p/go-wiki/wiki/HttpStaticFiles
  - laumars 14y ago
    Thanks for that. The issue I have is that I would prefer to keep Apache around if I can (I have a few sites - some of which I host for friends). While my web server does have a few IPs attached, I'd rather not have to buy more IPs just to separate Apache from Go. And to be perfectly honest, I do quite like Apache. (each to their own I know, but I've had little reason to complain about it).
    - carbocation 14y ago
      Right, you're describing my situation exactly. I used to run Apache solo. Now I run Nginx, which forwards requests to Apache or to my various go apps based on the domain name. Apache and the go apps just listen on different ports internally. The server only has 1 IP address.
      - laumars 14y ago
        You are an absolute star. If I could up vote you a hundred times I would. Thank you
jscottmiller 14y ago
Can you describe your workload a bit as well as any benchmarking you did to determine (and perhaps optimize) hotspots in the ruby code before you started the port? Personally, I've been interested in moving to Go on a python-based project of mine. Thus far, I've avoided it because 1) the extra work required to self-implement a few third party libs I rely on and 2) I've been able to eek out sufficient performance using c extensions and cython.
- treeder 14y ago
  This first rewrite in Go was for the IronWorker API so all the operations are here: http://dev.iron.io/worker/reference/api/ http://dev.iron.io/worker/reference/api/ There were less endpoints back then, but you can get the idea. The most heavily used operation being queuing up tasks/jobs.
slem311 14y ago
great stuff!!
orangethirty 14y ago
What changed in the architecture? It is rather interesting that by just having a language change you could remove 28 servers from the system.
- ikeepforgetting 14y ago
  My guess is that removing 2-5 layers of abstraction can get you pretty far along.
  - buro9 14y ago
    We're writing most of the systems for our company in Go, and of the many reasons this ranked highly. Just removing the multiple unnecessary abstractions gets you a hell of a lot of simplicity and performance back.
    - ikeepforgetting 14y ago
      I work in an area that requires a lot of high performance computing and there is a reason people in this area like code simplicity. Stacking layer upon layer of complex algorithms is a lot more difficult to reason about than having a simple and elegant system that's close to the metal. There is a recent trend of building more and more complex systems, have less coupling and 'good oop'. That's all good and nice but simple code is: - more maintainable (a complex nest of objects & function calls all loosely coupled isn't) - a lot more readable - easier to optimize - better for your sanity Some languages have build a culture of extreme complexity and are notable for building huge structures and systems to do simple tasks. Having a 50K lines of code dedicated to just inversion of control or O/R mapping makes a problem a lot more complex than it probably needs to be.
      - martinced 14y ago
        "Having a 50K lines of code dedicated to just inversion of control or O/R mapping makes a problem a lot more complex than it probably needs to be." Exactly. That's the terrible stuff with all this "enterprisey" mindset: people are working on medium-sized codebase made of this special kind of hell that Java/C# + ORM ([N]hibernate) + XML + SQL is and these app often run into the 200K / 300 KLOC lines if not more. Yet what does these applications really do? Actually not very much. Yet these programmers are sure to be working on super-advanced stuff because their codebase is big. When several companies reported a drop in LOC of 90% by switching to something else than Java then at one point you have to at least consider that maybe most of your Java codebase is hot air.
zobzu 14y ago
It's not really "go" that makes the difference. it's how the runtimes and frameworks are used and/or made. frameworks on top of frameworks, all being over engineered, with poor understanding of what the system actually does, result in super slow apps on top, that you generally go to aws to scale. It's not the first time that I see people reducing a dozen servers that were "always maxed out" by a couple of servers "that can barely feel the load". Yeah throwing hardware at the issues is fine'n all but we're been way over the limit too many times in too many directions. Go provides a clear start/APIs/framework, and the language enforces good habits. Also, it doesnt have things like global interpreter locks. The lesson? Stop using cool techs because there's blogs about them. UNDERSTAND the tech before using it.
- pjscott 14y ago
  Nice rant, and I mostly agree with it, but there are a few slow things about the ways that some languages are usually implemented that don't really have much to do with over-engineering. Let's take this line of Python code, for example, and look at how the CPython interpreter runs it: return x + 42 This turns into the following bytecode instructions: LOAD_FAST 0 (x) LOAD_CONST 1 (42) BINARY_ADD RETURN_VALUE First we get the x variable from the local namespace. This is an array of PyObject pointers; the LOAD_FAST oepration simply does an array lookup and puts the result on the stack, incrementing the reference count. Pretty fast. Next is LOAD_CONST, which is even faster; it takes an already-allocated PyObject and puts a pointer to it on the stack, incrementing the reference count. BINARY_ADD removes two numbers from the stack, dereferences the pointers to get the integer values, allocates a new PyObject with the resulting integer inside, bumps up its reference count, decrements the reference count of the two operands, and pushes the result on the stack. Finally, RETURN_VALUE jumps back to the caller of the function. In Go, the corresponding code would be compiled to two, maybe three machine code instructions. There are good reasons why Python does it this way, but it does suffer some inherent slowdown.
  - kevingadd 14y ago
    A more interesting comparison would be the PyPy-generated native code for the expression versus the compiled code from Go. Comparing an interpreter with a compiler is generally uninteresting; the compiler almost always wins. If you look at the code that comes out of various modern JITs you often will find really interesting differences in the native code they produce, even for simple constructs like arithmetic. Type checks, barriers, bailouts, etc - in addition to more mundane differences, like one using SSE to do floating point and the other using x87, and one JIT having a better register allocator.
conorh 14y ago
I agree with the 'Go is fun' (and easy to pick up) sentiment. I spent a few weeks recently writing an NTLM library for Go for a consulting client (I hope they'll let us open source it at some point). All the parsing and bit manipulation was extremely straightforward due to the excellent standard libraries and I was able to quickly move from not knowing Go, to a fully implemented NTLMv1/v2 library!
deleted 14y ago
[deleted]
jroseattle 14y ago
Would be great to hear more specifics of the rewrite. What did the API do, what pieces were built in RR, what got replaced in Go, etc. At least as much as you're willing to share. :-)
rmoriz 14y ago
How about BDD, test-driven development and quality insurance? Does Go provide an ecosystem that supports agile refactorings that are common in lean startups? You can say anything bad about the performance of Ruby and Rails etc. but rspec, cucumber, capybara, vcr, factorygirl are really important features to start from zero and reach a viable product.
- burntsushi 14y ago
  > test-driven development `go test`. > Does Go provide an ecosystem that supports agile refactorings that are common in lean startups? Go has a compiler and is staticly and strongly typed. Which automatically makes it at least an order of magnitude easier to refactor safely. Not really sure about the other stuff you mentioned. I've never even heard of BDD before.
  - gregors 14y ago
    I've heard this argument before e.g. we don't need to test because the compiler catches errors. The compiler catches the mot trivial of error conditions. Unfortunately it doesn't address faulty logic/behavior.
- GhotiFish 14y ago
  I rather like what go has set up for test's. They made it as small and functional as they could. Out of curiosity. How does a language specifically support "agile refactoring"? What features permit (or hinder) stopping every couple of weeks and removing the accumulated WTF's in the codebade? Want to buy a left handed stapler?
- paddyforan 14y ago
  Go's interfaces make it really, really easy to change things with very little code changing. Go's testing tools are superb.
  - rmoriz 14y ago
    Can you blog about it in the future? Couldn't find good resources and tools about testing in Go. Especially with concurrency this seems to be very important.
waterside81 14y ago
Go's definitely picking up momentum. I know that Mozilla is shifting much of its services infrastructure to Go. We're porting over our Arabic sentiment engine, currently in Python/Cython to Go. If you're dealing with simple data structures, going from Python to Go is almost a line-for-line port, but the performance benefits are, of course, massive. Our benchmarks show a 40x speed improvement so far. Lastly, for anyone thinking about taking the plunge, use Go tip (from their source code repo) - don't use their "stable" releases. They fix bugs so fast you'll always want to be current.
- DASD 14y ago
  Do you have a citation for Mozilla using Go? I would be a bit surprised given their(Mozilla) development of Rust.
  - waterside81 14y ago
    Sure, check out their github pages (here's a few): * https://github.com/mozilla-services/heka-mozsvc-plugins https://github.com/mozilla-services/heka-mozsvc-plugins * https://github.com/mozilla-services/heka https://github.com/mozilla-services/heka My friend works for Mozilla - that's how I knew.
    - DASD 14y ago
      Thanks waterside81. Can anyone explain the difference between Mozilla and Mozilla services? A cursory glance seems some of the Github repositories overlap.
      - rfk 14y ago
        Mozilla Services is the team within Mozilla that builds/runs much of the backend infrastructure, e.g. the firefox sync servers, marketplace servers etc. The existence of separate "mozilla" and "mozilla-services" github projects is largely a historical accident, since different teams started moving to github organically at different times. (Source: I work for Mozilla, on the Services team)
TakeTwo 14y ago
"We decided to rewrite the API" and now it is over 300x faster. Go had nothing to do with this, it was obviously some boneheaded original design. Probably something like fork+exec on a Ruby VM on each request to enter a chroot. I know HN has a hard-on for Go but come on.
- drivebyacct2 14y ago
  Are you the same asshole making throwaways to shit on all of the Go postings or are there actually more than one of you going around making assumptions, accusations and spreading falsehoods about Go. I can't get over how pathetic it is to make a throwaway for that comment, I'm more embarrassed five people validated your comment and attitude.
saosebastiao 14y ago
This could easily be rephrased as "How We Went from 30 Servers to 2: Static Typing and Compiled Code".
s_baby 14y ago
Why Go over VisualBasic?
rcb 14y ago
Why did go "win" over erlang, scala, and clojure? Because it's becoming trendy and "fun?" And, why are "fun" and "joy" terms ruby bloggers use for languages? Is this code for "easy" and "familiar?" As in, "this language is easy to learn.. It does not require us to learn difficult but mind-expanding concepts to become proficient." Replacing ruby/rails with something (just about anything!) results in far fewer servers. Isn't this obvious to all? Is this really news to the average HN reader? If that's an I/O bound API server, bet those two servers could be brought to one in a language/runtime that's more productive but a little less "fun" to learn.
- rektide 14y ago
  One could adopt some kind of event-based system in scala or clojure, but Erlang and Go are alike in being the lone runtimes where running millions and millions of very small messaging processes is AOK no problem for the runtime, and not something one has to work really hard for. Fun and joy are terms applicable here because the alternative is Erlang. Zing! Go is rather ideal for these guys use case: if it wasn't an easy win, given what a snug fit their use case is, there would've had to have been red flags abound, as this really is a nearly idealized work load for Go's use: a hell of a lot of processes which sit around doing nothing, where one occasionally gets a message and forwards it along. Perfect Go story, as Go's lightweight processes (goroutines) are ideal for this kind of Communicating Sequential Processes routing workload.
  - Peaker 14y ago
    Haskell's runtime makes for cheaper threads than Erlang. Possibly cheaper than Go's, too.
    - rektide 14y ago
      Are they also in Haskell as in Erlang pre-emptable by the runtime? There's certainly some cost in this design decision, more context that has to be swapped in and out and more state kept, but it's important capability to allow blind design & use- think Node, where everyone has to keep re-iterating how important it is to keep yielding to the event loop, to not do a lot of CPU work in a handler: irrelevant in Erlang world- in spite of the threading model not being OS threads, Erlang is happy to drop your lightweight thread on the fly when it sees fit.
raphaelj 14y ago
I'm getting the same experience here by entirely rewriting an old PHP web application [1] in Haskell using the Yesod web framework. I've been using JMeter to benchmark both versions of the application. On a 10€/month dedicated server [2], the Haskell one was able to generate 220 dynamic pages per second [3] whereas the PHP one tops at 35 pages per second on a equivalent page. Moreover, concurrence capabilities of Haskell are also pretty sweet : while I was benchmarking the web app using 2,000 concurrent connections, the application server was only using around 90MiB of RAM. I was not able to increase the number of concurrent connections as the client application I was using started to kill my quad core desktop, I suspect Haskell to be able to manage A LOT MORE concurrent connections as I didn't see any decrease in the throughput of the application as I was increasing the number of concurrent connections. [1] http://files.getwebb.org http://files.getwebb.org [2] http://www.kimsufi.com/fr/ http://www.kimsufi.com/fr/ [3] Static content has been ignored as modern servers like NGinx seem to be able to carry the static content (CSS, images, ...) at more than 5,000 req/sec on the same machine.
riobard 14y ago
“the entire process started up with only a few hundred KB's of memory (on startup)” There is no way this could be true: last time I checked, a bare minimal Go HTTP server requires at least 2.8MB of memory on 64-bit machine. Are you using the default net/http library?
- codygman 14y ago
  Yeah, my HTTP server made in Go usually starts up at around 6MB or so.
topbanana 14y ago
Alternatively... why we needed 30 servers: Rails
programminggeek 14y ago
I wonder how far they could have got going more of a bare-bones Sinatra app using JRuby. It's still not nearly as fast as Go obviously, but take away Rails and some of the more dynamic usages of Ruby on a faster VM and I'd venture to guess you might have been able to go from 30 servers to 10-15? That's still not 2 mind you...
rektide 14y ago
p.s. come disrupt our business: with nearly no hardware, no costs, and no maintence burden you too can pass OMG huge amounts of message traffic. this is one bold as fuck post.
rachelbythebay 14y ago
This story sounds mighty familiar. I realize the topic is about one language over another, and most of the comments here seem to be focusing on that, but there's another part which I find fascinating: the cascading failure mode when they'd lose a web head. "This would in turn cause the load balancer to think it failed and take it out of the pool, thereby applying the load that the unresponsive server would have been handling to the remaining servers. And since the remaining servers are now handling the load of the lost server plus the spike, inevitably a second server would go down, the load balancer would take it out of the pool and so on." It was 8 years ago, and the customer was running PHP, not Ruby, but otherwise it's the same basic story that lead to the creation of "Surge Protector", or its actual name, "Suicide Pact". http://rachelbythebay.com/w/2011/06/28/sp/ http://rachelbythebay.com/w/2011/06/28/sp/
- pilsetnieks 14y ago
  Keeping in theme with the suicide pact, STONITH, the other kind of pact for high-availability machines comes to mind - Shoot The Other Node In The Head.
dubcanada 14y ago
You have an error in Opera 'SyntaxHighlighter.config.bloggerMode = true;' Undefined variable: SyntaxHighlighterError
stalcottsmith 14y ago
I've seen a lot of dumb Rails server configs and even dumber usages of Rails without any tuning at all. With a couple weeks or a month of work I could shrink the hosting fees or resources consumed by a factor of 5 out of most Rails apps that are sitting up around 30 servers... and probably a factor of 10. Leaving aside whatever rookie or even intermediate mistakes were made in their Ruby code or their database, this post indicates a lack of understanding of what happened when their server fell over. Proper tuning of a deployment should not trigger a 100% failure mode like this. These folks were itching to get off of Ruby for whatever reason... after all their roots were in Java. If your goal is to do a rewrite and learn a new language and gain some notoriety why waste time learning what you did wrong with Ruby or your server config?
- dkkkdkdk 14y ago
  TLDR: I'm butt-hurt that people are dumping Ruby. And because I hate Java I'll blame it on it also.
  - sabat 14y ago
    Not quite. He's saying that people who think Ruby is slow are often people who have no idea how to performance-tune it.
    - bpicolo 14y ago
      Well, more of the time it's because Ruby actually is, in fact, slow.
      - pjmlp 14y ago
        I though the implementations were slow or fast, not the languages.
        codygman 14y ago
        Typically a language is considered slow or fast based on it's most popular implementation, sadly.
taproot 14y ago
> with Ruby. Title should have been: "How we went from 30 servers to 2: Replacing Ruby"
saidajigumi 14y ago
I'm missing the connection between poor API performance under Rails and the decision to do a ground-up rewrite. Was the initial performance problem due to an API problem or a platform problem? Was there any profiling of the poorly performing API server? I'd love to hear the story behind that analysis and how it impacted the tale told in TFA. I ask all this since I've lost count of how many times I've participated in or witnessed an averted rewrite via a good dose of profiling and a few key bugfixes. I'll acknowledge that's not as fun as a clean-slate project, but vast amounts of engineering time were saved.
zenocon 14y ago
It's an interesting tale, and I honestly have no bias toward Ruby or Go, but the cause/effect relationship is poorly illustrated (we switched languages). We all know runtime / performance / scalability is more complex than that. I'd like to hear where the bottleneck(s) were and how Go solved them.
trungonnews 14y ago
I wish Go has named arguments like Python and Scala. Reading and debugging code could be so much simpler.
drivebyacct2 14y ago
Man, what is it about Go that just brings out the pricks of HN? There are over three throwaway accounts created exclusively for this thread to shit-post about Go, to steal an old term I have never, ever used on HN before.
- jff 14y ago
  I'm not exactly sure, I've noticed the same thing. Go appeared a few years ago, quickly became pretty well-known, and continues to grow in popularity. Maybe it's seen as "threatening" to older projects like Ruby and Python? The most vitriolic comments always seem to come from Ruby, Python, and Rust partisans. Of course, I'm not sure why they're so threatened by it, they themselves explain in great depth how the lack of generics or exceptions will prevent anyone from writing any real code in it, or even trying to in the first place!
jebblue 14y ago
http://www.dartlang.org/support/faq.html http://www.dartlang.org/support/faq.html >> Q. How does Dart relate to Go?# Dart and Go are both language projects started at Google, but they are independent and have different goals. As a result, they make different choices, and the languages have very different natures, even while we all try to learn from each others' work. Hmm, don't think I want to bet the farm on that.
- drivebyacct2 14y ago
  Don't want to bet the farm on what, do you mean Dart or Go sticking around? Go is used extensively through Google. It powers a significant DB layer that powers YouTube. It serves up all of Google's static downloads and several employees hint at other significant projects they can't speak about. I've only dabbled in Dart, I can't speak to how vital it is, currently it only seems viable when translated to JS or when using in a server-side VM, which is certainly an option if you want. (sorry, I originally had written Google where I meant YouTube, see "Vitess" for more info)
rurounijones 14y ago
Ok, this bit has left me confused. They were Java devs that liked ruby, they wrote applications in Ruby on Rails and the ruby apps were hitting limits so they immediately started looking at other languages. But they don't mention the most obviously (to my mind) simple option. JRuby It is ruby (They like ruby). Most ruby apps can be run on JRuby with very very little changes (No need for a big rewrite) and it runs on the JVM with which they are familiar and it is very fast (and true multi-threading). Maybe they did look into it but I would have thought that would be #1 on their "We tried this but discounted it" list.
- oscardelben 14y ago
  I'd like to see some real benchmarks with a real app, but I doubt moving to jruby would bring the magnitude of improvement mentioned in the post.
- pkulak 14y ago
  In my experience, rewriting for JRuby is as much work as re-writing in a new language. Last I looked was a year or two ago, so it may have all changed, but it's not as simple as just moving your code over. Many really important gems don't work, or don't work properly.
  - boundlessdreamz 14y ago
    The only gems that do not work are those which are not pure ruby. For the popular gems I think there have been rewrites. I haven't looked into JRuby but it has changed quite a bit in the last year from what I read
    - johnb 14y ago
      I don't think that's true. There are plenty of threadsafe gems with C extensions, and plenty of non-threadsafe pure ruby gems that mutate class/class-instance variable willy-nilly.
      - boundlessdreamz 14y ago
knodi 14y ago
Go 1.1RC just around the corner in April. A lot of good stuff in Go 1.1RC.
trungonnews 14y ago
I was originally very excited about Go when I first learned about it. But then I got tired and frustrated quickly after having to listen to the other Gophers telling me that I don't need this or that feature because there is a better way to do it in Go. Like. I don't need exceptions because Go function can return multiple values. I don't need a mocking framework like Mockito because Go has interfaces. I don't need an interactive debugger because I can debug with command line using gdb. I don't need named arguments because I can instantiate a struct and call my function with it. (Have you seen Ruby or Javascript code? Almost every function takes 'opts' as a single argument. Go is probably going down this path too.) Then I learned about Scala. I'm not saying that Scala is better than Go. However, it has everything that I need. :)
- yannk 14y ago
  tl;dr: I tried Go, but it wasn't like Ruby, so now I'm trying Scala.
  - trungonnews 14y ago
    Ha. Is there a high performance, compiled, and concurrent friendly language that looks like Ruby? Please share. :)
    - burntsushi 14y ago
      Rust might be your best bet. I think better advice for you is to just stay away from opinionated languages. I don't think Rust falls into that category, but I'm not sure.
    - qznc 14y ago
      Looks like Ruby? You mean syntactically like "end" instead of "}", I assume. Sorry, but the C syntax is dominating the native-compiled world. You could try Pascal, they have "end". Or use C with preprocessor magic: #define END ;} If you mean Ruby like terse, generic code, then try Rust, Go, and D.
    - pjmlp 14y ago
fernandezpablo 14y ago
There's simply no technology that makes you switch from 30 machines to 1 (you say you keep 2 just for reliance). Obviously there must be other architectural changes in play here (I'm guessing RoRs thread-per-request vs some kind of event loop on Go?). Please be more specific.
- genwin 14y ago
  Go is good at keeping all the cores on the machine pumping. Maybe they switched to machines with more cores.
moron4hire 14y ago
There are a lot of things that can be said about the "average" programmer in such-and-such languages, what they do and don't do, what habits they have, etc. But I'm not interested in working with merely average people.
- smrtinsert 14y ago
  That's right. People like you and me work with the top 5% of programmers! It's because I'm smrt.
free652 14y ago
Uh I think I love it. Easy to learn language and awesome performance.
fwee 14y ago
Rails is not for API,why not use eventmachine?
desireco42 14y ago
I went briefly to through the comments, it seems that no-one is bothered with lack of details in this post, things are not logical at all, almost like it was written by PR agency in charge of promoting Go. Except it is established business so I would assume things are true, but real motives are hidden. Go is obviously good language for some specific things, but it's not like ruby is pure trash, how come you did 'everything' in ruby, and then didn't know about eventmachine, but just had to rewrite things in Go. And rewrite happened overnight, right? A lot is missing from this story. I will definitely look more into Go, mostly because someone compared it in comments with TurboPascal, and I have fond memories of Borland tools, Pascal especially. Go and Rails are so different that there is almost no point in comparing them.
- burntsushi 14y ago
  > but real motives are hidden Could you elaborate on this please? Are you referring to the OP's motives of publishing the article? Or of switching to Go? > Go and Rails are so different that there is almost no point in comparing them. Except when one solves the same problem better than the other.
  - lysium 14y ago
    - They did not say why they did not choose one of the other alternatives. - They did not say how they used the concurrent features of Go, only mentioned that they were there. - They did not say how long it took to rewrite. - They did not say what they changed in the API. I'm not saying the story isn't true, but for a true story it lacks a lot. You could summarize the article with the title and you won't be missing much.
    - bascule 14y ago
      They didn't say why Ruby was using a lot of CPU either. Some basic investigation into the root cause might've revealed some intractable problems that are tied to Ruby (or at least the MRI interpreter) like long GC pauses (something which can be mitigated by going to a different interpreter like JRuby versus a complete rewrite in a different language) Instead, it's "OMG we're maxing out CPU, time to completely rewrite our app in a different language" And how big is this app that it needs 30 servers? We probably serve an order of magnitude of traffic where I work than these guys do (just guessing) and don't need that many.
dschiptsov 14y ago
Science and engineering from Bell Labs behind Go is what make this possible. It is not a "different language", it is carefully selected design decisions and ideas behind it. It relies on a principle of being good-enough, not to stuff everything in, as a "feature sellers" and "buzzword shouters" used to do. There are also lot of work of great minds behind Lisps or Erlang, and same principles in a foundation.
neya 14y ago
It's funny that they argue that they dumped the JVM derivatives like Scala, but wouldn't exactly tell why, nor support their argument with any sort of data/logs. I am genuinely curious to know why they chose Go over Scala. If it was the syntax, etc. I can partially agree because it's one of scala's weak points, but then they pitch the main reason citing performance, so I'm genuinely curious to know.
- fauigerzigerk 14y ago
  He mentions JVM memory usage.
  - neya 14y ago
    I think he edited it after me and many other posted similar comments, coz I didn't notice it the first time...
- masklinn 14y ago
  I got the same feeling from the odd dismissal of Erlang.
  - neya 14y ago
    Exactly, you know, my advice would be to take these blog posts with a pinch of salt. There is something called as the 'mob' mindset. The mob in general can be easily manipulated to believe in something that is not true. (Remember Julius Caesar?). For example, a year or two ago we saw the whole world of start-ups adopt MongoDB with so much vigor and almost everyone started writing "Why we moved away from MySQL to MongoDB". And two years later, we have now a bunch of posts saying the opposite - "Why we moved away from MongoDB..", etc. The same thing applies to Node.JS and Go too. They are still new technologies, so I think they deserve some time to be tried and battle-tested, instead of writing zero-data driven blogposts like these at an earlier stage. I would have been much happier to have read something like "After x years of using Go, Erlang and Scala, here is our comparison on what works well and what doesn't and how each one of them perform under different conditions" instead of "Hey we just reduced our server count to 2 using Go, hence it's better than everything out there.." Personally, just like you, I think these guys could have achieved more with Erlang or Scala, but given the fact that they chose Go because it works well for their architecture, I am not complaining.
johncarpinelli 14y ago
OK, I am sold on Go as my next language for web applications. Unfortunately, I can't throw away the existing PHP/MySQL code. Any suggestions on how to integrate Go and PHP into a single web server? Should I be using FastCGI for Apache to call the Go programs?
seivan 14y ago
The problem was Ruby, not Rails. Did you give ever give Jruby a chance?
seivan 14y ago
Hmm, wouldn't it be better to rewrite the ruby services to Go, the actual workers and not the entire API? I don't think the API HTTP requests where your main issues here...
deleted 14y ago
[deleted]
pjmlp 14y ago
Another day, another Go PR post on HN. What is new about porting code running in one of the slowest interpreters around to a compiled implementation of another language? I was already porting Perl/TCL code to C++ back in 2002 with similar performance results. Don't kids nowadays learn anything about performance in their CS degrees?
stmartin 14y ago
Or maybe you have no idea how to write performant code in Java, because if you did, it would actually work just as fast and like every other dickless wonder on HN you actually thought your post was interesting.
vitalie 14y ago
I rewrote my DNS checking tool (http://www.dnsinspect.com/ http://www.dnsinspect.com/) in Go and I saw huge differences in resource usage (previously it was implemented with Ruby on Rails + EventMachine), my memory usage went down from 128MB per background worker to a few KB. Now I'm able to run hundreds to thousands of concurrent reports using a small VPS, the Go application is using 36MB of RAM (24MB front end + 12MB for background workers). Go language is well suited for my particular case (many concurrent IO operations). This was my first Go project, in a week I was comfortable with it, I had many alternatives but I really liked the simplicity of Go, fast compilation and easy of deployment. Because I've missed some pieces from the Ruby's world I've combined Jekyll (Compass, HAML SASS, RedCarpet, etc) with Go. :)
- julien_c 14y ago
  Looks great, bookmarked.
- swah 14y ago
  Do you also used the default net/http support libraries?
  - vitalie 14y ago
    Yes, I'm using net/http.
chmike 14y ago
If you look for a Golang programmer today, it's more likely you'll find a true programmer instead of a me too programmer. So if you're a true programmer, learn Golang. ;)
Tzunamitom 14y ago
Was anyone else expecting this to be a boast from EA about how they cut their operating costs for the SimCity launch?!?
ajanuary 14y ago
"So then the decision came down to which language to use." I find this tendency to constantly jackknife between talking about languages and talking about frameworks a little dizzying. Shouldn't the decision have been what web stack to use? I'm sure you could get half way there with a bespoke Ruby web stack, maybe built on top of JRuby or with a sprinkling of C extensions. You could get the same poor scalability with some badly designed framework on top of Go [1]. There's a lack of meat in the post (hence why people are calling it out as PR). How do these features of Go make it easier to write applications or frameworks that cater to this particular scalability need? Could extra effort be put in up front to get something similar out of Ruby or any of the other languages, or will they always be sub-par? What is it about Go that makes it light up the rest of the stack in a way other languages don't? Or have you just traded one trendy technology for another because it promises to be the magic bullet for your current itch? [1] This isn't saying Rails is badly designed.
S_A_P 14y ago
Now I know that the dynamic languages such as ruby and python are not known for performance, but the benchmarks I have seen with Go look like it isnt exactly the fastest language available either. After reading this article I am more apt to think that something is/was VERY wrong with their Ruby code than to think that Go is that much better. Maybe my lack of experience with either GO or Ruby is showing here, but Ruby cant be that slow can it????
daderam 14y ago
Will be great if you can throw some light on "where" and in "what kind of work load" Ruby did not do well? I wonder why Ruby had hogged CPU? As I understand there are 4 kinds of workloads in your system (please correct me if I am wrong) 1. Handling several concurrent client connections. * How many concurrent clients are we talking about here? * Use of blocking sockets will require a separate thread for each connected client? Was Ruby not able to support many threads? * Non-Blocking sockets will require lesser number of threads. Note: I am from a Java background and Java's NIO scales well in handling volumes of concurrent sockets) 2. Handling volumes of data. (Data mining/aggregation) This is an IO/CPU intensive work load. * I dont think you would had done data-processing in ruby. Please correct me if I am wrong. * Scaling the "periodic" process to mine data can be efficiently done by choosing an appropriate storage system with efficient indexes. (MySQL/Postgres/Columnar-Storage/Cassandra etc etc) * Did you run a database? Did it run as a separate tier in a different server? 3. A queue/scheduling framework which would trigger the data-processing in periodic intervals(like google app-engine cron service) * How many "cron-entries" do you have in your system? 4. A CPU intensive algorithm which parses the incoming client's data before persisting in the database. * The incoming data from the client may require CPU intensive processing. (Example: parsing the data, eliminating duplicates etc etc). Did the CPU shoot up because of this? Will be great if you can throw some light on the volume/scale of the processing involved. Would like to know what kind of work load is "Go" better than Ruby. * Should I consider "Go" if I have lots of String (stream) processing to do? * Should I consider "Go" for handling several concurrent sockets? Would appreciate any help in understanding this.
ColinWright 14y ago
I know I'm late to the party, and no one will read this, but ... "... Java derivatives like Scala and Clojure ..." Someone help me out here: In what way is Clojure a Javav derivative?
- willvarfar 14y ago
  Perhaps in the sense that they are languages that target the JVM and that rely on interoperability with the Java standard library to bootstrap people making apps with them?
  - ColinWright 14y ago
    I can't see how it could mean anything else, but I find that, well, bizarre. Oh well, different point of view, I guess.
ksec 14y ago
No Love for Lua and LuaJit.
st3fan 14y ago
We decided to rewrite the API. This was an easy decision, clearly our Ruby on Rails API wasn't going to scale well and coming from many years of Java development and having written a bunch of things that handled tons of load with way less resources than what this Ruby on Rails setup could handle, I knew we could do a lot better

688 points by treeder 14y ago

Hey that's my photo of the GOpher! :) [1] http://www.flickr.com/photos/jianshen/8080852738/in/photostream http://www.flickr.com/photos/jianshen/8080852738/in/photostr...

drivebyacct2 14y ago
:) I don't know if the small plastic or larger plushy ones are cuter. Love having them on my desk.
- treeder 14y ago
  We got a bunch of the little plastic ones sitting behind my desk: https://plus.google.com/u/0/101022900381697718949/posts/1DbeLzTVzvu https://plus.google.com/u/0/101022900381697718949/posts/1Dbe...
treeder 14y ago
Hah, nice photo! Are you ok with that? Found it on google images.
- jianshen 14y ago
  It's cool. :)
carimura 14y ago
Monitor Stands!! [1] https://plus.google.com/photos/100264837158995116368/albums/5854596778257472465 https://plus.google.com/photos/100264837158995116368/albums/...
martinced 14y ago
+1 for the cool monitor stands but geez... Invest some $$$ in a real keyboard: scissor switches are the most painful type of keyboard switches ever : (
- redbad 14y ago
  The Apple wireless keyboard is the best keyboard on the market right now IMO.

phasevar 14y ago

Excellent success story with Go. Love this quote... "We also weren't sure if we would be able hire top talent if we chose Go, but we soon found out that we could get top talent because we chose Go."

GhotiFish 14y ago
That's really obvious but overlooked. Developers just looking for their paycheck will only have learnt the established languages. Developers who enjoy their profession will look to the future, excitement gleaming in their eyes. I've heard of business developing in Haskell a good amount of success. I suspect, for the same reason.
- jmilkbal 14y ago
  It could also be said that developers who use established languages are excited to get things done rather than reinvent wheels continuously.
  - thirsteh 14y ago
    No, it can't. People don't use only Java because they "just want to get things done." They use only Java because that's all they learned, and all they need for their day job. If not, then they will invariably step over languages that are very interesting but obscure.
    - jmilkbal 14y ago
      I'll give an anecdote. I've spent the last 7 years using Ada, a language that is /both/ interesting and obscure, almost exclusively. It's really a fantastic language that is frequently neglected much like poor Go, but with more merit. Anyway, in preparing the launch of my company's first iteration of its podcast network (http://76streetnetwork.com http://76streetnetwork.com), I absolutely wanted to use Ada, but the allure of the utterly fantastic tool support and breadth of libraries available for Java had me up and running in a day or two despite having never written anymore than basic hello world-like crap with it in the past. There's really nothing crazy about Java, you're right, but I live for coding and I have a deep interest in quality software development, but I guess I'm just looking for a paycheck for this side project that will never make me any money! Ironically, Ada is what pays my bills.

jabagonuts 14y ago

Out of curiousity, I would be interested to know which version/patches of ruby you were using. With ruby I'm used to running out of memory on servers long before running out of CPU. Of course, most of our workloads are IO bound, so that may be the diffent.

duaneb 14y ago
while true; end
treeder 14y ago
This was on Ruby 1.9, 2010 era. No patches. Memory was definitely a concern and probably contributed to some of it. Just to note though, after a machine was taken offline by the load balancer, it would generally come back to life.
- TresAmiga 14y ago
  Memory "probably contributed to it"? You don't even know why the old version was slow but you want to credit golang with making it fast? Hype much? The lack of critical thinking here on Hacker News is astounding.
FooBarWidget 14y ago
When running I/O bound workloads you really should be using a multithreading-capable app server. For example, if you're using Unicorn then that is extremely bad for I/O bound workloads (the Unicorn website's Philosophy page documents this under section "Just Worse in Some Cases": http://unicorn.bogomips.org/PHILOSOPHY.html http://unicorn.bogomips.org/PHILOSOPHY.html). On the other hand, something like Phusion Passenger Enterprise 4 with multithreading turned on is excellent at handling I/O bound workloads and reduce your memory by a significant factor.

bokglobule 14y ago

Why was Go chosen over NodeJS ?

treeder 14y ago
Mostly because I can't stand Javascript. It makes me cringe just thinking about it. Go is much nicer to work with.
- thibaut_barrere 14y ago
  Although I personally like CoffeeScript, I must say that Go fills me with joy for some reason. It feels "fresh" and lightweight to the beginner, and reminds me of TurboPascal in some ways.
- methehack 14y ago
  Hey OP! I appreciate your sharing. Since you came from ruby and we're on the topic of the language itself, I'd appreciate your impression of how well Go supports collections. Is there or could one write something like http://underscorejs.org/ http://underscorejs.org/? Can you do this kind of thing? [1, 2, 3, 4, 5].reject {|i| i < 3}.map {|i| i + 9} I did the go tutorial the other day and I became a little worried that one would not be able to do this kind of thing without a bunch of unwieldy declarations. I think that would be a showstopper for me. Thanks for any insight you might have!
  - jbooth 14y ago
    Go has first-class functions, so you could write a library do the same thing with, yes, a few more declarations around it. In your example, .map { |i| i + 9 } is .map( func(i int) int { return i + 9 }), or .map(myFuncDefinedElsewhere) in most real-world scenarios. As a commenter upthread pointed out, Go and serverside Javascript are aimed at very different use cases and audiences.
    - deleted 14y ago
      [deleted]

jewel 14y ago

For large projects, it'd probably make more sense to rewrite the critical code sections in C. rubyinline makes this really easy, and ruby C extensions aren't that bad either. One time on a contracting gig I had some code that was interacting with cairo drawing and imagemagick. They didn't have a compatible raw image format at the time (one was RGBA and the other was ABGR, IIRC.) It's trivial to convert between the two in ruby, but it was taking 60 seconds per image. Once I switched that one loop to C, it took 0.02 seconds. I've got nothing against Go, just suggesting an alternative that might make more sense in some situations.

bad_user 14y ago
For web servers there are other things more important than the raw processing speed of a single routine. What you want is to get the CPU usage to 100%, CPU resources used preferably in actual processing and less on things like garbage collecting or blocking for I/O. For web servers in particular it is tricky since processing requests also involves a fair bit of I/O interwoven with CPU processing (in the case of apps that aren't just CRUD over a DB).
dyselon 14y ago
This definitely does make a lot more sense than throwing out your whole app and rewriting it in many situations, but I'm always a little hinky about moving stuff into C just because it has so many ways to shoot yourself in the foot. You can obviously create bugs in any language (I know I certainly have!), but Go really does hit a nice spot between safety, productivity, and speed, so I can definitely see the attraction.

siavosh 14y ago

I've read this domino-effect on server clusters on several HN postmortems and I've seen various flavors of this on our own web servers. I'd like to think there's a simple configuration in most servers that prevent 100% cpu utilization from taking place and preventing the server from telling its cluster that it's still alive. Anyone have any experience with this?

laumars 14y ago
It's not really possible without adding resources (ie a new server). The problem is, once your servers get saturated, requests queue up, processing slows down and users start to hit refresh which artificially and exponentially increases traffic. Then to compound things, servers buckle, which means you lose a resource right when you needed it the most. A domino effect is one analogy, another might be how a small hole in a damn can buckle and blow open from the force of the water pushing through.
- ef4 14y ago
  > It's not really possible without adding resources (ie a new server). That's not quite true. With the right architecture, performance will degrade gracefully. You'll get slower and slower responses as the load goes up, and eventually start dropping requests, but the servers will not die and there will be no cascading failure. One way to achieve this is to make sure the queuing happens at the load balancer, and no large queues are allowed to build up in the individual application servers.
  - laumars 14y ago
    Well yeah. You can do all sorts of tricks from TCP/IP hacks to streamline HTTP requests through to disabling queuing entirely. But my point is you cannot entirely prevent your site from saturation without adding extra servers to your web farm (and more so, that siavosh's method of taking servers out of service has the inverse effect of what he was trying to achieve). Thus all you can do is slow the escalation in the hope that the traffic peaks before your resources buckle. The only method I'd found that is "guaranteed" to prevent such outages is the use of sorry pages (ie a static page stating "We're experiencing high volumes" which users are directed too if the dynamic page connections are maxed out). However even that is just essentially a prettier version of a page time out - and I mean this in terms of usability rather than technicality. ie the site is still unavailable, but you're killing the connection in a user friendly way rather than allowing connections to stack or just flat out disallowing "> n" active TCP/IP connections.

grey-area 14y ago

Interesting to hear of Go being used in production. It'd be great to hear some more details on your setup when deploying the go processes - how are you managing failover, what's your load balancer, and how are you handling swapping out processes etc? Are you compiling on the server or local machines before deployment? Most other languages have lots of solutions on the deployment side now but Go is so new there isn't much info out there. Also, as you are running an API presumably your app in Rails was pretty simple, did you have the impression things would be more complex in Go if you were writing an app with an extensive front end and UI and using sql? I'd miss all the view helpers etc available at present I think. Going from 30 servers to 2 certainly sounds like a huge improvement, so it was definitely worth it for you, are you thinking of writing any front-end apps in Go? I've been playing around with Go recently and it is a fantastic language for someone coming from dynamic languages like Ruby. I particularly liked interfaces as a way to define a contract for implementations to follow, and the simple package system which encourages you to make your code modular.

carbocation 14y ago
For my (toy) apps, I've been compiling locally and pushing to the server. It's trivial to compile for a target platform and architecture, and you get a single compiled binary. Certainly my response isn't getting all the way to what you're looking for, but it's not difficult to write shell scripts that manage the deploy process from here.
- grey-area 14y ago
  Thanks, I use rsync wrapped in a script usually (because I'm dealing with more files than just a binary), haven't tried cross compiling yet but was going to give that a go. I suppose I was more concerned about what happens when things are on the server and how multiple processes can be managed. As soon as you have more than one and want to swap out server workers seamlessly and load balance etc it gets a little more complicated.
  - carbocation 14y ago
    Yes, I think you're right on the mark re: swapping out workers, etc. Those nuances I haven't had to deal with due to these being toy apps.

donebizkit 14y ago

Last month I picked NodeJS and build couple of sites with it. This month, I want to pick up something new. I was hesitating between Python and Go. Can you answer couple of questions about Go from someone coming from NodeJS: 1- What is the state of the external Go libraries, especially DB (MySql), caching libraries (memcached), protocol libraries (Oauth). Are they stable 100% 2- How easy is logging and tracing in go? Thanks.

paddyforan 14y ago
Memcached's author (Brad Fitzpatrick) is a Go user, and wrote a memcache client for Go. Go has a bunch of good stuff surrounding SQL. I haven't used MySQL, personally, so I can't comment on which MySQL driver is best, but I know YouTube uses Go for something related to MySQL, though I can't say for certain exactly what. http://code.google.com/p/vitess http://code.google.com/p/vitess The goauth2 (http://code.google.com/p/goauth2 http://code.google.com/p/goauth2) library is written by Brad Fitzpatrick and Andrew Gerrand (two members of the Go team), and is stable. Logging is great. Interfaces make it really, really flexible. I'm not sure what you mean by tracing. Stacktraces? Those are easy to retrieve: http://golang.org/pkg/runtime/debug/#Stack http://golang.org/pkg/runtime/debug/#Stack
- tptacek 14y ago
  Well, Go has a bunch of libraries for different SQL servers. I wouldn't go so far as to say it has lots of good SQL stuff; it's in approximately the same place as C is w.r.t. databases. Having to interface with complicated SQL is a reason not to use Go (but not an insurmountable one).
  - paddyforan 14y ago
    I think it has lots of good SQL stuff just because I think it laid a nice foundation for developers to build SQL libraries on, and I think there's a lot of potential for using SQL in Go to get dramatically better in the next year or so. That said, I'm not an SQL expert or even very competent, so I'm probably not the best to comment on it--I just included it for the sake of addressing all the points. I would agree, though--working with SQL is the worst part of the software I'm currently writing in Go. I think that's more a praise of working with Go than a condemnation of its SQL support, though.

goatslacker 14y ago

It would be interesting to hear some of the reasons you went with Go vs Clojure (and also the other languages you mentioned)

shurcooL 14y ago

> "We also weren't sure if we would be able hire top talent if we chose Go, but we soon found out that we could get top talent because we chose Go." I feel[1] that a smart/talented C/C++/anything developer can go from someone who has never seen or heard of golang to a proficient and productive Go developer in a matter of a few weeks, maybe even _days_, if not less. That's how long it takes to go through the following materials (and fav some for later reference) and play with the language a bit. http://golang.org/ref/spec http://golang.org/ref/spec http://www.youtube.com/watch?v=ytEkHepK08c http://www.youtube.com/watch?v=ytEkHepK08c http://commandcenter.blogspot.com.au/2012/06/less-is-exponentially-more.html http://commandcenter.blogspot.com.au/2012/06/less-is-exponen... http://tour.golang.org/ http://tour.golang.org/ http://golang.org/doc/install http://golang.org/doc/install http://www.youtube.com/watch?v=XCsL89YtqCs http://www.youtube.com/watch?v=XCsL89YtqCs http://golang.org/doc/code.html http://golang.org/doc/code.html http://golang.org/doc/effective_go.html http://golang.org/doc/effective_go.html http://golang.org/pkg/ http://golang.org/pkg/ - use as reference http://www.youtube.com/watch?v=f6kdp27TYZs http://www.youtube.com/watch?v=f6kdp27TYZs http://talks.golang.org/2012/splash.article http://talks.golang.org/2012/splash.article https://gobyexample.com/ https://gobyexample.com/ http://golang-examples.tumblr.com/ http://golang-examples.tumblr.com/ And a some more similar things that you can mostly get to from golang.org site. The beauty of how concise the language and even its website are, is that you can literally just go through everything there one thing after another. [1] This is my personal opinion based on playing with go the last few weeks/months. I'd love to verify this theory. It's not yet the primary language in which I do things in (I use C++11 atm), but for all my side tasks[2] it proved to be indispensable. And I found it very easy to pick up. I can't wait until I start doing all my work in Go, that will be a true test of its productivity efficiency. [2] https://gist.github.com/shurcooL https://gist.github.com/shurcooL

TillE 14y ago
It's quite strange to me that people would identify as or look for a "[language] programmer". Sure, I happen to write more C++, Python, and C than anything else, but I've dabbled in just about everything and could reach comfortable proficiency in a matter of weeks. Most of programming and all of computer science is universal. Any serious programmer should be a polyglot by default.

petrohi 14y ago

Why Go over Erlang?

Ixiaus 14y ago
(Note: I'm a big Erlang supporter and 90% of my startup's code is written in Erlang) Erlang is not memory efficient (compared to Go? I don't have benchmarks to back this up) AND the learning curve for developers not exposed to functional programming (which is a lot) is quite a bit higher than it is in Go.

mrwnmonm 14y ago

man, you just made me worry very much now - we are making taxi dispatching api using ROR - and we will lunch it soon

thibaut_barrere 14y ago
Why worry? If you can handle the initial load, you can still optimize later on, rewriting some functions or parts of the system with something faster.
- mrwnmonm 14y ago
  great
carimura 14y ago
don't worry... our Ruby servers kept us churning for a long time.
programminggeek 14y ago
If perf is that much of a worry, you can look at JRuby, that will get you a bit (or a lot) farther from what I've seen.

laumars 14y ago

Has anyone had an experience of running websites from Go; or more specifically, how you handle none-HTML content? I've been considering porting my CMS from mod_perl to Go, but I'm not sure how you work with the other files (CSS et al). I did read somewhere that you run Go from Apache but it's not recommended.

carbocation 14y ago
I've just started creating toy projects in Go. I put these Go projects (and Apache, which runs older PHP projects) behind nginx. For hosting CSS, you either generate it programmatically and send the right Header, or you can use http.FileServe from the standard library [1]. (Surely other approaches are possible, but those are the two I've played with so far.) [1] https://code.google.com/p/go-wiki/wiki/HttpStaticFiles https://code.google.com/p/go-wiki/wiki/HttpStaticFiles
- laumars 14y ago
  Thanks for that. The issue I have is that I would prefer to keep Apache around if I can (I have a few sites - some of which I host for friends). While my web server does have a few IPs attached, I'd rather not have to buy more IPs just to separate Apache from Go. And to be perfectly honest, I do quite like Apache. (each to their own I know, but I've had little reason to complain about it).
  - carbocation 14y ago
    Right, you're describing my situation exactly. I used to run Apache solo. Now I run Nginx, which forwards requests to Apache or to my various go apps based on the domain name. Apache and the go apps just listen on different ports internally. The server only has 1 IP address.
    - laumars 14y ago
      You are an absolute star. If I could up vote you a hundred times I would. Thank you

jscottmiller 14y ago

Can you describe your workload a bit as well as any benchmarking you did to determine (and perhaps optimize) hotspots in the ruby code before you started the port? Personally, I've been interested in moving to Go on a python-based project of mine. Thus far, I've avoided it because 1) the extra work required to self-implement a few third party libs I rely on and 2) I've been able to eek out sufficient performance using c extensions and cython.

treeder 14y ago
This first rewrite in Go was for the IronWorker API so all the operations are here: http://dev.iron.io/worker/reference/api/ http://dev.iron.io/worker/reference/api/ There were less endpoints back then, but you can get the idea. The most heavily used operation being queuing up tasks/jobs.

slem311 14y ago

great stuff!!

orangethirty 14y ago

What changed in the architecture? It is rather interesting that by just having a language change you could remove 28 servers from the system.

ikeepforgetting 14y ago
My guess is that removing 2-5 layers of abstraction can get you pretty far along.
- buro9 14y ago
  We're writing most of the systems for our company in Go, and of the many reasons this ranked highly. Just removing the multiple unnecessary abstractions gets you a hell of a lot of simplicity and performance back.
  - ikeepforgetting 14y ago
    I work in an area that requires a lot of high performance computing and there is a reason people in this area like code simplicity. Stacking layer upon layer of complex algorithms is a lot more difficult to reason about than having a simple and elegant system that's close to the metal. There is a recent trend of building more and more complex systems, have less coupling and 'good oop'. That's all good and nice but simple code is: - more maintainable (a complex nest of objects & function calls all loosely coupled isn't) - a lot more readable - easier to optimize - better for your sanity Some languages have build a culture of extreme complexity and are notable for building huge structures and systems to do simple tasks. Having a 50K lines of code dedicated to just inversion of control or O/R mapping makes a problem a lot more complex than it probably needs to be.
    - martinced 14y ago
      "Having a 50K lines of code dedicated to just inversion of control or O/R mapping makes a problem a lot more complex than it probably needs to be." Exactly. That's the terrible stuff with all this "enterprisey" mindset: people are working on medium-sized codebase made of this special kind of hell that Java/C# + ORM ([N]hibernate) + XML + SQL is and these app often run into the 200K / 300 KLOC lines if not more. Yet what does these applications really do? Actually not very much. Yet these programmers are sure to be working on super-advanced stuff because their codebase is big. When several companies reported a drop in LOC of 90% by switching to something else than Java then at one point you have to at least consider that maybe most of your Java codebase is hot air.

zobzu 14y ago

It's not really "go" that makes the difference. it's how the runtimes and frameworks are used and/or made. frameworks on top of frameworks, all being over engineered, with poor understanding of what the system actually does, result in super slow apps on top, that you generally go to aws to scale. It's not the first time that I see people reducing a dozen servers that were "always maxed out" by a couple of servers "that can barely feel the load". Yeah throwing hardware at the issues is fine'n all but we're been way over the limit too many times in too many directions. Go provides a clear start/APIs/framework, and the language enforces good habits. Also, it doesnt have things like global interpreter locks. The lesson? Stop using cool techs because there's blogs about them. UNDERSTAND the tech before using it.

pjscott 14y ago
Nice rant, and I mostly agree with it, but there are a few slow things about the ways that some languages are usually implemented that don't really have much to do with over-engineering. Let's take this line of Python code, for example, and look at how the CPython interpreter runs it: return x + 42 This turns into the following bytecode instructions: LOAD_FAST 0 (x) LOAD_CONST 1 (42) BINARY_ADD RETURN_VALUE First we get the x variable from the local namespace. This is an array of PyObject pointers; the LOAD_FAST oepration simply does an array lookup and puts the result on the stack, incrementing the reference count. Pretty fast. Next is LOAD_CONST, which is even faster; it takes an already-allocated PyObject and puts a pointer to it on the stack, incrementing the reference count. BINARY_ADD removes two numbers from the stack, dereferences the pointers to get the integer values, allocates a new PyObject with the resulting integer inside, bumps up its reference count, decrements the reference count of the two operands, and pushes the result on the stack. Finally, RETURN_VALUE jumps back to the caller of the function. In Go, the corresponding code would be compiled to two, maybe three machine code instructions. There are good reasons why Python does it this way, but it does suffer some inherent slowdown.
- kevingadd 14y ago
  A more interesting comparison would be the PyPy-generated native code for the expression versus the compiled code from Go. Comparing an interpreter with a compiler is generally uninteresting; the compiler almost always wins. If you look at the code that comes out of various modern JITs you often will find really interesting differences in the native code they produce, even for simple constructs like arithmetic. Type checks, barriers, bailouts, etc - in addition to more mundane differences, like one using SSE to do floating point and the other using x87, and one JIT having a better register allocator.

conorh 14y ago

I agree with the 'Go is fun' (and easy to pick up) sentiment. I spent a few weeks recently writing an NTLM library for Go for a consulting client (I hope they'll let us open source it at some point). All the parsing and bit manipulation was extremely straightforward due to the excellent standard libraries and I was able to quickly move from not knowing Go, to a fully implemented NTLMv1/v2 library!

deleted 14y ago

[deleted]

jroseattle 14y ago

Would be great to hear more specifics of the rewrite. What did the API do, what pieces were built in RR, what got replaced in Go, etc. At least as much as you're willing to share. :-)

rmoriz 14y ago

How about BDD, test-driven development and quality insurance? Does Go provide an ecosystem that supports agile refactorings that are common in lean startups? You can say anything bad about the performance of Ruby and Rails etc. but rspec, cucumber, capybara, vcr, factorygirl are really important features to start from zero and reach a viable product.

burntsushi 14y ago
> test-driven development `go test`. > Does Go provide an ecosystem that supports agile refactorings that are common in lean startups? Go has a compiler and is staticly and strongly typed. Which automatically makes it at least an order of magnitude easier to refactor safely. Not really sure about the other stuff you mentioned. I've never even heard of BDD before.
- gregors 14y ago
  I've heard this argument before e.g. we don't need to test because the compiler catches errors. The compiler catches the mot trivial of error conditions. Unfortunately it doesn't address faulty logic/behavior.
GhotiFish 14y ago
I rather like what go has set up for test's. They made it as small and functional as they could. Out of curiosity. How does a language specifically support "agile refactoring"? What features permit (or hinder) stopping every couple of weeks and removing the accumulated WTF's in the codebade? Want to buy a left handed stapler?
paddyforan 14y ago
Go's interfaces make it really, really easy to change things with very little code changing. Go's testing tools are superb.
- rmoriz 14y ago
  Can you blog about it in the future? Couldn't find good resources and tools about testing in Go. Especially with concurrency this seems to be very important.

waterside81 14y ago

Go's definitely picking up momentum. I know that Mozilla is shifting much of its services infrastructure to Go. We're porting over our Arabic sentiment engine, currently in Python/Cython to Go. If you're dealing with simple data structures, going from Python to Go is almost a line-for-line port, but the performance benefits are, of course, massive. Our benchmarks show a 40x speed improvement so far. Lastly, for anyone thinking about taking the plunge, use Go tip (from their source code repo) - don't use their "stable" releases. They fix bugs so fast you'll always want to be current.

DASD 14y ago
Do you have a citation for Mozilla using Go? I would be a bit surprised given their(Mozilla) development of Rust.
- waterside81 14y ago
  Sure, check out their github pages (here's a few): * https://github.com/mozilla-services/heka-mozsvc-plugins https://github.com/mozilla-services/heka-mozsvc-plugins * https://github.com/mozilla-services/heka https://github.com/mozilla-services/heka My friend works for Mozilla - that's how I knew.
  - DASD 14y ago
    Thanks waterside81. Can anyone explain the difference between Mozilla and Mozilla services? A cursory glance seems some of the Github repositories overlap.
    - rfk 14y ago
      Mozilla Services is the team within Mozilla that builds/runs much of the backend infrastructure, e.g. the firefox sync servers, marketplace servers etc. The existence of separate "mozilla" and "mozilla-services" github projects is largely a historical accident, since different teams started moving to github organically at different times. (Source: I work for Mozilla, on the Services team)

TakeTwo 14y ago

"We decided to rewrite the API" and now it is over 300x faster. Go had nothing to do with this, it was obviously some boneheaded original design. Probably something like fork+exec on a Ruby VM on each request to enter a chroot. I know HN has a hard-on for Go but come on.

drivebyacct2 14y ago
Are you the same asshole making throwaways to shit on all of the Go postings or are there actually more than one of you going around making assumptions, accusations and spreading falsehoods about Go. I can't get over how pathetic it is to make a throwaway for that comment, I'm more embarrassed five people validated your comment and attitude.

saosebastiao 14y ago

This could easily be rephrased as "How We Went from 30 Servers to 2: Static Typing and Compiled Code".

s_baby 14y ago

Why Go over VisualBasic?

rcb 14y ago

Why did go "win" over erlang, scala, and clojure? Because it's becoming trendy and "fun?" And, why are "fun" and "joy" terms ruby bloggers use for languages? Is this code for "easy" and "familiar?" As in, "this language is easy to learn.. It does not require us to learn difficult but mind-expanding concepts to become proficient." Replacing ruby/rails with something (just about anything!) results in far fewer servers. Isn't this obvious to all? Is this really news to the average HN reader? If that's an I/O bound API server, bet those two servers could be brought to one in a language/runtime that's more productive but a little less "fun" to learn.

rektide 14y ago
One could adopt some kind of event-based system in scala or clojure, but Erlang and Go are alike in being the lone runtimes where running millions and millions of very small messaging processes is AOK no problem for the runtime, and not something one has to work really hard for. Fun and joy are terms applicable here because the alternative is Erlang. Zing! Go is rather ideal for these guys use case: if it wasn't an easy win, given what a snug fit their use case is, there would've had to have been red flags abound, as this really is a nearly idealized work load for Go's use: a hell of a lot of processes which sit around doing nothing, where one occasionally gets a message and forwards it along. Perfect Go story, as Go's lightweight processes (goroutines) are ideal for this kind of Communicating Sequential Processes routing workload.
- Peaker 14y ago
  Haskell's runtime makes for cheaper threads than Erlang. Possibly cheaper than Go's, too.
  - rektide 14y ago
    Are they also in Haskell as in Erlang pre-emptable by the runtime? There's certainly some cost in this design decision, more context that has to be swapped in and out and more state kept, but it's important capability to allow blind design & use- think Node, where everyone has to keep re-iterating how important it is to keep yielding to the event loop, to not do a lot of CPU work in a handler: irrelevant in Erlang world- in spite of the threading model not being OS threads, Erlang is happy to drop your lightweight thread on the fly when it sees fit.

raphaelj 14y ago

I'm getting the same experience here by entirely rewriting an old PHP web application [1] in Haskell using the Yesod web framework. I've been using JMeter to benchmark both versions of the application. On a 10€/month dedicated server [2], the Haskell one was able to generate 220 dynamic pages per second [3] whereas the PHP one tops at 35 pages per second on a equivalent page. Moreover, concurrence capabilities of Haskell are also pretty sweet : while I was benchmarking the web app using 2,000 concurrent connections, the application server was only using around 90MiB of RAM. I was not able to increase the number of concurrent connections as the client application I was using started to kill my quad core desktop, I suspect Haskell to be able to manage A LOT MORE concurrent connections as I didn't see any decrease in the throughput of the application as I was increasing the number of concurrent connections. [1] http://files.getwebb.org http://files.getwebb.org [2] http://www.kimsufi.com/fr/ http://www.kimsufi.com/fr/ [3] Static content has been ignored as modern servers like NGinx seem to be able to carry the static content (CSS, images, ...) at more than 5,000 req/sec on the same machine.

riobard 14y ago

“the entire process started up with only a few hundred KB's of memory (on startup)” There is no way this could be true: last time I checked, a bare minimal Go HTTP server requires at least 2.8MB of memory on 64-bit machine. Are you using the default net/http library?

codygman 14y ago
Yeah, my HTTP server made in Go usually starts up at around 6MB or so.

topbanana 14y ago

Alternatively... why we needed 30 servers: Rails

programminggeek 14y ago

I wonder how far they could have got going more of a bare-bones Sinatra app using JRuby. It's still not nearly as fast as Go obviously, but take away Rails and some of the more dynamic usages of Ruby on a faster VM and I'd venture to guess you might have been able to go from 30 servers to 10-15? That's still not 2 mind you...

rektide 14y ago

p.s. come disrupt our business: with nearly no hardware, no costs, and no maintence burden you too can pass OMG huge amounts of message traffic. this is one bold as fuck post.

rachelbythebay 14y ago

This story sounds mighty familiar. I realize the topic is about one language over another, and most of the comments here seem to be focusing on that, but there's another part which I find fascinating: the cascading failure mode when they'd lose a web head. "This would in turn cause the load balancer to think it failed and take it out of the pool, thereby applying the load that the unresponsive server would have been handling to the remaining servers. And since the remaining servers are now handling the load of the lost server plus the spike, inevitably a second server would go down, the load balancer would take it out of the pool and so on." It was 8 years ago, and the customer was running PHP, not Ruby, but otherwise it's the same basic story that lead to the creation of "Surge Protector", or its actual name, "Suicide Pact". http://rachelbythebay.com/w/2011/06/28/sp/ http://rachelbythebay.com/w/2011/06/28/sp/

pilsetnieks 14y ago
Keeping in theme with the suicide pact, STONITH, the other kind of pact for high-availability machines comes to mind - Shoot The Other Node In The Head.

dubcanada 14y ago

You have an error in Opera 'SyntaxHighlighter.config.bloggerMode = true;' Undefined variable: SyntaxHighlighterError

stalcottsmith 14y ago

I've seen a lot of dumb Rails server configs and even dumber usages of Rails without any tuning at all. With a couple weeks or a month of work I could shrink the hosting fees or resources consumed by a factor of 5 out of most Rails apps that are sitting up around 30 servers... and probably a factor of 10. Leaving aside whatever rookie or even intermediate mistakes were made in their Ruby code or their database, this post indicates a lack of understanding of what happened when their server fell over. Proper tuning of a deployment should not trigger a 100% failure mode like this. These folks were itching to get off of Ruby for whatever reason... after all their roots were in Java. If your goal is to do a rewrite and learn a new language and gain some notoriety why waste time learning what you did wrong with Ruby or your server config?

dkkkdkdk 14y ago
TLDR: I'm butt-hurt that people are dumping Ruby. And because I hate Java I'll blame it on it also.
- sabat 14y ago
  Not quite. He's saying that people who think Ruby is slow are often people who have no idea how to performance-tune it.
  - bpicolo 14y ago
    Well, more of the time it's because Ruby actually is, in fact, slow.
    - pjmlp 14y ago
      I though the implementations were slow or fast, not the languages.
      - codygman 14y ago
        Typically a language is considered slow or fast based on it's most popular implementation, sadly.

taproot 14y ago

> with Ruby. Title should have been: "How we went from 30 servers to 2: Replacing Ruby"

saidajigumi 14y ago

I'm missing the connection between poor API performance under Rails and the decision to do a ground-up rewrite. Was the initial performance problem due to an API problem or a platform problem? Was there any profiling of the poorly performing API server? I'd love to hear the story behind that analysis and how it impacted the tale told in TFA. I ask all this since I've lost count of how many times I've participated in or witnessed an averted rewrite via a good dose of profiling and a few key bugfixes. I'll acknowledge that's not as fun as a clean-slate project, but vast amounts of engineering time were saved.

zenocon 14y ago

It's an interesting tale, and I honestly have no bias toward Ruby or Go, but the cause/effect relationship is poorly illustrated (we switched languages). We all know runtime / performance / scalability is more complex than that. I'd like to hear where the bottleneck(s) were and how Go solved them.

trungonnews 14y ago

I wish Go has named arguments like Python and Scala. Reading and debugging code could be so much simpler.

drivebyacct2 14y ago

Man, what is it about Go that just brings out the pricks of HN? There are over three throwaway accounts created exclusively for this thread to shit-post about Go, to steal an old term I have never, ever used on HN before.

jff 14y ago
I'm not exactly sure, I've noticed the same thing. Go appeared a few years ago, quickly became pretty well-known, and continues to grow in popularity. Maybe it's seen as "threatening" to older projects like Ruby and Python? The most vitriolic comments always seem to come from Ruby, Python, and Rust partisans. Of course, I'm not sure why they're so threatened by it, they themselves explain in great depth how the lack of generics or exceptions will prevent anyone from writing any real code in it, or even trying to in the first place!

jebblue 14y ago

http://www.dartlang.org/support/faq.html http://www.dartlang.org/support/faq.html >> Q. How does Dart relate to Go?# Dart and Go are both language projects started at Google, but they are independent and have different goals. As a result, they make different choices, and the languages have very different natures, even while we all try to learn from each others' work. Hmm, don't think I want to bet the farm on that.

drivebyacct2 14y ago
Don't want to bet the farm on what, do you mean Dart or Go sticking around? Go is used extensively through Google. It powers a significant DB layer that powers YouTube. It serves up all of Google's static downloads and several employees hint at other significant projects they can't speak about. I've only dabbled in Dart, I can't speak to how vital it is, currently it only seems viable when translated to JS or when using in a server-side VM, which is certainly an option if you want. (sorry, I originally had written Google where I meant YouTube, see "Vitess" for more info)

rurounijones 14y ago

Ok, this bit has left me confused. They were Java devs that liked ruby, they wrote applications in Ruby on Rails and the ruby apps were hitting limits so they immediately started looking at other languages. But they don't mention the most obviously (to my mind) simple option. JRuby It is ruby (They like ruby). Most ruby apps can be run on JRuby with very very little changes (No need for a big rewrite) and it runs on the JVM with which they are familiar and it is very fast (and true multi-threading). Maybe they did look into it but I would have thought that would be #1 on their "We tried this but discounted it" list.

oscardelben 14y ago
I'd like to see some real benchmarks with a real app, but I doubt moving to jruby would bring the magnitude of improvement mentioned in the post.
pkulak 14y ago
In my experience, rewriting for JRuby is as much work as re-writing in a new language. Last I looked was a year or two ago, so it may have all changed, but it's not as simple as just moving your code over. Many really important gems don't work, or don't work properly.
- boundlessdreamz 14y ago
  The only gems that do not work are those which are not pure ruby. For the popular gems I think there have been rewrites. I haven't looked into JRuby but it has changed quite a bit in the last year from what I read
  - johnb 14y ago
    I don't think that's true. There are plenty of threadsafe gems with C extensions, and plenty of non-threadsafe pure ruby gems that mutate class/class-instance variable willy-nilly.
    - boundlessdreamz 14y ago

knodi 14y ago

Go 1.1RC just around the corner in April. A lot of good stuff in Go 1.1RC.

trungonnews 14y ago

I was originally very excited about Go when I first learned about it. But then I got tired and frustrated quickly after having to listen to the other Gophers telling me that I don't need this or that feature because there is a better way to do it in Go. Like. I don't need exceptions because Go function can return multiple values. I don't need a mocking framework like Mockito because Go has interfaces. I don't need an interactive debugger because I can debug with command line using gdb. I don't need named arguments because I can instantiate a struct and call my function with it. (Have you seen Ruby or Javascript code? Almost every function takes 'opts' as a single argument. Go is probably going down this path too.) Then I learned about Scala. I'm not saying that Scala is better than Go. However, it has everything that I need. :)

yannk 14y ago
tl;dr: I tried Go, but it wasn't like Ruby, so now I'm trying Scala.
- trungonnews 14y ago
  Ha. Is there a high performance, compiled, and concurrent friendly language that looks like Ruby? Please share. :)
  - burntsushi 14y ago
    Rust might be your best bet. I think better advice for you is to just stay away from opinionated languages. I don't think Rust falls into that category, but I'm not sure.
  - qznc 14y ago
    Looks like Ruby? You mean syntactically like "end" instead of "}", I assume. Sorry, but the C syntax is dominating the native-compiled world. You could try Pascal, they have "end". Or use C with preprocessor magic: #define END ;} If you mean Ruby like terse, generic code, then try Rust, Go, and D.
  - pjmlp 14y ago

fernandezpablo 14y ago

There's simply no technology that makes you switch from 30 machines to 1 (you say you keep 2 just for reliance). Obviously there must be other architectural changes in play here (I'm guessing RoRs thread-per-request vs some kind of event loop on Go?). Please be more specific.

genwin 14y ago
Go is good at keeping all the cores on the machine pumping. Maybe they switched to machines with more cores.

moron4hire 14y ago

There are a lot of things that can be said about the "average" programmer in such-and-such languages, what they do and don't do, what habits they have, etc. But I'm not interested in working with merely average people.

smrtinsert 14y ago
That's right. People like you and me work with the top 5% of programmers! It's because I'm smrt.

free652 14y ago

Uh I think I love it. Easy to learn language and awesome performance.

fwee 14y ago

Rails is not for API,why not use eventmachine?

desireco42 14y ago

I went briefly to through the comments, it seems that no-one is bothered with lack of details in this post, things are not logical at all, almost like it was written by PR agency in charge of promoting Go. Except it is established business so I would assume things are true, but real motives are hidden. Go is obviously good language for some specific things, but it's not like ruby is pure trash, how come you did 'everything' in ruby, and then didn't know about eventmachine, but just had to rewrite things in Go. And rewrite happened overnight, right? A lot is missing from this story. I will definitely look more into Go, mostly because someone compared it in comments with TurboPascal, and I have fond memories of Borland tools, Pascal especially. Go and Rails are so different that there is almost no point in comparing them.

burntsushi 14y ago
> but real motives are hidden Could you elaborate on this please? Are you referring to the OP's motives of publishing the article? Or of switching to Go? > Go and Rails are so different that there is almost no point in comparing them. Except when one solves the same problem better than the other.
- lysium 14y ago
  - They did not say why they did not choose one of the other alternatives. - They did not say how they used the concurrent features of Go, only mentioned that they were there. - They did not say how long it took to rewrite. - They did not say what they changed in the API. I'm not saying the story isn't true, but for a true story it lacks a lot. You could summarize the article with the title and you won't be missing much.
  - bascule 14y ago
    They didn't say why Ruby was using a lot of CPU either. Some basic investigation into the root cause might've revealed some intractable problems that are tied to Ruby (or at least the MRI interpreter) like long GC pauses (something which can be mitigated by going to a different interpreter like JRuby versus a complete rewrite in a different language) Instead, it's "OMG we're maxing out CPU, time to completely rewrite our app in a different language" And how big is this app that it needs 30 servers? We probably serve an order of magnitude of traffic where I work than these guys do (just guessing) and don't need that many.

dschiptsov 14y ago

Science and engineering from Bell Labs behind Go is what make this possible. It is not a "different language", it is carefully selected design decisions and ideas behind it. It relies on a principle of being good-enough, not to stuff everything in, as a "feature sellers" and "buzzword shouters" used to do. There are also lot of work of great minds behind Lisps or Erlang, and same principles in a foundation.

neya 14y ago

It's funny that they argue that they dumped the JVM derivatives like Scala, but wouldn't exactly tell why, nor support their argument with any sort of data/logs. I am genuinely curious to know why they chose Go over Scala. If it was the syntax, etc. I can partially agree because it's one of scala's weak points, but then they pitch the main reason citing performance, so I'm genuinely curious to know.

fauigerzigerk 14y ago
He mentions JVM memory usage.
- neya 14y ago
  I think he edited it after me and many other posted similar comments, coz I didn't notice it the first time...
masklinn 14y ago
I got the same feeling from the odd dismissal of Erlang.
- neya 14y ago
  Exactly, you know, my advice would be to take these blog posts with a pinch of salt. There is something called as the 'mob' mindset. The mob in general can be easily manipulated to believe in something that is not true. (Remember Julius Caesar?). For example, a year or two ago we saw the whole world of start-ups adopt MongoDB with so much vigor and almost everyone started writing "Why we moved away from MySQL to MongoDB". And two years later, we have now a bunch of posts saying the opposite - "Why we moved away from MongoDB..", etc. The same thing applies to Node.JS and Go too. They are still new technologies, so I think they deserve some time to be tried and battle-tested, instead of writing zero-data driven blogposts like these at an earlier stage. I would have been much happier to have read something like "After x years of using Go, Erlang and Scala, here is our comparison on what works well and what doesn't and how each one of them perform under different conditions" instead of "Hey we just reduced our server count to 2 using Go, hence it's better than everything out there.." Personally, just like you, I think these guys could have achieved more with Erlang or Scala, but given the fact that they chose Go because it works well for their architecture, I am not complaining.

johncarpinelli 14y ago

OK, I am sold on Go as my next language for web applications. Unfortunately, I can't throw away the existing PHP/MySQL code. Any suggestions on how to integrate Go and PHP into a single web server? Should I be using FastCGI for Apache to call the Go programs?

seivan 14y ago

The problem was Ruby, not Rails. Did you give ever give Jruby a chance?

seivan 14y ago

Hmm, wouldn't it be better to rewrite the ruby services to Go, the actual workers and not the entire API? I don't think the API HTTP requests where your main issues here...

deleted 14y ago

[deleted]

pjmlp 14y ago

Another day, another Go PR post on HN. What is new about porting code running in one of the slowest interpreters around to a compiled implementation of another language? I was already porting Perl/TCL code to C++ back in 2002 with similar performance results. Don't kids nowadays learn anything about performance in their CS degrees?

stmartin 14y ago

Or maybe you have no idea how to write performant code in Java, because if you did, it would actually work just as fast and like every other dickless wonder on HN you actually thought your post was interesting.

vitalie 14y ago

I rewrote my DNS checking tool (http://www.dnsinspect.com/ http://www.dnsinspect.com/) in Go and I saw huge differences in resource usage (previously it was implemented with Ruby on Rails + EventMachine), my memory usage went down from 128MB per background worker to a few KB. Now I'm able to run hundreds to thousands of concurrent reports using a small VPS, the Go application is using 36MB of RAM (24MB front end + 12MB for background workers). Go language is well suited for my particular case (many concurrent IO operations). This was my first Go project, in a week I was comfortable with it, I had many alternatives but I really liked the simplicity of Go, fast compilation and easy of deployment. Because I've missed some pieces from the Ruby's world I've combined Jekyll (Compass, HAML SASS, RedCarpet, etc) with Go. :)

julien_c 14y ago
Looks great, bookmarked.
swah 14y ago
Do you also used the default net/http support libraries?
- vitalie 14y ago
  Yes, I'm using net/http.

chmike 14y ago

If you look for a Golang programmer today, it's more likely you'll find a true programmer instead of a me too programmer. So if you're a true programmer, learn Golang. ;)

Tzunamitom 14y ago

Was anyone else expecting this to be a boast from EA about how they cut their operating costs for the SimCity launch?!?

ajanuary 14y ago

"So then the decision came down to which language to use." I find this tendency to constantly jackknife between talking about languages and talking about frameworks a little dizzying. Shouldn't the decision have been what web stack to use? I'm sure you could get half way there with a bespoke Ruby web stack, maybe built on top of JRuby or with a sprinkling of C extensions. You could get the same poor scalability with some badly designed framework on top of Go [1]. There's a lack of meat in the post (hence why people are calling it out as PR). How do these features of Go make it easier to write applications or frameworks that cater to this particular scalability need? Could extra effort be put in up front to get something similar out of Ruby or any of the other languages, or will they always be sub-par? What is it about Go that makes it light up the rest of the stack in a way other languages don't? Or have you just traded one trendy technology for another because it promises to be the magic bullet for your current itch? [1] This isn't saying Rails is badly designed.

S_A_P 14y ago

Now I know that the dynamic languages such as ruby and python are not known for performance, but the benchmarks I have seen with Go look like it isnt exactly the fastest language available either. After reading this article I am more apt to think that something is/was VERY wrong with their Ruby code than to think that Go is that much better. Maybe my lack of experience with either GO or Ruby is showing here, but Ruby cant be that slow can it????

daderam 14y ago

Will be great if you can throw some light on "where" and in "what kind of work load" Ruby did not do well? I wonder why Ruby had hogged CPU? As I understand there are 4 kinds of workloads in your system (please correct me if I am wrong) 1. Handling several concurrent client connections. * How many concurrent clients are we talking about here? * Use of blocking sockets will require a separate thread for each connected client? Was Ruby not able to support many threads? * Non-Blocking sockets will require lesser number of threads. Note: I am from a Java background and Java's NIO scales well in handling volumes of concurrent sockets) 2. Handling volumes of data. (Data mining/aggregation) This is an IO/CPU intensive work load. * I dont think you would had done data-processing in ruby. Please correct me if I am wrong. * Scaling the "periodic" process to mine data can be efficiently done by choosing an appropriate storage system with efficient indexes. (MySQL/Postgres/Columnar-Storage/Cassandra etc etc) * Did you run a database? Did it run as a separate tier in a different server? 3. A queue/scheduling framework which would trigger the data-processing in periodic intervals(like google app-engine cron service) * How many "cron-entries" do you have in your system? 4. A CPU intensive algorithm which parses the incoming client's data before persisting in the database. * The incoming data from the client may require CPU intensive processing. (Example: parsing the data, eliminating duplicates etc etc). Did the CPU shoot up because of this? Will be great if you can throw some light on the volume/scale of the processing involved. Would like to know what kind of work load is "Go" better than Ruby. * Should I consider "Go" if I have lots of String (stream) processing to do? * Should I consider "Go" for handling several concurrent sockets? Would appreciate any help in understanding this.

ColinWright 14y ago

I know I'm late to the party, and no one will read this, but ... "... Java derivatives like Scala and Clojure ..." Someone help me out here: In what way is Clojure a Javav derivative?

willvarfar 14y ago
Perhaps in the sense that they are languages that target the JVM and that rely on interoperability with the Java standard library to bootstrap people making apps with them?
- ColinWright 14y ago
  I can't see how it could mean anything else, but I find that, well, bizarre. Oh well, different point of view, I guess.

ksec 14y ago

No Love for Lua and LuaJit.

st3fan 14y ago

We decided to rewrite the API. This was an easy decision, clearly our Ruby on Rails API wasn't going to scale well and coming from many years of Java development and having written a bunch of things that handled tons of load with way less resources than what this Ruby on Rails setup could handle, I knew we could do a lot better