10 ms·
Handling 1M Requests per Minute with Go
- ExpiredLink 11y agoGo astroturfing continues ...
- de_dave 11y agoIndeed, think the solution probably did most of the work here, although Go's channels would have made it a little easier to implement.
- fixxer 11y agoBased on the write-up? Regardless of language choice, I thought the analysis was pretty honest and the data suggests they did indeed engineer a solution worth being proud of. Did you get beyond the title?
- aikah 11y agoWould be interesting to have an app that display stats and trends about technologies hitting HN homepage. Regardless of any astro-turfing going on or other shenanigans. But it's true Go somehow gets a lot of exposure on HN, not unlike Nodejs a few years ago, or Rails before. Funny how people seem to discover hot water each time a new technology/hype hit the IT world. Soon one will read "How we went from 100000 LOC to 1000 by switching from Go to nim/crystal".
- thwd 11y agoArguably, UploadToS3 should not be a method of *Payload. Suggestion: Make an S3-uploader package with _internal_ connection pooling, upload queueing and concurrency handling.
- zimbatm 11y agoThis. Each S3-uploader can then hold a http.Client instance that keeps the connection open to S3 between uploads.
- placeybordeaux 11y agoI haven't used this library, but this appears to be what you want. https://github.com/rlmcpherson/s3gof3r https://github.com/rlmcpherson/s3gof3r Nice features include: > Retry Everything: All http requests and every part is retried on both uploads and downloads. > Configurable conncurrency > Uses an io.Writer (you could actually start posting to S3 before all of the data gets in on your side.) etc.
- th0br0 11y agoOther than the worker/concurrency mechanism being part of the language, what's the difference to a RabbitMQ-Worker architecture? You might even argue that the lack of persistence (in the given example) is a potential source for data loss.
- fixxer 11y agoI'm guessing persistence is not a priority. The difference is management of a simple process (behind elastic load balancer, of course) vs a more complicated architecture with three distinct, load balanced process types (webserver -> queue -> worker).
- fasteo 11y ago>> But since the beginning, our team knew that we should do this in Go because during the discussion phases we saw this could be potentially a very large traffic system I don't get this reasoning.
- fixxer 11y agoThey're going from Ruby, so Go is a pretty easy transition with respect to skill set and helped them accomplish scale (as seen from the analysis). I agree that other languages could accomplish the same effect, but I'm not surprised by their sentiment given where they were coming from.
- Ao7bei3s 11y agoGo is 1-2 orders of magnitude faster than Ruby, and much easier to write concurrent code in. It makes perfect sense to me. What would you have recommended them, for a reasonably-high-performance server implementation? (Please don't say C.) https://benchmarksgame.alioth.debian.org/u64q/benchmark.php?test=all&lang=go&lang2=yarv&data=u64q https://benchmarksgame.alioth.debian.org/u64q/benchmark.php?...
- plydatbk 11y agoLua would beat go every day of the week.
- jbeja 11y agoAs Clojure would beat Lua every second of a day.
- plydatbk 11y agoshow me the clojure firewall and I'll take you seriously. Or the even something simple as an interesting clojure ML project. Java gross.
- bitcrusher 11y agoIf you're talking about raw performance, I'm afraid you're mistaken. If you're perhaps referring to the 'aesthetics' of the language syntax, that's pretty subjective, in which case you're right for your tastes.
- mstump 11y ago16k requests per second isn't fast.
- fixxer 11y ago16k requests per second is not fast if you're talking about fetching a page or doing a minimal amount of I/O. 16k requests per second is worth writing about if you're talking about a process with substantial side effects (S3 I/O).
- mstump 11y agoIt's just pass through. The limit in this instance is probably packets per second of the legacy AWS network. Why is this difficult?
- fixxer 11y agoRead the analysis. First attempt was "just a pass through".
- mstump 11y agoNo, it is just pass through, they just did a crappy job of handling concurrency.
- fixxer 11y agoMaybe they should hire somebody smart like you.
- mstump 11y agoTo be snide, maybe they should, I am an expert in this topic this is what I do all day every day. They're doing development without understanding how computers work, where the bottlenecks are, or what the maximum theoretical throughput for the use-case is. They ended up with something slightly better than the horrible situation they were in, and are celebrating a inefficient solution as a technical triumph.
- wpeterson 11y agoIf this system is merely decoding JSON and writing payloads to S3 for asynchronous data processing, why not have your clients write directly to S3?
- blakesmith 11y agoI'm not the author of the article, but if I'd have to guess: Data encapsulation, and interface control. If all your clients are talking directly to S3 instead of your encapsulated service interface, you can't inject any business logic at all and must design around the fact that you don't control the service interface, Amazon does.
- melling 11y agoWhat's the trick to resubmit without getting stuck going to the first post? For example, I submitted this story 2 hours before this one: https://news.ycombinator.com/item?id=9844826 https://news.ycombinator.com/item?id=9844826 In the past, when I try to submit a story, even if it's a couple days old, the submission is ignored, and my vote is added to the original.
- thwd 11y agoThe slash at the end of the URL made the difference in this case.
- vruiz 11y agoOuch, it really shows that getting to the HN frontpage can be just a lottery.
- shocks 11y agoOne trick is to add an anchor tag to the end of the URL.
- aw3c2 11y agoYou can always cheat by appending garbage with ? or # Please only do this if you really believe that you should resubmit.
- mrfusion 11y agoIt's interesting this came up today. I'm looking for a new language to migrate my flask/uwsgi web service to. I'm having a terrible time making it scale. Are there any tutorials/templates/best practices for writing a small web service in Golang?
- ihsw 11y agoThe best advice I can give is to write a minimal web server using net/http[0], get a feel for how to write a RESTful HTTP API by hand, and then start experimenting with web frameworks like Gin[1], Martini[2], and Beego[3]. [0] https://golang.org/doc/articles/wiki/ https://golang.org/doc/articles/wiki/ [1] https://github.com/gin-gonic/gin https://github.com/gin-gonic/gin [2] https://github.com/go-martini/martini https://github.com/go-martini/martini [3] https://github.com/astaxie/beego https://github.com/astaxie/beego
- placeybordeaux 11y agoI'd recommend against both martini and gin. Neither of which properly use the HTTPHandler interface and the owner of martini has officially stated that martini is not idiomatic. I loved it when I started, then really hated it when I needed to refactor, there is just too much magic and you pay a speed cost for that magic.
- plydatbk 11y agothe only reason to write something like that in Go is if you're looking to impress a recruiter at (Go). Advise: pick a better tool for your problem.
- amencarini 11y agoA better tool such as?
- felixgallo 11y agodepends a lot on what your constraints are and what goal you're trying to achieve. Go could be the right choice, but I believe the prior poster was alluding to the fact that it does not fit the sinatra/flask/etc. use case very well for most use cases.
- deleted 11y ago[deleted]
- mrfusion 11y agoMods, can we change "go" in the title to "golang"? That's the official name, and it makes it searchable by search engines?
- f2f 11y agoI'm sorry but the official name is Go. You can search for "Golang" or "Go Language" with no issue.
- mrfusion 11y agoWould this thread come up under one of your suggested searches?
- f2f 11y agoboth the original article and the Reddit discussion come up in my results (as 1 and 2). the reddit discussion is only 16 hours old. https://www.google.com/search?q=golang%201%20million&rct=j https://www.google.com/search?q=golang%201%20million&rct=j this HN discussion is too recent to show up in my results, however yesterday's thread about Qihoo and golang does show up as number 4 or 5 when searching for "golang qihoo". no need to panic.
- matthewbauer 11y agoI still see it written as "Go" most of the time. It's unfortunate because I've picked up the habit of searching with "golang".
- ktsmith 11y agoGo is the official name, golang is the only way to get reliable search results however.
- darksaints 11y agoDoes anybody here have any experience with Go's garbage collection pauses with large stack sizes? I've got a scala app that regularly consumes about 48G of ram, and I'm very happy with the response times during heavy loads like this, but the P99.5 is abysmal because of garbage collection. I've tried tuning it, but it doesn't seem like anything I do helps. I'll probably end up using an Azul JVM but I'm curious how other languages end up handling this problem.
- f2f 11y agothis article discusses how they handled 69GB of heap and up to 6 seconds of pause times: http://blog.golang.org/qihoo http://blog.golang.org/qihoo the new garbage collector in 1.5 should improve things.
- baijum 11y agoMore info about GC changes in 1.5: https://talks.golang.org/2015/state-of-go-may.slide#6 https://talks.golang.org/2015/state-of-go-may.slide#6 (Slides 6 to 11) https://golang.org/s/go14gc https://golang.org/s/go14gc
- thrownaway2424 11y agoIt's really more about how many objects are on the heap, less about how long the heap is.
- cdelsolar 11y agoWhat is P99.5?
- deleted 11y ago[deleted]
- ihsw 11y ago99.5th percentile -- below that threshold, requests are generally fine, but there's a small fraction of requests that take way too long to go through due to GC kicking in.
- bpicolo 11y agoCool article, not necessarily because of the language specifics but because of the thought process involved. Thanks!
- peterwaller 11y agoThere are two other solutions that spring to mind, which might require quite a bit less code: 1) Take the original code, do the upload exactly in place in the original request (not even spawning a goroutine). However: protect the upload with a semaphore which only allows N-in-flight. My reasoning is, well, if the system operates with low latency when operating nominally, blocking the incoming request isn't too painful. The reason there was a problem in the first place that there were too many requests in flight and the system hit a meta-stable state where no requests could complete efficiently. 2) (or instead of (1)): If you're going to have a worker pool, why have that complicated chan-chan-Job business? It seems that `func StartProcessor` was close to being a viable solution. All you need is to start a few of those in parallel, each reading from the same `Queue`. Was there a reason to introduce the `WorkerPool chan chan Job`? That looks quite a bit more complicated than it needs to be. The queues don't need to be separate per worker unless there is some other substantial reason. -- The next thing one would need to take care of is to ensure that the whole system doesn't stall due to a broken/laggy network, so, to put some timeouts on the S3 uploads, for example, to ensure the system can return to a stable state on its own when the thundering herd has passed.
- placeybordeaux 11y agoYeah I thought the double chan was a bit strange. It's like mutliple lines at checkout instead of one big line, it just increases the variance.
- eva1984 11y agoTimeout/Retry/ExponentialBackOff
- ignoramous 11y agoRe: Semaphore: Not dropping the connection might mean we might starve others incoming msgs of resources (which might be a good thing) [0]. Also, releasing the semaphore back into the pool in case of failures takes on a happy complication of having to deal with errors beyond one's control. Re: Queue: Wouldn't the queue involve locking lest two workers end up trying to work on the same request? To be completely concurrent, I guess one could use a lock-free data structure instead (or implement one on top of something like RocksDB)? [0] http://ferd.ca/queues-don-t-fix-overload.html http://ferd.ca/queues-don-t-fix-overload.html [0] http://engineering.voxer.com/2013/09/16/backpressure-in-nodejs/ http://engineering.voxer.com/2013/09/16/backpressure-in-node...
- AYBABTME 11y agoYou can put up any large number of request if you make the period in 'per {{period}}' large enough.
- Udo 11y agoI was confused by the numbers at first, so: that's 17k requests per second, spread out over 4 dual core Xeon (Haswell) machines, which works out to just over 4000 requests/s per machine. It's still a respectable number, but it's much closer to what one would expect given the task. Don't get me wrong, the most interesting part is definitely the implementation and as a Go noob I found it very useful - it's just a bit misleading for the headline to sum your request rate across all parallelized machines.
- Spien 11y agoCan already do this with a single thread using epoll. In fact, even a simple epoll implementation can handle 1M HTTP requests in ~30 seconds on a single thread juggling 10k connections.
- deleted 11y ago[deleted]
- eva1984 11y agoLittle confused here.So the third solution mentioned in the post is just a worker pool? I think if we just slightly modified the second solution, it will totally work. 1).Initialize a job channel 2).Initialize a set of workers that listen to this channel to pull the jobs indefinitely. In this case just call go StartProcessor() for fixed number of times. What confuses me is that IMO workPoolChannel isn't necessary here. What is the consideration behind to use a channel for workers?
- deleted 11y ago[deleted]
- meir_yanovich 11y agoCan you please explain why to use go and not c++/c forget about language syntax / compilation complexity. say i know both very well , now why to go with "GO" ? thanks
- nindalf 11y agoThere are a few benefits of Go I can think of * Fewer lines of code, fewer gotchas and hence easier to reason about and maintain. * Powerful concurrency primitives (channels, select) built right into the language, rather than a library. A scalable producer-consumer implementation would probably be 100 lines of Go code. * If your application isn't too latency sensitive (game server, frequency trading etc) then the GC simplifies matters. Its guaranteed to run for a maximum of 10ms out of every 50ms which is good enough for most applications. (but typically runs for around 1ms) * Some of the tooling around the language is great. There are some great articles (I remember one posted to HN yesterday) about how people wrangled a lot more performance out of their code using the profile tool, for instance. * Miscellaneous goodies like testing out of the box, an extensive standard library and being able to compile in 1/10th of the time. An example of a service migrated from C++ to Go - dl.google.com - http://talks.golang.org/2013/oscon-dl.slide#1 http://talks.golang.org/2013/oscon-dl.slide#1 These are the benefits I could think of if a programmer knows C++ and Go equally well. However, suppose he has to work with fellow programmers who aren't comfortable with either, Go would be a superior choice. It would take a week to learn most of Go and perhaps a month to grok it. I think C++ takes much, much longer than that to learn properly.
- meir_yanovich 11y agoThank you for the reasoned reply.
- nindalf 11y agoYou're welcome :)
- azth 11y agoWould have been much simpler and more straight forward to use a library like Akka instead of manually coding all of this.
- sinzone 11y agoHave you considered in putting KONG [1] which is basically OpenResty (nginx) in the front? With LuaJIT performances [2] are outstanding. [1] https://github.com/mashape/kong https://github.com/mashape/kong [2] https://github.com/mashape/kong#benchmarks https://github.com/mashape/kong#benchmarks
- avitzurel 11y agoIMHO this solution is nice but wrong. The point is not just creating a "cool" program with Go that will handle HTTP requests. Without really knowing the company's needs, I am relying on this paragraph from the post: While working on a piece of our anonymous telemetry and analytics system, our goal was to be able to handle a large amount of POST requests from millions of endpoints. The web handler would receive a JSON document that may contain a collection of many payloads that needed to be written to Amazon S3, in order for our map-reduce systems to later operate on this data. Knowing this, I would build it differently. 1. Clients post to S3 Directly 2. Lambda -> Overload business logic, private data, cleanup, spam control etc... 3. Prepare files (64M) for Hadoop 4. Hadoop There's no reason to have that proxy in the middle, Amazon S3 will handle those millions of requests with no real trouble, I wouldn't throw machines on this process.
- istvan__ 11y agoIs this supposed to be great performance? I think Netty does ~30K/s (1800000 req/min) out of the box. It thought Go has more out of the box performance, maybe I am missing something.
- anonyfox 11y agoOh, performance? Let's throw numbers around! Here, Elixir/Phoenix beats them all! https://twitter.com/julianobs/status/614416512825323520 https://twitter.com/julianobs/status/614416512825323520 Hey, and Elixir is already way more expressive than Go and it's incredibly easy to build fault-tolerant and distributed systems, not to mention the productivity gains when using the phoenix framework! Seriously, posting requests/second metric without any context about hardware and sample code doesn't help anyone.
- istvan__ 11y agoCool it is only 10-16x slower than Aleph. https://github.com/ptaoussanis/clojure-web-server-benchmarks/tree/master/results/60k-keepalive#60k-keepalive https://github.com/ptaoussanis/clojure-web-server-benchmarks...
- anonyfox 11y agofull blown mvc framework vs communication layer, seems legit :) but hey, as long as stuff responds in microseconds with zero errors under load, just use it ! (Also, clojure is a way better language than go, too.)
- istvan__ 11y agoYes I like to throw meaningless numbers around as much as the other guy. :)