10 ms·

Web Framework Benchmarks - Round 8

174 points by curiousAl 13y ago

curiousAl 13y ago
I've been following these for most of the rounds, and Go has been improving impressively. Whether that's because of improvements in the language itself or a more zealous crowd sending pull requests, I don't know, but it made me want to try go, so I did. It's not as comforting as the scripting (PHP, Python, JS) languages I'm used to. Having no REPL and having to think about types takes a bit more getting used to than I thought (arrays vs slices/maps, and having no REPL). I find having a quick build script (mine's in vim) so you can compile+run and go back to the code quickly helps a lot. Also, http://play.golang.org/ http://play.golang.org/ isn't too shabby either. It would be fun to see this project (https://github.com/TechEmpower/FrameworkBenchmarks https://github.com/TechEmpower/FrameworkBenchmarks) become more and more popular, with formidable developers squeezing out performance from their framework of choice.
- dylandrop 13y ago
  Yeah but I'm kind of confused as it's my understanding Go is not a web framework so much as a language. Is this just testing how fast Go can print out the string "{message: 'Hello World'}"? Or are they testing a specific component/library in Go? I mean obviously having a language just spit out a line is going to be faster than having a fully blown framework such as Rails work through all of the query parsing, view building, etc. so it doesn't seem like a very fair or useful comparison.
  - dangrossman 13y ago
    It's included, alongside Go frameworks, for the same reason PHP/Ruby/ASP.NET are included -- so that you can see how much overhead the frameworks are adding compared to a minimal implementation in the language they're built on. The code behind every benchmark is available under the source code tab up top. The Go benchmark is using some JSON library, not just printing a string.
  - curiousAl 13y ago
    Right, Go is a language. It does have a decent (built in) HTTP library, and is actually pretty fast as a webserver in itself. On the more conventionally robust framework end of things, I suppose Revel/Falcore would be a more appropriate comparison. (both of which have impressive performance of their own).
  - mrweasel 13y ago
    The issue is that Go comes with enough included in the standard library that it could be considered a web framework. If you're just build a few rest endpoint or a simply site, you may not need to leave the standard library. Compare that to a language like Python or Ruby, where you need something "extra" to make it easier to do a web application. You could certainly do with just the standard library in other languages, but very few would choose that option, because it would involve write a lot of additional code. I think it fair to include Go, because it's a language/programming environment, that comes with it's one built in web framework. A framework that's actually advanced enough that many don't need to look else where.
  - Shamanmuni 13y ago
    Go is a language, but its standard library is very comprehensive for most things Internet related and you can build web applications quite easily using just that. You also have similar examples in the benchmark using nodejs and php, which aren't exactly frameworks. I know there should be some overhead when using a framework, but sometimes the cost is too high and it's useful to know it (compare php and symfony2, for example). You can look at the Revel benchmark, a web framework written in Go which did quite well in the benchmark.
atonse 13y ago
Interesting to see go moving up there. Curious - any reason why you guys don't have ASP.NET tests in Windows with SQL Server? I fiddled with the filters and found none. Update: Never mind. I see it now. You don't have Windows tests on EC2.
- deleted 13y ago
  [deleted]
nikentic 13y ago
I cannot find Flask in the list. Any specific reason?
- krg 13y ago
  The submitters didn't create a JSON test, but all the other tests are present. Switch to the Plaintext test to see Flask.
- deleted 13y ago
  [deleted]
- pekk 13y ago
  Python has never been a priority in these benchmarks
riquito 13y ago
I'd love to see how many lines of code each test required, but it's probably impossible to do in a fair way edit: I meant in the chart, at a glance
- jwcrux 13y ago
  I believe the tests are open-sourced on GitHub.
- steveklabnik 13y ago
  https://github.com/TechEmpower/FrameworkBenchmarks https://github.com/TechEmpower/FrameworkBenchmarks
- bhauer 13y ago
  In fact, we have some work in progress on that front, along with the number of commits to the test implementation directory. Combined, these will give a rough idea of code length and the amount of community input/review each test has received.
  - riquito 13y ago
    Thank you for your work, it's really interesting
mmucklo 13y ago
Regarding symfony2 at the bottom - I submitted a simple pull request to try and fix some issues with the setup, but it's been sitting and sitting there... https://github.com/TechEmpower/FrameworkBenchmarks/pull/650 https://github.com/TechEmpower/FrameworkBenchmarks/pull/650
- bhauer 13y ago
  Hi mmucklo. We'll get that merged in for Round 9!
shijie 13y ago
This is a fascinating round for WFB, with drastically different results from round 7. I'm impressed with the strides Go has made, and also quite impressed with JRuby. I know the banking app Simple chose it as its language/runtime of choice, and they seem to leverage it well. I'd still like to see a good showing from Django, maybe using uWSGI + Nginx. I might submit a pull request and see if I can't get that included in the next round. Gunicorn is great and incredibly easy to set up, but pales in comparison to other platforms when it comes to raw speed.
- ninjay 13y ago
  As far as Django goes, there hasn't been much tuning in general[0]. The only thing I see them doing is template caching. At the least they should be running 1.6 with persistent DB connections. Beyond that they have a lot of middleware enabled that isn't being used. [0]https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/django https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast...
saltvedt 13y ago
Whats up with the number of Rails errors?
sker 13y ago
HHVM and Dart seem to be the two new fast performers in town showing impressive performance in some tests. JS has been falling off the charts compared to some of the first rounds, but still a good option performance-wise. C# keeps sucking badly. I miss Nimrod/Jester, I always wanted to see it in the top 10.
- pjmlp 13y ago
  > C# keeps sucking badly I wonder why, the Fortune 500 sites we have built are handling the load quite well.
  - ebbv 13y ago
    Probably because you threw Fortune 500 sized budgets and hardware at them. This is only really relevant to people who need to maximize limited hardware.
    - pjmlp 13y ago
      Might be, the servers are actually quite beefy.
  - asdasf 13y ago
    Fortune 500 has little to no relation to demanding website. Plenty of fortune 500 companies aren't even in the top 1000 busiest sites.
    - pjmlp 13y ago
      The sites we have built have pretty high demand, I just mentioned like that because of NDA.
  - bigtones 13y ago
    These benchmark tests for C# are run against MySQL or PostgreSQL on Linux. In the Fortune 500 setup you're probably connecting to SQL Server or Oracle in the back end for which Microsoft and DB vendors have optimized OLE-DB drivers. That, and JSON serialization on .Net using default MS serializer is super slow. Everyone uses JSON.NET or another faster serializer in the real world.
ritchiea 13y ago
In the past I've noticed posters on HN picking on Rails by lazily linking to these benchmarks but click over to the average latency tab and Rails looks pretty solid with an average response latency of 1.8 ms, which is not at the very top but far better than Django, which is a comparable framework and is near the bottom of the average latency table. If anything to me this data confirms that Rails is an amazing tool because not only do you get to develop quickly, but you also get pretty good average latency (or at least the potential depending on what you add to your app in terms of 3rd party libraries). And what Rails isn't good at is throughput, which is almost never a problem for an early stage company. Working at a startup it's a huge success if I ever have to handle a lot of connections to my app, but today and everyday, I want fast response times on a page load.
- bhauer 13y ago
  Fair warning: as far I know, Wrk's latency measurement does not distinguish between 500s and 200s. For some frameworks, you will see unnaturally low latency because the front-end web server is providing a 500 response very quickly.
- Segmentation 13y ago
  > Working at a startup it's a huge success if I ever have to handle a lot of connections to my app, but today and everyday, I want fast response times on a page load. Realistically, I doubt many humans can distinguish between 1ms and 100-200ms. response times.
  - bhauer 13y ago
    True, especially when accounting for Internet latency. However, the purpose of this project is not actually to measure how quickly platforms and frameworks can execute fundamental/trivial operations. Rather, these tasks are a proxy for real-world applications. Across the board, we can reasonably assume that real applications will perform 10x, 50x, 100x, or even slower than these tests. The question is, where does that put your application? If your application runs 100x slower than your platform/framework, does that put your application's response time at 200ms or 2,000ms? That's a difference users do notice.
pfraze 13y ago
Cppsp (top of the i7 charts) is some mad science http://xa.us.to/cppsp/index.cppsp http://xa.us.to/cppsp/index.cppsp
anilmujagic 13y ago
I'm really surprised with ASP.NET/C# results :-S
- faraazin 13y ago
  its asp.net mono and i am not surprised.
- friism 13y ago
  Also note that those tests have tons of errors, so they're probably not representative.
- sergiotapia 13y ago
  It's Mono implementation, not .NET.
RyanZAG 13y ago
Interesting observation regarding the differences between the EC2 and i7 results: the platforms at the top of the EC2 benchmarks are generally MongoDB+async io java, while the ones at the top of the i7 results are MySQL+heavily threaded (go, servlet, openresty). I think it's a pretty interesting result because it shows how much your choice of available hardware has on which platform would be best - and it's not a small difference either. If you're going for an EC2/digital ocean setup with a lot of small instances, then you want to go with something like vert.x or node or whatever - while if you are deploying directly onto bare metal high core/ram servers, you'd be better off with something that is better at handling high thread counts - something like Golang.
- piranha 13y ago
  Go has low number of threads, usually a number of your cores or close to it. Goroutines are not threads.
  - RyanZAG 13y ago
    If you have 8 cores, you'll get 8 threads and all your requests will be nicely distributed across the cores. This is why Go is up near the top for the i7 benchmarks. On the EC2 ones, there are far fewer cores, so the overhead of distributing them is much more pronounced. As you add more cores, you'd probably see Go pull further ahead of some of the competition. However if you're only ever going to be running Go on small instances as many people do, this advantage is actually a hindrance because of the added overhead. Not that it's necessarily a big issue or anything, it's just interesting to consider. The point I was making is that your actual hardware and workload can turn this benchmark on its head. You may naively think you are upgrading performance by switching to a different framework/language, yet if you don't understand why each platform is getting the numbers it does you might end up rewriting your app and actually decreasing performance because of your server hardware.
brickcap 13y ago
Looks like erlang frameworks are not represented...
- kainsavage 13y ago
  We have been having trouble with Erlang frameworks since before Round 7. Unfortunately, I was still getting up to speed and improving the suite mostly for Round 7/8 and did not get to fix this yet. I do have it topping my todo list for round 9, with the hope being to get them all back in and working soon.
neya 13y ago
I know benchmarks should be taken with a pinch of salt, but by round 5 I was totally into Scala (Scalatra), trying to write my own framework, so I could get better bang for buck from my EC2 instances, which to be honest, aren't cheap when compared to say, Digital Ocean. Around round 6 of these benchmarks, I ditched Scala altogether (and also my framework).The reason I ditched Scala was not because of it's performance, etc. But it was because I was the only developer in my company who knew and learnt Scala after reading a couple of books (one was around 800 pages). Obviously, I needed a language that any other developer should have no problem taking over, and Scala developers are 1)expensive 2)not easy to find. Also, Slick (the database interacting code for Scala by Typesafe) wasn't mature yet. For this reason, around Round 6, I started writing my own framework in GoLang and used it internally as an 'auxiliary framework'. I will explain more about this framework soon soon. In my company, we have about a handful of backend programmers and a couple of frontend devs. I found that GoLang was much much easier to teach my programmers, than say I could teach Scala. Please note - Scala is a brilliant functional programming language, but if you are thinking switching from Ruby/Python/etc would be easy, then you are wrong. Now, we have a workflow that allows us to deliver as quickly as possible, but without missing out on performance - We write our entire V1 in Rails. We implement all the UI/frontend related code and then port it to our GoLang framework. We have an internal generator where we just feed our rails app, and the code for our framework is just 'ported'/generated on the fly based on our framework and we just deploy it. So far, our productivity is slightly lost while handling the type conversions, bugs, etc. But it's totally worth it. Go outperforms Rails by a huge margin. I noticed that using something like Puma helps a lot, but it still is no way comparable to our GoLang framework. As for our framework, it's just pretty simple - Just organize all the files as you would in a Rails application (Models/Views/Controllers/Config) and everything just works without much performance hiccups. We use Gorilla components for stuff like routing and cookies. The rest of the stuff is slightly adapted from other frameworks (like Martini). All in all, I love the ability to have JVM like performance with the productivity of Ruby with a language like GoLang. And this round 8 benchmark is nothing short of impressive. If you haven't tried GoLang yet, you should try writing your own framework, not only do you learn about all the trade-offs for the 'magic' that rails makes under the hood, you also learn about some new stuff and thus become a better programmer. I think GoLang is pretty impressive if someone as average as me can even write a framework like Rails, except for better performance. Give it a try, people, you won't be disappointed.
WoodenChair 13y ago
It's amazing how well a young language like Dart and its frameworks performs in the multi-query benchmarks. There's still so much more optimization to go; at this stage it feels optimistically like the sky is the limit!
desireco42 13y ago
What always impresses me and leaves impression, is just how fast raw PHP is. At times it seems PHP has been obsoleted by new platforms, but benchmarks like these make a case for it's use. Especially because it is really easy for beginners to pick this up.
- kvtrew76557 13y ago
  The first PHP result for the JSON test comes in at 31.7% of the performance of the top performer. PHP also occupies the bottom 3 worst slots.
  - krapp 13y ago
    That's disappointing. One thing PHP should be really good at is serializing/deserializing JSON.
zaroth 13y ago
It's interesting to compare what the code looks like. CPPSP (C++ Server Pages) which is putting up ridiculous numbers... here is the Single Query test: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/cpoll_cppsp/www/db https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... It's quite different from the more typical implementations, where they all sort of look the same... (Go) https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/go/src/hello/hello.go https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... (NodeJS) https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/nodejs/hello.js https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... (Gemini) https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/gemini/Source/hello/home/handler/HelloHandler.java https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... Also interesting to compare it to C# / HttpListener... which would benefit from moving all the framework code out into a separate library; (C#/HTTP.sys) https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/HttpListener/HttpListener/Program.cs https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
stesch 13y ago
I like the benchmark and I appreciate the work that was put into, but Erlang is missing again. If you don't even consider Erlang you won't miss it. But if you know it has some strengths for this kind of job and you don't mind the syntax, you'd like to see it compared to other solutions.
- agilebyte 13y ago
  I have heard that string operations (and I thus suspect JSON parsing too) is slower on Erlang. Maybe it is not included as it was not built for raw speed but rather stability, hotswapping code etc?
  - stesch 13y ago
    Erlang was included in round 6 of the benchmark.
    - agilebyte 13y ago
      Ah thanks, under Cowboy and Elli.
  - rdtsc 13y ago
    String equivalents in Erlang are binaries quite often. There are also iolists, those are pretty efficient. This blog post describes this in a lot more depth: http://jlouisramblings.blogspot.com/2009/01/common-erlang-misconceptions.html http://jlouisramblings.blogspot.com/2009/01/common-erlang-mi...
- kainsavage 13y ago
  I just posted another response to this - we had some trouble with the package manager for Erlang after round 6. Additionally, I had been working on improvements for the suite specifically (better logging/reporting, etc) and did not get a chance to resolve the Erlang problems. Rest assured, "get erlang running again" tops my 'todo' list for round 9.
  - 13y ago
mrinterweb 13y ago
I'm rather surprised to see rack-jruby up as high as it was. I discounted ruby as an option for a very high performance http service, but I guess I'd be wrong to do that. Don't get me wrong, I love ruby and I use it every day. I just didn't expect to see it in the top performance contenders list.
- bhauer 13y ago
  That is principally thanks to TorqBox, the codename for Torquebox 3, which is built on Undertow. Undertow is the web server that is scrapping with Netty and Vert.x on the plaintext tests. Also note that the particular Rack test that performs very well is running a very small amount of Ruby code. Thanks to these improvements, however, rails-jruby now consistently tops rails-ruby, if only by a small amount. See more on TorqBox: http://torquebox.org/news/2013/12/04/torquebox-next-generation/ http://torquebox.org/news/2013/12/04/torquebox-next-generati...
banachtarski 13y ago
None of these numbers are significant! Give me something that tries hundreds if not thousands or tens of thousands of simultaneous requests. Then we have a real benchmark that will probably push a lot of these over the edge in terms of mean latency and especially tail/peak latency.
- MetaCosm 13y ago
  There have been a group of us -- consistently pushing for exactly this. The maintainers of the benchmark are exceptional resistant to this idea... https://github.com/TechEmpower/FrameworkBenchmarks/issues/49 https://github.com/TechEmpower/FrameworkBenchmarks/issues/49 ... https://github.com/TechEmpower/FrameworkBenchmarks/issues/36 https://github.com/TechEmpower/FrameworkBenchmarks/issues/36 ... https://github.com/TechEmpower/FrameworkBenchmarks/issues/48 https://github.com/TechEmpower/FrameworkBenchmarks/issues/48 ... there are even more issues asking for concurrency increase, just search for concurrency. It it silly that such an rich and awesome set of benchmarks never pushes on concurrency, one of the major points of failure "in the wild" -- more common as you become the go-between for your users and some set of APIs -- users stack up on one side, waiting connections stack up on the other.
  - bhauer 13y ago
    There is a very simple reason for this: we do not yet have a test that is designed to include idling. One of the future test types [1], number 12 on the list, is designed to allow the request to idle while waiting on an external service. Until we have such a test type, there is no value in exercising higher concurrency levels. Outside of a few frameworks that have systemic difficulty utilizing all available CPU cores, all tests are fully CPU saturated by the existing tests. With that condition, additional concurrency would only stress-test servers' inbound request queue capacity and cause some with shorter queues to generate 500 responses. Even at our 256 concurrency (maximum for all but the plaintext test), many servers' request queues are tapped out and they cope with this by responding with 500s. The existing tests are all about processing requests as quickly as possible and moving onto the next request. When we have a future test type that by design allows requests to idle for a period of time, higher concurrency levels will be necessary to fully saturate the CPU. Presently, the Plaintext test spans to higher concurrency levels because the workload is utterly trivial and some frameworks are not CPU constrained at 256 concurrency on our i7 hardware. As for the EC2 instances, their much smaller CPU capacity means the higher-concurrency tests are fairly moot. If you switch to the data-table for Plaintext, you can see that the higher concurrency levels are roughly equivalent to 256 concurrency on EC2. For example, jetty-servlet on EC2 m1.large: 256 concurrency: 51,418 1,024 concurrency: 44,615 4,096 concurrency: 49,903 16,384 concurrency: 50,117 The EC2 m1.large virtual CPU cores are saturated at all tested concurrency levels. jetty-servlet on i7: 256 concurrency: 320,543 1,024 concurrency: 396,285 4,096 concurrency: 432,456 16,384 concurrency: 448,947 The i7 CPU cores are not saturated at 256 concurrency, and reach saturation at 16,384 concurrency. We are not against high-concurrency tests; we are just not interested in high-concurrency tests where they would add no value. We're trying to find where the maximum capacity of frameworks is, not how frameworks behave after they reach maximum capacity. We know that they tend to send 500s after they reach maximum capacity. That's not very interesting. All that said, once we have an environment set up that can do continuous running of the tests, I'll be more amenable to a wider variety of test variables (such as higher concurrency for already CPU-saturated test types) because the amount of time to execute a full run will no longer matter as much. [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/133 https://github.com/TechEmpower/FrameworkBenchmarks/issues/13...
bsaul 13y ago
Anybody could explain what gemini is ? I've been on the eclipse project home, and i really don't see the link with a web framework benchmark.
- mutagen 13y ago
  Gemini is the private Java framework Techempower uses on their client projects. I believe questions regarding its performance relative to various open source and enterprise JVM frameworks inspired the first Techempower benchmarks. http://www.techempower.com/blog/2013/03/28/frameworks-round-1/#section=questions http://www.techempower.com/blog/2013/03/28/frameworks-round-...
- abvdasker 13y ago
  Second on this question. Extra points if you've done any development with Gemini. I'm really curious: in every category it was the highest-scoring full-stack framework, which is especially surprising given its apparent obscurity. Edit: every category but the plaintext benchmark.
- kainsavage 13y ago
  Gemini is our internal Java web framework; it has no relation to the Eclipse project (other than I use Eclipse when I work on Gemini) ^_^
matrix 13y ago
These results are tempting me to do my next project in a modern lightweight Java framework. No Hibernate, bloated frameworks of yore, or weird complex build and dependency management. Play is ruled out - it's Scala (Java is a second-class citizen in Play). Maybe something that ties together things like ebean ORM, Jetty, Jersey, Jackson, Guice. Dropwizard is the right idea, but is geared towards building REST backends. Any suggestions on a pure Java framework that has critical mass and would fit the bill?
- matrix 13y ago
  Following up on my own question, there doesn't appear to be any that quite fit the bill right now, if we define the ideal framework as having the following characteristics: * Java as a first-class citizen * Strong core of basic web app functionality * REST and Search engine friendly URLs * Action oriented – basic framework for routes, MVC etc * Stateless * Good documentation, active community If we look at action frameworks only: * Play 2: Great except it's Scala. Ruled out. * Spring MVC: Spring is bloated old-school Java with Hibernate. Out. * Stripes: hasn’t had a commit in over a year… which is unfortunate because it looks interesting. Out. * Spark: appears to be a one-person project. Out. * Google Sitebricks – ditto * Ninja: Ditto
  - stusmall 13y ago
    Why did you find lacking in java on Play? I've just started playing with it and aside from the template engine (which I don't count) I haven't found any part of the java support lacking vs the scala.
    - matrix 13y ago
      In my brief try-out of it, it felt to me like if you want to do anything different (e.g. your own implementation of something), you pretty much need to switch to Scala. It's a Scala framework first and foremost. A minor demerit was getting SBT and Play Java to work correctly in IDEA was enough of a pain to make me wonder how much overhead that was going to incur in the long run. If someone who has used Play Java in production on a large project can weigh in on whether these points are true or not in real production use, I'd love to hear it.
rartichoke 13y ago
Benchmarks are fun but I'll stick with rails and its simple ways of letting you cache data. I'm ok with getting out the door response times of 8-15ms while serving 20,000 unique hits a day on a $5/month VPS. The server does not even break a sweat too and it's doing more than serving the app too.
- matthewking 13y ago
  What kind of response times do you get on a cache miss though?
  - rartichoke 13y ago
    80ms-350ms is normal under typical traffic conditions. It depends on the complexity of the page. That's still not terrible though and it could easily improve by massive amounts with a stronger server. I have not gone crazy with profiling either. Just using fairly basic cache blocks when applicable.
veto64 13y ago
i'm interested in who is financing this benchmarks. really sorry, but for me it looks like a new way of doing seo marketing
- bhauer 13y ago
  This comment makes me dream of putting together an Indiegogo campaign for the project so that we can stop using our workstations and finally get some proper 10 gigabit Ethernet hardware. It sure would be nice if the JSON and Plaintext tests weren't network-limited.
riffraff 13y ago
dumb question: are we sure these things are doing the same thing? AFAICT some of the larger frameworks by default do a bunch of stuff (csrf and ip spoof checks, session management, etag generation based on content etc) that simpler solutions don't, but this things can usually be turned off.
- nacs 13y ago
  The source code to all of the benchmarks they do is available on Github. You can look at/submit patches for any improvements that are needed.
- jsmeaton 13y ago
  Exactly the same things? No, of course not. The non-framework code is the same, but the framework specific code (and features/functions) is going to be very very different. A lot of pull requests have been sent that turn off certain features (like unnecessary django middleware). Barebones frameworks of the same language are generally going to out perform heavier frameworks. Feature count/matrixes are not taken into consideration for these benchmarks.
  - riffraff 13y ago
    I'm sorry, I do not understand how this was obvious, I'll see if I can send pull requests. Of course barebones platforms will be faster, but doing unnecessary work is a different thing.
    - jsmeaton 13y ago
      It's more obvious if you read the blog posts linked to each of the rounds (but not this one), since they describe some of the changes that were made to each framework test to bring them closer to parity.
guotie 13y ago
which version of go is used?
- nickpresta 13y ago
  Go 1.2rc3
optymizer1 13y ago
I find the JSON benchmark misleading a bit. I posted this before, but I'll say it again: JSON serialization in Go is slow (2.5x slower than Node.js for example [1]). The web server, however, is very fast. When they measure webserver+json, Go wins because of its webserver, not because it serializes JSON faster. If you want to parse a lot of JSON objects with 1 request (or 1 script), or if you have a large JSON object to parse, Node.js will outperform Go. That said, I rewrote my app in Go and I'm very happy with the performance, stability and testability. The recently announced go 'cover' tool is very useful and a breeze to use. [1] Here are my benchmarks: https://docs.google.com/spreadsheet/ccc?key=0AhlslT1P32MzdGREdGl1X0pHWmU0d2xLcHNjbE9Yc0E&usp=drive_web#gid=0 https://docs.google.com/spreadsheet/ccc?key=0AhlslT1P32MzdGR... (includes codepad.org links to the source for each benchmark)
- bradfitz 13y ago
  I optimized the Go JSON serialization in Go 1.2. See https://code.google.com/p/go/source/detail?r=5a51d54e34bb https://code.google.com/p/go/source/detail?r=5a51d54e34bb ... it went from 30% to 500% faster. It uses much less stack space now, so the hot stack splits are no longer an issue (also Go defaults to 8KB stacks for new goroutines now).
lazyshit 13y ago
I'm curious as to why Finagle has 0's across the board for everything.
Horusiath 13y ago
I'm curious why ServiceStack.net has fall out so badly, since their own benchmarks shows a lot higher performance than ASP.NET web applications.
hit8run 13y ago
I started a conversation in #python on freenode and people were a bit outraged by the way frameworks are compared. Some open Database connections and never close them (example: GO) and others open and close DB connections for every request (example: flask). The guys at techempowered should review every pull request and check if it is implemented in a fair way.
liquidcool 13y ago
Am I the only one shocked to see Grails beat Spring? I mean, I think it's awesome, but part of me wonders if something went awry in the Spring code. I know a last minute (breaking) change kept Grails out of Round 7, so perhaps whatever that was made a big impact.
- ZoFreX 13y ago
  Spring has dropped quite dramatically in most of the tests, I wonder what changed.
  - bhauer 13y ago
    The following was the last notable PR processed for Spring: https://github.com/TechEmpower/FrameworkBenchmarks/pull/606 https://github.com/TechEmpower/FrameworkBenchmarks/pull/606
agnsaft 13y ago
Why is Python doing so much worse than PHP?

174 points by curiousAl 13y ago

I've been following these for most of the rounds, and Go has been improving impressively. Whether that's because of improvements in the language itself or a more zealous crowd sending pull requests, I don't know, but it made me want to try go, so I did. It's not as comforting as the scripting (PHP, Python, JS) languages I'm used to. Having no REPL and having to think about types takes a bit more getting used to than I thought (arrays vs slices/maps, and having no REPL). I find having a quick build script (mine's in vim) so you can compile+run and go back to the code quickly helps a lot. Also, http://play.golang.org/ http://play.golang.org/ isn't too shabby either. It would be fun to see this project (https://github.com/TechEmpower/FrameworkBenchmarks https://github.com/TechEmpower/FrameworkBenchmarks) become more and more popular, with formidable developers squeezing out performance from their framework of choice.

dylandrop 13y ago
Yeah but I'm kind of confused as it's my understanding Go is not a web framework so much as a language. Is this just testing how fast Go can print out the string "{message: 'Hello World'}"? Or are they testing a specific component/library in Go? I mean obviously having a language just spit out a line is going to be faster than having a fully blown framework such as Rails work through all of the query parsing, view building, etc. so it doesn't seem like a very fair or useful comparison.
- dangrossman 13y ago
  It's included, alongside Go frameworks, for the same reason PHP/Ruby/ASP.NET are included -- so that you can see how much overhead the frameworks are adding compared to a minimal implementation in the language they're built on. The code behind every benchmark is available under the source code tab up top. The Go benchmark is using some JSON library, not just printing a string.
- curiousAl 13y ago
  Right, Go is a language. It does have a decent (built in) HTTP library, and is actually pretty fast as a webserver in itself. On the more conventionally robust framework end of things, I suppose Revel/Falcore would be a more appropriate comparison. (both of which have impressive performance of their own).
- mrweasel 13y ago
  The issue is that Go comes with enough included in the standard library that it could be considered a web framework. If you're just build a few rest endpoint or a simply site, you may not need to leave the standard library. Compare that to a language like Python or Ruby, where you need something "extra" to make it easier to do a web application. You could certainly do with just the standard library in other languages, but very few would choose that option, because it would involve write a lot of additional code. I think it fair to include Go, because it's a language/programming environment, that comes with it's one built in web framework. A framework that's actually advanced enough that many don't need to look else where.
- Shamanmuni 13y ago
  Go is a language, but its standard library is very comprehensive for most things Internet related and you can build web applications quite easily using just that. You also have similar examples in the benchmark using nodejs and php, which aren't exactly frameworks. I know there should be some overhead when using a framework, but sometimes the cost is too high and it's useful to know it (compare php and symfony2, for example). You can look at the Revel benchmark, a web framework written in Go which did quite well in the benchmark.

atonse 13y ago

Interesting to see go moving up there. Curious - any reason why you guys don't have ASP.NET tests in Windows with SQL Server? I fiddled with the filters and found none. Update: Never mind. I see it now. You don't have Windows tests on EC2.

deleted 13y ago
[deleted]

nikentic 13y ago

I cannot find Flask in the list. Any specific reason?

krg 13y ago
The submitters didn't create a JSON test, but all the other tests are present. Switch to the Plaintext test to see Flask.
deleted 13y ago
[deleted]
pekk 13y ago
Python has never been a priority in these benchmarks

riquito 13y ago

I'd love to see how many lines of code each test required, but it's probably impossible to do in a fair way edit: I meant in the chart, at a glance

jwcrux 13y ago
I believe the tests are open-sourced on GitHub.
steveklabnik 13y ago
https://github.com/TechEmpower/FrameworkBenchmarks https://github.com/TechEmpower/FrameworkBenchmarks
bhauer 13y ago
In fact, we have some work in progress on that front, along with the number of commits to the test implementation directory. Combined, these will give a rough idea of code length and the amount of community input/review each test has received.
- riquito 13y ago
  Thank you for your work, it's really interesting

mmucklo 13y ago

Regarding symfony2 at the bottom - I submitted a simple pull request to try and fix some issues with the setup, but it's been sitting and sitting there... https://github.com/TechEmpower/FrameworkBenchmarks/pull/650 https://github.com/TechEmpower/FrameworkBenchmarks/pull/650

bhauer 13y ago
Hi mmucklo. We'll get that merged in for Round 9!

shijie 13y ago

This is a fascinating round for WFB, with drastically different results from round 7. I'm impressed with the strides Go has made, and also quite impressed with JRuby. I know the banking app Simple chose it as its language/runtime of choice, and they seem to leverage it well. I'd still like to see a good showing from Django, maybe using uWSGI + Nginx. I might submit a pull request and see if I can't get that included in the next round. Gunicorn is great and incredibly easy to set up, but pales in comparison to other platforms when it comes to raw speed.

ninjay 13y ago
As far as Django goes, there hasn't been much tuning in general[0]. The only thing I see them doing is template caching. At the least they should be running 1.6 with persistent DB connections. Beyond that they have a lot of middleware enabled that isn't being used. [0]https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/django https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast...

saltvedt 13y ago

Whats up with the number of Rails errors?

sker 13y ago

HHVM and Dart seem to be the two new fast performers in town showing impressive performance in some tests. JS has been falling off the charts compared to some of the first rounds, but still a good option performance-wise. C# keeps sucking badly. I miss Nimrod/Jester, I always wanted to see it in the top 10.

pjmlp 13y ago
> C# keeps sucking badly I wonder why, the Fortune 500 sites we have built are handling the load quite well.
- ebbv 13y ago
  Probably because you threw Fortune 500 sized budgets and hardware at them. This is only really relevant to people who need to maximize limited hardware.
  - pjmlp 13y ago
    Might be, the servers are actually quite beefy.
- asdasf 13y ago
  Fortune 500 has little to no relation to demanding website. Plenty of fortune 500 companies aren't even in the top 1000 busiest sites.
  - pjmlp 13y ago
    The sites we have built have pretty high demand, I just mentioned like that because of NDA.
- bigtones 13y ago
  These benchmark tests for C# are run against MySQL or PostgreSQL on Linux. In the Fortune 500 setup you're probably connecting to SQL Server or Oracle in the back end for which Microsoft and DB vendors have optimized OLE-DB drivers. That, and JSON serialization on .Net using default MS serializer is super slow. Everyone uses JSON.NET or another faster serializer in the real world.

ritchiea 13y ago

In the past I've noticed posters on HN picking on Rails by lazily linking to these benchmarks but click over to the average latency tab and Rails looks pretty solid with an average response latency of 1.8 ms, which is not at the very top but far better than Django, which is a comparable framework and is near the bottom of the average latency table. If anything to me this data confirms that Rails is an amazing tool because not only do you get to develop quickly, but you also get pretty good average latency (or at least the potential depending on what you add to your app in terms of 3rd party libraries). And what Rails isn't good at is throughput, which is almost never a problem for an early stage company. Working at a startup it's a huge success if I ever have to handle a lot of connections to my app, but today and everyday, I want fast response times on a page load.

bhauer 13y ago
Fair warning: as far I know, Wrk's latency measurement does not distinguish between 500s and 200s. For some frameworks, you will see unnaturally low latency because the front-end web server is providing a 500 response very quickly.
Segmentation 13y ago
> Working at a startup it's a huge success if I ever have to handle a lot of connections to my app, but today and everyday, I want fast response times on a page load. Realistically, I doubt many humans can distinguish between 1ms and 100-200ms. response times.
- bhauer 13y ago
  True, especially when accounting for Internet latency. However, the purpose of this project is not actually to measure how quickly platforms and frameworks can execute fundamental/trivial operations. Rather, these tasks are a proxy for real-world applications. Across the board, we can reasonably assume that real applications will perform 10x, 50x, 100x, or even slower than these tests. The question is, where does that put your application? If your application runs 100x slower than your platform/framework, does that put your application's response time at 200ms or 2,000ms? That's a difference users do notice.

pfraze 13y ago

Cppsp (top of the i7 charts) is some mad science http://xa.us.to/cppsp/index.cppsp http://xa.us.to/cppsp/index.cppsp

anilmujagic 13y ago

I'm really surprised with ASP.NET/C# results :-S

faraazin 13y ago
its asp.net mono and i am not surprised.
friism 13y ago
Also note that those tests have tons of errors, so they're probably not representative.
sergiotapia 13y ago
It's Mono implementation, not .NET.

RyanZAG 13y ago

Interesting observation regarding the differences between the EC2 and i7 results: the platforms at the top of the EC2 benchmarks are generally MongoDB+async io java, while the ones at the top of the i7 results are MySQL+heavily threaded (go, servlet, openresty). I think it's a pretty interesting result because it shows how much your choice of available hardware has on which platform would be best - and it's not a small difference either. If you're going for an EC2/digital ocean setup with a lot of small instances, then you want to go with something like vert.x or node or whatever - while if you are deploying directly onto bare metal high core/ram servers, you'd be better off with something that is better at handling high thread counts - something like Golang.

piranha 13y ago
Go has low number of threads, usually a number of your cores or close to it. Goroutines are not threads.
- RyanZAG 13y ago
  If you have 8 cores, you'll get 8 threads and all your requests will be nicely distributed across the cores. This is why Go is up near the top for the i7 benchmarks. On the EC2 ones, there are far fewer cores, so the overhead of distributing them is much more pronounced. As you add more cores, you'd probably see Go pull further ahead of some of the competition. However if you're only ever going to be running Go on small instances as many people do, this advantage is actually a hindrance because of the added overhead. Not that it's necessarily a big issue or anything, it's just interesting to consider. The point I was making is that your actual hardware and workload can turn this benchmark on its head. You may naively think you are upgrading performance by switching to a different framework/language, yet if you don't understand why each platform is getting the numbers it does you might end up rewriting your app and actually decreasing performance because of your server hardware.

brickcap 13y ago

Looks like erlang frameworks are not represented...

kainsavage 13y ago
We have been having trouble with Erlang frameworks since before Round 7. Unfortunately, I was still getting up to speed and improving the suite mostly for Round 7/8 and did not get to fix this yet. I do have it topping my todo list for round 9, with the hope being to get them all back in and working soon.

neya 13y ago

I know benchmarks should be taken with a pinch of salt, but by round 5 I was totally into Scala (Scalatra), trying to write my own framework, so I could get better bang for buck from my EC2 instances, which to be honest, aren't cheap when compared to say, Digital Ocean. Around round 6 of these benchmarks, I ditched Scala altogether (and also my framework).The reason I ditched Scala was not because of it's performance, etc. But it was because I was the only developer in my company who knew and learnt Scala after reading a couple of books (one was around 800 pages). Obviously, I needed a language that any other developer should have no problem taking over, and Scala developers are 1)expensive 2)not easy to find. Also, Slick (the database interacting code for Scala by Typesafe) wasn't mature yet. For this reason, around Round 6, I started writing my own framework in GoLang and used it internally as an 'auxiliary framework'. I will explain more about this framework soon soon. In my company, we have about a handful of backend programmers and a couple of frontend devs. I found that GoLang was much much easier to teach my programmers, than say I could teach Scala. Please note - Scala is a brilliant functional programming language, but if you are thinking switching from Ruby/Python/etc would be easy, then you are wrong. Now, we have a workflow that allows us to deliver as quickly as possible, but without missing out on performance - We write our entire V1 in Rails. We implement all the UI/frontend related code and then port it to our GoLang framework. We have an internal generator where we just feed our rails app, and the code for our framework is just 'ported'/generated on the fly based on our framework and we just deploy it. So far, our productivity is slightly lost while handling the type conversions, bugs, etc. But it's totally worth it. Go outperforms Rails by a huge margin. I noticed that using something like Puma helps a lot, but it still is no way comparable to our GoLang framework. As for our framework, it's just pretty simple - Just organize all the files as you would in a Rails application (Models/Views/Controllers/Config) and everything just works without much performance hiccups. We use Gorilla components for stuff like routing and cookies. The rest of the stuff is slightly adapted from other frameworks (like Martini). All in all, I love the ability to have JVM like performance with the productivity of Ruby with a language like GoLang. And this round 8 benchmark is nothing short of impressive. If you haven't tried GoLang yet, you should try writing your own framework, not only do you learn about all the trade-offs for the 'magic' that rails makes under the hood, you also learn about some new stuff and thus become a better programmer. I think GoLang is pretty impressive if someone as average as me can even write a framework like Rails, except for better performance. Give it a try, people, you won't be disappointed.

WoodenChair 13y ago

It's amazing how well a young language like Dart and its frameworks performs in the multi-query benchmarks. There's still so much more optimization to go; at this stage it feels optimistically like the sky is the limit!

desireco42 13y ago

What always impresses me and leaves impression, is just how fast raw PHP is. At times it seems PHP has been obsoleted by new platforms, but benchmarks like these make a case for it's use. Especially because it is really easy for beginners to pick this up.

kvtrew76557 13y ago
The first PHP result for the JSON test comes in at 31.7% of the performance of the top performer. PHP also occupies the bottom 3 worst slots.
- krapp 13y ago
  That's disappointing. One thing PHP should be really good at is serializing/deserializing JSON.

zaroth 13y ago

It's interesting to compare what the code looks like. CPPSP (C++ Server Pages) which is putting up ridiculous numbers... here is the Single Query test: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/cpoll_cppsp/www/db https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... It's quite different from the more typical implementations, where they all sort of look the same... (Go) https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/go/src/hello/hello.go https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... (NodeJS) https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/nodejs/hello.js https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... (Gemini) https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/gemini/Source/hello/home/handler/HelloHandler.java https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... Also interesting to compare it to C# / HttpListener... which would benefit from moving all the framework code out into a separate library; (C#/HTTP.sys) https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/HttpListener/HttpListener/Program.cs https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

stesch 13y ago

I like the benchmark and I appreciate the work that was put into, but Erlang is missing again. If you don't even consider Erlang you won't miss it. But if you know it has some strengths for this kind of job and you don't mind the syntax, you'd like to see it compared to other solutions.

agilebyte 13y ago
I have heard that string operations (and I thus suspect JSON parsing too) is slower on Erlang. Maybe it is not included as it was not built for raw speed but rather stability, hotswapping code etc?
- stesch 13y ago
  Erlang was included in round 6 of the benchmark.
  - agilebyte 13y ago
    Ah thanks, under Cowboy and Elli.
- rdtsc 13y ago
  String equivalents in Erlang are binaries quite often. There are also iolists, those are pretty efficient. This blog post describes this in a lot more depth: http://jlouisramblings.blogspot.com/2009/01/common-erlang-misconceptions.html http://jlouisramblings.blogspot.com/2009/01/common-erlang-mi...
kainsavage 13y ago
I just posted another response to this - we had some trouble with the package manager for Erlang after round 6. Additionally, I had been working on improvements for the suite specifically (better logging/reporting, etc) and did not get a chance to resolve the Erlang problems. Rest assured, "get erlang running again" tops my 'todo' list for round 9.
- 13y ago

mrinterweb 13y ago

I'm rather surprised to see rack-jruby up as high as it was. I discounted ruby as an option for a very high performance http service, but I guess I'd be wrong to do that. Don't get me wrong, I love ruby and I use it every day. I just didn't expect to see it in the top performance contenders list.

bhauer 13y ago
That is principally thanks to TorqBox, the codename for Torquebox 3, which is built on Undertow. Undertow is the web server that is scrapping with Netty and Vert.x on the plaintext tests. Also note that the particular Rack test that performs very well is running a very small amount of Ruby code. Thanks to these improvements, however, rails-jruby now consistently tops rails-ruby, if only by a small amount. See more on TorqBox: http://torquebox.org/news/2013/12/04/torquebox-next-generation/ http://torquebox.org/news/2013/12/04/torquebox-next-generati...

banachtarski 13y ago

None of these numbers are significant! Give me something that tries hundreds if not thousands or tens of thousands of simultaneous requests. Then we have a real benchmark that will probably push a lot of these over the edge in terms of mean latency and especially tail/peak latency.

MetaCosm 13y ago
There have been a group of us -- consistently pushing for exactly this. The maintainers of the benchmark are exceptional resistant to this idea... https://github.com/TechEmpower/FrameworkBenchmarks/issues/49 https://github.com/TechEmpower/FrameworkBenchmarks/issues/49 ... https://github.com/TechEmpower/FrameworkBenchmarks/issues/36 https://github.com/TechEmpower/FrameworkBenchmarks/issues/36 ... https://github.com/TechEmpower/FrameworkBenchmarks/issues/48 https://github.com/TechEmpower/FrameworkBenchmarks/issues/48 ... there are even more issues asking for concurrency increase, just search for concurrency. It it silly that such an rich and awesome set of benchmarks never pushes on concurrency, one of the major points of failure "in the wild" -- more common as you become the go-between for your users and some set of APIs -- users stack up on one side, waiting connections stack up on the other.
- bhauer 13y ago
  There is a very simple reason for this: we do not yet have a test that is designed to include idling. One of the future test types [1], number 12 on the list, is designed to allow the request to idle while waiting on an external service. Until we have such a test type, there is no value in exercising higher concurrency levels. Outside of a few frameworks that have systemic difficulty utilizing all available CPU cores, all tests are fully CPU saturated by the existing tests. With that condition, additional concurrency would only stress-test servers' inbound request queue capacity and cause some with shorter queues to generate 500 responses. Even at our 256 concurrency (maximum for all but the plaintext test), many servers' request queues are tapped out and they cope with this by responding with 500s. The existing tests are all about processing requests as quickly as possible and moving onto the next request. When we have a future test type that by design allows requests to idle for a period of time, higher concurrency levels will be necessary to fully saturate the CPU. Presently, the Plaintext test spans to higher concurrency levels because the workload is utterly trivial and some frameworks are not CPU constrained at 256 concurrency on our i7 hardware. As for the EC2 instances, their much smaller CPU capacity means the higher-concurrency tests are fairly moot. If you switch to the data-table for Plaintext, you can see that the higher concurrency levels are roughly equivalent to 256 concurrency on EC2. For example, jetty-servlet on EC2 m1.large: 256 concurrency: 51,418 1,024 concurrency: 44,615 4,096 concurrency: 49,903 16,384 concurrency: 50,117 The EC2 m1.large virtual CPU cores are saturated at all tested concurrency levels. jetty-servlet on i7: 256 concurrency: 320,543 1,024 concurrency: 396,285 4,096 concurrency: 432,456 16,384 concurrency: 448,947 The i7 CPU cores are not saturated at 256 concurrency, and reach saturation at 16,384 concurrency. We are not against high-concurrency tests; we are just not interested in high-concurrency tests where they would add no value. We're trying to find where the maximum capacity of frameworks is, not how frameworks behave after they reach maximum capacity. We know that they tend to send 500s after they reach maximum capacity. That's not very interesting. All that said, once we have an environment set up that can do continuous running of the tests, I'll be more amenable to a wider variety of test variables (such as higher concurrency for already CPU-saturated test types) because the amount of time to execute a full run will no longer matter as much. [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/133 https://github.com/TechEmpower/FrameworkBenchmarks/issues/13...

bsaul 13y ago

Anybody could explain what gemini is ? I've been on the eclipse project home, and i really don't see the link with a web framework benchmark.

mutagen 13y ago
Gemini is the private Java framework Techempower uses on their client projects. I believe questions regarding its performance relative to various open source and enterprise JVM frameworks inspired the first Techempower benchmarks. http://www.techempower.com/blog/2013/03/28/frameworks-round-1/#section=questions http://www.techempower.com/blog/2013/03/28/frameworks-round-...
abvdasker 13y ago
Second on this question. Extra points if you've done any development with Gemini. I'm really curious: in every category it was the highest-scoring full-stack framework, which is especially surprising given its apparent obscurity. Edit: every category but the plaintext benchmark.
kainsavage 13y ago
Gemini is our internal Java web framework; it has no relation to the Eclipse project (other than I use Eclipse when I work on Gemini) ^_^

matrix 13y ago

These results are tempting me to do my next project in a modern lightweight Java framework. No Hibernate, bloated frameworks of yore, or weird complex build and dependency management. Play is ruled out - it's Scala (Java is a second-class citizen in Play). Maybe something that ties together things like ebean ORM, Jetty, Jersey, Jackson, Guice. Dropwizard is the right idea, but is geared towards building REST backends. Any suggestions on a pure Java framework that has critical mass and would fit the bill?

matrix 13y ago
Following up on my own question, there doesn't appear to be any that quite fit the bill right now, if we define the ideal framework as having the following characteristics: * Java as a first-class citizen * Strong core of basic web app functionality * REST and Search engine friendly URLs * Action oriented – basic framework for routes, MVC etc * Stateless * Good documentation, active community If we look at action frameworks only: * Play 2: Great except it's Scala. Ruled out. * Spring MVC: Spring is bloated old-school Java with Hibernate. Out. * Stripes: hasn’t had a commit in over a year… which is unfortunate because it looks interesting. Out. * Spark: appears to be a one-person project. Out. * Google Sitebricks – ditto * Ninja: Ditto
- stusmall 13y ago
  Why did you find lacking in java on Play? I've just started playing with it and aside from the template engine (which I don't count) I haven't found any part of the java support lacking vs the scala.
  - matrix 13y ago
    In my brief try-out of it, it felt to me like if you want to do anything different (e.g. your own implementation of something), you pretty much need to switch to Scala. It's a Scala framework first and foremost. A minor demerit was getting SBT and Play Java to work correctly in IDEA was enough of a pain to make me wonder how much overhead that was going to incur in the long run. If someone who has used Play Java in production on a large project can weigh in on whether these points are true or not in real production use, I'd love to hear it.

rartichoke 13y ago

Benchmarks are fun but I'll stick with rails and its simple ways of letting you cache data. I'm ok with getting out the door response times of 8-15ms while serving 20,000 unique hits a day on a $5/month VPS. The server does not even break a sweat too and it's doing more than serving the app too.

matthewking 13y ago
What kind of response times do you get on a cache miss though?
- rartichoke 13y ago
  80ms-350ms is normal under typical traffic conditions. It depends on the complexity of the page. That's still not terrible though and it could easily improve by massive amounts with a stronger server. I have not gone crazy with profiling either. Just using fairly basic cache blocks when applicable.

veto64 13y ago

i'm interested in who is financing this benchmarks. really sorry, but for me it looks like a new way of doing seo marketing

bhauer 13y ago
This comment makes me dream of putting together an Indiegogo campaign for the project so that we can stop using our workstations and finally get some proper 10 gigabit Ethernet hardware. It sure would be nice if the JSON and Plaintext tests weren't network-limited.

riffraff 13y ago

dumb question: are we sure these things are doing the same thing? AFAICT some of the larger frameworks by default do a bunch of stuff (csrf and ip spoof checks, session management, etag generation based on content etc) that simpler solutions don't, but this things can usually be turned off.

nacs 13y ago
The source code to all of the benchmarks they do is available on Github. You can look at/submit patches for any improvements that are needed.
jsmeaton 13y ago
Exactly the same things? No, of course not. The non-framework code is the same, but the framework specific code (and features/functions) is going to be very very different. A lot of pull requests have been sent that turn off certain features (like unnecessary django middleware). Barebones frameworks of the same language are generally going to out perform heavier frameworks. Feature count/matrixes are not taken into consideration for these benchmarks.
- riffraff 13y ago
  I'm sorry, I do not understand how this was obvious, I'll see if I can send pull requests. Of course barebones platforms will be faster, but doing unnecessary work is a different thing.
  - jsmeaton 13y ago
    It's more obvious if you read the blog posts linked to each of the rounds (but not this one), since they describe some of the changes that were made to each framework test to bring them closer to parity.

guotie 13y ago

which version of go is used?

nickpresta 13y ago
Go 1.2rc3

optymizer1 13y ago

I find the JSON benchmark misleading a bit. I posted this before, but I'll say it again: JSON serialization in Go is slow (2.5x slower than Node.js for example [1]). The web server, however, is very fast. When they measure webserver+json, Go wins because of its webserver, not because it serializes JSON faster. If you want to parse a lot of JSON objects with 1 request (or 1 script), or if you have a large JSON object to parse, Node.js will outperform Go. That said, I rewrote my app in Go and I'm very happy with the performance, stability and testability. The recently announced go 'cover' tool is very useful and a breeze to use. [1] Here are my benchmarks: https://docs.google.com/spreadsheet/ccc?key=0AhlslT1P32MzdGREdGl1X0pHWmU0d2xLcHNjbE9Yc0E&usp=drive_web#gid=0 https://docs.google.com/spreadsheet/ccc?key=0AhlslT1P32MzdGR... (includes codepad.org links to the source for each benchmark)

bradfitz 13y ago
I optimized the Go JSON serialization in Go 1.2. See https://code.google.com/p/go/source/detail?r=5a51d54e34bb https://code.google.com/p/go/source/detail?r=5a51d54e34bb ... it went from 30% to 500% faster. It uses much less stack space now, so the hot stack splits are no longer an issue (also Go defaults to 8KB stacks for new goroutines now).

lazyshit 13y ago

I'm curious as to why Finagle has 0's across the board for everything.

Horusiath 13y ago

I'm curious why ServiceStack.net has fall out so badly, since their own benchmarks shows a lot higher performance than ASP.NET web applications.

hit8run 13y ago

I started a conversation in #python on freenode and people were a bit outraged by the way frameworks are compared. Some open Database connections and never close them (example: GO) and others open and close DB connections for every request (example: flask). The guys at techempowered should review every pull request and check if it is implemented in a fair way.

liquidcool 13y ago

Am I the only one shocked to see Grails beat Spring? I mean, I think it's awesome, but part of me wonders if something went awry in the Spring code. I know a last minute (breaking) change kept Grails out of Round 7, so perhaps whatever that was made a big impact.

ZoFreX 13y ago
Spring has dropped quite dramatically in most of the tests, I wonder what changed.
- bhauer 13y ago
  The following was the last notable PR processed for Spring: https://github.com/TechEmpower/FrameworkBenchmarks/pull/606 https://github.com/TechEmpower/FrameworkBenchmarks/pull/606

agnsaft 13y ago

Why is Python doing so much worse than PHP?