14 ms·

Web Framework Benchmarks Round 7

227 points by mangeletti 13y ago

ddnfcodx 13y ago
No JSF or plain JavaEE?
- krg 13y ago
  JSF may not be a good fit for these benchmarks, as discussed in this issue on the project: https://github.com/TechEmpower/FrameworkBenchmarks/issues/307 https://github.com/TechEmpower/FrameworkBenchmarks/issues/30...
- fedesilva 13y ago
  There are several servlet containers listed, that is as plain as JEE can get, isn't it?
sker 13y ago
Slightly glad to see ASP.NET improved a little (at least the stripped/raw/barebones version). Slightly sad to see the Mono performance is still abysmal. Edit: link to the blog. It was submitted earlier, but apparently got killed: http://www.techempower.com/blog/2013/10/31/framework-benchmarks-round-7/ http://www.techempower.com/blog/2013/10/31/framework-benchma...
- cmircea 13y ago
  ASP.NET`s performance can be improved further, but that requires application specific tuning. That said, it's pretty damn good, for a full framework. I'm impressed by the Java options. Definitely something to consider if absolute performance is the requirement.
  - platz 13y ago
    I agree - I wonder if the Entity Framework results could be improved by using Compiled Queries.
  - radicalbyte 13y ago
    Performance should improve massively with the use of async..
    - MalcolmEvershed 13y ago
      There were a variety of experiments with async and I don't think there has yet been any data to show that it improved performance of this benchmark. [0] https://github.com/TechEmpower/FrameworkBenchmarks/pull/272#issuecomment-18108704 https://github.com/TechEmpower/FrameworkBenchmarks/pull/272#... [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/315#issuecomment-18938468 https://github.com/TechEmpower/FrameworkBenchmarks/issues/31... [2] https://github.com/TechEmpower/FrameworkBenchmarks/pull/339#issuecomment-19427936 https://github.com/TechEmpower/FrameworkBenchmarks/pull/339#...
- redact207 13y ago
  Benchmarks aside, I'm really impressed with the whole One ASP.NET thing. Seeing the whole MVC with web api and SignalR all now coming together now is making me finally feel excited about developing on the MS stack again.
- bigdubs 13y ago
  The .net benchmarks, the http listener one in particular, uses threadpool tuning that actually hurts performance. In my own testing there was another ~60% performance to gain with better tweaks.
  - MalcolmEvershed 13y ago
    I implemented the threadpool tuning originally first for the ASP.NET tests (https://github.com/TechEmpower/FrameworkBenchmarks/issues/399 https://github.com/TechEmpower/FrameworkBenchmarks/issues/39...) because it improved performance 5-10%. If you've seen data showing that it actually hurts performance, please open an issue or send a pull request (especially for the additional ~60% performance gain). Thanks.
- mwsherman 13y ago
  I’d like to see ASP.net MVC natively on Windows, as well as ServiceStack. I think they’d compete. Mono is just not a thing for those of us in the .Net world, frankly.
  - mhixson 13y ago
    We are running several .NET tests on Windows. Some are using SQL Server as their database, too. See: http://www.techempower.com/benchmarks/#section=data-r7&hw=i7&test=db&s=1 http://www.techempower.com/benchmarks/#section=data-r7&hw=i7...
- omphalos 13y ago
  Maybe there is one an I'm just missing it, but I'd love to see an OWIN-based benchmark in that list.
  - bhauer 13y ago
    I'm not familiar with OWIN. However, a GitHub user named @damianh is working on an OWIN implementation. Perhaps reach out to him and offer to help? :) [1] https://github.com/damianh/FrameworkBenchmarks/blob/nowin/nowin/src/Program.cs https://github.com/damianh/FrameworkBenchmarks/blob/nowin/no...
integraton 13y ago
Summary: if you want performance, use Java, Scala, Go, Clojure, Lua, or C++. Honestly, now with all the great Scala frameworks, Clojure, and the ability to run Rails, plus Cassandra, Storm, etc, I'm a little creeped out that I'm actually strongly considering building my current new project completely on the JVM.
- deleted 13y ago
  [deleted]
  - cgh 13y ago
    Here is the environment info: "As of Round 7, three Intel Sandy Bridge Core i7-2600K workstations with 8 GB memory each (early 2011 vintage) for the i7 tests; database server equipped with Samsung 840 Pro SSD Two Amazon EC2 m1.large instances for the EC2 tests Switched gigabit Ethernet" So 8GB workstations. Even assuming full memory usage, the JVM, Go and C++ frameworks destroyed all comers.
    - rallison 13y ago
      Just for reference, the deleted comment used to say: Raw performance is nothing without memory usage information.
  - TomasEkeli 13y ago
    That said - memory is dirt cheap
    - Thaxll 13y ago
      Not really, on a desktop it is, on Amazon it's very expensive.
      - deleted 13y ago
        [deleted]
      - meepmorp 13y ago
cgh 13y ago
With all the nice alternative frameworks available for the JVM these days, including some pretty micro/bare-bones alternatives, I wonder if Java will find new favour with startups or if they'll stick with Rails. The old "the Java ecosystem is too complex/over-engineered" excuse falls down with stuff like Play or Grizzly. I fully realise the language itself is annoying and clunky and that Scala is not necessarily the answer. I'm just saying that there are some serious upsides to consider beyond language aesthetics.
- ollysb 13y ago
  The quickstart for grizzly[0] contains a staggering amount of boilerplate code. I don't think anyone's going to be running back to the JVM for that. https://grizzly.java.net/quickstart.html https://grizzly.java.net/quickstart.html
  - meepmorp 13y ago
    Ok, but that's not code for an HTTP app, it's a raw socket echo client and server.
    - cgh 13y ago
      Yes, and a lot of it is comments.
- numbsafari 13y ago
  Which is more expensive, servers or developers? Which is more expensive, server or opportunity cost? In a lot of cases, performance of the server isn't the determining variable in the overall cost calculation. In addition, the relationships between those variables likely changes over time. I'm guessing that for most startups, HR expenses and opportunity costs are pretty important relative to actual server performance. Obviously, if the point of your startup is some kind of high-performance computing thing, the equation is tipped more towards server performance. But, you probably aren't using off-the-shelf frameworks in that case, either.
  - 13y ago
nivertech 13y ago
Why Erlang-based frameworks removed in Round 7?
- strmpnk 13y ago
  Looks like they had some issues building with rebar: https://github.com/TechEmpower/FrameworkBenchmarks/issues/495 https://github.com/TechEmpower/FrameworkBenchmarks/issues/49...
- bhauer 13y ago
  We were unable to correct a freezing issue that we ran into with the Rebar package manager. We removed Cowboy and Elli until this can be resolved. See https://github.com/TechEmpower/FrameworkBenchmarks/issues/495 https://github.com/TechEmpower/FrameworkBenchmarks/issues/49... This less forgiving stance with respect to glitches comes from our intent to publish a round of tests every month from this point forward (assuming we have the manpower to do so). In order to pull that off, we need to be less forgiving with problems that we don't have the know-how to fix immediately. We investigated Rebar for a bit but ultimately conceded defeat, skipped the tests, and posted the issue linked above. We'll revisit it again for Round 8.
mayhew 13y ago
Go's numbers are quite impressive considering how young the language is. That combined with the concurrency primitives and ease of deployment makes Go very appealing.
Spiritus 13y ago
I'm impressed by bottle-pypy, and a bit surprised about the gap between it and flask-pypy.
- cvburgess 13y ago
  I use bottle in production and it's fantastic. I'm curious to see what happens to performance when v.12 is released.
rallison 13y ago
For all those interested in hackernews comments from the previous rounds, enjoy: Round 6: https://news.ycombinator.com/item?id=5979766 https://news.ycombinator.com/item?id=5979766 Round 5: https://news.ycombinator.com/item?id=5727012 https://news.ycombinator.com/item?id=5727012 Round 4: https://news.ycombinator.com/item?id=5644880 https://news.ycombinator.com/item?id=5644880 Round 3: https://news.ycombinator.com/item?id=5573532 https://news.ycombinator.com/item?id=5573532 Round 2: https://news.ycombinator.com/item?id=5498869 https://news.ycombinator.com/item?id=5498869 Round 1: https://news.ycombinator.com/item?id=5454775 https://news.ycombinator.com/item?id=5454775 Lots of solid discussion.
mcv 13y ago
I have honestly never heard of any of those fastest frameworks. I thought I knew a lot, but most of these are pretty obscure.
phillmv 13y ago
Do they list the ruby versions they use? Also, what's up with the "errors" column?
- bhauer 13y ago
  Updating the versions in the Environment details became a game a whack-a-mole, so I removed that information from the Environment section for now. Maybe some day I'll come up with a nice way to automatically pull it from the GitHub repo or meta-data about each test. For now, you can find some details in the Rack script [1]: ruby-2.0.0-p0. We'd be happy to receive a PR to update these versions if anyone is interested in crafting one. The errors column is a sum of non-200 HTTP responses and socket connect, read, and write errors. In practice, it tends to be predominantly 500-series HTTP responses, meaning the web server acknowledged the request but was too busy to assign it to a worker process or thread. This can cause the latency of some frameworks to appear artificially low because the 500 responses tend to come back very quickly once they start. Incidentally, if you hover over the error values, you'll get a success rate percentage. [1] https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/rack/setup_ruby.py https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
gesman 13y ago
ASP.NET/C# definitely improving. Rails is disappointing. Slow to be expected for interpreted language framework but that is really sh slow.
- keithwarren 13y ago
  I have a feeling it is less about ASP.NET/C# improving and more likely their tests improving. Baseline perf has not went up much in years.
  - bhauer 13y ago
    That is correct. We have received many contributions from the community (I would call them subject matter experts, but they seem very humble to me, so I'm not sure what they'd call themselves) to improve the C# tests. It looks as if they have helped quite a bit, as I pointed out in the blog entry that accompanies Round 7.
Continuous 13y ago
According to the charts, PHP performs well when there are multiple queries and in the case of Data updates. I wonder of they setup PHP with APC cache (opcode cache) which is basic and easy setup to speed up PHP.
- krg 13y ago
  Yes: http://www.techempower.com/benchmarks/#section=motivation http://www.techempower.com/benchmarks/#section=motivation "Have you enabled APC for the PHP tests?" Yes, the PHP tests run with APC and PHP-FPM on nginx.
buremba 13y ago
It's strange that Vertx has very poor performance when it performs database queries.
- bhauer 13y ago
  I noticed the same, and I suspect this has more to do with the implementation of the Mongo driver it's using than with Vert.x itself. Vert.x is a very high-performance server.
- clwk 13y ago
  I thought so too, so I looked at the source. It's communicating with a MongoDB persistor over the event bus rather than querying the database directly. So the poor performance is because there's an extra layer of communication happening, which is kind of misleading for a benchmark like this.
  - mhixson 13y ago
    Vert.x was in our original batch of tests (round 1). At the time, its documentation showed that using the persistor and event bus was the standard intended mode of communicating with a database. If someone can point us to documentation showing that is no longer how you're supposed to use a database with Vert.x, we'd be happy to accept a pull request that changes the test. Otherwise, I think it's valid as is, and not misleading.
jsnk 13y ago
Can you add Meteor for Round 8?
- rallison 13y ago
  We'd love to have a Meteor test. Any Meteor developers interested in submitting a pull request with a Meteor test? https://github.com/TechEmpower/FrameworkBenchmarks https://github.com/TechEmpower/FrameworkBenchmarks
- bhauer 13y ago
  There is a category of web frameworks that imply specific web application architectures which make them somewhat difficult to shoehorn to our test types. Meteor is one of those, as are JSF and GWT. We'd welcome an implementation that attempts to match Meteor to these test types, but acknowledge that might be awkward.
tjeerdnet 13y ago
In the previous tests Grails was also tested, I don't see it back in this round anymore?
- bhauer 13y ago
  See my reply elsewhere in this thread [1] concerning a new less-forgiving approach we've adopted to glitches/errors. It's not the most polite way to run the project, and I apologize for any oversights and errors, but it's just a necessity since we don't have expertise in everything. With Grails specifically, a very late PR arrived that unfortunately broke the test implementation for us, so we removed it from the Round 7 test. We aim to push new rounds on a monthly cycle, so I hope that anything that is missing in one round won't be missing very long. [1] https://news.ycombinator.com/item?id=6650136 https://news.ycombinator.com/item?id=6650136
  - jacques_chester 13y ago
    Perhaps you could add a second table on each page of "Did Not Complete" entries. Because people just aren't going to read the notes. They'll skip to the results.
    - bhauer 13y ago
      That is a good idea. I'll add that to the GitHub issues.
jakejake 13y ago
I have the dubious honor of the slowest framework on the test on page #1! For some reason the i7 tests perform really badly for Phreeze, whereas the EC2 tests it performs near the top of all the PHP frameworks. I've been too busy with work to look at the tests but my goal is to make a proper showing in round #8!
- bhauer 13y ago
  Thanks, jakejake. Your participation is always appreciated, especially your good spirit. Yes, if you have time, please do help us resolve the issue (and I am guessing it is some configuration issue) for Round 8.
koblas 13y ago
Be very suspect of the results - for instance in python land you have: wsgi: import ujson ... response = {"message": "Hello, World!"} ... tornado: obj = dict(message="Hello, World!") self.write(obj) What you have is the slower way to contruct a dictionary and then passing to the Python native JSON vs. a C optimized JSON for output.
- bhauer 13y ago
  Although this is Round 7, we have no delusions that there is not a lot of room for improvement (particularly in the toolset, although that's a separate matter). If there are tuning tweaks, we'd love to receive a pull request.
- e12e 13y ago
  Just to expand on that (I had to check it for myself) - in my ipython (ageing core2, two cores at around 6k "bogomips"): %timeit d={"message": "Hello, world"} 10000000 loops, best of 3: 103 ns per loop %timeit d=dict(message="Hello, world") 1000000 loops, best of 3: 298 ns per loop sys.version Out[24]: '2.7.3 (default, Jan 2 2013, 13:56:14) \n[GCC 4.7.2]' So, pretty big difference.
  - rallison 13y ago
    Yes, but! At 103ns per iteration, you can do 9.7 million iterations per second. At 298ns per iteration, you can do 3.4 million iterations per second. Let's look at the wsgi numbers for json serialization: wsgi-nginx-uWSGI: 109,882 rps bottle-nginx-uWSGI: 65,793 rps wsgi: 65,755 .. Now, let's look at the best case comparison: 109,882 rps vs 3.4 million iterations per second. Is cutting iteration time down to 1/3 significant (298ns vs 103ns)? Yes. Is it significant in the overall context? No. With a 31x difference between 109,882 and 3.4 million* (and, since this is comparing to an i7, that 31x is probably closer to the ballpark of 100x), this simply isn't likely to be the place where optimization will help much. Put simply, the {"message": "Hello, world"} vs dict(message="Hello, world") cost difference is likely insignificant when compared to the cost of the rest of the request. With that said, this sounds like a change that should be committed as part of a pull request! Why? A couple of reasons: - It may still help the performance in a minor way (at the very least, it shouldn't hurt it) - We prefer idiomatic code and, assuming the literal {} notation - vs dict() - is idiomatic for such a case, this would be a good change. Your point about this is also interesting: What you have is the slower way to contruct a dictionary and then passing to the Python native JSON vs. a C optimized JSON for output. Please do consider a pull request for both. We love community contributions. * Yes, I know these aren't directly comparable numbers, but they compare well enough in this case. Edit: Fixed the literal {} vs dict() mixup that e12e pointed out in my comment.
rhornberger 13y ago
I'd like to see what this test looks like once you add event-machine to the ruby frameworks.
haberman 13y ago
From the "JSON serialization" requirements: > For each request, an object mapping the key message to Hello, World! must be instantiated. Why mandate how the benchmark is implemented? What if some frameworks could be faster by not instantiating an object? For maximum speed, you'd want to stream-process the input into output while materializing as few intermediate objects as possible. Benchmarks should be defined in terms of their observable inputs/outputs. Otherwise it's like defining a car's 0-60 measurement with a rule that says "the car must then take gas from its tank and inject it into the engine." And then a Tesla could only compete by adding a gas engine that it doesn't need or want.
- meepmorp 13y ago
  For JSON serialization, I'd assume it's because they're testing object serialization.
  - haberman 13y ago
    Clients of a system don't know or care if the server actually constructed an object and serialized it. They care that they get the reply they expect, and do not care how it was constructed.
    - meepmorp 13y ago
      Right, but then you're just testing how fast a framework can vomit out a small string and some headers. I'm not saying it's an especially useful benchmark, but it's intended to test something besides raw response speed.
      - haberman 13y ago
        See: https://news.ycombinator.com/item?id=6650499 https://news.ycombinator.com/item?id=6650499
        meepmorp 13y ago
babuskov 13y ago
PHP and node.js neck to neck? I'd really like to see the chart showing RAM usage for these tests.
- nacs 13y ago
  This has puzzled/surprised me over multiple benchmarks. I had assumed that node.js would be faster at pretty much everything than PHP (especially for concurrency). Time and time again though, these benchmarks show PHP outperforming node.js. I'm sure node.js will have lower RAM usage than PHP but as I'm more interested in reqs/s, I'm going to reconsider using node.
  - hpaavola 13y ago
    PHP is just a thin layer on top of C. Of course it will perform decently. PHP frameworks are usually designed in a dumb way and thus perform badly. Usually they load all or most of their files into memory with every request, even though only couple files would be enough.
    - babuskov 13y ago
      I'd really like to see Yii framework included in the test as they claim to only load what's needed.
dubcanada 13y ago
According to this nodejs is barely faster then PHP. Does this have to do with the nodejs http server vs nginx or more to do with the language itself.
- ricardobeat 13y ago
  That's node vs bare PHP, which has limited utility. Performance drops like a rock if you use any kind of framework - see cake, CI, symfony all at the very bottom.
- hpaavola 13y ago
  Plain PHP without any frameworks is just a really thin layer on top of C. So it's not a suprise that it performs well.
HN_Master_Race 13y ago
PHP benchmarks are extremelly outdated and not even using PHP 5.5 with OPCache (that is now native). Python and Ruby implementation choices are also really weird. - Laravel Version 3.2.14 - PHP Version 5.4.13 with FPM and APC Don't waste your time with this crap.
- jmagoon 13y ago
  they seem pretty open to pull requests. so feel free to fix the "crap", as you call it.
- rubyn00bie 13y ago
  Not to be an ass, I don't know much about their approaches and Php (APC in particular)... But wouldn't enabling caching sort of defeat a portion of the test? Wouldn't it just be better to compare caches/caching mechanisms?
  - nercury 13y ago
    While other sensible language interpreters have a feature like byte-code caching built-in in them, PHP for quite a while did not, because someone was monetizing a proprietary cache add-on named ZendOptimizerPlus! They released it as part of the language in 5.5 because alternative APC cache got just good enough and everyone was installing it by default. So, to answer your question: yes, APC should be used on any PHP benchmark if performance is in question.
raveli 13y ago
I was surprised not to see Revel included in the round 7 results. You can filter results by it but nothing is shown. This is significant because, on round 6, Revel was performing better than raw Go in at least some of the test scenarios, which is not something you can say about many frameworks. Was there ever an explanation why Go without any frameworks was performing worse than Revel? Simply because not much effort had been put into optimizing the Go code?
- wheaties 13y ago
  Good question, I would like to know this as well.
- iends 13y ago
  I was also confused that when I switched to EC2 instances I didn't see ANY go.
shijie 13y ago
I'm curious as to why Django was paired with gUnicorn and only gUnicorn for its tests. I would have loved to see it benchmarked with uWSGI, gEvent or something with a little more gusto.
- meepmorp 13y ago
  Probably because the testers aren't experts in all frameworks and languages and deployment options, and don't have infinite time and resources to try all combinations of options to test stuff. Maybe suggest some updates to the people doing the benchmarking. Or, god forbid, do the testing yourself and post some results for people.
  - shijie 13y ago
    Looking through all your comments, I see one vein of similarity between a majority of your words: you're especially, unnecessarily, and consistently impolite. Please, a little internet civility. My question was merely curiosity, and I'm (gasp) well aware that the testers at techempower don't possess infinite time or resources. Your reply was useless and in no way appropriate. Have a bit of respect for yourself.
    - meepmorp 13y ago
      Ok? You wanted to know why they didn't test whatever you thought was the optimal setup for your use case, and the answer should've been obvious - there's dozens of frameworks across multiple languages and OS platforms. If you look around on this thread and those for previous iterations of the benchmarks, a consistent response from the benchmarkers is that they didn't test some specific scenario because they lack the expertise to do so. The tests are on github. Go write one that suits you and send a pull request, or, god forbid, do it yourself and make a post about it on HN and get yourself some karma. But as it stands, your post was lazy and trivial. I've said similarly dumb things in the past, gotten downvoted or called out, and we all moved on. I'm sorry your fee-fees got hurt because you said something dumb, and I pointed it out in terms that failed to indicate respect for you. I upvoted your response as a gesture of love and understanding.
nkg 13y ago
Can someone tell me what is the difference between php and php-raw?
- krg 13y ago
  In the tests that use a database, this distinguishes using an ORM vs. direct SQL queries. From the guide on the filters panel: We classify object-relational mappers as follows: Full, meaning an ORM that provides wide functionality, possibly including a query language. Micro, meaning a less comprehensive abstraction of the relational model. Raw, meaning no ORM is used at all; the platform's raw database connectivity is used.
hiddentaco 13y ago
Is it sad that I all I care about is if Django beats Rails?
- steveklabnik 13y ago
  Yes. Don't define yourself by the tools you use. It creates silly divides that don't matter, and only harm.
- jsmeaton 13y ago
  Honestly, that's what I began looking at also. Django and Rails are compared frequently - even if those comparisons aren't hostile. This is the first time I've seen this benchmark, and it was really really interesting to see how Djangos performance degraded as more queries were executed. They mention that the lack of connection pooling is likely a big factor. I never realised how much that could affect an application. These results are really interesting!
- jbeja 13y ago
  Now that is so 2008!
cies 13y ago
In an era when VMs are bought by the RAM size, this benchmark still does not show RAM usage, or performance at a specific RAM limit. In such a benchmark JVM is allowed to take advantage of its weakness. :)
- cies 13y ago
  One more thing: I love the benchmark! I love to see every new issue...
- bhauer 13y ago
  We do have a fairly long-standing issue to add collection of resource statistics while tests are running [1]. It has not yet been implemented, however. I do want to get that implemented sometime soon because I too would like to see CPU utilization (some frameworks do not successfully saturate all of the cores!), memory utilization, and IO utilization. [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/108 https://github.com/TechEmpower/FrameworkBenchmarks/issues/10...
oscargrouch 13y ago
I would like to see Casablanca in the next rounds as a good C++ contender: http://casablanca.codeplex.com/ http://casablanca.codeplex.com/
- jbeja 13y ago
  I just read the word "microsoft" and instantly kill my tab.
TheMagicHorsey 13y ago
What happened to the Go on EC2 data? It appears to be missing.
- bhauer 13y ago
  With Round 7 we have adopted a less forgiving approach to the test runs. In previous rounds, we would expend a non-trivial amount of effort attempting to ensure all of the tests worked correctly on both i7 and EC2. But when we're busy with other projects, this becomes a real bottleneck. We wanted to get Round 7 out and be able to do future rounds on a monthly basis. In order to pull that off, we simply need to reduce our own bottleneck and ask the community to do more of that work. Round 7 was the first with a preview round and the community submitted a bunch of PRs prior to the final run. To be as clear as possible, I absolutely do not blame the missing Go data on EC2 on the community. It's just the nature of attempting to herd all of these cats, er frameworks. :) I'm fairly sure it's a configuration problem either in the Go test implementation -or- our test-suite. And this part may be obvious, but it bears repeating: except in very rare cases, test absence should not reflect poorly on the framework itself.
  - TheMagicHorsey 13y ago
    Please post a link to where we can go to help contribute to your benchmark work. Thank you.
- deleted 13y ago
  [deleted]
Vektorweg 13y ago
Well, if every language would have a sufficiently smart compiler, the world would be fine.
rubyn00bie 13y ago
I am absolutely shocked that Ruby kicked so much ass-- in the latency department. It's actually a pretty significant margin when you think about how long some of those simpler requests probably took the server to complete and then the latency to return the request. Also, anyone out there have any good real world experience with other scala persistence frameworks? I was surprised to see play-anorm do so "relatively" poor compared to other scala frameworks. Though I'm quite, admittedly, naive in how "big" each of the other frameworks are (only used play and scalatra). Please forgive any misspellings or grammar errors. This was typed on my iphone.
waitingkuo 13y ago
What's wrong with rails data update?
arahaya 13y ago
why no template rendering?
- mhixson 13y ago
  The Fortunes test exercises server-side templates. You can see the requirements for each test in the "Source code & requirements" section: http://www.techempower.com/benchmarks/#section=code http://www.techempower.com/benchmarks/#section=code I think we used to have a brief description of each test in the results section as well, and they somehow got lost in this round. We should add them back; how is a new reader supposed to know what "Fortunes" means?
  - bhauer 13y ago
    The requirement summary is still there for each test, but it has been moved below the results table. I am experimenting with this re-arrangement to see what people think.
  - deleted 13y ago
    [deleted]
stesch 13y ago
Nice work, but a bit useless without Erlang. :-(
hayksaakian 13y ago
One issue I have with these frameworks is the scaling pattern/path With Rails I might do: 1. Fresh rails app 2. Add page caching / CDN 3. Add database/query caching (I have no apps large enough to require anything past here) 4. Sharding the DB? Rails is awesome because each step is concise (via the language, ruby) and many other people have demos / documentation on how to do them. Sure, your framework of choice may be fast at stage 1, but how easy is it to go to 2, 3 and so on when I need to?
- nikcub 13y ago
  Those are all basic optimizations that most make as their apps grow, I doubt there is a single framework in that benchmark that couldn't handle all of them easily.
virtualwhys 13y ago
Just saw Matthius give a talk at Scala.io in Paris on his meisterwerk, Spray. Surprised it came in so high (6), no wonder Spray was acquired by Typesafe...and why Spray will take place of Netty in Play stack.
stormpat 13y ago
Again, a totally useless benchmark. I really feel all these benchmark tests are useless, if they are not in the realm of any real world app. So why not create a crud app mimicing benchmark, test writes and reads, and calculations instead. In the real world it comes down to the ease of development, deployment and maintenance. This is where the traditional web languages still win, and ill want to bet you can ship software faster than ever these days when using a proved web framework. Speed is not the only factor, and usually not even a factor that has to be accounted for, lets be reasonable, 99% of any web apps dont reach the state that they even need to scale to 10000s of user actions per second. Id be interested to see how hhvm/php whould compare though.
- gizzlon 13y ago
  Useless? Not at all, but it's just data. It has to be interpreted, and how you do that depends on your needs. If nothing else, it's interesting from a technical standpoint to see the huge difference it performance. To think that a benchmark like this tells the whole story of the "best" framework is wrong. Of course it is. It's also wrong to think that it tells the whole performance story. Of course it doesn't. But that does not make it useless. Id be interested to see how hhvm/php whould compare though Now you're just arguing yourself :)
- hrjet 13y ago
  Aren't you assuming that the benchmark is targeted at all web-apps? They are just creating a benchmark. If an app you design doesn't need to scale, then you can ignore the benchmark. To the rest who are building apps that need to scale, the benchmarks are definitely useful.
porker 13y ago
Are there enough HN readers interested in contributing new benchmark scenarios for their favourite language? Ideas I had were: - Generating a nested JSON structure weighing in at 30kb (from a database) - Other more-real-word scenarios which I haven't thought up yet ;)
- bhauer 13y ago
  I like your way of thinking. You may want to read and contribute to our "Additional Test Types" issue at GitHub [1]. A test type with a larger JSON payload is on the list. [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/133 https://github.com/TechEmpower/FrameworkBenchmarks/issues/13...
tbarbugli 13y ago
such a waste of time and money, I really dont get it...
- nercury 13y ago
  It is a SEO booster for that company, but don't tell anyone.
  - tbarbugli 13y ago
    I even got punished with bad karma for that :D
kasperbn 13y ago
Caching is always faster than any web framework, so I don't think a faster web framework add much value to most companies. Choose a framework that makes it easy to add caching.
- bhauer 13y ago
  I assume you mean reverse-proxy caching and not back-end caching. Not all use cases are suitable for reverse-proxy caching. Blogs, news sites, and the like are very suitable. Applications that are heavily personalized or work with private data are unsuitable for reverse proxying. Our test cases are explicitly concerned with exercising performance when reverse proxying is not suitable, for whatever reason. If reverse proxying works for your use-case, definitely consider using it.
sfjailbird 13y ago
84 frameworks and no GWT? The original cross-compiling web framework, being used for Adwords, Adsense, Google Play store, Amazon AWS, Google Groups, and lots of other huge deployments? JRebel's web framework comparison this summer had GWT as the front runner, pitted against Spring MVC, Play, Grails, etc.: http://zeroturnaround.com/rebellabs/the-curious-coders-java-web-frameworks-comparison-spring-mvc-grails-vaadin-gwt-wicket-play-struts-and-jsf/ http://zeroturnaround.com/rebellabs/the-curious-coders-java-... Glaring omission for a comprehensive comparison, IMHO.
- axelf 13y ago
  Then send a pull request
- bhauer 13y ago
  See my reply elsewhere in this thread concerning Meteor [1]. Some frameworks are opinionated, for lack of a better term, to a degree that makes it difficult to shoehorn them to provide implementations of our very fundamental/simple test types. If you or anyone is willing to give it a try, however, we would gladly accept a pull request. [1] https://news.ycombinator.com/item?id=6650262 https://news.ycombinator.com/item?id=6650262
espeed 13y ago
For Round 8, it would interesting to see Cognitect's (http://cognitect.com/ http://cognitect.com/) new Pedestal framework (http://pedestal.io/ http://pedestal.io/) for Clojure. Pedestal is Ring compatible but it doesn't use Ring. Instead, Pedestal's new interceptors abstraction decouples http requests from threads. This provides better concurrency support because it enables processing a single request across multiple threads (http://pedestal.io/documentation/service-interceptors/ http://pedestal.io/documentation/service-interceptors/).
- bhauer 13y ago
  I agree. I'd like to see Pedestal added. I've created a new GitHub issue to let contributors know we're looking for an implementation [1]. Incidentally, the Pedestal team gave their fans a free idea that I really liked in August [2]. Do it for fun and glory. [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/583 https://github.com/TechEmpower/FrameworkBenchmarks/issues/58... [2] https://twitter.com/pedestal_team/status/364945233814884352 https://twitter.com/pedestal_team/status/364945233814884352
ExpiredLink 13y ago
> We use the word framework loosely to refer to any HTTP stack—a full-stack framework, a micro-framework, or even a web platform such as Rack, Servlet, or plain PHP. Yep, that's the problem of these benchmarks. Apples and oranges.
- iends 13y ago
  Wait, why is this a problem? Nobody is suggesting you blindly pick the #1 framework on the list without thought. It IS however very interesting to see how things compare on one benchmark. It also provides some empirical evidence that JVM languages are still king if you care primarily about performance.
- bhauer 13y ago
  You can open the filters panel to slice the data however you see fit, including by removing all of the platforms from the view. I have answered this sort of question many times. I eventually wrote a blog entry about it: http://tiamat.tsotech.com/unfair-comparisons http://tiamat.tsotech.com/unfair-comparisons
akrymski 13y ago
So the first framework ever (Java Servlets) is still pretty much the fastest? I wonder what that says about where development is heading. In my previous project I've actually used straight servlets and found them quite RESTful and elegant - HTTP GET calls a function, you run a SQL query, render results using a template. What else does a framework really need? I much prefer that to the beast that is Spring. In fact, that's pretty much what people use tornado+express for these days - map a URL to a function.
rcarmo 13y ago
I'm rather unimpressed by the implementations of the Python benchmarks, and surprised there isn't a single mention of gevent - that alone would have speeded up, say, Bottle by nearly an order of magnitude. We use uWSGI + bottle + gevent for REST APIs and a relatively humble box can reach flood a gig Ethernet link with _useful_ replies. And, of course, any decent implementation will cache computed results, etc. (but toss Varnish in - which we haven't, yet, since there's no need - and you'll outperform anything else for cacheable replies).
neya 13y ago
http://www.techempower.com/benchmarks/#section=data-r7&hw=i7&test=query&f=rpfnk-35t-bxu8 http://www.techempower.com/benchmarks/#section=data-r7&hw=i7... If you notice the specific filters I've set, PHP actually outperforms Go for many cases. That's actually nice to see (PHP gets better with each release version). Well, if there was something like Coffeescript, but for PHP, then there's no better time than now to make it. I really love the simplicity of Ruby, the performance of Golang/JVM and the massive popularity of PHP and it's compatibility with budget hardware (Shared hosting, etc). If there was a "Write your code in Ruby and we'll compile it to PHP" kind of generator/converter/service, that would be so awesome :D (If anyone knows something like that that already exists, then I would be extremely thankful).
- iends 13y ago
  This is interesting, but why is there such a huge gap between raw php, and most of the php frameworks? The one PHP framework that seems to do well is yaf. However, Yaf is actually written in C and is a php extension. I would expect it to do better than all the php frameworks, but that doesn't account for the huge gap between raw php and the other frameworks.
  - plorkyeran 13y ago
    Assuming that the PHP frameworks are not just all terribly written (which seems unlikely), what this shows us is that the PHP interpreter is very slow, but the libraries that PHP uses are fast. The raw PHP benchmark doesn't actually have very much PHP code, so the slowness of the interpreter doesn't matter much, while the PHP frameworks (other than Yaf) all have a great deal of PHP code.
  - neya 13y ago
    Yeah, me neither, I don't understand why the huge gap. Would be interesting to see some technical background on this, though.
  - deleted 13y ago
    [deleted]
- 13y ago
kvinnako 13y ago
tree-frog is also a framework that needs to be included. It is based on c++ and follows rails thinking. So you get both rails way of doing things and c++ performance.
- bhauer 13y ago
  Treefrog is included. It's one of the new frameworks in Round 7. http://www.techempower.com/benchmarks/#section=data-r7&hw=i7&test=json&f=0-0-9zlds http://www.techempower.com/benchmarks/#section=data-r7&hw=i7...

227 points by mangeletti 13y ago

ddnfcodx 13y ago

No JSF or plain JavaEE?

krg 13y ago
JSF may not be a good fit for these benchmarks, as discussed in this issue on the project: https://github.com/TechEmpower/FrameworkBenchmarks/issues/307 https://github.com/TechEmpower/FrameworkBenchmarks/issues/30...
fedesilva 13y ago
There are several servlet containers listed, that is as plain as JEE can get, isn't it?

sker 13y ago

Slightly glad to see ASP.NET improved a little (at least the stripped/raw/barebones version). Slightly sad to see the Mono performance is still abysmal. Edit: link to the blog. It was submitted earlier, but apparently got killed: http://www.techempower.com/blog/2013/10/31/framework-benchmarks-round-7/ http://www.techempower.com/blog/2013/10/31/framework-benchma...

cmircea 13y ago
ASP.NET`s performance can be improved further, but that requires application specific tuning. That said, it's pretty damn good, for a full framework. I'm impressed by the Java options. Definitely something to consider if absolute performance is the requirement.
- platz 13y ago
  I agree - I wonder if the Entity Framework results could be improved by using Compiled Queries.
- radicalbyte 13y ago
  Performance should improve massively with the use of async..
  - MalcolmEvershed 13y ago
    There were a variety of experiments with async and I don't think there has yet been any data to show that it improved performance of this benchmark. [0] https://github.com/TechEmpower/FrameworkBenchmarks/pull/272#issuecomment-18108704 https://github.com/TechEmpower/FrameworkBenchmarks/pull/272#... [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/315#issuecomment-18938468 https://github.com/TechEmpower/FrameworkBenchmarks/issues/31... [2] https://github.com/TechEmpower/FrameworkBenchmarks/pull/339#issuecomment-19427936 https://github.com/TechEmpower/FrameworkBenchmarks/pull/339#...
redact207 13y ago
Benchmarks aside, I'm really impressed with the whole One ASP.NET thing. Seeing the whole MVC with web api and SignalR all now coming together now is making me finally feel excited about developing on the MS stack again.
bigdubs 13y ago
The .net benchmarks, the http listener one in particular, uses threadpool tuning that actually hurts performance. In my own testing there was another ~60% performance to gain with better tweaks.
- MalcolmEvershed 13y ago
  I implemented the threadpool tuning originally first for the ASP.NET tests (https://github.com/TechEmpower/FrameworkBenchmarks/issues/399 https://github.com/TechEmpower/FrameworkBenchmarks/issues/39...) because it improved performance 5-10%. If you've seen data showing that it actually hurts performance, please open an issue or send a pull request (especially for the additional ~60% performance gain). Thanks.
mwsherman 13y ago
I’d like to see ASP.net MVC natively on Windows, as well as ServiceStack. I think they’d compete. Mono is just not a thing for those of us in the .Net world, frankly.
- mhixson 13y ago
  We are running several .NET tests on Windows. Some are using SQL Server as their database, too. See: http://www.techempower.com/benchmarks/#section=data-r7&hw=i7&test=db&s=1 http://www.techempower.com/benchmarks/#section=data-r7&hw=i7...
omphalos 13y ago
Maybe there is one an I'm just missing it, but I'd love to see an OWIN-based benchmark in that list.
- bhauer 13y ago
  I'm not familiar with OWIN. However, a GitHub user named @damianh is working on an OWIN implementation. Perhaps reach out to him and offer to help? :) [1] https://github.com/damianh/FrameworkBenchmarks/blob/nowin/nowin/src/Program.cs https://github.com/damianh/FrameworkBenchmarks/blob/nowin/no...

integraton 13y ago

Summary: if you want performance, use Java, Scala, Go, Clojure, Lua, or C++. Honestly, now with all the great Scala frameworks, Clojure, and the ability to run Rails, plus Cassandra, Storm, etc, I'm a little creeped out that I'm actually strongly considering building my current new project completely on the JVM.

deleted 13y ago
[deleted]
- cgh 13y ago
  Here is the environment info: "As of Round 7, three Intel Sandy Bridge Core i7-2600K workstations with 8 GB memory each (early 2011 vintage) for the i7 tests; database server equipped with Samsung 840 Pro SSD Two Amazon EC2 m1.large instances for the EC2 tests Switched gigabit Ethernet" So 8GB workstations. Even assuming full memory usage, the JVM, Go and C++ frameworks destroyed all comers.
  - rallison 13y ago
    Just for reference, the deleted comment used to say: Raw performance is nothing without memory usage information.
- TomasEkeli 13y ago
  That said - memory is dirt cheap
  - Thaxll 13y ago
    Not really, on a desktop it is, on Amazon it's very expensive.
    - deleted 13y ago
      [deleted]
    - meepmorp 13y ago

cgh 13y ago

With all the nice alternative frameworks available for the JVM these days, including some pretty micro/bare-bones alternatives, I wonder if Java will find new favour with startups or if they'll stick with Rails. The old "the Java ecosystem is too complex/over-engineered" excuse falls down with stuff like Play or Grizzly. I fully realise the language itself is annoying and clunky and that Scala is not necessarily the answer. I'm just saying that there are some serious upsides to consider beyond language aesthetics.

ollysb 13y ago
The quickstart for grizzly[0] contains a staggering amount of boilerplate code. I don't think anyone's going to be running back to the JVM for that. https://grizzly.java.net/quickstart.html https://grizzly.java.net/quickstart.html
- meepmorp 13y ago
  Ok, but that's not code for an HTTP app, it's a raw socket echo client and server.
  - cgh 13y ago
    Yes, and a lot of it is comments.
numbsafari 13y ago
Which is more expensive, servers or developers? Which is more expensive, server or opportunity cost? In a lot of cases, performance of the server isn't the determining variable in the overall cost calculation. In addition, the relationships between those variables likely changes over time. I'm guessing that for most startups, HR expenses and opportunity costs are pretty important relative to actual server performance. Obviously, if the point of your startup is some kind of high-performance computing thing, the equation is tipped more towards server performance. But, you probably aren't using off-the-shelf frameworks in that case, either.
- 13y ago

nivertech 13y ago

Why Erlang-based frameworks removed in Round 7?

strmpnk 13y ago
Looks like they had some issues building with rebar: https://github.com/TechEmpower/FrameworkBenchmarks/issues/495 https://github.com/TechEmpower/FrameworkBenchmarks/issues/49...
bhauer 13y ago
We were unable to correct a freezing issue that we ran into with the Rebar package manager. We removed Cowboy and Elli until this can be resolved. See https://github.com/TechEmpower/FrameworkBenchmarks/issues/495 https://github.com/TechEmpower/FrameworkBenchmarks/issues/49... This less forgiving stance with respect to glitches comes from our intent to publish a round of tests every month from this point forward (assuming we have the manpower to do so). In order to pull that off, we need to be less forgiving with problems that we don't have the know-how to fix immediately. We investigated Rebar for a bit but ultimately conceded defeat, skipped the tests, and posted the issue linked above. We'll revisit it again for Round 8.

mayhew 13y ago

Go's numbers are quite impressive considering how young the language is. That combined with the concurrency primitives and ease of deployment makes Go very appealing.

Spiritus 13y ago

I'm impressed by bottle-pypy, and a bit surprised about the gap between it and flask-pypy.

cvburgess 13y ago
I use bottle in production and it's fantastic. I'm curious to see what happens to performance when v.12 is released.

rallison 13y ago

For all those interested in hackernews comments from the previous rounds, enjoy: Round 6: https://news.ycombinator.com/item?id=5979766 https://news.ycombinator.com/item?id=5979766 Round 5: https://news.ycombinator.com/item?id=5727012 https://news.ycombinator.com/item?id=5727012 Round 4: https://news.ycombinator.com/item?id=5644880 https://news.ycombinator.com/item?id=5644880 Round 3: https://news.ycombinator.com/item?id=5573532 https://news.ycombinator.com/item?id=5573532 Round 2: https://news.ycombinator.com/item?id=5498869 https://news.ycombinator.com/item?id=5498869 Round 1: https://news.ycombinator.com/item?id=5454775 https://news.ycombinator.com/item?id=5454775 Lots of solid discussion.

mcv 13y ago

I have honestly never heard of any of those fastest frameworks. I thought I knew a lot, but most of these are pretty obscure.

phillmv 13y ago

Do they list the ruby versions they use? Also, what's up with the "errors" column?

bhauer 13y ago
Updating the versions in the Environment details became a game a whack-a-mole, so I removed that information from the Environment section for now. Maybe some day I'll come up with a nice way to automatically pull it from the GitHub repo or meta-data about each test. For now, you can find some details in the Rack script [1]: ruby-2.0.0-p0. We'd be happy to receive a PR to update these versions if anyone is interested in crafting one. The errors column is a sum of non-200 HTTP responses and socket connect, read, and write errors. In practice, it tends to be predominantly 500-series HTTP responses, meaning the web server acknowledged the request but was too busy to assign it to a worker process or thread. This can cause the latency of some frameworks to appear artificially low because the 500 responses tend to come back very quickly once they start. Incidentally, if you hover over the error values, you'll get a success rate percentage. [1] https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/rack/setup_ruby.py https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

gesman 13y ago

ASP.NET/C# definitely improving. Rails is disappointing. Slow to be expected for interpreted language framework but that is really sh slow.

keithwarren 13y ago
I have a feeling it is less about ASP.NET/C# improving and more likely their tests improving. Baseline perf has not went up much in years.
- bhauer 13y ago
  That is correct. We have received many contributions from the community (I would call them subject matter experts, but they seem very humble to me, so I'm not sure what they'd call themselves) to improve the C# tests. It looks as if they have helped quite a bit, as I pointed out in the blog entry that accompanies Round 7.

Continuous 13y ago

According to the charts, PHP performs well when there are multiple queries and in the case of Data updates. I wonder of they setup PHP with APC cache (opcode cache) which is basic and easy setup to speed up PHP.

krg 13y ago
Yes: http://www.techempower.com/benchmarks/#section=motivation http://www.techempower.com/benchmarks/#section=motivation "Have you enabled APC for the PHP tests?" Yes, the PHP tests run with APC and PHP-FPM on nginx.

buremba 13y ago

It's strange that Vertx has very poor performance when it performs database queries.

bhauer 13y ago
I noticed the same, and I suspect this has more to do with the implementation of the Mongo driver it's using than with Vert.x itself. Vert.x is a very high-performance server.
clwk 13y ago
I thought so too, so I looked at the source. It's communicating with a MongoDB persistor over the event bus rather than querying the database directly. So the poor performance is because there's an extra layer of communication happening, which is kind of misleading for a benchmark like this.
- mhixson 13y ago
  Vert.x was in our original batch of tests (round 1). At the time, its documentation showed that using the persistor and event bus was the standard intended mode of communicating with a database. If someone can point us to documentation showing that is no longer how you're supposed to use a database with Vert.x, we'd be happy to accept a pull request that changes the test. Otherwise, I think it's valid as is, and not misleading.

jsnk 13y ago

Can you add Meteor for Round 8?

rallison 13y ago
We'd love to have a Meteor test. Any Meteor developers interested in submitting a pull request with a Meteor test? https://github.com/TechEmpower/FrameworkBenchmarks https://github.com/TechEmpower/FrameworkBenchmarks
bhauer 13y ago
There is a category of web frameworks that imply specific web application architectures which make them somewhat difficult to shoehorn to our test types. Meteor is one of those, as are JSF and GWT. We'd welcome an implementation that attempts to match Meteor to these test types, but acknowledge that might be awkward.

tjeerdnet 13y ago

In the previous tests Grails was also tested, I don't see it back in this round anymore?

bhauer 13y ago
See my reply elsewhere in this thread [1] concerning a new less-forgiving approach we've adopted to glitches/errors. It's not the most polite way to run the project, and I apologize for any oversights and errors, but it's just a necessity since we don't have expertise in everything. With Grails specifically, a very late PR arrived that unfortunately broke the test implementation for us, so we removed it from the Round 7 test. We aim to push new rounds on a monthly cycle, so I hope that anything that is missing in one round won't be missing very long. [1] https://news.ycombinator.com/item?id=6650136 https://news.ycombinator.com/item?id=6650136
- jacques_chester 13y ago
  Perhaps you could add a second table on each page of "Did Not Complete" entries. Because people just aren't going to read the notes. They'll skip to the results.
  - bhauer 13y ago
    That is a good idea. I'll add that to the GitHub issues.

jakejake 13y ago

I have the dubious honor of the slowest framework on the test on page #1! For some reason the i7 tests perform really badly for Phreeze, whereas the EC2 tests it performs near the top of all the PHP frameworks. I've been too busy with work to look at the tests but my goal is to make a proper showing in round #8!

bhauer 13y ago
Thanks, jakejake. Your participation is always appreciated, especially your good spirit. Yes, if you have time, please do help us resolve the issue (and I am guessing it is some configuration issue) for Round 8.

koblas 13y ago

Be very suspect of the results - for instance in python land you have: wsgi: import ujson ... response = {"message": "Hello, World!"} ... tornado: obj = dict(message="Hello, World!") self.write(obj) What you have is the slower way to contruct a dictionary and then passing to the Python native JSON vs. a C optimized JSON for output.

bhauer 13y ago
Although this is Round 7, we have no delusions that there is not a lot of room for improvement (particularly in the toolset, although that's a separate matter). If there are tuning tweaks, we'd love to receive a pull request.
e12e 13y ago
Just to expand on that (I had to check it for myself) - in my ipython (ageing core2, two cores at around 6k "bogomips"): %timeit d={"message": "Hello, world"} 10000000 loops, best of 3: 103 ns per loop %timeit d=dict(message="Hello, world") 1000000 loops, best of 3: 298 ns per loop sys.version Out[24]: '2.7.3 (default, Jan 2 2013, 13:56:14) \n[GCC 4.7.2]' So, pretty big difference.
- rallison 13y ago
  Yes, but! At 103ns per iteration, you can do 9.7 million iterations per second. At 298ns per iteration, you can do 3.4 million iterations per second. Let's look at the wsgi numbers for json serialization: wsgi-nginx-uWSGI: 109,882 rps bottle-nginx-uWSGI: 65,793 rps wsgi: 65,755 .. Now, let's look at the best case comparison: 109,882 rps vs 3.4 million iterations per second. Is cutting iteration time down to 1/3 significant (298ns vs 103ns)? Yes. Is it significant in the overall context? No. With a 31x difference between 109,882 and 3.4 million* (and, since this is comparing to an i7, that 31x is probably closer to the ballpark of 100x), this simply isn't likely to be the place where optimization will help much. Put simply, the {"message": "Hello, world"} vs dict(message="Hello, world") cost difference is likely insignificant when compared to the cost of the rest of the request. With that said, this sounds like a change that should be committed as part of a pull request! Why? A couple of reasons: - It may still help the performance in a minor way (at the very least, it shouldn't hurt it) - We prefer idiomatic code and, assuming the literal {} notation - vs dict() - is idiomatic for such a case, this would be a good change. Your point about this is also interesting: What you have is the slower way to contruct a dictionary and then passing to the Python native JSON vs. a C optimized JSON for output. Please do consider a pull request for both. We love community contributions. * Yes, I know these aren't directly comparable numbers, but they compare well enough in this case. Edit: Fixed the literal {} vs dict() mixup that e12e pointed out in my comment.

rhornberger 13y ago

I'd like to see what this test looks like once you add event-machine to the ruby frameworks.

haberman 13y ago

From the "JSON serialization" requirements: > For each request, an object mapping the key message to Hello, World! must be instantiated. Why mandate how the benchmark is implemented? What if some frameworks could be faster by not instantiating an object? For maximum speed, you'd want to stream-process the input into output while materializing as few intermediate objects as possible. Benchmarks should be defined in terms of their observable inputs/outputs. Otherwise it's like defining a car's 0-60 measurement with a rule that says "the car must then take gas from its tank and inject it into the engine." And then a Tesla could only compete by adding a gas engine that it doesn't need or want.

meepmorp 13y ago
For JSON serialization, I'd assume it's because they're testing object serialization.
- haberman 13y ago
  Clients of a system don't know or care if the server actually constructed an object and serialized it. They care that they get the reply they expect, and do not care how it was constructed.
  - meepmorp 13y ago
    Right, but then you're just testing how fast a framework can vomit out a small string and some headers. I'm not saying it's an especially useful benchmark, but it's intended to test something besides raw response speed.
    - haberman 13y ago
      See: https://news.ycombinator.com/item?id=6650499 https://news.ycombinator.com/item?id=6650499
      - meepmorp 13y ago

babuskov 13y ago

PHP and node.js neck to neck? I'd really like to see the chart showing RAM usage for these tests.

nacs 13y ago
This has puzzled/surprised me over multiple benchmarks. I had assumed that node.js would be faster at pretty much everything than PHP (especially for concurrency). Time and time again though, these benchmarks show PHP outperforming node.js. I'm sure node.js will have lower RAM usage than PHP but as I'm more interested in reqs/s, I'm going to reconsider using node.
- hpaavola 13y ago
  PHP is just a thin layer on top of C. Of course it will perform decently. PHP frameworks are usually designed in a dumb way and thus perform badly. Usually they load all or most of their files into memory with every request, even though only couple files would be enough.
  - babuskov 13y ago
    I'd really like to see Yii framework included in the test as they claim to only load what's needed.

dubcanada 13y ago

According to this nodejs is barely faster then PHP. Does this have to do with the nodejs http server vs nginx or more to do with the language itself.

ricardobeat 13y ago
That's node vs bare PHP, which has limited utility. Performance drops like a rock if you use any kind of framework - see cake, CI, symfony all at the very bottom.
hpaavola 13y ago
Plain PHP without any frameworks is just a really thin layer on top of C. So it's not a suprise that it performs well.

HN_Master_Race 13y ago

PHP benchmarks are extremelly outdated and not even using PHP 5.5 with OPCache (that is now native). Python and Ruby implementation choices are also really weird. - Laravel Version 3.2.14 - PHP Version 5.4.13 with FPM and APC Don't waste your time with this crap.

jmagoon 13y ago
they seem pretty open to pull requests. so feel free to fix the "crap", as you call it.
rubyn00bie 13y ago
Not to be an ass, I don't know much about their approaches and Php (APC in particular)... But wouldn't enabling caching sort of defeat a portion of the test? Wouldn't it just be better to compare caches/caching mechanisms?
- nercury 13y ago
  While other sensible language interpreters have a feature like byte-code caching built-in in them, PHP for quite a while did not, because someone was monetizing a proprietary cache add-on named ZendOptimizerPlus! They released it as part of the language in 5.5 because alternative APC cache got just good enough and everyone was installing it by default. So, to answer your question: yes, APC should be used on any PHP benchmark if performance is in question.

raveli 13y ago

I was surprised not to see Revel included in the round 7 results. You can filter results by it but nothing is shown. This is significant because, on round 6, Revel was performing better than raw Go in at least some of the test scenarios, which is not something you can say about many frameworks. Was there ever an explanation why Go without any frameworks was performing worse than Revel? Simply because not much effort had been put into optimizing the Go code?

wheaties 13y ago
Good question, I would like to know this as well.
iends 13y ago
I was also confused that when I switched to EC2 instances I didn't see ANY go.

shijie 13y ago

I'm curious as to why Django was paired with gUnicorn and only gUnicorn for its tests. I would have loved to see it benchmarked with uWSGI, gEvent or something with a little more gusto.

meepmorp 13y ago
Probably because the testers aren't experts in all frameworks and languages and deployment options, and don't have infinite time and resources to try all combinations of options to test stuff. Maybe suggest some updates to the people doing the benchmarking. Or, god forbid, do the testing yourself and post some results for people.
- shijie 13y ago
  Looking through all your comments, I see one vein of similarity between a majority of your words: you're especially, unnecessarily, and consistently impolite. Please, a little internet civility. My question was merely curiosity, and I'm (gasp) well aware that the testers at techempower don't possess infinite time or resources. Your reply was useless and in no way appropriate. Have a bit of respect for yourself.
  - meepmorp 13y ago
    Ok? You wanted to know why they didn't test whatever you thought was the optimal setup for your use case, and the answer should've been obvious - there's dozens of frameworks across multiple languages and OS platforms. If you look around on this thread and those for previous iterations of the benchmarks, a consistent response from the benchmarkers is that they didn't test some specific scenario because they lack the expertise to do so. The tests are on github. Go write one that suits you and send a pull request, or, god forbid, do it yourself and make a post about it on HN and get yourself some karma. But as it stands, your post was lazy and trivial. I've said similarly dumb things in the past, gotten downvoted or called out, and we all moved on. I'm sorry your fee-fees got hurt because you said something dumb, and I pointed it out in terms that failed to indicate respect for you. I upvoted your response as a gesture of love and understanding.

nkg 13y ago

Can someone tell me what is the difference between php and php-raw?

krg 13y ago
In the tests that use a database, this distinguishes using an ORM vs. direct SQL queries. From the guide on the filters panel: We classify object-relational mappers as follows: Full, meaning an ORM that provides wide functionality, possibly including a query language. Micro, meaning a less comprehensive abstraction of the relational model. Raw, meaning no ORM is used at all; the platform's raw database connectivity is used.

hiddentaco 13y ago

Is it sad that I all I care about is if Django beats Rails?

steveklabnik 13y ago
Yes. Don't define yourself by the tools you use. It creates silly divides that don't matter, and only harm.
jsmeaton 13y ago
Honestly, that's what I began looking at also. Django and Rails are compared frequently - even if those comparisons aren't hostile. This is the first time I've seen this benchmark, and it was really really interesting to see how Djangos performance degraded as more queries were executed. They mention that the lack of connection pooling is likely a big factor. I never realised how much that could affect an application. These results are really interesting!
jbeja 13y ago
Now that is so 2008!

cies 13y ago

In an era when VMs are bought by the RAM size, this benchmark still does not show RAM usage, or performance at a specific RAM limit. In such a benchmark JVM is allowed to take advantage of its weakness. :)

cies 13y ago
One more thing: I love the benchmark! I love to see every new issue...
bhauer 13y ago
We do have a fairly long-standing issue to add collection of resource statistics while tests are running [1]. It has not yet been implemented, however. I do want to get that implemented sometime soon because I too would like to see CPU utilization (some frameworks do not successfully saturate all of the cores!), memory utilization, and IO utilization. [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/108 https://github.com/TechEmpower/FrameworkBenchmarks/issues/10...

oscargrouch 13y ago

I would like to see Casablanca in the next rounds as a good C++ contender: http://casablanca.codeplex.com/ http://casablanca.codeplex.com/

jbeja 13y ago
I just read the word "microsoft" and instantly kill my tab.

TheMagicHorsey 13y ago

What happened to the Go on EC2 data? It appears to be missing.

bhauer 13y ago
With Round 7 we have adopted a less forgiving approach to the test runs. In previous rounds, we would expend a non-trivial amount of effort attempting to ensure all of the tests worked correctly on both i7 and EC2. But when we're busy with other projects, this becomes a real bottleneck. We wanted to get Round 7 out and be able to do future rounds on a monthly basis. In order to pull that off, we simply need to reduce our own bottleneck and ask the community to do more of that work. Round 7 was the first with a preview round and the community submitted a bunch of PRs prior to the final run. To be as clear as possible, I absolutely do not blame the missing Go data on EC2 on the community. It's just the nature of attempting to herd all of these cats, er frameworks. :) I'm fairly sure it's a configuration problem either in the Go test implementation -or- our test-suite. And this part may be obvious, but it bears repeating: except in very rare cases, test absence should not reflect poorly on the framework itself.
- TheMagicHorsey 13y ago
  Please post a link to where we can go to help contribute to your benchmark work. Thank you.
deleted 13y ago
[deleted]

Vektorweg 13y ago

Well, if every language would have a sufficiently smart compiler, the world would be fine.

rubyn00bie 13y ago

I am absolutely shocked that Ruby kicked so much ass-- in the latency department. It's actually a pretty significant margin when you think about how long some of those simpler requests probably took the server to complete and then the latency to return the request. Also, anyone out there have any good real world experience with other scala persistence frameworks? I was surprised to see play-anorm do so "relatively" poor compared to other scala frameworks. Though I'm quite, admittedly, naive in how "big" each of the other frameworks are (only used play and scalatra). Please forgive any misspellings or grammar errors. This was typed on my iphone.

waitingkuo 13y ago

What's wrong with rails data update?

arahaya 13y ago

why no template rendering?

mhixson 13y ago
The Fortunes test exercises server-side templates. You can see the requirements for each test in the "Source code & requirements" section: http://www.techempower.com/benchmarks/#section=code http://www.techempower.com/benchmarks/#section=code I think we used to have a brief description of each test in the results section as well, and they somehow got lost in this round. We should add them back; how is a new reader supposed to know what "Fortunes" means?
- bhauer 13y ago
  The requirement summary is still there for each test, but it has been moved below the results table. I am experimenting with this re-arrangement to see what people think.
- deleted 13y ago
  [deleted]

stesch 13y ago

Nice work, but a bit useless without Erlang. :-(

hayksaakian 13y ago

One issue I have with these frameworks is the scaling pattern/path With Rails I might do: 1. Fresh rails app 2. Add page caching / CDN 3. Add database/query caching (I have no apps large enough to require anything past here) 4. Sharding the DB? Rails is awesome because each step is concise (via the language, ruby) and many other people have demos / documentation on how to do them. Sure, your framework of choice may be fast at stage 1, but how easy is it to go to 2, 3 and so on when I need to?

nikcub 13y ago
Those are all basic optimizations that most make as their apps grow, I doubt there is a single framework in that benchmark that couldn't handle all of them easily.

virtualwhys 13y ago

Just saw Matthius give a talk at Scala.io in Paris on his meisterwerk, Spray. Surprised it came in so high (6), no wonder Spray was acquired by Typesafe...and why Spray will take place of Netty in Play stack.

stormpat 13y ago

Again, a totally useless benchmark. I really feel all these benchmark tests are useless, if they are not in the realm of any real world app. So why not create a crud app mimicing benchmark, test writes and reads, and calculations instead. In the real world it comes down to the ease of development, deployment and maintenance. This is where the traditional web languages still win, and ill want to bet you can ship software faster than ever these days when using a proved web framework. Speed is not the only factor, and usually not even a factor that has to be accounted for, lets be reasonable, 99% of any web apps dont reach the state that they even need to scale to 10000s of user actions per second. Id be interested to see how hhvm/php whould compare though.

gizzlon 13y ago
Useless? Not at all, but it's just data. It has to be interpreted, and how you do that depends on your needs. If nothing else, it's interesting from a technical standpoint to see the huge difference it performance. To think that a benchmark like this tells the whole story of the "best" framework is wrong. Of course it is. It's also wrong to think that it tells the whole performance story. Of course it doesn't. But that does not make it useless. Id be interested to see how hhvm/php whould compare though Now you're just arguing yourself :)
hrjet 13y ago
Aren't you assuming that the benchmark is targeted at all web-apps? They are just creating a benchmark. If an app you design doesn't need to scale, then you can ignore the benchmark. To the rest who are building apps that need to scale, the benchmarks are definitely useful.

porker 13y ago

Are there enough HN readers interested in contributing new benchmark scenarios for their favourite language? Ideas I had were: - Generating a nested JSON structure weighing in at 30kb (from a database) - Other more-real-word scenarios which I haven't thought up yet ;)

bhauer 13y ago
I like your way of thinking. You may want to read and contribute to our "Additional Test Types" issue at GitHub [1]. A test type with a larger JSON payload is on the list. [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/133 https://github.com/TechEmpower/FrameworkBenchmarks/issues/13...

tbarbugli 13y ago

such a waste of time and money, I really dont get it...

nercury 13y ago
It is a SEO booster for that company, but don't tell anyone.
- tbarbugli 13y ago
  I even got punished with bad karma for that :D

kasperbn 13y ago

Caching is always faster than any web framework, so I don't think a faster web framework add much value to most companies. Choose a framework that makes it easy to add caching.

bhauer 13y ago
I assume you mean reverse-proxy caching and not back-end caching. Not all use cases are suitable for reverse-proxy caching. Blogs, news sites, and the like are very suitable. Applications that are heavily personalized or work with private data are unsuitable for reverse proxying. Our test cases are explicitly concerned with exercising performance when reverse proxying is not suitable, for whatever reason. If reverse proxying works for your use-case, definitely consider using it.

sfjailbird 13y ago

84 frameworks and no GWT? The original cross-compiling web framework, being used for Adwords, Adsense, Google Play store, Amazon AWS, Google Groups, and lots of other huge deployments? JRebel's web framework comparison this summer had GWT as the front runner, pitted against Spring MVC, Play, Grails, etc.: http://zeroturnaround.com/rebellabs/the-curious-coders-java-web-frameworks-comparison-spring-mvc-grails-vaadin-gwt-wicket-play-struts-and-jsf/ http://zeroturnaround.com/rebellabs/the-curious-coders-java-... Glaring omission for a comprehensive comparison, IMHO.

axelf 13y ago
Then send a pull request
bhauer 13y ago
See my reply elsewhere in this thread concerning Meteor [1]. Some frameworks are opinionated, for lack of a better term, to a degree that makes it difficult to shoehorn them to provide implementations of our very fundamental/simple test types. If you or anyone is willing to give it a try, however, we would gladly accept a pull request. [1] https://news.ycombinator.com/item?id=6650262 https://news.ycombinator.com/item?id=6650262

espeed 13y ago

For Round 8, it would interesting to see Cognitect's (http://cognitect.com/ http://cognitect.com/) new Pedestal framework (http://pedestal.io/ http://pedestal.io/) for Clojure. Pedestal is Ring compatible but it doesn't use Ring. Instead, Pedestal's new interceptors abstraction decouples http requests from threads. This provides better concurrency support because it enables processing a single request across multiple threads (http://pedestal.io/documentation/service-interceptors/ http://pedestal.io/documentation/service-interceptors/).

bhauer 13y ago
I agree. I'd like to see Pedestal added. I've created a new GitHub issue to let contributors know we're looking for an implementation [1]. Incidentally, the Pedestal team gave their fans a free idea that I really liked in August [2]. Do it for fun and glory. [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/583 https://github.com/TechEmpower/FrameworkBenchmarks/issues/58... [2] https://twitter.com/pedestal_team/status/364945233814884352 https://twitter.com/pedestal_team/status/364945233814884352

ExpiredLink 13y ago

> We use the word framework loosely to refer to any HTTP stack—a full-stack framework, a micro-framework, or even a web platform such as Rack, Servlet, or plain PHP. Yep, that's the problem of these benchmarks. Apples and oranges.

iends 13y ago
Wait, why is this a problem? Nobody is suggesting you blindly pick the #1 framework on the list without thought. It IS however very interesting to see how things compare on one benchmark. It also provides some empirical evidence that JVM languages are still king if you care primarily about performance.
bhauer 13y ago
You can open the filters panel to slice the data however you see fit, including by removing all of the platforms from the view. I have answered this sort of question many times. I eventually wrote a blog entry about it: http://tiamat.tsotech.com/unfair-comparisons http://tiamat.tsotech.com/unfair-comparisons

akrymski 13y ago

So the first framework ever (Java Servlets) is still pretty much the fastest? I wonder what that says about where development is heading. In my previous project I've actually used straight servlets and found them quite RESTful and elegant - HTTP GET calls a function, you run a SQL query, render results using a template. What else does a framework really need? I much prefer that to the beast that is Spring. In fact, that's pretty much what people use tornado+express for these days - map a URL to a function.

rcarmo 13y ago

I'm rather unimpressed by the implementations of the Python benchmarks, and surprised there isn't a single mention of gevent - that alone would have speeded up, say, Bottle by nearly an order of magnitude. We use uWSGI + bottle + gevent for REST APIs and a relatively humble box can reach flood a gig Ethernet link with _useful_ replies. And, of course, any decent implementation will cache computed results, etc. (but toss Varnish in - which we haven't, yet, since there's no need - and you'll outperform anything else for cacheable replies).

neya 13y ago

http://www.techempower.com/benchmarks/#section=data-r7&hw=i7&test=query&f=rpfnk-35t-bxu8 http://www.techempower.com/benchmarks/#section=data-r7&hw=i7... If you notice the specific filters I've set, PHP actually outperforms Go for many cases. That's actually nice to see (PHP gets better with each release version). Well, if there was something like Coffeescript, but for PHP, then there's no better time than now to make it. I really love the simplicity of Ruby, the performance of Golang/JVM and the massive popularity of PHP and it's compatibility with budget hardware (Shared hosting, etc). If there was a "Write your code in Ruby and we'll compile it to PHP" kind of generator/converter/service, that would be so awesome :D (If anyone knows something like that that already exists, then I would be extremely thankful).

iends 13y ago
This is interesting, but why is there such a huge gap between raw php, and most of the php frameworks? The one PHP framework that seems to do well is yaf. However, Yaf is actually written in C and is a php extension. I would expect it to do better than all the php frameworks, but that doesn't account for the huge gap between raw php and the other frameworks.
- plorkyeran 13y ago
  Assuming that the PHP frameworks are not just all terribly written (which seems unlikely), what this shows us is that the PHP interpreter is very slow, but the libraries that PHP uses are fast. The raw PHP benchmark doesn't actually have very much PHP code, so the slowness of the interpreter doesn't matter much, while the PHP frameworks (other than Yaf) all have a great deal of PHP code.
- neya 13y ago
  Yeah, me neither, I don't understand why the huge gap. Would be interesting to see some technical background on this, though.
- deleted 13y ago
  [deleted]
13y ago

kvinnako 13y ago

tree-frog is also a framework that needs to be included. It is based on c++ and follows rails thinking. So you get both rails way of doing things and c++ performance.

bhauer 13y ago
Treefrog is included. It's one of the new frameworks in Round 7. http://www.techempower.com/benchmarks/#section=data-r7&hw=i7&test=json&f=0-0-9zlds http://www.techempower.com/benchmarks/#section=data-r7&hw=i7...