9 ms·

Frameworks Round 3

92 points by amarsahinovic 14y ago

zix 14y ago
Honest benchmarks. Open and gracious group of developers behind it. Disclaimer - I'm married to one of these guys. I'm not a programmer, but I can see how hard this group has worked to provide something useful to the community and I've enjoyed watching it grow.
- pekk 14y ago
  I'm sure they're great guys, but I found it interesting how many obvious suggested changes were not incorporated at all. What's honest about it?
  - bhauer 14y ago
    Pekk, we'd love to address any problems you can point us to. Obviously we have limited time to work on this, so the best way to get a fix in is via a pull request. Are there any issues in particular you think are especially high-priority?
  - wereHamster 14y ago
    Open a pull request.
  - gphreak 14y ago
    They have been extremely responsive to feedback so far and always carefully explained why they are not incorporating everything the community suggested.
- chromatic 14y ago
  Honest benchmarks. "Select random number from MySQL"? I'd like to see the stochastic control for that one.
  - bhauer 14y ago
    We have considered alternatives [1] but randomly selecting from 10,000 rows was selected because it's easy to implement in a threadsafe manner without introducing even more complicated requirements such as a concurrent queue or atomic counter. Note that after the warm-up, MySQL is essentially guaranteed to have the entire table in its own cache, so the exercise is testing the ORM (if applicable), database connection pool, and the database driver. Honestly, assuming none of the implementations are using an expensive random number generator (e.g., a random number generator intended for cryptographic use), I don't believe there would be different results if it were feasible to sequentially retrieve rows without the need to contend with concurrency. [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/111 https://github.com/TechEmpower/FrameworkBenchmarks/issues/11...
bhauer 14y ago
Thanks amarsahinovic for sharing this here. This is a brief blog entry with some observations about Round 3 of our web frameworks benchmark project. The community contributions continue unabated and I think the impressively long charts demonstrate just how much the community has given us. Thank you to everyone who has participated and we are looking forward to Round 4 already! Please reply to let us know if you have any comments, questions, or criticisms. We'd love to hear your feedback.
- kbd 14y ago
  Would still like Ruby to be tested with app servers other than Passenger. Is this planned for Round 4?
  - bhauer 14y ago
    Oops! Thanks for pointing that out. This is an oversight in the readme for the Ruby tests. They are on Unicorn for Round 3 [1]. We'll get the readme updated and the Environment Details clarified as well. [1] https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/rails/setup_ruby.py https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... Edit: I've edited the Environment Details page accordingly.
    - kbd 14y ago
      This is the readme I looked at before I commented: https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/rack https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast... It still references Passenger.
- orclev 14y ago
  Really glad to see the performance of Yesod has moved up closer to where I'd expect now that some of the mistakes in the original benchmark have been fixed. One thing I do notice is that Yesod is using MySQL as the DB backend for the DB tests, but it does have support for MongoDB as well. Looked at from that perspective it's actually one of the higher performing frameworks for all the DB tasks, but it's hard to say for sure because it's never tested with MongoDB (it usually lands somewhere above all the MySQL based tests, but below all the MongoDB tests). Would be interested in seeing how it performs with the MongoDB backend used instead. Maybe if I get some time in the next couple weeks I'll try to swap out the backends and make a pull request.
zalew 14y ago
what's "django stripped"?
- cmsimike 14y ago
  A stripped down version of Django - no unnecessary apps installed. This was recommended to test Django's full speed but really wouldn't be reflective of a full deployment.
  - mixmastamyk 14y ago
    It's disappointing that it's so slow. I suppose it doesn't matter a lot of the time, but would prefer it to be more competitive.
    - amarsahinovic 14y ago
      In the previous benchmark it was pointed out that they don't use connection pooling [1], I don't know is it fixed in this round https://news.ycombinator.com/item?id=5499490 https://news.ycombinator.com/item?id=5499490
      - andybak 14y ago
        They don't say either way but it requires them to switch to Postgres so I think they would have said if they had.
        bhauer 14y ago
        That's correct; we are still testing on MySQL, so the Django tests are still configured without a connection pool. We would love a pull request that incorporates a connection pool for Django+MySQL. We do have Postgres as a target for future rounds of this project, but do not yet have an ETA for that.
        andybak 14y ago
baby 14y ago
I just don't understand how could Symfony2 be so fast considering it's pretty huge (twig itself is already huge). I guess it's time I consider switching from CI.
- nakovet 14y ago
  It's a matter of cache, APC cache, memcached, "compiled" autoload cache, there are many levels of caching in symfony, and boom, you have speed because many of your hits are hitting the ram memory.
- josh-j 14y ago
  I think you've got the charts backwards. Symfony2 is slower than CI in every chart.
  - baby 14y ago
    oh...
- camus 14y ago
  Symfony caches everything , dumping resources ( configs , templates , ... ) into PHP files. So when you are using Twig on a production server you are not actually reading twig files but basic PHP files with echo statements.
deadA1ias 14y ago
Why is ASP.NET never included in these type of benchmarks?
- Sodaware 14y ago
  They have someone working on ASP.net tests for the next round.
- mixmastamyk 14y ago
  Windows is clumsy on the network unless you've built up a suite of tools and know-how to manage it. The extra cost doesn't help much either. As someone is said to be working on .net benchmarks, you may get your wish.
  - WayneDB 14y ago
    Oh gawd. Please, tell us how Windows is "clumsy" to get setup on a network. Also, the "extra cost" doesn't even figure into a benchmark. Grab a 120 day trial ISO from Microsoft and install.
    - mixmastamyk 14y ago
      Simple really. Its heritage is single-user desktops. Therefore the security model is often "Don't look in here!" rather than having it done right the first time. Hides filenames; needs antivirus. Industry standard network/admin/deployment tools don't work as well, if supported. Until a couple of years ago, the only viable way to manage it was from a GUI... they are still figuring out how to run it headless. It's not always compatible with open standards. Shall I continue? Meanwhile Unix (and others) began life as time-sharing systems that became the original nodes of the internet, a scalable model that Windows has come back to forty years later. That's not to say that it doesn't have any strengths or isn't improving. The extra cost includes per minute charges on EC2.
      - WayneDB 14y ago
        That's great except the only thing you've proven here is your own ignorance. Shall I go on? You're spouting off about ancient history...and that somehow translates to Windows is (currently) "clumsy" on the network? Gee, I wonder why it only takes me 10 minutes flat to setup a headless ASP.Net server on EC2? And I can do that even with the GUI version of Windows because there's this little thing called RDP - maybe you've heard of it. Oh and ASP.Net also runs on Linux. It didn't a few years ago though, so maybe it won't work for you since apparently you are living in the past.
bdesimone 14y ago
Something seems off about "Results from dedicated hardware." Play and Go went from 1000s to <10 respectively. Care to give any insight there?
- bhauer 14y ago
  I assume by "Play" you mean Play1, which is a newly community-contributed test. Since the completion of the Round 3 tests, the Play community resolved the Play1 database test's problems [1] and the Go community is investigating the Go database tests [2]. Assuming the Go fix is in soon, we anticipate that both should be resolved in Round 4. [1] https://groups.google.com/forum/?fromgroups=#!topic/play-framework/HxgvitzGO50 https://groups.google.com/forum/?fromgroups=#!topic/play-fra... [2] https://groups.google.com/forum/?fromgroups=#!topic/golang-nuts/-YbB2Qjg41g https://groups.google.com/forum/?fromgroups=#!topic/golang-n...
  - bdesimone 14y ago
    Thank you for the response, and for the project in general.
error54 14y ago
Just curious but why are the major php frameworks so abysmally slow? It seems like php-raw does pretty well in all the tests but most of the big frameworks (cake, symfony, etc) are orders of magnitude slower.
- umsm 14y ago
  Wow, I was floored at how slow Symfony is... I have a feeling it's ORM that's killing these frameworks in the DB portion.
  - beberlei 14y ago
    it is used without caching, leading to unnecessary parsing of a metadata structure. This is explicitly not recommended for production use and kills the request time here.
  - Skamander 13y ago
    Hi, I'm the user who contributed symfony 2 to the benchmark. My intention was to get as much php frameworks in the next round as possible (Symfony 2, Codeigniter, Laravel etc.), so I didn't pay much attention on the possible optimisations of the frameworks. On top of that I'm not a php guy, so it's possible that every php framework I contributed to the benchmark runs on breaks. ;o) In Symfony's case I posted on the mailing list in order to get users with symfony experience to improve the performance, but this post was sadly not activated/moderated.
- 1880 14y ago
  A PHP framework has to initialize all its systems from zero for each request. It may because of that, it may not, it's just a guess.
  - Joeri 14y ago
    You would think the frameworks would be designed around that, loading the absolute minimum of code to handle the request.
jcoby 14y ago
A few things really stand out for me: 1) I am shocked how slow EC2 is and how expensive. A m1.large is $0.240/hr or ~$175/month. And it's 7-10x slower than a $350/mo dedicated box. You would be spending $1500/mo to equal one dedicated box (not including bandwidth and S3 fees). A reserved instance is cheaper of course. 2) The multiple queries test would seem to be the best one to really simulate real-world usage. JSON-serialization is mostly testing the language. This test pretty much puts most of the interpreted languages on a full stack framework together at the bottom. Although django seems to come out at the bottom of the pack for some reason. Then come the "raw" tests and JIT languages running full stack frameworks. And at the top are the compiled languages. Not really surprising there. 3) I'm surprised php-raw did so well. And that go did so poorly.
- pdog 14y ago
  It's more accurate to compare dedicated boxes to reserved instances.
  - cedrichurst 14y ago
    Reserved instances are simply a pricing construct. You're paying upfront for up to 24/7 usage, but you're running on the same pool as the standard instance. Dedicated instances, on the other hand, do what you're talking about but they're much pricier: http://aws.amazon.com/dedicated-instances/ http://aws.amazon.com/dedicated-instances/
    - Terretta 14y ago
      The comment you replied to didn't talk about what they do. I took it to mean it's more accurate from a price perspective.
- xt 14y ago
  A little too generalized, LuaJIT is winning some of the benchmarks
- orclev 14y ago
  The main selling point for EC2 isn't that it's cheap, it's that it's easy and fast to scale up/down and that you get managed failover. For that ease you pay a hefty premium. A lot of companies that have highly variable demand will often go with EC2 because if they get a sudden spike in demand they can quickly spin up 3 or 4 new instances to load balance across, and then when the demand dies off they can spin them all back down to save money. Doing something similar with dedicated hardware doesn't really work, you either end up without enough hardware to meet demand, or else you've got a bunch of hardware sitting around idle most of the time.
nakovet 14y ago
Where can I find this gemini? I went to environment and it's the only one that doesn't have a link.
- krg 14y ago
  It's our internal framework. From our first benchmarks post: http://www.techempower.com/blog/2013/03/28/framework-benchmarks/ http://www.techempower.com/blog/2013/03/28/framework-benchma... "Why include this Gemini framework I've never heard of?" We have included our in-house Java web framework, Gemini, in our tests. We've done so because it's of interest to us. You can consider it a stand-in for any relatively lightweight minimal-locking Java framework. While we're proud of how it performs among the well-established field, this exercise is not about Gemini. We routinely use other frameworks on client projects and we want this data to inform our recommendations for new projects.
  - continuations 14y ago
    Any plan to open source it? Would love to learn more about it.
    - bhauer 14y ago
      We are considering it. Although, fair warning: it is not documented (save for our internal documentation) and we don't have the capacity to support its usage by third-parties. The reason we are considering open sourcing it anyway is that we feel it's the fair thing to do since we've included it in this project.
      - continuations 14y ago
        Please do. I'd very much like to learn about it.
  - deleted 14y ago
    [deleted]
g3rald 14y ago
Wow I'm surprised that Rails has obtained better performance than PHP frameworks, and on many occasions has proved better than DJango...
- g3rald 14y ago
  Consider that we are talking about web oriented programming languages and frameworks.
benhanks040888 14y ago
Is the laravel on the benchmark Laravel 3 or Laravel 4? Still, CodeIgniter is the fastest PHP framework? Wow.
- bhauer 14y ago
  The Larevel test was community-contributed, but according to its readme [1], this is 3.2.14. Is Laravel 4 production-ready? [1] https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/php-laravel https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast...
  - benhanks040888 14y ago
    I figured so too. Laravel 4 is still on beta as of now.
g3rald 14y ago
I want to try Gemini. Can you give me some tips to start?
dbecker 14y ago
Not being a web programmer, I have a naive question: Are many websites running into speed or scalability issues? I see programmers in other areas spend a lot of man-hours optimizing code that isn't a meaningful bottleneck. Is the same thing happening here?
- baby 14y ago
  If you don't have intelligent queries and a proper cache system. At least. You will quickly crash your database or saturate your traffic. It can even hurt small traffic websites (I remember coding a webchat with Mysql and crashing my database with only 5 people chatting).
josh-j 14y ago
Is the following a fair generalization? Java should be the conservative choice for your web framework's language (rather than PHP). Scala, Clojure, Nodejs, Erlang, Lua, and Haskell should be in your list of workable yet "cool" languages (rather than Ruby or Python).
- jerf 14y ago
  I'd add the caveat that Java should be the conservative choice only if you are expecting it to matter that you can serve 200,000 simpleish (but nontrivial) requests per second. In the vast majority of cases, development speed and things other than raw framework performance will dominate as concerns. However, that is a non-zero niche.
  - josh-j 14y ago
    I was considering other factors in that generalization, although they're implicit. E.g., a lot of the "cool" language frameworks I mentioned are comparable with the Java frameworks in speed, but Java has a considerably larger ecosystem and number of hirable developers (although I'm not sure if that translates well into web development). And I assume PHP is as verbose as Java from what I've heard from other people (I've never used PHP). What language should be the conservative choice based on the many factors and ignoring the specific case for the general case?
    - camus 14y ago
      " And I assume PHP is as verbose as Java from what I've heard from other people (I've never used PHP)". So you dont even know what you are talking about?
      - josh-j 14y ago
        What do you think questions are for?
    - jtreminio 14y ago
      PHP can be as verbose as you want it to be. If you're using something like Symfony2, there'll be more abstractions and layers than using something like Silex - but they still don't come close to what Java looks like.
shaydoc 14y ago
Do you have any plans to run asp.net MVC webapi on .net 4.5 using the task parallel library ?
juiceandjuice 14y ago
That's some amazing performance out of the JVM.
- kokoloko 14y ago
  Pretty amazing. I would like to see a raw C benchmark to see how far is the JVM from the limits.
  - continuations 14y ago
    Onion is C
  - Poyeyo 14y ago
    I want to see a Wt (witty) benchmark. C++ is welcome here too.
    - krg 14y ago
      Wt was suggested before, and we'd love to include it. Care to submit a pull request? https://github.com/TechEmpower/FrameworkBenchmarks/issues/92 https://github.com/TechEmpower/FrameworkBenchmarks/issues/92
pilgrim689 14y ago
I'm having a hard time making conclusions based off of this data, except that Python/Ruby/JavaScript are slow, JVM is fast, and the rest are in between... It's odd how Haskell (Yesod) and C (Onion) are both slower than the JVM frameworks. Is it just a maturity issue? I know Yesod is very new relative to some of these JVM frameworks. Also, minor typo: the link to Erlang from the Environment tab is not the right URL. You probably meant to reference http://www.erlang.org/ http://www.erlang.org/, not http://www.erlang.com/ http://www.erlang.com/
- bhauer 14y ago
  Thanks pilgrim689. I'll get that fixed up.
- mdasen 14y ago
  Because the benchmarks are quite narrow, sometimes you get something that has simply been optimized or isn't doing the same work across platforms. For example, the last time I checked their php-raw code, it was executing a database query and appending the raw results to an array. Then, it took that array and passed it to json_encode which I'm pretty sure is implemented in C (PHP's built-in functions are in C, yes?). Anyway, comparing that to something like Rails' ActiveRecord, the work is quite different. With Rails, it's going to go through and call the accessor methods on the objects to get the data because we often override the default data stored in the database for the representation we want on an object level. So, they aren't doing the same thing. Rails at least used to have a way to get back an array of hashes rather than getting the AR objects, but I haven't been able to find it recently. That could be an interesting comparison. So, in that case, it just isn't the same functionality. Similarly, it's hard to know exactly how they're setting things up. Django will perform significantly better if you set up something like pgBouncer for connection pooling. Rails includes connection pooling, IIRC. Without knowing something like that, it's hard to gauge whether it reflects real-world usage (I mean, I'd argue that defaults matter, but if you're looking to scale Django, this isn't an onerous addition). Similarly, with process-based concurrency, it's important to get settings that maximize CPU usage while not overloading the RAM. Since they don't talk about memory or CPU usage, it's hard to figure out whether they've dealt with this. It might be in their repository, but I haven't been able to go through it yet. Heck, the multiple-query test in go seems really odd. I mean, coming in last place, serving less than half the requests of Rails? Similarly, Play 1 hits a nice 28% on EC2 and then can't serve a single request on the dedicated hardware. Plus, the tests really stress JSON performance and don't use things like a framework's templating system. One doesn't need to argue that JSON performance is useful, but so is template performance and that just isn't being tested. So, yeah, it would be hard to make conclusions off of this data except possibly that benchmarking is hard. None of this comment is meant as a dig against the people doing these benchmarks. They're improving them, they're making their code open, they're doing the kind of stuff that allows good, open questioning on their results. But I think it's still early to consider these to be really meaningful. I think a test that combined a bunch of different types of requests with database access, JSON, templates, etc. would be interesting and it seems like they might go there as they have more time.
camus 14y ago
It's a bit strange to include node with mongodb in the tests. Because it changes 2 variables in the test ( the web solution and the database ). You should stick to mysql(or whatever DB you are using) with every tests or it doesnt sound that serious.
xb95 14y ago
I am not acquainted with what language every framework uses. It would be useful for me in interpreting the data if you could annotate -- Java/JVM, PHP, Python, Perl, etc. This is great work, though. Thank you for doing it -- and double thank you for taking submissions from each of the communities so they can try to submit their best shot. That really helps make sure the tests are as ideal as possible.
- bhauer 14y ago
  xb95, I agree. In the next round, I'll add more color-coded columns to indicate language and platform information.
mronge 14y ago
I'd love to see these benchmarks on a beefier ec2 instance type. I'm curious how it would compare to the dedicated hardware
- bhauer 14y ago
  Agreed. A future enhancement I plan to make on the results viewer is the capacity to select from a menu of results files (the results are rendered from results.json files). Once that is in place, anyone who cares to could run the full suite on EC2 extra-large or, say, Rackspace Cloud, and submit those results files for us to include as options.
stcredzero 14y ago
I'd be more interested in benchmarks that reflect production configurations.
- n1c 14y ago
  From what I can tell the guys are trying quite hard to get close. If you spotted things that seem off you should contribute to the repo or let them know.
  - bhauer 14y ago
    Indeed, a key objective of the project is to simulate a production environment, within reason. We are now at a point where about half of the tests are community-contributed and we have to take it for granted that the community has similarly aimed for production-readiness within their tests. As n1c points out, if there is something wrong with any of the configuration, please submit a Github issue or pull request.
bimr 14y ago
In order to have an apples to apples comparison, wouldn't you at least need to have an opcode cache for the PHP frameworks? PHP has to live and die upon each request, but Java culls all of its classes together on the first request and from then on it is running from RAM. Many PHP deployments, and most serious ones, will opcode cache in addition to database and page cache. You could rerun these tests with opcode cache only, leaving page and database caches for your "cache" round of tests. That will at least approximate the dramatic differences in architecture between Java and PHP.
- krg 14y ago
  All the PHP tests were done with the APC opcode cache, running within PHP-FPM. http://www.techempower.com/benchmarks/#section=environment http://www.techempower.com/benchmarks/#section=environment If this can be improved, please let us know. A pull request would be ideal.
  - bimr 14y ago
    Oh. Didn't make the connection that APC was on for Codeigniter, Laravel and the other frameworks.
whalesalad 14y ago
A few things that I noticed, just random notes, take 'em with a grain of salt. I'm not running any of the code myself, I'm just browsing around the Github repo. This is a birds-eye analysis. I was a Django developer for a long time and recently went to the dark side and love Rails at the moment. It's important to remember that database connections are a killer in most web apps. This is why a lot of people use connection pooling (in the postgresql world, for example, you have pgbouncer and pgpool). When you can pool and persist connections, you'll have a faster web app (assuming it interfaces with a db, as many of their tests do). Last time I checked, Django does NOT provide built-in connection pooling, whereas Rails does. This could be part of the reason that Rails outperforms in most of the tests. That says something ... but it would be cool to see both frameworks strutting their stuff on an identical connection scenario. Also note that the MySQL connection limit for nodejs is set to 256 connections, whereas Rails set to use a pool of 5. Since these tests are hitting the DB, I'd like to see some more consistency with the way connections are setup. Not that 256 connections is better than 5, but if all those connections are open, it's no wonder that an async framework can hammer the DB so much faster than Ruby chugging along on a pool of 5. In the Node examples, it's important to note the difference between the mysql and mysql-raw tests. And to remember this when comparing to things like Django and Rails which use ORM's like ActiveRecord. In the raw msql test, node is really quick because there is no overhead there for mapping a row to a model. Also, a lot of the viewers of these benchmarks might just look at the big bar for X framework and think, "fuck it, that's my next project". Sure, go for it. But also remember that the type of app you're building comes into play a lot. These tests do not have any state. That's why these bare metal frameworks like Gemini are doing so well. Node does great here because it's async. But certain apps (like your future project) isn't going to fit into the mold of an async hello world test. I'm not knocking the framework tests here, they're awesome, but just realize this. One area where Django outperforms Rails is in the realm of JSON serialization. I'm asusming here that Python's native JSON engine is quicker than Ruby's. I learned this with Rails the hard way. With an out-of-the-box Gemfile, you're gonan have a bad time. In Ruby-land there are a few ways to boost your octane here. You start with MultiJSON and then take your pick of the many JSON engines in the world today. MultiJSON being an adaptor that lets developers put the choice of a JSON engine in your hands. Popular ones are yajl-ruby (ruby bindings for yajl, which is C) and oj. If you're doing JRuby, I hear that Jackson is super effing fast so that's nice. I'd love to see the JSON tests with various JSON parsers (at some point these guys are going to get funding for being the benchmark company, having every framework with every possible configuration on display) Django Protip: read up on select_related and prefetch_related. Similarly, in Rails land, you'll want to learn about eager loading with things like includes() and joins(). This will let you do one or few queries up front, rather than doing a bunch of subqueries inside of a big loop. One big query is usually better than a billion tiny ones. A reason why the PHP apps are getting murdered is because in a typical configuration your application initialized on every single request. Opcode caching can help here since that reduces that boot period. This why in a production environment you can modify a PHP file and instantly see the results. Versus (depending on the application) a Ruby or Python app, where the app is booted initially and until that worker or process is restarted, is running the code it started out with. This is for speed. This is also why, for example, a big Rails project (or better yet, a Java project) has a significantly longer warm-up period. It's loading the entire app. This is a big reason the PHP frameworks are suffering here, since they have a lot of overhead to load for each request. Whereas the raw PHP doesn't have that overhead. PHP protip: Roll with opcode caching (APC is a favorite of mine, it's very simple to install and configure, and provides a handy web gui for viewing various details of things like cache hit so you can optimize it). Also, please throw apache and mod_php away if you can and run with Nginx + php_fpm ... although if you're an HN reader then you're probably already doing this. The JVM is balls-quick because, again, unlike something like PHP, it gets booted up and then does it's thing. The JVM is infamous for it's slow startup process (which really varies based on a billion things) but there is a reason for that: the JIT. The slow startup is due to bytecode generation, I believe. It churns that bytecode into machine code and during that process it can make optimizations like unrolling loops. I'm no Java pro though. I don't like working with the language (the verbosity kills me, and XML is painful), but I have an immense deal of respect for its performance. To be honest I am really going to take a solid look at the Play framework for my next side project. It's got a lot of the pragmatism of Rails but it runs on the JVM and comes in a Scala variant. These are things that I am stoked about dipping my toes into. Finally, my hat is off to the folks running these. Great work!
actf 14y ago
These are really great - one additional test I would love to see would be Mono Asp.net MVC. I think it would interesting to see how it compares to the JVM.
AlexSerov51 13y ago
It would be useful to try the HybridJava framework as well.

92 points by amarsahinovic 14y ago

zix 14y ago

Honest benchmarks. Open and gracious group of developers behind it. Disclaimer - I'm married to one of these guys. I'm not a programmer, but I can see how hard this group has worked to provide something useful to the community and I've enjoyed watching it grow.

pekk 14y ago
I'm sure they're great guys, but I found it interesting how many obvious suggested changes were not incorporated at all. What's honest about it?
- bhauer 14y ago
  Pekk, we'd love to address any problems you can point us to. Obviously we have limited time to work on this, so the best way to get a fix in is via a pull request. Are there any issues in particular you think are especially high-priority?
- wereHamster 14y ago
  Open a pull request.
- gphreak 14y ago
  They have been extremely responsive to feedback so far and always carefully explained why they are not incorporating everything the community suggested.
chromatic 14y ago
Honest benchmarks. "Select random number from MySQL"? I'd like to see the stochastic control for that one.
- bhauer 14y ago
  We have considered alternatives [1] but randomly selecting from 10,000 rows was selected because it's easy to implement in a threadsafe manner without introducing even more complicated requirements such as a concurrent queue or atomic counter. Note that after the warm-up, MySQL is essentially guaranteed to have the entire table in its own cache, so the exercise is testing the ORM (if applicable), database connection pool, and the database driver. Honestly, assuming none of the implementations are using an expensive random number generator (e.g., a random number generator intended for cryptographic use), I don't believe there would be different results if it were feasible to sequentially retrieve rows without the need to contend with concurrency. [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/111 https://github.com/TechEmpower/FrameworkBenchmarks/issues/11...

bhauer 14y ago

Thanks amarsahinovic for sharing this here. This is a brief blog entry with some observations about Round 3 of our web frameworks benchmark project. The community contributions continue unabated and I think the impressively long charts demonstrate just how much the community has given us. Thank you to everyone who has participated and we are looking forward to Round 4 already! Please reply to let us know if you have any comments, questions, or criticisms. We'd love to hear your feedback.

kbd 14y ago
Would still like Ruby to be tested with app servers other than Passenger. Is this planned for Round 4?
- bhauer 14y ago
  Oops! Thanks for pointing that out. This is an oversight in the readme for the Ruby tests. They are on Unicorn for Round 3 [1]. We'll get the readme updated and the Environment Details clarified as well. [1] https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/rails/setup_ruby.py https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... Edit: I've edited the Environment Details page accordingly.
  - kbd 14y ago
    This is the readme I looked at before I commented: https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/rack https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast... It still references Passenger.
orclev 14y ago
Really glad to see the performance of Yesod has moved up closer to where I'd expect now that some of the mistakes in the original benchmark have been fixed. One thing I do notice is that Yesod is using MySQL as the DB backend for the DB tests, but it does have support for MongoDB as well. Looked at from that perspective it's actually one of the higher performing frameworks for all the DB tasks, but it's hard to say for sure because it's never tested with MongoDB (it usually lands somewhere above all the MySQL based tests, but below all the MongoDB tests). Would be interested in seeing how it performs with the MongoDB backend used instead. Maybe if I get some time in the next couple weeks I'll try to swap out the backends and make a pull request.

zalew 14y ago

what's "django stripped"?

cmsimike 14y ago
A stripped down version of Django - no unnecessary apps installed. This was recommended to test Django's full speed but really wouldn't be reflective of a full deployment.
- mixmastamyk 14y ago
  It's disappointing that it's so slow. I suppose it doesn't matter a lot of the time, but would prefer it to be more competitive.
  - amarsahinovic 14y ago
    In the previous benchmark it was pointed out that they don't use connection pooling [1], I don't know is it fixed in this round https://news.ycombinator.com/item?id=5499490 https://news.ycombinator.com/item?id=5499490
    - andybak 14y ago
      They don't say either way but it requires them to switch to Postgres so I think they would have said if they had.
      - bhauer 14y ago
        That's correct; we are still testing on MySQL, so the Django tests are still configured without a connection pool. We would love a pull request that incorporates a connection pool for Django+MySQL. We do have Postgres as a target for future rounds of this project, but do not yet have an ETA for that.
        andybak 14y ago

baby 14y ago

I just don't understand how could Symfony2 be so fast considering it's pretty huge (twig itself is already huge). I guess it's time I consider switching from CI.

nakovet 14y ago
It's a matter of cache, APC cache, memcached, "compiled" autoload cache, there are many levels of caching in symfony, and boom, you have speed because many of your hits are hitting the ram memory.
josh-j 14y ago
I think you've got the charts backwards. Symfony2 is slower than CI in every chart.
- baby 14y ago
  oh...
camus 14y ago
Symfony caches everything , dumping resources ( configs , templates , ... ) into PHP files. So when you are using Twig on a production server you are not actually reading twig files but basic PHP files with echo statements.

deadA1ias 14y ago

Why is ASP.NET never included in these type of benchmarks?

Sodaware 14y ago
They have someone working on ASP.net tests for the next round.
mixmastamyk 14y ago
Windows is clumsy on the network unless you've built up a suite of tools and know-how to manage it. The extra cost doesn't help much either. As someone is said to be working on .net benchmarks, you may get your wish.
- WayneDB 14y ago
  Oh gawd. Please, tell us how Windows is "clumsy" to get setup on a network. Also, the "extra cost" doesn't even figure into a benchmark. Grab a 120 day trial ISO from Microsoft and install.
  - mixmastamyk 14y ago
    Simple really. Its heritage is single-user desktops. Therefore the security model is often "Don't look in here!" rather than having it done right the first time. Hides filenames; needs antivirus. Industry standard network/admin/deployment tools don't work as well, if supported. Until a couple of years ago, the only viable way to manage it was from a GUI... they are still figuring out how to run it headless. It's not always compatible with open standards. Shall I continue? Meanwhile Unix (and others) began life as time-sharing systems that became the original nodes of the internet, a scalable model that Windows has come back to forty years later. That's not to say that it doesn't have any strengths or isn't improving. The extra cost includes per minute charges on EC2.
    - WayneDB 14y ago
      That's great except the only thing you've proven here is your own ignorance. Shall I go on? You're spouting off about ancient history...and that somehow translates to Windows is (currently) "clumsy" on the network? Gee, I wonder why it only takes me 10 minutes flat to setup a headless ASP.Net server on EC2? And I can do that even with the GUI version of Windows because there's this little thing called RDP - maybe you've heard of it. Oh and ASP.Net also runs on Linux. It didn't a few years ago though, so maybe it won't work for you since apparently you are living in the past.

bdesimone 14y ago

Something seems off about "Results from dedicated hardware." Play and Go went from 1000s to <10 respectively. Care to give any insight there?

bhauer 14y ago
I assume by "Play" you mean Play1, which is a newly community-contributed test. Since the completion of the Round 3 tests, the Play community resolved the Play1 database test's problems [1] and the Go community is investigating the Go database tests [2]. Assuming the Go fix is in soon, we anticipate that both should be resolved in Round 4. [1] https://groups.google.com/forum/?fromgroups=#!topic/play-framework/HxgvitzGO50 https://groups.google.com/forum/?fromgroups=#!topic/play-fra... [2] https://groups.google.com/forum/?fromgroups=#!topic/golang-nuts/-YbB2Qjg41g https://groups.google.com/forum/?fromgroups=#!topic/golang-n...
- bdesimone 14y ago
  Thank you for the response, and for the project in general.

error54 14y ago

Just curious but why are the major php frameworks so abysmally slow? It seems like php-raw does pretty well in all the tests but most of the big frameworks (cake, symfony, etc) are orders of magnitude slower.

umsm 14y ago
Wow, I was floored at how slow Symfony is... I have a feeling it's ORM that's killing these frameworks in the DB portion.
- beberlei 14y ago
  it is used without caching, leading to unnecessary parsing of a metadata structure. This is explicitly not recommended for production use and kills the request time here.
- Skamander 13y ago
  Hi, I'm the user who contributed symfony 2 to the benchmark. My intention was to get as much php frameworks in the next round as possible (Symfony 2, Codeigniter, Laravel etc.), so I didn't pay much attention on the possible optimisations of the frameworks. On top of that I'm not a php guy, so it's possible that every php framework I contributed to the benchmark runs on breaks. ;o) In Symfony's case I posted on the mailing list in order to get users with symfony experience to improve the performance, but this post was sadly not activated/moderated.
1880 14y ago
A PHP framework has to initialize all its systems from zero for each request. It may because of that, it may not, it's just a guess.
- Joeri 14y ago
  You would think the frameworks would be designed around that, loading the absolute minimum of code to handle the request.

jcoby 14y ago

A few things really stand out for me: 1) I am shocked how slow EC2 is and how expensive. A m1.large is $0.240/hr or ~$175/month. And it's 7-10x slower than a $350/mo dedicated box. You would be spending $1500/mo to equal one dedicated box (not including bandwidth and S3 fees). A reserved instance is cheaper of course. 2) The multiple queries test would seem to be the best one to really simulate real-world usage. JSON-serialization is mostly testing the language. This test pretty much puts most of the interpreted languages on a full stack framework together at the bottom. Although django seems to come out at the bottom of the pack for some reason. Then come the "raw" tests and JIT languages running full stack frameworks. And at the top are the compiled languages. Not really surprising there. 3) I'm surprised php-raw did so well. And that go did so poorly.

pdog 14y ago
It's more accurate to compare dedicated boxes to reserved instances.
- cedrichurst 14y ago
  Reserved instances are simply a pricing construct. You're paying upfront for up to 24/7 usage, but you're running on the same pool as the standard instance. Dedicated instances, on the other hand, do what you're talking about but they're much pricier: http://aws.amazon.com/dedicated-instances/ http://aws.amazon.com/dedicated-instances/
  - Terretta 14y ago
    The comment you replied to didn't talk about what they do. I took it to mean it's more accurate from a price perspective.
xt 14y ago
A little too generalized, LuaJIT is winning some of the benchmarks
orclev 14y ago
The main selling point for EC2 isn't that it's cheap, it's that it's easy and fast to scale up/down and that you get managed failover. For that ease you pay a hefty premium. A lot of companies that have highly variable demand will often go with EC2 because if they get a sudden spike in demand they can quickly spin up 3 or 4 new instances to load balance across, and then when the demand dies off they can spin them all back down to save money. Doing something similar with dedicated hardware doesn't really work, you either end up without enough hardware to meet demand, or else you've got a bunch of hardware sitting around idle most of the time.

nakovet 14y ago

Where can I find this gemini? I went to environment and it's the only one that doesn't have a link.

krg 14y ago
It's our internal framework. From our first benchmarks post: http://www.techempower.com/blog/2013/03/28/framework-benchmarks/ http://www.techempower.com/blog/2013/03/28/framework-benchma... "Why include this Gemini framework I've never heard of?" We have included our in-house Java web framework, Gemini, in our tests. We've done so because it's of interest to us. You can consider it a stand-in for any relatively lightweight minimal-locking Java framework. While we're proud of how it performs among the well-established field, this exercise is not about Gemini. We routinely use other frameworks on client projects and we want this data to inform our recommendations for new projects.
- continuations 14y ago
  Any plan to open source it? Would love to learn more about it.
  - bhauer 14y ago
    We are considering it. Although, fair warning: it is not documented (save for our internal documentation) and we don't have the capacity to support its usage by third-parties. The reason we are considering open sourcing it anyway is that we feel it's the fair thing to do since we've included it in this project.
    - continuations 14y ago
      Please do. I'd very much like to learn about it.
- deleted 14y ago
  [deleted]

g3rald 14y ago

Wow I'm surprised that Rails has obtained better performance than PHP frameworks, and on many occasions has proved better than DJango...

g3rald 14y ago
Consider that we are talking about web oriented programming languages and frameworks.

benhanks040888 14y ago

Is the laravel on the benchmark Laravel 3 or Laravel 4? Still, CodeIgniter is the fastest PHP framework? Wow.

bhauer 14y ago
The Larevel test was community-contributed, but according to its readme [1], this is 3.2.14. Is Laravel 4 production-ready? [1] https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/php-laravel https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast...
- benhanks040888 14y ago
  I figured so too. Laravel 4 is still on beta as of now.

g3rald 14y ago

I want to try Gemini. Can you give me some tips to start?

dbecker 14y ago

Not being a web programmer, I have a naive question: Are many websites running into speed or scalability issues? I see programmers in other areas spend a lot of man-hours optimizing code that isn't a meaningful bottleneck. Is the same thing happening here?

baby 14y ago
If you don't have intelligent queries and a proper cache system. At least. You will quickly crash your database or saturate your traffic. It can even hurt small traffic websites (I remember coding a webchat with Mysql and crashing my database with only 5 people chatting).

josh-j 14y ago

Is the following a fair generalization? Java should be the conservative choice for your web framework's language (rather than PHP). Scala, Clojure, Nodejs, Erlang, Lua, and Haskell should be in your list of workable yet "cool" languages (rather than Ruby or Python).

jerf 14y ago
I'd add the caveat that Java should be the conservative choice only if you are expecting it to matter that you can serve 200,000 simpleish (but nontrivial) requests per second. In the vast majority of cases, development speed and things other than raw framework performance will dominate as concerns. However, that is a non-zero niche.
- josh-j 14y ago
  I was considering other factors in that generalization, although they're implicit. E.g., a lot of the "cool" language frameworks I mentioned are comparable with the Java frameworks in speed, but Java has a considerably larger ecosystem and number of hirable developers (although I'm not sure if that translates well into web development). And I assume PHP is as verbose as Java from what I've heard from other people (I've never used PHP). What language should be the conservative choice based on the many factors and ignoring the specific case for the general case?
  - camus 14y ago
    " And I assume PHP is as verbose as Java from what I've heard from other people (I've never used PHP)". So you dont even know what you are talking about?
    - josh-j 14y ago
      What do you think questions are for?
  - jtreminio 14y ago
    PHP can be as verbose as you want it to be. If you're using something like Symfony2, there'll be more abstractions and layers than using something like Silex - but they still don't come close to what Java looks like.

shaydoc 14y ago

Do you have any plans to run asp.net MVC webapi on .net 4.5 using the task parallel library ?

juiceandjuice 14y ago

That's some amazing performance out of the JVM.

kokoloko 14y ago
Pretty amazing. I would like to see a raw C benchmark to see how far is the JVM from the limits.
- continuations 14y ago
  Onion is C
- Poyeyo 14y ago
  I want to see a Wt (witty) benchmark. C++ is welcome here too.
  - krg 14y ago
    Wt was suggested before, and we'd love to include it. Care to submit a pull request? https://github.com/TechEmpower/FrameworkBenchmarks/issues/92 https://github.com/TechEmpower/FrameworkBenchmarks/issues/92

pilgrim689 14y ago

I'm having a hard time making conclusions based off of this data, except that Python/Ruby/JavaScript are slow, JVM is fast, and the rest are in between... It's odd how Haskell (Yesod) and C (Onion) are both slower than the JVM frameworks. Is it just a maturity issue? I know Yesod is very new relative to some of these JVM frameworks. Also, minor typo: the link to Erlang from the Environment tab is not the right URL. You probably meant to reference http://www.erlang.org/ http://www.erlang.org/, not http://www.erlang.com/ http://www.erlang.com/

bhauer 14y ago
Thanks pilgrim689. I'll get that fixed up.
mdasen 14y ago
Because the benchmarks are quite narrow, sometimes you get something that has simply been optimized or isn't doing the same work across platforms. For example, the last time I checked their php-raw code, it was executing a database query and appending the raw results to an array. Then, it took that array and passed it to json_encode which I'm pretty sure is implemented in C (PHP's built-in functions are in C, yes?). Anyway, comparing that to something like Rails' ActiveRecord, the work is quite different. With Rails, it's going to go through and call the accessor methods on the objects to get the data because we often override the default data stored in the database for the representation we want on an object level. So, they aren't doing the same thing. Rails at least used to have a way to get back an array of hashes rather than getting the AR objects, but I haven't been able to find it recently. That could be an interesting comparison. So, in that case, it just isn't the same functionality. Similarly, it's hard to know exactly how they're setting things up. Django will perform significantly better if you set up something like pgBouncer for connection pooling. Rails includes connection pooling, IIRC. Without knowing something like that, it's hard to gauge whether it reflects real-world usage (I mean, I'd argue that defaults matter, but if you're looking to scale Django, this isn't an onerous addition). Similarly, with process-based concurrency, it's important to get settings that maximize CPU usage while not overloading the RAM. Since they don't talk about memory or CPU usage, it's hard to figure out whether they've dealt with this. It might be in their repository, but I haven't been able to go through it yet. Heck, the multiple-query test in go seems really odd. I mean, coming in last place, serving less than half the requests of Rails? Similarly, Play 1 hits a nice 28% on EC2 and then can't serve a single request on the dedicated hardware. Plus, the tests really stress JSON performance and don't use things like a framework's templating system. One doesn't need to argue that JSON performance is useful, but so is template performance and that just isn't being tested. So, yeah, it would be hard to make conclusions off of this data except possibly that benchmarking is hard. None of this comment is meant as a dig against the people doing these benchmarks. They're improving them, they're making their code open, they're doing the kind of stuff that allows good, open questioning on their results. But I think it's still early to consider these to be really meaningful. I think a test that combined a bunch of different types of requests with database access, JSON, templates, etc. would be interesting and it seems like they might go there as they have more time.

camus 14y ago

It's a bit strange to include node with mongodb in the tests. Because it changes 2 variables in the test ( the web solution and the database ). You should stick to mysql(or whatever DB you are using) with every tests or it doesnt sound that serious.

xb95 14y ago

I am not acquainted with what language every framework uses. It would be useful for me in interpreting the data if you could annotate -- Java/JVM, PHP, Python, Perl, etc. This is great work, though. Thank you for doing it -- and double thank you for taking submissions from each of the communities so they can try to submit their best shot. That really helps make sure the tests are as ideal as possible.

bhauer 14y ago
xb95, I agree. In the next round, I'll add more color-coded columns to indicate language and platform information.

mronge 14y ago

I'd love to see these benchmarks on a beefier ec2 instance type. I'm curious how it would compare to the dedicated hardware

bhauer 14y ago
Agreed. A future enhancement I plan to make on the results viewer is the capacity to select from a menu of results files (the results are rendered from results.json files). Once that is in place, anyone who cares to could run the full suite on EC2 extra-large or, say, Rackspace Cloud, and submit those results files for us to include as options.

stcredzero 14y ago

I'd be more interested in benchmarks that reflect production configurations.

n1c 14y ago
From what I can tell the guys are trying quite hard to get close. If you spotted things that seem off you should contribute to the repo or let them know.
- bhauer 14y ago
  Indeed, a key objective of the project is to simulate a production environment, within reason. We are now at a point where about half of the tests are community-contributed and we have to take it for granted that the community has similarly aimed for production-readiness within their tests. As n1c points out, if there is something wrong with any of the configuration, please submit a Github issue or pull request.

bimr 14y ago

In order to have an apples to apples comparison, wouldn't you at least need to have an opcode cache for the PHP frameworks? PHP has to live and die upon each request, but Java culls all of its classes together on the first request and from then on it is running from RAM. Many PHP deployments, and most serious ones, will opcode cache in addition to database and page cache. You could rerun these tests with opcode cache only, leaving page and database caches for your "cache" round of tests. That will at least approximate the dramatic differences in architecture between Java and PHP.

krg 14y ago
All the PHP tests were done with the APC opcode cache, running within PHP-FPM. http://www.techempower.com/benchmarks/#section=environment http://www.techempower.com/benchmarks/#section=environment If this can be improved, please let us know. A pull request would be ideal.
- bimr 14y ago
  Oh. Didn't make the connection that APC was on for Codeigniter, Laravel and the other frameworks.

whalesalad 14y ago

A few things that I noticed, just random notes, take 'em with a grain of salt. I'm not running any of the code myself, I'm just browsing around the Github repo. This is a birds-eye analysis. I was a Django developer for a long time and recently went to the dark side and love Rails at the moment. It's important to remember that database connections are a killer in most web apps. This is why a lot of people use connection pooling (in the postgresql world, for example, you have pgbouncer and pgpool). When you can pool and persist connections, you'll have a faster web app (assuming it interfaces with a db, as many of their tests do). Last time I checked, Django does NOT provide built-in connection pooling, whereas Rails does. This could be part of the reason that Rails outperforms in most of the tests. That says something ... but it would be cool to see both frameworks strutting their stuff on an identical connection scenario. Also note that the MySQL connection limit for nodejs is set to 256 connections, whereas Rails set to use a pool of 5. Since these tests are hitting the DB, I'd like to see some more consistency with the way connections are setup. Not that 256 connections is better than 5, but if all those connections are open, it's no wonder that an async framework can hammer the DB so much faster than Ruby chugging along on a pool of 5. In the Node examples, it's important to note the difference between the mysql and mysql-raw tests. And to remember this when comparing to things like Django and Rails which use ORM's like ActiveRecord. In the raw msql test, node is really quick because there is no overhead there for mapping a row to a model. Also, a lot of the viewers of these benchmarks might just look at the big bar for X framework and think, "fuck it, that's my next project". Sure, go for it. But also remember that the type of app you're building comes into play a lot. These tests do not have any state. That's why these bare metal frameworks like Gemini are doing so well. Node does great here because it's async. But certain apps (like your future project) isn't going to fit into the mold of an async hello world test. I'm not knocking the framework tests here, they're awesome, but just realize this. One area where Django outperforms Rails is in the realm of JSON serialization. I'm asusming here that Python's native JSON engine is quicker than Ruby's. I learned this with Rails the hard way. With an out-of-the-box Gemfile, you're gonan have a bad time. In Ruby-land there are a few ways to boost your octane here. You start with MultiJSON and then take your pick of the many JSON engines in the world today. MultiJSON being an adaptor that lets developers put the choice of a JSON engine in your hands. Popular ones are yajl-ruby (ruby bindings for yajl, which is C) and oj. If you're doing JRuby, I hear that Jackson is super effing fast so that's nice. I'd love to see the JSON tests with various JSON parsers (at some point these guys are going to get funding for being the benchmark company, having every framework with every possible configuration on display) Django Protip: read up on select_related and prefetch_related. Similarly, in Rails land, you'll want to learn about eager loading with things like includes() and joins(). This will let you do one or few queries up front, rather than doing a bunch of subqueries inside of a big loop. One big query is usually better than a billion tiny ones. A reason why the PHP apps are getting murdered is because in a typical configuration your application initialized on every single request. Opcode caching can help here since that reduces that boot period. This why in a production environment you can modify a PHP file and instantly see the results. Versus (depending on the application) a Ruby or Python app, where the app is booted initially and until that worker or process is restarted, is running the code it started out with. This is for speed. This is also why, for example, a big Rails project (or better yet, a Java project) has a significantly longer warm-up period. It's loading the entire app. This is a big reason the PHP frameworks are suffering here, since they have a lot of overhead to load for each request. Whereas the raw PHP doesn't have that overhead. PHP protip: Roll with opcode caching (APC is a favorite of mine, it's very simple to install and configure, and provides a handy web gui for viewing various details of things like cache hit so you can optimize it). Also, please throw apache and mod_php away if you can and run with Nginx + php_fpm ... although if you're an HN reader then you're probably already doing this. The JVM is balls-quick because, again, unlike something like PHP, it gets booted up and then does it's thing. The JVM is infamous for it's slow startup process (which really varies based on a billion things) but there is a reason for that: the JIT. The slow startup is due to bytecode generation, I believe. It churns that bytecode into machine code and during that process it can make optimizations like unrolling loops. I'm no Java pro though. I don't like working with the language (the verbosity kills me, and XML is painful), but I have an immense deal of respect for its performance. To be honest I am really going to take a solid look at the Play framework for my next side project. It's got a lot of the pragmatism of Rails but it runs on the JVM and comes in a Scala variant. These are things that I am stoked about dipping my toes into. Finally, my hat is off to the folks running these. Great work!

actf 14y ago

These are really great - one additional test I would love to see would be Mono Asp.net MVC. I think it would interesting to see how it compares to the JVM.

AlexSerov51 13y ago

It would be useful to try the HybridJava framework as well.