3 ms·
A few things that I noticed, just random notes, take 'em with a grain of salt. I'm not running any of the code myself, I'm just browsing around the Github repo.
by whalesalad 14y ago
A few things that I noticed, just random notes, take 'em with a grain of salt. I'm not running any of the code myself, I'm just browsing around the Github repo. This is a birds-eye analysis. I was a Django developer for a long time and recently went to the dark side and love Rails at the moment.
It's important to remember that database connections are a killer in most web apps. This is why a lot of people use connection pooling (in the postgresql world, for example, you have pgbouncer and pgpool). When you can pool and persist connections, you'll have a faster web app (assuming it interfaces with a db, as many of their tests do).
Last time I checked, Django does NOT provide built-in connection pooling, whereas Rails does. This could be part of the reason that Rails outperforms in most of the tests. That says something ... but it would be cool to see both frameworks strutting their stuff on an identical connection scenario.
Also note that the MySQL connection limit for nodejs is set to 256 connections, whereas Rails set to use a pool of 5. Since these tests are hitting the DB, I'd like to see some more consistency with the way connections are setup. Not that 256 connections is better than 5, but if all those connections are open, it's no wonder that an async framework can hammer the DB so much faster than Ruby chugging along on a pool of 5.
In the Node examples, it's important to note the difference between the mysql and mysql-raw tests. And to remember this when comparing to things like Django and Rails which use ORM's like ActiveRecord. In the raw msql test, node is really quick because there is no overhead there for mapping a row to a model.
Also, a lot of the viewers of these benchmarks might just look at the big bar for X framework and think, "fuck it, that's my next project". Sure, go for it. But also remember that the type of app you're building comes into play a lot. These tests do not have any state. That's why these bare metal frameworks like Gemini are doing so well. Node does great here because it's async. But certain apps (like your future project) isn't going to fit into the mold of an async hello world test. I'm not knocking the framework tests here, they're awesome, but just realize this.
One area where Django outperforms Rails is in the realm of JSON serialization. I'm asusming here that Python's native JSON engine is quicker than Ruby's. I learned this with Rails the hard way. With an out-of-the-box Gemfile, you're gonan have a bad time. In Ruby-land there are a few ways to boost your octane here. You start with MultiJSON and then take your pick of the many JSON engines in the world today. MultiJSON being an adaptor that lets developers put the choice of a JSON engine in your hands. Popular ones are yajl-ruby (ruby bindings for yajl, which is C) and oj. If you're doing JRuby, I hear that Jackson is super effing fast so that's nice. I'd love to see the JSON tests with various JSON parsers (at some point these guys are going to get funding for being the benchmark company, having every framework with every possible configuration on display)
Django Protip: read up on select_related and prefetch_related. Similarly, in Rails land, you'll want to learn about eager loading with things like includes() and joins(). This will let you do one or few queries up front, rather than doing a bunch of subqueries inside of a big loop. One big query is usually better than a billion tiny ones.
A reason why the PHP apps are getting murdered is because in a typical configuration your application initialized on every single request. Opcode caching can help here since that reduces that boot period. This why in a production environment you can modify a PHP file and instantly see the results. Versus (depending on the application) a Ruby or Python app, where the app is booted initially and until that worker or process is restarted, is running the code it started out with. This is for speed. This is also why, for example, a big Rails project (or better yet, a Java project) has a significantly longer warm-up period. It's loading the entire app. This is a big reason the PHP frameworks are suffering here, since they have a lot of overhead to load for each request. Whereas the raw PHP doesn't have that overhead.
PHP protip: Roll with opcode caching (APC is a favorite of mine, it's very simple to install and configure, and provides a handy web gui for viewing various details of things like cache hit so you can optimize it). Also, please throw apache and mod_php away if you can and run with Nginx + php_fpm ... although if you're an HN reader then you're probably already doing this.
The JVM is balls-quick because, again, unlike something like PHP, it gets booted up and then does it's thing. The JVM is infamous for it's slow startup process (which really varies based on a billion things) but there is a reason for that: the JIT. The slow startup is due to bytecode generation, I believe. It churns that bytecode into machine code and during that process it can make optimizations like unrolling loops. I'm no Java pro though. I don't like working with the language (the verbosity kills me, and XML is painful), but I have an immense deal of respect for its performance.
To be honest I am really going to take a solid look at the Play framework for my next side project. It's got a lot of the pragmatism of Rails but it runs on the JVM and comes in a Scala variant. These are things that I am stoked about dipping my toes into.
Finally, my hat is off to the folks running these. Great work!
- bhauer 14y agoHi whalesalad. First, thank you for such a thoughtful and long reply. Second, you have an awesome handle. Indeed, database connections are a huge pain point for web applications. In our tests, Django does not yet use a connection pool. We hope that is addressed soon either (a) by usage of a MySQL connection pool or (b) by implementation of a Postgres suite of tests, which are planned. We don't have an ETA on either of those, but option (a) could be completed by a pull request if anyone has time. We've attempted to be consistent with database connection pooling. The Rails test runs in production mode and uses 256 connections in its pool [1]. Gemini is our in-house closed-source framework. We have been debating internally whether we are comfortable open sourcing it. It is not documented sufficiently for third-party use and we do not have the capacity to support its use by third-parties. On the other hand, we believe open sourcing it is the fair thing to do because we have included it here and other frameworks' fans should be able to examine it. We're still hemming and hawing over that. All that said, it is not a bare-metal framework [2]. My very strong opinion is that Node is not doing well because of its support for asynchronous application design, but rather because the V8 runtime that Node executes on is simply fast. I say that because the client-side concurrency is already sufficient to saturate the CPU cores using most frameworks (the exceptions being those that appear to have some lock or resource contention preventing full CPU utilization). Server-side concurrency fan-out strategies such as asynchronous evented loops may help superficially for our database tests, but pale in comparison to simply having a fast platform. I've written my opinions on that matter on my personal blog [3] (it's a later section of that blog entry). None of the JSON tests are asynchronous. All respond immediately within the handler code. I agree that it would be still more valuable to have additional dimensions for attributes such as the particular JSON serializer, the particular ORM, the particular database driver. With enough time, I'd love to have that breadth. The project has been steadily growing in scope, but handling that level of minor variations I fear remains quite a ways out. The priority right now is introducing new tests that include more representative functionality such as working with collections and server-side templates [4]. I'm not sure if your comment about PHP opcode caching is meant to suggest that we are not using it, but based on feedback prior to Round 2, we enabled APC for PHP, and that has remained enabled in this round [5]. I am not a part of the Play project, but I agree with your sentiment that Play deserves a good look by anyone starting a new web application. I've been especially impressed by their community. They have embraced this benchmark project and are helping us make sure Play performs as well as possible--with a fair and reasonable production-grade configuration. Their fans have given us some good advice and constructive criticism. Finally, thanks so much for the kind words at the end of your comment. It means a lot to us that the community finds value in this project. We've been thrilled with the response. On the other hand, we've been absolutely murdered by the moderators here. This item had a massive HN Slapdown score, and we weren't even responsible for posting it. :) [1] https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/rails/config/database-ruby.yml https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [2] https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/gemini/Source/hello/home/handler/HelloHandler.java https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [3] http://tiamat.tsotech.com/rps-vs-connections http://tiamat.tsotech.com/rps-vs-connections [4] https://github.com/TechEmpower/FrameworkBenchmarks/issues/133 https://github.com/TechEmpower/FrameworkBenchmarks/issues/13... [5] http://www.techempower.com/benchmarks/#section=environment http://www.techempower.com/benchmarks/#section=environment