7 ms·

Drupal Benchmarks

60 points by geerlingguy 11y ago

radicalbyte 11y ago
What is the value in using a system that you admit is in beta and not optimized to benchmark two different hardware platforms?
- supergreg 11y ago
  To see how it changes as you work on it?
- geerlingguy 11y ago
  I've been benchmarking Drupal 8 since early alphas (not all benchmarks are published on that Wiki page... I really need to update it), as I'm interested in seeing how different infrastructure, different PHP versions, and different caching architectures can help or hurt performance. See, for instance: https://twitter.com/geerlingguy/status/702909750405517312 https://twitter.com/geerlingguy/status/702909750405517312 Drupal 8.0.4 is the current release, and I'm going to be rerunning all benchmarks on that version soon.
- Ensorceled 11y ago
  To call attention to the fact that it needs that optimization?
  - radicalbyte 11y ago
    He's benchmarking the difference between a Pi2 and a server, so it makes no sense to introduce another variable - unoptimized, unfinished code.
    - geerlingguy 11y ago
      As long as the code itself is the same... It's not a variable. I always benchmark against the same point release or git hash. It's nice to have comparisons of e.g. Beta 1 vs Beta 2, so I can find out where performance regressions are introduced into the code.
      - radicalbyte 11y ago
        The problem is the increased risk of nonsense benchmarks. I.e. if the unoptimized code is super slow because the naive implementation compiles to something unsuited to, say, the x86 microarchitecture. Or that performance is limited by lock contention. That's way more likely with pre-release then production-grate software. If you want to compare B1 and B2 that's great - but then you post to hackernews a comparison between two Drupal betas, not a comparison between two hardware platforms.
      - radicalbyte 11y ago
        Benchmarking beta software is of course great and very valuable - it focuses attention and will hopefully motivate projects to improve performance. It's just strange to do it in a piece which is predominantly arguing for the effectiveness of a cheap hardware solution :)
- TickleSteve 11y ago
  Because he's benchmarking the system, not drupal....
geerlingguy 11y ago
From the Benchmarks Wiki page: Benchmarks are extremely important when making infrastructure and configuration changes. The Raspberry Pi is by no means a 'high performance' computer, but even with it's power-starved architecture, we can optimize many different parts of the stack to make the Pi perform well. I've been hosting www.pidramble.com on a cluster of 6 Raspberry Pi 2s with an Nginx load balancer, 4 Nginx + PHP web servers, and one MySQL database server for about 6 months, with two periods of downtime when I had to switch off the circuit in my office for some electrical work. The Pi 2 isn't a speed demon, but it's actually not so bad for a casual on-premise hosting solution, especially if you add in better failover modes. I'm going to be moving the architecture that direction for a presentation I'll be giving at php[tek] this summer.
brianwawok 11y ago
Is anyone else kind of shocked in 2016 that a webserver for a language can only serve 10 requests per second? I agree that a super optimized stack that can serve 15k requests per second is a bit much for hobby projects but 10? If 1 user pulls up a page with a bunch of images, he has a 3 second wait to get everything. Sure you can static host some stuff with nginx.... but still.
- geerlingguy 11y ago
  Keep in mind this was running on a beta; the current version of Drupal 8, running on PHP 7, gets something closer to 60 req/s authenticated, 300+ anonymous.
  - nkozyra 11y ago
    Even so, that strikes me as very low. I've always wondered where the bottlenecks are in Drupal (I haven't looked at it in many years). Getting that kind of throughput even on paltry hardware would give me pause about doing this in production.
    - emergentcypher 11y ago
      - PHP is slow. Unlike JS, it's generally interpreted and not JIT (unless you're Facebook). - Drupal has to bootstrap itself and all modules on every request. - Drupal makes waaaaay too many SQL queries - Drupal generates horribly performing SQL queries
      - brianwawok 11y ago
        How many SQL queries end up getting run if you are loading a simple page with 1 row read out of 1 table as the actual payload?
        knieveltech 11y ago
        That never happens though. Drupal has to bootstrap itself so it loads it's registry cache, variables cache, current user record (split across at least four tables to pull in the user account, associated roles, and user access permissions), any dynamic menu builds are going to the menu table, access callbacks and other function calls associated with dynamic menu rendering may also trigger other database calls, uncached content blocks have to be pulled out of the database, and we haven't even gotten to loading the node for the page which could be split across a bunch of tables if the content type has fields added to it...
miahi 11y ago
That's not an i7, it's a MacBook Air really low-power CPU. And it's not quad core, it's dual core + HT (i7 4650U [1]). A "real" mobile i7 quad has twice the performance, and a "real" desktop i7 is about three times faster. So a better title for this would be "6 Pi2 vs MBA vs DigitalOcean". [1] http://ark.intel.com/products/75114/Intel-Core-i7-4650U-Processor-4M-Cache-up-to-3_30-GHz http://ark.intel.com/products/75114/Intel-Core-i7-4650U-Proc...
- sciurus 11y ago
  Also, instead of testing directly on the Macbook Air the author runs the test against a "cluster" of 6 virtual machines.
  - geerlingguy 11y ago
    I wanted to compare apples to apples in terms of the DO vs local VM infrastructure; the VT-X and KVM structure make the overhead of running the individual VMs highly efficient. I'd like to also do some testing against a single local AMP stack when I get time.
geerlingguy 11y ago
Just an aside/question; the title was edited to "Drupal Benchmarks" which loses the contextual information. While, yes, the page title corresponds, this is part of a wiki, and the editorialized title adds contextual data that shows why this post is not (barely at all) about "Drupal benchmarks". Is there an official policy on titles / ninja edits by HN moderators? If so, next time I'll edit the Wiki page instead of leaving it in a hierarchical format, if that's required.
- brudgers 11y ago
  Higher up the thread there are comments regarding the use of "i7" in reference to a particular model of MacBook Air. I suspect that the change in title derails that sort of nitpicking...by which I mean that anyone who reads the page learns that it is that computer and can form their own judgments. Anyway, the mysterious ways of Hacker News are mysterious by definition, and since the story is on the front page as I write this, I think it's safe to conclude that the format is fine and the HN elves changed the title for their own purposes...i.e. it ain't broke, don't fix it.
  - geerlingguy 11y ago
    K, thanks for the explanation. The laptop does have an i7, btw... But pedantic comments get more upvotes in this community :) The main thing I wanted to compare, regardless, was DigitalOcean vs a common local development environment vs the Pi 2. It seems the thread turned into a storm of Drupal vitriol, so maybe for my next post I'll focus more on other benchmarks, and toss up one of my Flask apps, or an Express app instead.
    - brudgers 11y ago
      I think that it's often hit or miss with HN stories about more mainstream technology. I was sort of surprised about how well the Pi2 did versus more substantial hardware and the cloud...anyway, benchmarks have historically tended to evoke strong opinions. About the only one's I can think of that achieve widespread acceptance are Backblazes hard drive reviews and places like Anand's hardware reviews. It's probably because they're backed by significant data. Hopefully, your next review will also get widely read.

60 points by geerlingguy 11y ago

What is the value in using a system that you admit is in beta and not optimized to benchmark two different hardware platforms?

supergreg 11y ago
To see how it changes as you work on it?
geerlingguy 11y ago
I've been benchmarking Drupal 8 since early alphas (not all benchmarks are published on that Wiki page... I really need to update it), as I'm interested in seeing how different infrastructure, different PHP versions, and different caching architectures can help or hurt performance. See, for instance: https://twitter.com/geerlingguy/status/702909750405517312 https://twitter.com/geerlingguy/status/702909750405517312 Drupal 8.0.4 is the current release, and I'm going to be rerunning all benchmarks on that version soon.
Ensorceled 11y ago
To call attention to the fact that it needs that optimization?
- radicalbyte 11y ago
  He's benchmarking the difference between a Pi2 and a server, so it makes no sense to introduce another variable - unoptimized, unfinished code.
  - geerlingguy 11y ago
    As long as the code itself is the same... It's not a variable. I always benchmark against the same point release or git hash. It's nice to have comparisons of e.g. Beta 1 vs Beta 2, so I can find out where performance regressions are introduced into the code.
    - radicalbyte 11y ago
      The problem is the increased risk of nonsense benchmarks. I.e. if the unoptimized code is super slow because the naive implementation compiles to something unsuited to, say, the x86 microarchitecture. Or that performance is limited by lock contention. That's way more likely with pre-release then production-grate software. If you want to compare B1 and B2 that's great - but then you post to hackernews a comparison between two Drupal betas, not a comparison between two hardware platforms.
    - radicalbyte 11y ago
      Benchmarking beta software is of course great and very valuable - it focuses attention and will hopefully motivate projects to improve performance. It's just strange to do it in a piece which is predominantly arguing for the effectiveness of a cheap hardware solution :)
TickleSteve 11y ago
Because he's benchmarking the system, not drupal....

geerlingguy 11y ago

From the Benchmarks Wiki page: Benchmarks are extremely important when making infrastructure and configuration changes. The Raspberry Pi is by no means a 'high performance' computer, but even with it's power-starved architecture, we can optimize many different parts of the stack to make the Pi perform well. I've been hosting www.pidramble.com on a cluster of 6 Raspberry Pi 2s with an Nginx load balancer, 4 Nginx + PHP web servers, and one MySQL database server for about 6 months, with two periods of downtime when I had to switch off the circuit in my office for some electrical work. The Pi 2 isn't a speed demon, but it's actually not so bad for a casual on-premise hosting solution, especially if you add in better failover modes. I'm going to be moving the architecture that direction for a presentation I'll be giving at php[tek] this summer.

brianwawok 11y ago

Is anyone else kind of shocked in 2016 that a webserver for a language can only serve 10 requests per second? I agree that a super optimized stack that can serve 15k requests per second is a bit much for hobby projects but 10? If 1 user pulls up a page with a bunch of images, he has a 3 second wait to get everything. Sure you can static host some stuff with nginx.... but still.

geerlingguy 11y ago
Keep in mind this was running on a beta; the current version of Drupal 8, running on PHP 7, gets something closer to 60 req/s authenticated, 300+ anonymous.
- nkozyra 11y ago
  Even so, that strikes me as very low. I've always wondered where the bottlenecks are in Drupal (I haven't looked at it in many years). Getting that kind of throughput even on paltry hardware would give me pause about doing this in production.
  - emergentcypher 11y ago
    - PHP is slow. Unlike JS, it's generally interpreted and not JIT (unless you're Facebook). - Drupal has to bootstrap itself and all modules on every request. - Drupal makes waaaaay too many SQL queries - Drupal generates horribly performing SQL queries
    - brianwawok 11y ago
      How many SQL queries end up getting run if you are loading a simple page with 1 row read out of 1 table as the actual payload?
      - knieveltech 11y ago
        That never happens though. Drupal has to bootstrap itself so it loads it's registry cache, variables cache, current user record (split across at least four tables to pull in the user account, associated roles, and user access permissions), any dynamic menu builds are going to the menu table, access callbacks and other function calls associated with dynamic menu rendering may also trigger other database calls, uncached content blocks have to be pulled out of the database, and we haven't even gotten to loading the node for the page which could be split across a bunch of tables if the content type has fields added to it...

miahi 11y ago

That's not an i7, it's a MacBook Air really low-power CPU. And it's not quad core, it's dual core + HT (i7 4650U [1]). A "real" mobile i7 quad has twice the performance, and a "real" desktop i7 is about three times faster. So a better title for this would be "6 Pi2 vs MBA vs DigitalOcean". [1] http://ark.intel.com/products/75114/Intel-Core-i7-4650U-Processor-4M-Cache-up-to-3_30-GHz http://ark.intel.com/products/75114/Intel-Core-i7-4650U-Proc...

sciurus 11y ago
Also, instead of testing directly on the Macbook Air the author runs the test against a "cluster" of 6 virtual machines.
- geerlingguy 11y ago
  I wanted to compare apples to apples in terms of the DO vs local VM infrastructure; the VT-X and KVM structure make the overhead of running the individual VMs highly efficient. I'd like to also do some testing against a single local AMP stack when I get time.

geerlingguy 11y ago

Just an aside/question; the title was edited to "Drupal Benchmarks" which loses the contextual information. While, yes, the page title corresponds, this is part of a wiki, and the editorialized title adds contextual data that shows why this post is not (barely at all) about "Drupal benchmarks". Is there an official policy on titles / ninja edits by HN moderators? If so, next time I'll edit the Wiki page instead of leaving it in a hierarchical format, if that's required.

brudgers 11y ago
Higher up the thread there are comments regarding the use of "i7" in reference to a particular model of MacBook Air. I suspect that the change in title derails that sort of nitpicking...by which I mean that anyone who reads the page learns that it is that computer and can form their own judgments. Anyway, the mysterious ways of Hacker News are mysterious by definition, and since the story is on the front page as I write this, I think it's safe to conclude that the format is fine and the HN elves changed the title for their own purposes...i.e. it ain't broke, don't fix it.
- geerlingguy 11y ago
  K, thanks for the explanation. The laptop does have an i7, btw... But pedantic comments get more upvotes in this community :) The main thing I wanted to compare, regardless, was DigitalOcean vs a common local development environment vs the Pi 2. It seems the thread turned into a storm of Drupal vitriol, so maybe for my next post I'll focus more on other benchmarks, and toss up one of my Flask apps, or an Express app instead.
  - brudgers 11y ago
    I think that it's often hit or miss with HN stories about more mainstream technology. I was sort of surprised about how well the Pi2 did versus more substantial hardware and the cloud...anyway, benchmarks have historically tended to evoke strong opinions. About the only one's I can think of that achieve widespread acceptance are Backblazes hard drive reviews and places like Anand's hardware reviews. It's probably because they're backed by significant data. Hopefully, your next review will also get widely read.