7 ms·

Brightbox vs Amazon EC2: Phoronix Ubuntu 12.04 Benchmarks

32 points by jeremyjarvis 14y ago

comice 14y ago
We found comparing Xen to KVM on the same hardware showed similar results to this too (things doing lots of ram allocations, like redis, do very well with KVM).
RKearney 14y ago
Posted on brightbox.com. Surely this post won't be biased in the least.
- jeremyjarvis 14y ago
  They're open source benchmarks, try them yourself if not convinced :)
  - RKearney 14y ago
    Well also in the price comparison they forgot to mention EC2 offers reserved instances. A Brightbox instance would cost $466 USD a month, and an EC2 reserved c1.xlarge instance would cost $102 a month with a 3 year reserved instance. If you average out the cost to reserve an instance over 3 years, the total would be $188 a month, This means EC2 is actually 60% cheaper than Brightbox.
    - jeremyjarvis 14y ago
      This is clearly a comparison of on-demand instances, where the common unit is the hourly rate as with other IaaS. Reserved instances in themselves have a cost in terms of commitment or risk (if you aren't able to sell them back later).
- comice 14y ago
  hehe, sure, but the phoronix test suite is open source. Trivially easy to reproduce the results. (like literally a couple of commands and a few tens of pennies of server time).
- kev009 14y ago
  The pgbench score is conspicuously missing too, which is heavily disk dependent, which is universally heavily over-subscribed by VPS providers. There's a smokescreen at the end "our storage is hardware RAID6 with 15k rpm SAS disks" which means very little. (i.e. if that's internal storage across 8 spindles, it's barely going to sustain 1000 IOPS in anger. That could be monopolized by a just a handful of VMs) This link should be buried.
  - comice 14y ago
    I ran these benchmarks myself. The pgbench score just failed to run - I tried a few times but it wouldn't even start. I put it down to a bug in the benchmark. Should have explained. But yeah, no diskio tests here really (other than the pgbench one, but I don't know to what extent that would have tested diskio). The benchmark suite tests were chosen by Phoronix. We'll do a diskio one at some point too.
    - mryan 14y ago
      pgbench will only stress your disks if the test database is too large to fit entirely in memory (controlled by the scaling factor option). bonnie++ results would also be interesting.
s_henry_paulson 14y ago
From the comments: In your price comparison you forgot to mention EC2 offers reserved instances. Your instance would cost $466 USD a month, and an EC2 reserved c1.xlarge instance would cost $102 a month with a 3 year reserved instance. If you average out the cost to reserve an instance over 3 years, the total would be $188 a month, This means EC2 is actually 60% cheaper than Brightbox.
- jeremyjarvis 14y ago
  This is clearly a comparison of on-demand instances, where the common unit is the hourly rate as with other IaaS. Reserved instances in themselves have a cost in terms of commitment or risk (if you aren't able to sell them back later).
  - RKearney 14y ago
    Then why not go by EC2's spot instances? Those would still be cheaper with zero upfront commitment. A c1.xlarge spot instance is $0.07/hr (right now) compared to the on-demand pricing of $0.66/hr. That means, assuming pricing stays consistent, a c1.xlarge instance would cost $50 a month compared to $466 a month with Brightbox.
    - jeremyjarvis 14y ago
      Because it's not a common unit for comparison. The spot price can fluctuate. Obviously. (you edited your comment, so I'll edit mine :) ... How can you assume the spot price will stay the same?
      - jjm 14y ago
        How can we assume Brightbox won't raise prices? How can we assume Amazon won't raise base prices, possibly affecting the spot market, then again this is a spot market! It's like asking how can you assume stock prices (not exactly equiv, but thats not the point) will go up or down? How can we assume anything? We don't. I think the assumption on the wrong feature to market is the problem here. I only care that right now, EC2 spot are cheaper by a large margin. IF the price raises to match then I'll adjust. I'm not locked in.
gizzlon 14y ago
Looking at the pricing reminds me of how expensive cloud computing is if you keep them all running 24/7.. The small one would cost: server: £0.025 * 24 * 30 = 18.00 ip: £0.0035 * 24 * 30 = 2.52 20 GB /m out: £0.12 * 20 = 2.4 5 GB /m in : £0.08 * 05 = 0.4 = £23.32 Is this estimate correct? For this price you can get four and a half virtual 1-core servers, with 4 ip's, 4 TB of outgoing data and unlimited incoming data. Of course, cloud computing has additional features, so I'm not saying it's never worth it. But it seems like a bad deal for most uses. Am I missing something? or is this just the price you have to pay for the cloud "extras" ?
- giulianob 14y ago
  The advantage with Amazon is that if you get a reserved instance where you pay a bit upfront, your hourly price drops dramatically. I think you can get discounts upwards of 75% with reserved instances.
- eckyptang 14y ago
  Try HTTP://bigv.io if you are UK based.
ridruejo 14y ago
People keep comparing EC2 and other hosting providers in terms of performance and pricing. However, the biggest value in my mind is related to the agility it provides (you can basically model and control your infrastructure through a programming API) and the whole ecosystem around it (CloudFront, Route53, Beanstalk, S3). Some vendors, including Brightbox, have some of that functionality but it is a very small fraction of what is possible with AWS
- jeremyjarvis 14y ago
  Yeah - that's fair comment, we don't have the same breadth of products as AWS for sure. I agree absolutely that agility is what it's all about though.
- zurn 14y ago
  The EC2 cost premium over competitors is a factor, and so are the other AWS services. It's a meaningful comparison.
ck2 14y ago
If you are going to benchmark Apache, use 2.4 with event MPM which can be as fast as nginx.
jread 14y ago
You have to careful comparing one specific EC2 instance with other providers. Unlike most compute services, EC2 infrastructure is heterogenous. The c1.xlarge instance compared might deploy to 2 or 3 different chipsets in us-east-1, each with different performance characteristics. Based on the results they posted, this instance appears to have deployed to a 2.13GHz E5506 host. Most likely, Brightbox is using a homogenous infrastructure, and likely something a bit faster, like X55, X56 or E5s. For a more apples to apples comparison, they might have tested against an m2.4xlarge (8 cores) or cc1.4xlarge (8 cores hyperthreaded to 16), which are also 8 cores and deploy to X5550, X5570 or E5-2665. Based on my own benchmarking either of these instances would have performed comparably or better than the Brightbox 8GB instance. The point is, you have a lot of options with EC2 in terms of performance, and comparing a single instance deployment doesn't really do it justice. [Edit] Here are a few links to Phoronix results for EC2 instance types not included in their testing. m2s, cc1s and cc2s are the better CPU performers in EC2 due to faster processors. If you do a CPU performance comparison, these instance types should not be excluded (as I observe they often are): m2.4xlarge - 8 cores - X5550 http://openbenchmarking.org/result/1208019-SU-CLOUDHARM39 http://openbenchmarking.org/result/1208019-SU-CLOUDHARM39 m2.4xlarge - 8 cores - E5-2665 http://openbenchmarking.org/result/1208139-SU-CLOUDHARM43 http://openbenchmarking.org/result/1208139-SU-CLOUDHARM43 cc1.4xlarge - 8 cores/16 HT - X5570 http://openbenchmarking.org/result/1208127-SU-CLOUDHARM93 http://openbenchmarking.org/result/1208127-SU-CLOUDHARM93 c1.xlarge - 8 cores - E5506 http://openbenchmarking.org/result/1208134-SU-CLOUDHARM15 http://openbenchmarking.org/result/1208134-SU-CLOUDHARM15 These instances were tested using CentOS 6, which will be a bit slower than Ubuntu 12.04.
- jeremyjarvis 14y ago
  m2.4xlarge has 68.4 GB of RAM and costs more than 3 times as much per hour :)
- comice 14y ago
  If the m2.4xlarge goes only on X5550 cores, then that does work out to be about 26 compute units (as they claim). So I would expect a m2.4xlarge to perform the same as a Brightbox large (which is about 26 compute units too). The pricing (and ram spec) is very different though - not really a like for like comparison, but I see that it's important to note their use of cpu type. Interested in how xen vs. kvm is playing a part here too.

32 points by jeremyjarvis 14y ago

We found comparing Xen to KVM on the same hardware showed similar results to this too (things doing lots of ram allocations, like redis, do very well with KVM).

RKearney 14y ago

Posted on brightbox.com. Surely this post won't be biased in the least.

jeremyjarvis 14y ago
They're open source benchmarks, try them yourself if not convinced :)
- RKearney 14y ago
  Well also in the price comparison they forgot to mention EC2 offers reserved instances. A Brightbox instance would cost $466 USD a month, and an EC2 reserved c1.xlarge instance would cost $102 a month with a 3 year reserved instance. If you average out the cost to reserve an instance over 3 years, the total would be $188 a month, This means EC2 is actually 60% cheaper than Brightbox.
  - jeremyjarvis 14y ago
    This is clearly a comparison of on-demand instances, where the common unit is the hourly rate as with other IaaS. Reserved instances in themselves have a cost in terms of commitment or risk (if you aren't able to sell them back later).
comice 14y ago
hehe, sure, but the phoronix test suite is open source. Trivially easy to reproduce the results. (like literally a couple of commands and a few tens of pennies of server time).
kev009 14y ago
The pgbench score is conspicuously missing too, which is heavily disk dependent, which is universally heavily over-subscribed by VPS providers. There's a smokescreen at the end "our storage is hardware RAID6 with 15k rpm SAS disks" which means very little. (i.e. if that's internal storage across 8 spindles, it's barely going to sustain 1000 IOPS in anger. That could be monopolized by a just a handful of VMs) This link should be buried.
- comice 14y ago
  I ran these benchmarks myself. The pgbench score just failed to run - I tried a few times but it wouldn't even start. I put it down to a bug in the benchmark. Should have explained. But yeah, no diskio tests here really (other than the pgbench one, but I don't know to what extent that would have tested diskio). The benchmark suite tests were chosen by Phoronix. We'll do a diskio one at some point too.
  - mryan 14y ago
    pgbench will only stress your disks if the test database is too large to fit entirely in memory (controlled by the scaling factor option). bonnie++ results would also be interesting.

s_henry_paulson 14y ago

From the comments: In your price comparison you forgot to mention EC2 offers reserved instances. Your instance would cost $466 USD a month, and an EC2 reserved c1.xlarge instance would cost $102 a month with a 3 year reserved instance. If you average out the cost to reserve an instance over 3 years, the total would be $188 a month, This means EC2 is actually 60% cheaper than Brightbox.

jeremyjarvis 14y ago
This is clearly a comparison of on-demand instances, where the common unit is the hourly rate as with other IaaS. Reserved instances in themselves have a cost in terms of commitment or risk (if you aren't able to sell them back later).
- RKearney 14y ago
  Then why not go by EC2's spot instances? Those would still be cheaper with zero upfront commitment. A c1.xlarge spot instance is $0.07/hr (right now) compared to the on-demand pricing of $0.66/hr. That means, assuming pricing stays consistent, a c1.xlarge instance would cost $50 a month compared to $466 a month with Brightbox.
  - jeremyjarvis 14y ago
    Because it's not a common unit for comparison. The spot price can fluctuate. Obviously. (you edited your comment, so I'll edit mine :) ... How can you assume the spot price will stay the same?
    - jjm 14y ago
      How can we assume Brightbox won't raise prices? How can we assume Amazon won't raise base prices, possibly affecting the spot market, then again this is a spot market! It's like asking how can you assume stock prices (not exactly equiv, but thats not the point) will go up or down? How can we assume anything? We don't. I think the assumption on the wrong feature to market is the problem here. I only care that right now, EC2 spot are cheaper by a large margin. IF the price raises to match then I'll adjust. I'm not locked in.

gizzlon 14y ago

Looking at the pricing reminds me of how expensive cloud computing is if you keep them all running 24/7.. The small one would cost: server: £0.025 * 24 * 30 = 18.00 ip: £0.0035 * 24 * 30 = 2.52 20 GB /m out: £0.12 * 20 = 2.4 5 GB /m in : £0.08 * 05 = 0.4 = £23.32 Is this estimate correct? For this price you can get four and a half virtual 1-core servers, with 4 ip's, 4 TB of outgoing data and unlimited incoming data. Of course, cloud computing has additional features, so I'm not saying it's never worth it. But it seems like a bad deal for most uses. Am I missing something? or is this just the price you have to pay for the cloud "extras" ?

giulianob 14y ago
The advantage with Amazon is that if you get a reserved instance where you pay a bit upfront, your hourly price drops dramatically. I think you can get discounts upwards of 75% with reserved instances.
eckyptang 14y ago
Try HTTP://bigv.io if you are UK based.

ridruejo 14y ago

People keep comparing EC2 and other hosting providers in terms of performance and pricing. However, the biggest value in my mind is related to the agility it provides (you can basically model and control your infrastructure through a programming API) and the whole ecosystem around it (CloudFront, Route53, Beanstalk, S3). Some vendors, including Brightbox, have some of that functionality but it is a very small fraction of what is possible with AWS

jeremyjarvis 14y ago
Yeah - that's fair comment, we don't have the same breadth of products as AWS for sure. I agree absolutely that agility is what it's all about though.
zurn 14y ago
The EC2 cost premium over competitors is a factor, and so are the other AWS services. It's a meaningful comparison.

ck2 14y ago

If you are going to benchmark Apache, use 2.4 with event MPM which can be as fast as nginx.

jread 14y ago

You have to careful comparing one specific EC2 instance with other providers. Unlike most compute services, EC2 infrastructure is heterogenous. The c1.xlarge instance compared might deploy to 2 or 3 different chipsets in us-east-1, each with different performance characteristics. Based on the results they posted, this instance appears to have deployed to a 2.13GHz E5506 host. Most likely, Brightbox is using a homogenous infrastructure, and likely something a bit faster, like X55, X56 or E5s. For a more apples to apples comparison, they might have tested against an m2.4xlarge (8 cores) or cc1.4xlarge (8 cores hyperthreaded to 16), which are also 8 cores and deploy to X5550, X5570 or E5-2665. Based on my own benchmarking either of these instances would have performed comparably or better than the Brightbox 8GB instance. The point is, you have a lot of options with EC2 in terms of performance, and comparing a single instance deployment doesn't really do it justice. [Edit] Here are a few links to Phoronix results for EC2 instance types not included in their testing. m2s, cc1s and cc2s are the better CPU performers in EC2 due to faster processors. If you do a CPU performance comparison, these instance types should not be excluded (as I observe they often are): m2.4xlarge - 8 cores - X5550 http://openbenchmarking.org/result/1208019-SU-CLOUDHARM39 http://openbenchmarking.org/result/1208019-SU-CLOUDHARM39 m2.4xlarge - 8 cores - E5-2665 http://openbenchmarking.org/result/1208139-SU-CLOUDHARM43 http://openbenchmarking.org/result/1208139-SU-CLOUDHARM43 cc1.4xlarge - 8 cores/16 HT - X5570 http://openbenchmarking.org/result/1208127-SU-CLOUDHARM93 http://openbenchmarking.org/result/1208127-SU-CLOUDHARM93 c1.xlarge - 8 cores - E5506 http://openbenchmarking.org/result/1208134-SU-CLOUDHARM15 http://openbenchmarking.org/result/1208134-SU-CLOUDHARM15 These instances were tested using CentOS 6, which will be a bit slower than Ubuntu 12.04.

jeremyjarvis 14y ago
m2.4xlarge has 68.4 GB of RAM and costs more than 3 times as much per hour :)
comice 14y ago
If the m2.4xlarge goes only on X5550 cores, then that does work out to be about 26 compute units (as they claim). So I would expect a m2.4xlarge to perform the same as a Brightbox large (which is about 26 compute units too). The pricing (and ram spec) is very different though - not really a like for like comparison, but I see that it's important to note their use of cpu type. Interested in how xen vs. kvm is playing a part here too.