4 ms·

How "reliable" are those numbers in estimating real application performance? I've run nbench for a 1 CPU VM in EC2, and a 2 CPU VM in DO, and the former is a l

How "reliable" are those numbers in estimating real application performance? I've run nbench for a 1 CPU VM in EC2, and a 2 CPU VM in DO, and the former is a lot faster ( almost 2x ) than the latter! EC2 Ubuntu ( 1 CPU ) ==============================LINUX DATA BELOW=============================== CPU : GenuineIntel Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz 2400MHz L2 Cache : 30720 KB OS : Linux 3.13.0-74-generic C compiler : gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.1) libc : libc-2.19.so MEMORY INDEX : 39.370 INTEGER INDEX : 35.426 FLOATING-POINT INDEX: 53.665 DO CoreOS ( 2 CPU ) ==============================LINUX DATA BELOW=============================== CPU : Dual GenuineIntel Intel(R) Xeon(R) CPU E5-2630L 0 @ 2.00GHz 2000MHz L2 Cache : 15360 KB OS : Linux 4.2.2-coreos-r2 C compiler : gcc version 4.9.2 (Debian 4.9.2-10) libc : libc-2.19.so MEMORY INDEX : 20.668 INTEGER INDEX : 19.277 FLOATING-POINT INDEX: 28.370 Edit: Here's also a 2 CPU machine on Azure ==============================LINUX DATA BELOW=============================== CPU : Dual GenuineIntel Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz 2397MHz L2 Cache : 30720 KB OS : Linux 4.2.2-coreos-r2 C compiler : gcc version 4.9.2 (Debian 4.9.2-10) libc : libc-2.19.so MEMORY INDEX : 28.667 INTEGER INDEX : 24.351 FLOATING-POINT INDEX: 39.950

jamiesonbecker 11y ago
This is helpful but you're not mentioning what instance type at EC2.. t2 and t1 instances do heavy throttling after you use up your CPU credits[1] so it's likely that you were just using all or most of a full Xeon CPU. I got similar results on even a t2.nano.. until I used up the CPU credits. Then it slowed to a crawl! Also keep in mind that nbench only tests a single CPU[2]. Apples to apples would probably be a small m3 or m4 against a comparable DO instance. (but let's not go down that cost disparity rabbit hole.. esp EBS vs SSD or bandwidth!) 1. https://aws.amazon.com/ec2/instance-types/t2/ https://aws.amazon.com/ec2/instance-types/t2/ 2. https://en.wikipedia.org/wiki/NBench#Shortcomings https://en.wikipedia.org/wiki/NBench#Shortcomings
- antouank 11y ago
  If nbench only uses 1 core, yes, my comparison is wrong. Is there a similar benchmarking linux tool that can use many cores? edit: instance is a t2.micro
  - jamiesonbecker 11y ago
    I don't think the comparison is wrong per se; this actually makes it a bit easier, since you're just comparing one core to another. The only thing that I think is problematic is comparing a t2 to anything else, since that CPU performance is not sustainable as it might be elsewhere. m3 and m4 (and other instance types @ aws) are not explicitly throttled. (Source: AWS Solutions Architect) We're running a portion of our workload at Userify[1] on AWS, but not the biggest portion, but this is actually for bandwidth cost reasons, not CPU (even though our workload is almost entirely CPU and bandwidth -- almost zero disk; our infrastructure at AWS would cost 8x more!) 1. https://userify.com https://userify.com (SSH key management)
    - antouank 11y ago
      I assumed nbench uses all the available cores, that's why I said I'm wrong. On a 1 core basis, yes, it's useful, and I get that this t2 instance is not a stable baseline. I only have that available right now, as my free credit.
  - vmorgulis 11y ago
    > Is there a similar benchmarking linux tool that can use many cores? There is kcbench: http://linux.die.net/man/1/kcbench http://linux.die.net/man/1/kcbench It compiles a kernel.

jamiesonbecker 11y ago

This is helpful but you're not mentioning what instance type at EC2.. t2 and t1 instances do heavy throttling after you use up your CPU credits[1] so it's likely that you were just using all or most of a full Xeon CPU. I got similar results on even a t2.nano.. until I used up the CPU credits. Then it slowed to a crawl! Also keep in mind that nbench only tests a single CPU[2]. Apples to apples would probably be a small m3 or m4 against a comparable DO instance. (but let's not go down that cost disparity rabbit hole.. esp EBS vs SSD or bandwidth!) 1. https://aws.amazon.com/ec2/instance-types/t2/ https://aws.amazon.com/ec2/instance-types/t2/ 2. https://en.wikipedia.org/wiki/NBench#Shortcomings https://en.wikipedia.org/wiki/NBench#Shortcomings

antouank 11y ago
If nbench only uses 1 core, yes, my comparison is wrong. Is there a similar benchmarking linux tool that can use many cores? edit: instance is a t2.micro
- jamiesonbecker 11y ago
  I don't think the comparison is wrong per se; this actually makes it a bit easier, since you're just comparing one core to another. The only thing that I think is problematic is comparing a t2 to anything else, since that CPU performance is not sustainable as it might be elsewhere. m3 and m4 (and other instance types @ aws) are not explicitly throttled. (Source: AWS Solutions Architect) We're running a portion of our workload at Userify[1] on AWS, but not the biggest portion, but this is actually for bandwidth cost reasons, not CPU (even though our workload is almost entirely CPU and bandwidth -- almost zero disk; our infrastructure at AWS would cost 8x more!) 1. https://userify.com https://userify.com (SSH key management)
  - antouank 11y ago
    I assumed nbench uses all the available cores, that's why I said I'm wrong. On a 1 core basis, yes, it's useful, and I get that this t2 instance is not a stable baseline. I only have that available right now, as my free credit.
- vmorgulis 11y ago
  > Is there a similar benchmarking linux tool that can use many cores? There is kcbench: http://linux.die.net/man/1/kcbench http://linux.die.net/man/1/kcbench It compiles a kernel.