4 ms·

The numbers are confusing and doesn't seem to add up at all. The section "stat definitions" describes data not available in the table below.

KirinDave 14y ago
Part of the problem is that EC2 is such a whacky environment. Notice Erlang's median connection time was tiny but its average had huge outliers. Read the timing numbers with a reasonable portion of salt. It's EC2 we're talking about here.
- tedsuo 14y ago
  Yeah I wince a bit at any benchmarking being run in a shared environment like EC2, where other activity outside of your VM can affect performance.
- StavrosK 14y ago
  Hmm, why would he run a benchmark on a shared environment? Didn't he have a home computer to run this on?
  - wmf 14y ago
    Which is more repeatable but less representative. IMO a non-shared EC2 instance is probably the way to benchmark (if you can drive enough load to saturate it).
    - StavrosK 14y ago
      I agree, but does EC2 have non-shared instances?
      - gojomo 14y ago
        Hmm. Some cloud provider ought to provide (for a premium price) guaranteed-identical non-shared configurations... including small groups of machines with uncontended, identical cross-connects. (I know it's then close to 'dedicated' hosting, but people running benchmarks also want the quick setup and discard of virtualized instances. This would be a hybrid offering that ensures their cloud services is always chosen for such benchmarking comparisons. Of course this offering would not be good for cross-cloud comparisons, because it's not representative of their usual offerings.)
        wmf 14y ago
        guaranteed-identical non-shared configurations... including small groups of machines with uncontended, identical cross-connects. That's called EC2 cluster compute.
      - bartman 14y ago
        On EC2 there are two ways to get a dedicated machine: - Using VPC you can specify that your instance should run on dedicated hardware [1] - Cluster Compute instances [2] most likely (see [3]) run on dedicated hardware too [1] http://aws.amazon.com/dedicated-instances/ http://aws.amazon.com/dedicated-instances/ [2] http://aws.amazon.com/ec2/#instance http://aws.amazon.com/ec2/#instance [3] https://forums.aws.amazon.com/message.jspa?messageID=238197#238197 https://forums.aws.amazon.com/message.jspa?messageID=238197#...
- j2labs 14y ago
  I totally agree, but I'm not sure of any other places that offer such flexibility in pricing and hardware. If you were to conduct a test like this, where would you go for more reliable performance from hardware?
  - KirinDave 14y ago
    I think EC2 is great. It's just whacky.
  - deleted 14y ago
    [deleted]
- caf 14y ago
  This is true, but it's still interesting if you're intending to deploy onto EC2.

KirinDave 14y ago

Part of the problem is that EC2 is such a whacky environment. Notice Erlang's median connection time was tiny but its average had huge outliers. Read the timing numbers with a reasonable portion of salt. It's EC2 we're talking about here.

tedsuo 14y ago
Yeah I wince a bit at any benchmarking being run in a shared environment like EC2, where other activity outside of your VM can affect performance.
StavrosK 14y ago
Hmm, why would he run a benchmark on a shared environment? Didn't he have a home computer to run this on?
- wmf 14y ago
  Which is more repeatable but less representative. IMO a non-shared EC2 instance is probably the way to benchmark (if you can drive enough load to saturate it).
  - StavrosK 14y ago
    I agree, but does EC2 have non-shared instances?
    - gojomo 14y ago
      Hmm. Some cloud provider ought to provide (for a premium price) guaranteed-identical non-shared configurations... including small groups of machines with uncontended, identical cross-connects. (I know it's then close to 'dedicated' hosting, but people running benchmarks also want the quick setup and discard of virtualized instances. This would be a hybrid offering that ensures their cloud services is always chosen for such benchmarking comparisons. Of course this offering would not be good for cross-cloud comparisons, because it's not representative of their usual offerings.)
      - wmf 14y ago
        guaranteed-identical non-shared configurations... including small groups of machines with uncontended, identical cross-connects. That's called EC2 cluster compute.
    - bartman 14y ago
      On EC2 there are two ways to get a dedicated machine: - Using VPC you can specify that your instance should run on dedicated hardware [1] - Cluster Compute instances [2] most likely (see [3]) run on dedicated hardware too [1] http://aws.amazon.com/dedicated-instances/ http://aws.amazon.com/dedicated-instances/ [2] http://aws.amazon.com/ec2/#instance http://aws.amazon.com/ec2/#instance [3] https://forums.aws.amazon.com/message.jspa?messageID=238197#238197 https://forums.aws.amazon.com/message.jspa?messageID=238197#...
j2labs 14y ago
I totally agree, but I'm not sure of any other places that offer such flexibility in pricing and hardware. If you were to conduct a test like this, where would you go for more reliable performance from hardware?
- KirinDave 14y ago
  I think EC2 is great. It's just whacky.
- deleted 14y ago
  [deleted]
caf 14y ago
This is true, but it's still interesting if you're intending to deploy onto EC2.