3 ms·

Given how common it is for production applications to be running in the cloud, this test is both valid and useful. Therefore these tests are much more meaningfu

by teilo 17y ago

Given how common it is for production applications to be running in the cloud, this test is both valid and useful. Therefore these tests are much more meaningful than the first.

jacquesm 17y ago
I disagree. If you run a test on a virtual machine then you are testing a lot of other software besides the software under test so the picture gets distorted. For instance, in the graphs that are in the report you can see a bunch of drops to '0' traffic, these suggest serious problems with the software under test. But knowing that the tests were done in a VM environment it is possible that the cause of these drops is not in the software under test but with the VM, or alternatively, with other software running on the host the VM is running on. That makes it very difficult to assign meaning to all this. If you test software you do it in an environment that controls all the variables as much as possible otherwise the results will not help you in making decisions. If people run software 'in the cloud' then that's a different story altogether, then you test the same software side-by-side on your bare machine vs your 'in the cloud' setup and you benchmark those. This test is labelled 'twister vs tornado', not 'twister in the cloud' vs 'tornado in the cloud'.
- bmelton 17y ago
  But if you're testing them both in a VM, wouldn't you consider this a valid performance benchmark for those looking to run their instance in a cloud VM? I mean, I think I get your point, in that this benchmark doesn't apply to those running bare metal... but isn't the obvious counterpoint that this is an especially useful benchmark for those running in a VM?
  - jacquesm 17y ago
    If you compare two things running on a VM you have to have a play-by-play idea of what else is running on the machine the VM is running on otherwise your test could be seriously affected by something outside your view.
- teilo 17y ago
  If the tests were all over the map, I could agree with your original statement. But they are not. They are consistent. Do you honestly expect anyone to believe that when test after test is run on a VM, and one framework consistently outperforms another, that this is due to the VM, and not the frameworks themselves?
  - jacquesm 17y ago
    I'm more concerned with the drops to 0 and the overall throughput. The drops to 0 look quite bad, overall throughput is low compared to what I would expect from a setup like this.

by teilo 17y ago

Given how common it is for production applications to be running in the cloud, this test is both valid and useful. Therefore these tests are much more meaningful than the first.

jacquesm 17y ago

I disagree. If you run a test on a virtual machine then you are testing a lot of other software besides the software under test so the picture gets distorted. For instance, in the graphs that are in the report you can see a bunch of drops to '0' traffic, these suggest serious problems with the software under test. But knowing that the tests were done in a VM environment it is possible that the cause of these drops is not in the software under test but with the VM, or alternatively, with other software running on the host the VM is running on. That makes it very difficult to assign meaning to all this. If you test software you do it in an environment that controls all the variables as much as possible otherwise the results will not help you in making decisions. If people run software 'in the cloud' then that's a different story altogether, then you test the same software side-by-side on your bare machine vs your 'in the cloud' setup and you benchmark those. This test is labelled 'twister vs tornado', not 'twister in the cloud' vs 'tornado in the cloud'.

bmelton 17y ago
But if you're testing them both in a VM, wouldn't you consider this a valid performance benchmark for those looking to run their instance in a cloud VM? I mean, I think I get your point, in that this benchmark doesn't apply to those running bare metal... but isn't the obvious counterpoint that this is an especially useful benchmark for those running in a VM?
- jacquesm 17y ago
  If you compare two things running on a VM you have to have a play-by-play idea of what else is running on the machine the VM is running on otherwise your test could be seriously affected by something outside your view.
teilo 17y ago
If the tests were all over the map, I could agree with your original statement. But they are not. They are consistent. Do you honestly expect anyone to believe that when test after test is run on a VM, and one framework consistently outperforms another, that this is due to the VM, and not the frameworks themselves?
- jacquesm 17y ago
  I'm more concerned with the drops to 0 and the overall throughput. The drops to 0 look quite bad, overall throughput is low compared to what I would expect from a setup like this.