20 ms·

Except he isn't just testing one aspect in isolation of others. He's actually introducing an entirely new aspect (the client) which is a substantial load and wo

by robtoo 15y ago

Except he isn't just testing one aspect in isolation of others. He's actually introducing an entirely new aspect (the client) which is a substantial load and won't be there in production.

peterwwillis 15y ago
the extra load incurred is the same across each test (same client, same args, same tuning) thus the extra load is the same across all tests, so the results are still valid, just not the highest performance possible.
- robtoo 15y ago
  Complex systems have complex interactions. You can't just just hand-wave this away by claiming that the interactions will be identical for all cases without actually demonstrating it.
  - peterwwillis 15y ago
    Ok, so as the client processes more requests with the server the resource use increases, so in theory the higher the benchmark numbers the faster the server would actually respond without the extra load of the client. So (in theory) the server with the highest performance actually performs better than perceived (assuming that the tester is hitting resource bottlenecks somewhere on his server during the test, which isn't shown). Luckily this benchmark is incredibly simple. It's not a complex system as the test is using a single set of data with two pieces of software in a single contained environment; the only thing that changes is one piece of software and one configuration: the server. Separate the server/client and your test is still the same, only with extra resources for the server and client to take advantage of (and less network bandwidth and higher latency). Knowing how http clients work, and knowing how http servers work, is it possible that the client or server could be utilizing resources in such a different way after being separated as to skew the results in a significant way? I don't believe so. Even if you saturated a 1Gbps network link, you will see differences in CPU time between processing of requests and differences in memory use, and unless they are all fast enough to saturate that link you will see some servers process more requests than others. If you want to verify this you can follow the benchmark's set-up and try on two separate machines and let us know if there's a significant difference.

by robtoo 15y ago

Except he isn't just testing one aspect in isolation of others. He's actually introducing an entirely new aspect (the client) which is a substantial load and won't be there in production.

peterwwillis 15y ago

the extra load incurred is the same across each test (same client, same args, same tuning) thus the extra load is the same across all tests, so the results are still valid, just not the highest performance possible.

robtoo 15y ago
Complex systems have complex interactions. You can't just just hand-wave this away by claiming that the interactions will be identical for all cases without actually demonstrating it.
- peterwwillis 15y ago
  Ok, so as the client processes more requests with the server the resource use increases, so in theory the higher the benchmark numbers the faster the server would actually respond without the extra load of the client. So (in theory) the server with the highest performance actually performs better than perceived (assuming that the tester is hitting resource bottlenecks somewhere on his server during the test, which isn't shown). Luckily this benchmark is incredibly simple. It's not a complex system as the test is using a single set of data with two pieces of software in a single contained environment; the only thing that changes is one piece of software and one configuration: the server. Separate the server/client and your test is still the same, only with extra resources for the server and client to take advantage of (and less network bandwidth and higher latency). Knowing how http clients work, and knowing how http servers work, is it possible that the client or server could be utilizing resources in such a different way after being separated as to skew the results in a significant way? I don't believe so. Even if you saturated a 1Gbps network link, you will see differences in CPU time between processing of requests and differences in memory use, and unless they are all fast enough to saturate that link you will see some servers process more requests than others. If you want to verify this you can follow the benchmark's set-up and try on two separate machines and let us know if there's a significant difference.