5 ms·

previously: http://news.ycombinator.com/item?id=2629631 http://news.ycombinator.com/item?id=2629631 Also, "The client as well as the web server tested are host

by robtoo 15y ago

previously: http://news.ycombinator.com/item?id=2629631 http://news.ycombinator.com/item?id=2629631 Also, "The client as well as the web server tested are hosted on the same computer", which is pretty poor design, to be honest.

timc3 15y ago
Yes, a complete waste of time.
skbohra123 15y ago
why it is making to the front page again and again. seems like a foul play.
palish 15y ago
"Doing a correct benchmark is clearly not an easy task. There are many walls (TCP/IP stack, OS settings, the client itself, …) that may corrupt the results, and there is always the risk to compare apples with oranges (e.g. benchmarking the TCP/IP stack instead of the server itself)." By hosting the web server and the client on the same computer, he is testing one aspect in isolation of others. This is a good thing, and is generally the way in which scientific tests advance knowledge.
- robtoo 15y ago
  Except he isn't just testing one aspect in isolation of others. He's actually introducing an entirely new aspect (the client) which is a substantial load and won't be there in production.
  - peterwwillis 15y ago
    the extra load incurred is the same across each test (same client, same args, same tuning) thus the extra load is the same across all tests, so the results are still valid, just not the highest performance possible.
    - robtoo 15y ago
      Complex systems have complex interactions. You can't just just hand-wave this away by claiming that the interactions will be identical for all cases without actually demonstrating it.
      - peterwwillis 15y ago
        Ok, so as the client processes more requests with the server the resource use increases, so in theory the higher the benchmark numbers the faster the server would actually respond without the extra load of the client. So (in theory) the server with the highest performance actually performs better than perceived (assuming that the tester is hitting resource bottlenecks somewhere on his server during the test, which isn't shown). Luckily this benchmark is incredibly simple. It's not a complex system as the test is using a single set of data with two pieces of software in a single contained environment; the only thing that changes is one piece of software and one configuration: the server. Separate the server/client and your test is still the same, only with extra resources for the server and client to take advantage of (and less network bandwidth and higher latency). Knowing how http clients work, and knowing how http servers work, is it possible that the client or server could be utilizing resources in such a different way after being separated as to skew the results in a significant way? I don't believe so. Even if you saturated a 1Gbps network link, you will see differences in CPU time between processing of requests and differences in memory use, and unless they are all fast enough to saturate that link you will see some servers process more requests than others. If you want to verify this you can follow the benchmark's set-up and try on two separate machines and let us know if there's a significant difference.
- bxr 15y ago
  >he is testing one aspect in isolation of others. This is a good thing, and is generally the way in which scientific tests advance knowledge. Yes, maintaining constants and only altering one variable. But this test has an 2nd variable, that is altered into a special state for all of the experimentation, and then we make the leap into assuming that these results are anything other than pretty to look at when the 2nd variable is in any other position. This is a good test for seeing which server runs best with a load testing client competing for resources on the box and using the loopback network. Thats it. The problem is you can't test without a network stack at all, you can only test with a different network stack. Whats to say the particulars of the loopback network aren't causing more issues to be introduced to the validity of the results than the full TCP/IP stack you're trying to avoid influencing the results? Nothing, that's what.
  - palish 15y ago
    I'm happy people have begun to see that scientific tests generally have massive assumptions hidden deep within them, or quietly brushed underneath them. Personally, I find this to be a welcome change from the typical "oooh-look-at-the-shiny-graphs" mentality that has become so pervasive on HN with regards to performance testing. Keep up this spirit of questioning, and you'll discover a lot of interesting secrets. (For example, a placebo is 87% as effective as the 5 major antidepressant brands. Also, the CIE 1931 "color space" is a flawed system which is only roughly accurate.) In general, anyone who presents statistics (of anything) should be subjected to a lot of skepticism. Scientific knowledge can only advance by truly testing one variable and only one variable.

by robtoo 15y ago

timc3 15y ago

Yes, a complete waste of time.

skbohra123 15y ago

why it is making to the front page again and again. seems like a foul play.

palish 15y ago

"Doing a correct benchmark is clearly not an easy task. There are many walls (TCP/IP stack, OS settings, the client itself, …) that may corrupt the results, and there is always the risk to compare apples with oranges (e.g. benchmarking the TCP/IP stack instead of the server itself)." By hosting the web server and the client on the same computer, he is testing one aspect in isolation of others. This is a good thing, and is generally the way in which scientific tests advance knowledge.

robtoo 15y ago
Except he isn't just testing one aspect in isolation of others. He's actually introducing an entirely new aspect (the client) which is a substantial load and won't be there in production.
- peterwwillis 15y ago
  the extra load incurred is the same across each test (same client, same args, same tuning) thus the extra load is the same across all tests, so the results are still valid, just not the highest performance possible.
  - robtoo 15y ago
    Complex systems have complex interactions. You can't just just hand-wave this away by claiming that the interactions will be identical for all cases without actually demonstrating it.
    - peterwwillis 15y ago
      Ok, so as the client processes more requests with the server the resource use increases, so in theory the higher the benchmark numbers the faster the server would actually respond without the extra load of the client. So (in theory) the server with the highest performance actually performs better than perceived (assuming that the tester is hitting resource bottlenecks somewhere on his server during the test, which isn't shown). Luckily this benchmark is incredibly simple. It's not a complex system as the test is using a single set of data with two pieces of software in a single contained environment; the only thing that changes is one piece of software and one configuration: the server. Separate the server/client and your test is still the same, only with extra resources for the server and client to take advantage of (and less network bandwidth and higher latency). Knowing how http clients work, and knowing how http servers work, is it possible that the client or server could be utilizing resources in such a different way after being separated as to skew the results in a significant way? I don't believe so. Even if you saturated a 1Gbps network link, you will see differences in CPU time between processing of requests and differences in memory use, and unless they are all fast enough to saturate that link you will see some servers process more requests than others. If you want to verify this you can follow the benchmark's set-up and try on two separate machines and let us know if there's a significant difference.
bxr 15y ago
>he is testing one aspect in isolation of others. This is a good thing, and is generally the way in which scientific tests advance knowledge. Yes, maintaining constants and only altering one variable. But this test has an 2nd variable, that is altered into a special state for all of the experimentation, and then we make the leap into assuming that these results are anything other than pretty to look at when the 2nd variable is in any other position. This is a good test for seeing which server runs best with a load testing client competing for resources on the box and using the loopback network. Thats it. The problem is you can't test without a network stack at all, you can only test with a different network stack. Whats to say the particulars of the loopback network aren't causing more issues to be introduced to the validity of the results than the full TCP/IP stack you're trying to avoid influencing the results? Nothing, that's what.
- palish 15y ago
  I'm happy people have begun to see that scientific tests generally have massive assumptions hidden deep within them, or quietly brushed underneath them. Personally, I find this to be a welcome change from the typical "oooh-look-at-the-shiny-graphs" mentality that has become so pervasive on HN with regards to performance testing. Keep up this spirit of questioning, and you'll discover a lot of interesting secrets. (For example, a placebo is 87% as effective as the 5 major antidepressant brands. Also, the CIE 1931 "color space" is a flawed system which is only roughly accurate.) In general, anyone who presents statistics (of anything) should be subjected to a lot of skepticism. Scientific knowledge can only advance by truly testing one variable and only one variable.