4 ms·

That's exactly right. I'm not a big believer in synthetic benchmarks, and I'm not concerned with repeatability of my experiments. The purpose of my test was t

That's exactly right. I'm not a big believer in synthetic benchmarks, and I'm not concerned with repeatability of my experiments. The purpose of my test was to determine, for our specific workload, how much of a difference we would see if we went to dedicated hardware -- and even further, what is the difference between a couple of very specific hardware configurations with that workload. I think it accomplished that nicely.

omouse 17y ago
The purpose of my test was to determine, for our specific workload, Define "specific workload". The workload depends on how many users are using the system, what parts of the website they're hitting (some may require more backend processing than others), etc. By not telling us the specifics and by killing the repeatability of this benchmark, you can only say with a minimal amount of certainty that the dedicated hardware improved performance. I love how the industry functions on magic, whether it's on the software or hardware side.
- teej 17y ago
  > By not telling us the specifics and by killing the repeatability of this benchmark, you can only say with a minimal amount of certainty that the dedicated hardware improved performance. I can agree that this test only has a small amount of certainty that actual performance of the application was higher on the new hardware. So what? It's is possible to use correlated metrics, such as engagement, time on site, and pageviews per user to determine if the move was "successful". 37signals' end goal was most likely to -improve user experience-. If you can confidently say that a valid random sample of users on A servers have a better experience than those on B servers, raw speed numbers don't matter. I guess my point is: Test everything on your own. In the end, it's your users happiness that matters the most, not server speed.
- imbriaco 17y ago
  Synthetic benchmarks are far more subjective than actual production traffic, which is what we used to test. We ran the various configurations in production beside one another and compared the relative performance. Is it possible that the vagaries of load distribution through our load balancers skewed the results? Sure, it's possible but it's not terribly likely with the volume of traffic we receive on a daily basis. In the end, we wanted to see a response time comparison between a few different server configurations, using our production traffic as the load. That's exactly what we ended up with.

omouse 17y ago

The purpose of my test was to determine, for our specific workload, Define "specific workload". The workload depends on how many users are using the system, what parts of the website they're hitting (some may require more backend processing than others), etc. By not telling us the specifics and by killing the repeatability of this benchmark, you can only say with a minimal amount of certainty that the dedicated hardware improved performance. I love how the industry functions on magic, whether it's on the software or hardware side.

teej 17y ago
> By not telling us the specifics and by killing the repeatability of this benchmark, you can only say with a minimal amount of certainty that the dedicated hardware improved performance. I can agree that this test only has a small amount of certainty that actual performance of the application was higher on the new hardware. So what? It's is possible to use correlated metrics, such as engagement, time on site, and pageviews per user to determine if the move was "successful". 37signals' end goal was most likely to -improve user experience-. If you can confidently say that a valid random sample of users on A servers have a better experience than those on B servers, raw speed numbers don't matter. I guess my point is: Test everything on your own. In the end, it's your users happiness that matters the most, not server speed.
imbriaco 17y ago
Synthetic benchmarks are far more subjective than actual production traffic, which is what we used to test. We ran the various configurations in production beside one another and compared the relative performance. Is it possible that the vagaries of load distribution through our load balancers skewed the results? Sure, it's possible but it's not terribly likely with the volume of traffic we receive on a daily basis. In the end, we wanted to see a response time comparison between a few different server configurations, using our production traffic as the load. That's exactly what we ended up with.