4 ms·

When you knock 4 other companies and promote one, then make money off people buying the service based on your analysis, you create an impression of being non-ob

by hop 17y ago

When you knock 4 other companies and promote one, then make money off people buying the service based on your analysis, you create an impression of being non-objective.

pavs 17y ago
He is not giving you his opinion, he is giving you results of his benchmark. You can even download logs to check and run the patched benchmark util yourself to compare. I (and most others hopefully) already knew about prgmr and EC2 has horrible performance, even before I saw this benchmark, the only one new to me is Rackspace performance numbers (BTW Rackspace owns slicehost). I don't see any logic behind your objection.
- gojomo 17y ago
  The overall presentation convinces me this is a good-faith effort, but such a financial interest in the results would ideally be disclosed prominently up front. Even with no conscious/devious intent, this sort of interest can bias the results. For example, what if the relative results could vary arbitrarily from week to week (especially possible with the neighboring load on VPS nodes)? If you get a plausible result that shows the companies pay commissions do best, you are incented to publish right away. If you get a result that shows the non-commission-paying companies on top, you will dig deeper: maybe there are ways the commission-payers are better, in a longer analysis. (It's similar to the publication bias in other research, which also requires no scheming intent, just natural behavior in response to incentives which are each alone reasonable.)
  - pavs 17y ago
    Generally I agree with you, on the importance of being skeptical about analysis/benchmark comparison where financial incentives are at sake. IE, some research publications are sponsored by Microsoft who in return publishes glowing review of Internet Explorer 8. However, in this case, the author is not sponsored by or works for any VPS service. He provided affiliated links at the end of the article, where one of the linked service did not prove to be superior than the other according to his analysis. If he sneaked in an affiliated link within the article, the objection and skepticism could be somewhat justified (still I think its somewhat fair game).

by hop 17y ago

When you knock 4 other companies and promote one, then make money off people buying the service based on your analysis, you create an impression of being non-objective.

pavs 17y ago

He is not giving you his opinion, he is giving you results of his benchmark. You can even download logs to check and run the patched benchmark util yourself to compare. I (and most others hopefully) already knew about prgmr and EC2 has horrible performance, even before I saw this benchmark, the only one new to me is Rackspace performance numbers (BTW Rackspace owns slicehost). I don't see any logic behind your objection.

gojomo 17y ago
The overall presentation convinces me this is a good-faith effort, but such a financial interest in the results would ideally be disclosed prominently up front. Even with no conscious/devious intent, this sort of interest can bias the results. For example, what if the relative results could vary arbitrarily from week to week (especially possible with the neighboring load on VPS nodes)? If you get a plausible result that shows the companies pay commissions do best, you are incented to publish right away. If you get a result that shows the non-commission-paying companies on top, you will dig deeper: maybe there are ways the commission-payers are better, in a longer analysis. (It's similar to the publication bias in other research, which also requires no scheming intent, just natural behavior in response to incentives which are each alone reasonable.)
- pavs 17y ago
  Generally I agree with you, on the importance of being skeptical about analysis/benchmark comparison where financial incentives are at sake. IE, some research publications are sponsored by Microsoft who in return publishes glowing review of Internet Explorer 8. However, in this case, the author is not sponsored by or works for any VPS service. He provided affiliated links at the end of the article, where one of the linked service did not prove to be superior than the other according to his analysis. If he sneaked in an affiliated link within the article, the objection and skepticism could be somewhat justified (still I think its somewhat fair game).