3 ms·
The overall presentation convinces me this is a good-faith effort, but such a financial interest in the results would ideally be disclosed prominently up front.
by gojomo 17y ago
The overall presentation convinces me this is a good-faith effort, but such a financial interest in the results would ideally be disclosed prominently up front.
Even with no conscious/devious intent, this sort of interest can bias the results. For example, what if the relative results could vary arbitrarily from week to week (especially possible with the neighboring load on VPS nodes)?
If you get a plausible result that shows the companies pay commissions do best, you are incented to publish right away. If you get a result that shows the non-commission-paying companies on top, you will dig deeper: maybe there are ways the commission-payers are better, in a longer analysis.
(It's similar to the publication bias in other research, which also requires no scheming intent, just natural behavior in response to incentives which are each alone reasonable.)
- pavs 17y agoGenerally I agree with you, on the importance of being skeptical about analysis/benchmark comparison where financial incentives are at sake. IE, some research publications are sponsored by Microsoft who in return publishes glowing review of Internet Explorer 8. However, in this case, the author is not sponsored by or works for any VPS service. He provided affiliated links at the end of the article, where one of the linked service did not prove to be superior than the other according to his analysis. If he sneaked in an affiliated link within the article, the objection and skepticism could be somewhat justified (still I think its somewhat fair game).