3 ms·
Without a breakdown of the stats and methodology, we have no way to know if this result was statistically meaningful, nor whether Trivago ran any null hypothesi
by jbreckmckye 8y ago
Without a breakdown of the stats and methodology, we have no way to know if this result was statistically meaningful, nor whether Trivago ran any null hypothesis tests. We don't even know that there were no confounding factors - just none the author thought of.
The may be a spurious figure.
- jonas21 8y agoExcept that the author claims it's a statistically significant result immediately before reporting the 1% number. The full phrase (which was truncated in the other comment) is: "we were able to statistically verify a -1% negative impact on core business metrics including conversions"
- deleted 8y ago[deleted]
- jbreckmckye 8y agoEven if so: the relationship between page conversion and revenue is not direct. You don't know whether the lost leads were high quality, nor whether they were reacquired through other means. Ultimately though the issue is more fundamental: you can't make assertions about performance without publishing actual figures. All we have to go on is the author's assertion that the test methodology was sound and the results were out of the bounds of chance. That's not really enough. I have worked with many, many teams who earnestly believed they were running "scientific" MVTs, with solid statistical analysis free of bias. They weren't.