5 ms·
Except that the author claims it's a statistically significant result immediately before reporting the 1% number. The full phrase (which was truncated in the ot
by jonas21 8y ago
Except that the author claims it's a statistically significant result immediately before reporting the 1% number. The full phrase (which was truncated in the other comment) is:
"we were able to statistically verify a -1% negative impact on core business metrics including conversions"
- deleted 8y ago[deleted]
- jbreckmckye 8y agoEven if so: the relationship between page conversion and revenue is not direct. You don't know whether the lost leads were high quality, nor whether they were reacquired through other means. Ultimately though the issue is more fundamental: you can't make assertions about performance without publishing actual figures. All we have to go on is the author's assertion that the test methodology was sound and the results were out of the bounds of chance. That's not really enough. I have worked with many, many teams who earnestly believed they were running "scientific" MVTs, with solid statistical analysis free of bias. They weren't.