4 ms·
I've personally never been in a situation where analytics drove change. Even mere A/B testing seems like a ghost in (most?) organizations, where management or d
by mighmi 3y ago
I've personally never been in a situation where analytics drove change. Even mere A/B testing seems like a ghost in (most?) organizations, where management or designers decide to force through their personal preferences.
That said, I would love to join such an organization. Any suggestions on where to find data driven jobs?
- dstroot 3y ago> Any suggestions where to find data driven jobs? My experiences suggest a large measure of caution. Many organizations describe themselves as data-driven and many actually do try. However humans are humans and many will be “data driven” when it aligns with what they want to do, otherwise it’s off for “more data” or a “new perspective”. In other words sometimes data driven means use data when it supports my already staked out position. I hate to be so cynical but I can’t help it.
- exolymph 3y agoOne might argue that your cynicism is (drum roll please) data driven.
- tensor 3y agoIt may be less used in driving website design, but I assure you it is very very widely used in marketing to determine which campaigns are working.
- dustymcp 3y agoAmazon?
- fmajid 3y agoThe second half of this article explains Amazon's process around metrics, well worth a read: https://commoncog.com/goodharts-law-not-useful/ https://commoncog.com/goodharts-law-not-useful/
- nine_k 3y agoAt my previous job, A/B testing was widely used to check which of the small UX tweaks works better.
- exodust 3y agoThe problem there is many uncontrollable factors contributing to the data. If engagement is found to be better with A on the day, does that mean B will never be better? It doesn't. A/B testing is often conducted unscientifically or with insufficient sample size and timeframe. Sometimes the cards fall where they fall, and your small UX tweak wasn't involved. It's tempting to conclude A/B testing delivered a valid result. As an experiment, if you deliberately made A and B identical, then A/B test as normal, you will still get a winner. They won't be equal.
- nine_k 3y agoThis is fair; valid experiments need to run for weeks, not hours, to account for cyclic variations, should not be confounded by some events (like holidays, elections, etc) that severely change relevant user activity, etc, and need to bring large enough amounts of data points for the results to be able to be statistically significant.
- gfodor 3y agoWell yeah you have to actually do it right.