4 ms·

CEO of VWO here - a competitor to Optimizely. The article's title "A/B testing is dead" is very misleading. A/B testing is not a tool, it's mental model seekin

CEO of VWO here - a competitor to Optimizely. The article's title "A/B testing is dead" is very misleading. A/B testing is not a tool, it's mental model seeking learnings and improvements. Anytime you introduce a new change to your business, you're implicitly or explicitly trying to understand its impact. Most of the times, setting up an environment to tease out real effects of change is hard (experiments in strategy are usually very difficult) so you're left to guessing. However, for mediums that allow easy tests (like website, landing pages where you can set up scientific experiments with ease and have enough traffic to get results in a short time period), why wouldn't you? The way I see it, rejection of A/B testing is the rejection of the scientific method of conjecture and improvement. In the highly contextual business environment with tens and hundreds of conditional variables, best practices and "gut feeling" can only go so far. A/B testing for very small businesses (those who get attracted to free plans) doesn't make a lot of sense because they usually don't have enough traffic to get statistically significant results. Plus, optimization via A/B testing comes only after you've gotten your basics right and it's now time to scale. This is why at VWO we never had a free plan. So Optimizely's decision to ditch free plan makes sense.

greglindahl 9y ago
Normally I don't upvote "CEO of <competing thing>" but this is a good comment.
TeMPOraL 9y ago
> The way I see it, rejection of A/B testing is the rejection of the scientific method of conjecture and improvement. How much of A/B testing being done is actually scientific? The article touches on that a little bit, but I'm reminded of this gem from 2014: http://blog.sumall.com/journal/optimizely-got-me-fired.html http://blog.sumall.com/journal/optimizely-got-me-fired.html When it was discussed 2 years ago, I teased out[0] (with a dose of cynical attitude) some key points about structural problems with A/B testing tools; problems, which navigate people away from scientific methods, and into the realm of easy-exploitable bullshit artistry. I never had an opportunity to use or check out VWO, so I'd like to ask you: does your tool structurally helps users create sound statistical experiments? Can you share something about how you do it? -- [0] - https://news.ycombinator.com/item?id=10873226 https://news.ycombinator.com/item?id=10873226
- paraschopra 9y ago
  >: does your tool structurally helps users create sound statistical experiments? Can you share something about how you do it? Yes, check out the methodology we use: https://vwo.com/blog/smartstats-testing-for-truth/ https://vwo.com/blog/smartstats-testing-for-truth/
jbreckmckye 9y ago
> A/B testing is not a tool, it's mental model seeking learnings and improvements No, it's a validation phase most businesses use to sign off product decisions that are already made. They invest thousands in a new feature, run an A/B test, and then wait until they see an ambiguous 'win' in favour of the new feature. This is somewhat akin to flipping a coin until you get three heads in a row, and declaring yourself winner of the game. You can denounce this as A/B testing gone wrong, but it's a reality of the web industry that most testing is badly motivated and statistically illiterate. The mentality I describe isn't limited to SMEs and tinpoint organisations, either. I've seen it espoused by highly respected product managers at prestigious organisations. Where does this endemic of pseudoempiricism come from? Well, if I had to guess, I'd point to the marketing of companies like your own. Take a look at https://vwo.com/ab-testing/ https://vwo.com/ab-testing/ - "AB testing - The Complete Guide". There's not one mention of statistical significance either in word or spirit. There's no talk of setting a timeline in advance of the test. There's no hard numbers to set a context of what's an acceptable sample size - in fact the guide is specifically aimed at small business owners. So if anyone is undermining trust in the "scientific process", it's not Optimizely, but in a way - you.
- user5994461 9y ago
  That's not the fault of the tool if people want to throw money at it and ignore the results. Please let people who want to throw money throw money, while the rest of us can use A/B testing to improve our sites. Thank you.
- wastedhours 9y ago
  > No, it's a validation phase most businesses use to sign off product decisions that are already made. Interesting, could you elaborate further on this? I've only really worked with technically unsophisticated (i.e. non-tech companies), and they've always shown willing to do it properly and factor in results, and not set up leading experiments. Is it a symptom of technical companies that staff are more bought-in to their solutions?
- paraschopra 9y ago
  >There's no talk of setting a timeline in advance of the test. There's no hard numbers to set a context of what's an acceptable sample size - in fact the guide is specifically aimed at small business owners. You should take a trial of our product. Before setting up a test in VWO, we ensure you understand that the test will have to be run until we have statistically significant results. We've moved away from frequentist methodologies (where peeking at results was an issue) to bayesian. Check out what we do here: https://vwo.com/blog/smartstats-testing-for-truth/ https://vwo.com/blog/smartstats-testing-for-truth/ The guide is aimed at people who are just starting out, but I agree with you, we should have pointed them to the importance of getting statistically correct results.

greglindahl 9y ago

Normally I don't upvote "CEO of <competing thing>" but this is a good comment.

TeMPOraL 9y ago

> The way I see it, rejection of A/B testing is the rejection of the scientific method of conjecture and improvement. How much of A/B testing being done is actually scientific? The article touches on that a little bit, but I'm reminded of this gem from 2014: http://blog.sumall.com/journal/optimizely-got-me-fired.html http://blog.sumall.com/journal/optimizely-got-me-fired.html When it was discussed 2 years ago, I teased out[0] (with a dose of cynical attitude) some key points about structural problems with A/B testing tools; problems, which navigate people away from scientific methods, and into the realm of easy-exploitable bullshit artistry. I never had an opportunity to use or check out VWO, so I'd like to ask you: does your tool structurally helps users create sound statistical experiments? Can you share something about how you do it? -- [0] - https://news.ycombinator.com/item?id=10873226 https://news.ycombinator.com/item?id=10873226

paraschopra 9y ago
>: does your tool structurally helps users create sound statistical experiments? Can you share something about how you do it? Yes, check out the methodology we use: https://vwo.com/blog/smartstats-testing-for-truth/ https://vwo.com/blog/smartstats-testing-for-truth/

jbreckmckye 9y ago

> A/B testing is not a tool, it's mental model seeking learnings and improvements No, it's a validation phase most businesses use to sign off product decisions that are already made. They invest thousands in a new feature, run an A/B test, and then wait until they see an ambiguous 'win' in favour of the new feature. This is somewhat akin to flipping a coin until you get three heads in a row, and declaring yourself winner of the game. You can denounce this as A/B testing gone wrong, but it's a reality of the web industry that most testing is badly motivated and statistically illiterate. The mentality I describe isn't limited to SMEs and tinpoint organisations, either. I've seen it espoused by highly respected product managers at prestigious organisations. Where does this endemic of pseudoempiricism come from? Well, if I had to guess, I'd point to the marketing of companies like your own. Take a look at https://vwo.com/ab-testing/ https://vwo.com/ab-testing/ - "AB testing - The Complete Guide". There's not one mention of statistical significance either in word or spirit. There's no talk of setting a timeline in advance of the test. There's no hard numbers to set a context of what's an acceptable sample size - in fact the guide is specifically aimed at small business owners. So if anyone is undermining trust in the "scientific process", it's not Optimizely, but in a way - you.

user5994461 9y ago
That's not the fault of the tool if people want to throw money at it and ignore the results. Please let people who want to throw money throw money, while the rest of us can use A/B testing to improve our sites. Thank you.
wastedhours 9y ago
> No, it's a validation phase most businesses use to sign off product decisions that are already made. Interesting, could you elaborate further on this? I've only really worked with technically unsophisticated (i.e. non-tech companies), and they've always shown willing to do it properly and factor in results, and not set up leading experiments. Is it a symptom of technical companies that staff are more bought-in to their solutions?
paraschopra 9y ago
>There's no talk of setting a timeline in advance of the test. There's no hard numbers to set a context of what's an acceptable sample size - in fact the guide is specifically aimed at small business owners. You should take a trial of our product. Before setting up a test in VWO, we ensure you understand that the test will have to be run until we have statistically significant results. We've moved away from frequentist methodologies (where peeking at results was an issue) to bayesian. Check out what we do here: https://vwo.com/blog/smartstats-testing-for-truth/ https://vwo.com/blog/smartstats-testing-for-truth/ The guide is aimed at people who are just starting out, but I agree with you, we should have pointed them to the importance of getting statistically correct results.