4 ms·

Isn't it odd to announce the A/B test when you've just started it ? I mean, I don't think HN readers are in the bingo cards market, but still, it feels like he

Isn't it odd to announce the A/B test when you've just started it ? I mean, I don't think HN readers are in the bingo cards market, but still, it feels like he might be influencing the results by blogging about this beforehand.

paraschopra 16y ago
Somewhat agree with you. I recently blogged about why you shouldn't publicize that you are doing an A/B test http://visualwebsiteoptimizer.com/split-testing-blog/3-controversial-ab-testing-tips-that-you-may-not-agree-with/ http://visualwebsiteoptimizer.com/split-testing-blog/3-contr... Though as you say, the target market and HN crowd isn't exactly overlapping but there may be curious folks who can skew the results or the difference of performance between variations.
JoachimSchipper 16y ago
In what way would this influence things? Sure, it might draw some additional uninterested visitors, but that will just depress conversion rates all around, not exclusively for A or B. The results remain valid.
- paraschopra 16y ago
  Yes but that changes the real difference between conversion rates of A and B. Even though trend won't get affected but actual numbers will. Not that serious but given that this problem is easily avoidable one may want to not announce the test.
- fierarul 16y ago
  Assuming no actual possible customer reads this blog, this doesn't influences the number of new customers, but it does changes the overall numbers. But, if people know they are being A/B tested, this allows them to make a decision about what they think they prefer. Sometimes what people think and are willing to vote (say, by complaining they don't like their option, or by buying a product to support it) isn't what they would have actually picked.
Sukotto 16y ago
The chances of his userbase reading his tech blog and being impacted by something like this seems pretty small to me.
patio11 16y ago
If I woke up in the morning and had a shocking lack of worries, between two software products and client work I could probably find something real to worry about without having to manufacture worries out of the ether. "Several dozen HN users will go to the website, see which variation they get, clear their cookies until they get the right one, then pay me $30... and then cackle madly" is not a real worry.
- fierarul 16y ago
  Again, I don't think blogging about this will influence your A/B test, because your (future) customers and people reading your blog don't overlap very much (or at all). But, I do think publishing the A/B test does influence the outcome if they do overlap. You are talking about the $30 that you could get but miss the point: you might get some extra money but in the process corrupt the validity of your A/B test and make a bad decision afterwards, based on that result. Well, not that it matters anyhow, as I'm pretty sure you have actually decided already to stop doing the Java app. I would like to have somebody that knows more about sociology and statistical surveys explain if what I am saying makes sense or not.
  - ErrantX 16y ago
    The point of the A/B test is that, once devised, it is purely statistics and the software (from here on in) will tell patio11 which of the two options to adopt.
  - jules 16y ago
    > You are talking about the $30 that you could get but miss the point I think you missed his: it's pretty unlikely that anyone on HN will do that.
    - dhimes 16y ago
      Are you sure that only HN readers read the blog?
  - deleted 16y ago
    [deleted]

paraschopra 16y ago

Somewhat agree with you. I recently blogged about why you shouldn't publicize that you are doing an A/B test http://visualwebsiteoptimizer.com/split-testing-blog/3-controversial-ab-testing-tips-that-you-may-not-agree-with/ http://visualwebsiteoptimizer.com/split-testing-blog/3-contr... Though as you say, the target market and HN crowd isn't exactly overlapping but there may be curious folks who can skew the results or the difference of performance between variations.

JoachimSchipper 16y ago

In what way would this influence things? Sure, it might draw some additional uninterested visitors, but that will just depress conversion rates all around, not exclusively for A or B. The results remain valid.

paraschopra 16y ago
Yes but that changes the real difference between conversion rates of A and B. Even though trend won't get affected but actual numbers will. Not that serious but given that this problem is easily avoidable one may want to not announce the test.
fierarul 16y ago
Assuming no actual possible customer reads this blog, this doesn't influences the number of new customers, but it does changes the overall numbers. But, if people know they are being A/B tested, this allows them to make a decision about what they think they prefer. Sometimes what people think and are willing to vote (say, by complaining they don't like their option, or by buying a product to support it) isn't what they would have actually picked.

Sukotto 16y ago

The chances of his userbase reading his tech blog and being impacted by something like this seems pretty small to me.

patio11 16y ago

If I woke up in the morning and had a shocking lack of worries, between two software products and client work I could probably find something real to worry about without having to manufacture worries out of the ether. "Several dozen HN users will go to the website, see which variation they get, clear their cookies until they get the right one, then pay me $30... and then cackle madly" is not a real worry.

fierarul 16y ago
Again, I don't think blogging about this will influence your A/B test, because your (future) customers and people reading your blog don't overlap very much (or at all). But, I do think publishing the A/B test does influence the outcome if they do overlap. You are talking about the $30 that you could get but miss the point: you might get some extra money but in the process corrupt the validity of your A/B test and make a bad decision afterwards, based on that result. Well, not that it matters anyhow, as I'm pretty sure you have actually decided already to stop doing the Java app. I would like to have somebody that knows more about sociology and statistical surveys explain if what I am saying makes sense or not.
- ErrantX 16y ago
  The point of the A/B test is that, once devised, it is purely statistics and the software (from here on in) will tell patio11 which of the two options to adopt.
- jules 16y ago
  > You are talking about the $30 that you could get but miss the point I think you missed his: it's pretty unlikely that anyone on HN will do that.
  - dhimes 16y ago
    Are you sure that only HN readers read the blog?
- deleted 16y ago
  [deleted]