7 ms·

Why You Don't A/B Test, and How You Can Start This August

119 points by dangrossman 13y ago

ronaldx 13y ago
> It was a preventable accident. Was it really preventable? How should we know when to make a quick decision and when to A/B test? [Clarification: it seems to me that it's better to decisively get things done] Considering that the most widely known A/B tests are based on 41 shades of blue and pixel-perfection, I'm not sure that this is as obvious as the article claims.
- patio11 13y ago
  the most widely known A/B tests are based on 41 shades of blue and pixel-perfection This is not true among people who actually A/B test for a living. (The 41 shades of blue thing is a test cherry picked to suggest that testing is not material. The only reason the world knows about that test, as opposed to others conducted by Google/MSN, is because someone who believed they didn't fit in a culture of testing called out that test as the reductio ad absurdum of that culture.) Without saying exactly what it was that the client didn't know at the time, suffice it to say that if you got five A/B testing practitioners in a room and asked them for the top five things to try on that client's site, every last one would have listed the problematic area as something to test. I mean, it wasn't the H1 on the front page, but it could have been. This is similar to "How do we make our pages load faster?" Are there large amounts of subjectivity and risk involved here? Yes. Trying to outguess your favorite SQL query optimizer sometimes feels like reading chicken entrails. But, if you're not using gzip yet, then you should turn on gzip, because gzip always wins.
  - ronaldx 13y ago
    I accept that you are an expert and that there is value in this expertise, but I remain unconvinced that it's so obvious where to execute A/B tests (in contrast to page load time, where bottlenecks can be measurably identified).
patio11 13y ago
If you have any questions, feel free to ask.
- pisarzp 13y ago
  Very often your current users get used to your design and hate it when you change something. Is there a way to do A/B testing on your current users, or should you try it only on new ones?
  - patio11 13y ago
    You can, theoretically, only A/B test things against new users. Most A/B testing tools do not support this behavior out of the box. The one time I made a change which was drastic enough to consider doing that, I just put a revert button on the interface (on both sides) and got ready to tell people how to use it. It turns out nobody asked. In general, though, many of the things which are most valuable to test border on imperceptible for long-term users of the site. For example, do you use Dropbox? (Picking a well-known example locally.) Can you identify the H1 on dropbox.com? Can you identify the button copy on dropbox.com? Most seasoned users can't, and won't notice changes to these elements, yet they're strongly influential on free trial signups. Can you freehand sketch the Dropbox credit card form? How about identifying the button copy on it? These have substantial impact on purchases at the margins. People only see them once (well, typical case for software), and nobody remembers them for more than a few minutes. Think back to your first run experience with Dropbox. How many steps did it have? What were their names? I'm going to bet you that the onboarding process for Dropbox has had more dedicated optimization effort than everything else the company does combined, but the median number of exposures per user is one.
    - ampersandy 13y ago
      Great insight! Though I'd expand on your bet and claim that at almost every major site, a majority of testing is channeled into onboarding. It's certainly the case at Twitter and Facebook that a lot of work goes into optimizing a user's first steps.
- 13y ago
noelwelsh 13y ago
I'm interested to see the sample size calculations. Those numbers don't jibe with any calculations I've ever done.
- throwawayg99 13y ago
  Nor me, to collect a 1% change: > power.prop.test(p1=0.1, p2=0.11, power=0.8,sig.level=0.05) Two-sample comparison of proportions power calculation n = 14750.79 p1 = 0.1 p2 = 0.11 sig.level = 0.05 power = 0.8 alternative = two.sided NOTE: n is number in *each* group Edit: so that's 30,000 samples. To do the same with a 5% change in the same area is still 1300 samples (alpha=0.8, p=0.05)
  - noelwelsh 13y ago
    I normally consider relative minimum discernable effect. Your 5% absolute change is a 50% increase is the base rate. I also typically go for a higher power (e.g. 0.9). Under these conditions 60K samples is more typical. Sample size calculator here: http://www.evanmiller.org/ab-testing/sample-size.html http://www.evanmiller.org/ab-testing/sample-size.html You'll need to change the defaults to match the above to get the figures I mention.
    - throwawayg99 13y ago
      You're right. That's a bit more reasonable, so back to my 10% change in base rate (1% abs.) but with a 90% power: > power.prop.test(p1=0.1, p2=0.11, power=0.9,sig.level=0.05) Two-sample comparison of proportions power calculation n = 19746.62 p1 = 0.1 p2 = 0.11 sig.level = 0.05 power = 0.9 alternative = two.sided NOTE: n is number in *each* group Requires about 40,000 samples per test. I would strongly recommend anyone serious about doing this look in to MAB testing, as A-B testing is way too expensive for reasonable scale testing (unless you have a strong a priori hypothesis to test).
  - deleted 13y ago
    [deleted]
GhotiFish 13y ago
One thing that unnerves me about A/B testing is that you can't test for evilness. How much of this glossary[1] do you think is intentionally designed? I doubt that all of these changes were intentional, it's just that those changes showed objective results. A/B testing can't test for morality, and you may very well be implying something with your B design that you didn't mean to imply, which none the less rates higher in your test. In a business setting, it becomes awfully hard to argue morality against objective numbers. It's hard to do that anyway. Many business operate with a profit first motive. So once a dark pattern is in, how are you going to get it out? Don't get me wrong, A/B testing is a tool, and like all good tools it can be used for good or evil. It just worries me that A/B testing, despite good intentions, can lead to evil results. I don't see anything on this page about how to avoid evil results. 1. http://darkpatterns.org/ http://darkpatterns.org/
- wikwocket 13y ago
  I can see how you don't want to be in a situation where you are holding in your hand evidence that Shady Marketing Tactic X is proven to convert better. But I think the solution is simple: Don't A/B test shady features. If something makes you uncomfortable or seems like a dark pattern, don't test it, because you would never want it on your site anyway. I don't think this is a failing of A/B testing. That someone can get numbers that show "evil" tactics can make money is not materially different from someone coding "evil" HTML in Notepad and getting results. As you said, it is just a tool.
  - GhotiFish 13y ago
    The problem is you might not be aware you're making a shady feature. As an example, if you are making an ad for your app fooer. You A/B test some banners on an ad distribution network. You find that a simpler ad with a "download now" hyperlink graphic proves effective. No big deal right? People are going to be seeing these banners on places like news sites right? It's not tricking people. It's also pretty reasonable right? People are used to clicking on hyperlinks, it overcomes the resistance to click on images. A reasonable move for a banner ad to make. Except the ad distribution network serves content sites like softpedia as well. You saw the spike because you emulated the download button right under the mirror list, and people were confused. They downloaded your application under the pretext of getting something else. Or what about if I want a "Hey you should sign up for notifications for other job offerings in this field" message box? I'd love to have one of those in my application. I A/B test to see that this message box drastically increases signups. Except it was because people thought it was a paywall. The road to hell is paved with good intentions. Consider this case about the Weebly blogging service: http://minimaxir.com/2013/05/overly-attached-startup/ http://minimaxir.com/2013/05/overly-attached-startup/ They were A/B testing messaging for first week signups. This hyper aggressive and insane route was actually considered. Do you think they're black hat? I don't think so, but what if this had become standard practice?
noelwelsh 13y ago
I hope no-one minds me touting a free course I'm offering on bandit algorithms: http://noelwelsh.com/data/2013/08/05/why-you-should-know-about-bandit-algorithms/ http://noelwelsh.com/data/2013/08/05/why-you-should-know-abo... I was literally working on the post when Patrick's email arrived. It's more focused on the algorithms and will, I think, be a good compliment to what Patrick is offering.
- patio11 13y ago
  I upvoted you from zero, in the interest of HN remaining a supportive and congenial place for people who are doing stuff to move the industry's state of play forward.
  - noelwelsh 13y ago
    Thanks! I appreciate your support, especially given you don't agree with me on the benefits of bandit testing.
- badclient 13y ago
  I signed up as someone who has a deep need for this. We are trying to run through 4 months of cold calling data to identify the ideal customer profile / call frequency etc. that has the highest odds of converting into a sale. I vaguely remember learning some of it in my Marketing Research class but have since forgotten. From your post it sounded like a free online course but it seems like the free course will be presented in person? I wish you had a more formal online offering.
  - noelwelsh 13y ago
    The free course is online. Let me give the backstory, which I hope will clarify things: I'm giving a talk at Strata London (a "Big Data" conference) in a few months. I won't be able to go into details for the talk, so I want to create supplementary material. I seems sharing this material online will be good for others (and for me). I don't want to plonk it all on the web because 1) I don't want to make all my material available till after the talk and 2) I want to build a mailing list of people interested in this kind of stuff. I may build the material into a more formal course if there is sufficient interest.
rsync 13y ago
I know exactly why we[1] don't A/B test. It is because I don't want to serve the people that it would be effective on. If you're marginally buying / not buying my product based on color scheme or (buzz)word order or some other piece of puff, we're all better served if you just moved along. [1] You know who we are.
- baddox 13y ago
  Is your desire to only serve certain people based in the long run on profit motive (i.e. you think your fewer customers will be more valuable and stick around longer), or is it just out of principle even if it means less profit in the long run?
  - rsync 13y ago
    It is done out of principle.
- runako 13y ago
  Just a thought -- this POV implies that there is a single way to explain every piece of information on your site that is optimal for each person who might be a customer. (The corollary is that all your customers are the same.) A person doesn't have to be swayed by a buzzword, color scheme, or fluff to respond to different presentation of facts. You know this is true if you've ever had to explain anything to anyone. Some examples of how different presentations of your content might affect an individual: - English: first language? - Country/region of origin? - What did they read before coming to your site? A lot of things influence human behavior, and A/B testing isn't just to tease out the fluffy bits you can use to sell to the gullible.
- sethev 13y ago
  How do you know your current design is excluding the right people? The people who are buying your product now might just be reacting to your current color scheme (or lack thereof) or word order.
  - pc86 13y ago
shadeless 13y ago
The bit about 37signals reminded me of their timelapse video, "evolution of a homepage"[1], which shows just how many things they try before sticking to something (albeit shortly). [1] http://vimeo.com/29088090 http://vimeo.com/29088090
thenomad 13y ago
All very good stuff, but this line stood out in particular: "Video makes Waterfall software development look like bugs-in-your-teeth speed, though" As someone who is coming to the close of a 4-year work stint on a short film, I can tell you, you ain't wrong. Indeed, it has gotten to the point that I've started a computer game design project as light relief.
viggity 13y ago
what the hell is S/J testing? never heard of it and apparently neither has the googles.
- dangrossman 13y ago
  Steve Jobs.
qeorge 13y ago
If you're looking for things to A/B test, please check out my post: https://news.ycombinator.com/item?id=6163397 https://news.ycombinator.com/item?id=6163397 It details 19 A/B test ideas that have worked for us in the past. I hope HN finds it useful!
Silhouette 13y ago
I've now started flagging these kinds of submissions on HN, for the simple reason that they are little more than ads for something vaguely specified that may or may not even exist yet. There is absolutely no directly actionable content in this article that I can see, nor any other original ideas that anyone following HN doesn't see several times a week on the front page alone. I post this only so I can note that this is not intended as a criticism of Patrick, just a view that the hero worship and endless reposting of A/B evangelism has far overstayed its welcome IMHO. If people want to read the fluff as well, then as we get told every five minutes they can sign up for Patrick's newsletter. Please, for the love of all that is holy, spare those of us who have chosen not to do so the endless join-my-mailing-list spam that is starting to dominate HN, and save the HN submissions for substantial content.
- gus_massa 13y ago
  This email is more infomercial than the emails from the past weeks. I reread the last two emails and I think they are more interesting, but this got much more upvotes here. Anyway, this email convinced me to try to run an A/B test. I have a button that I don't like and I got negative feedback about that button in the past. So I will (probably) try to optimize the microcopy, but I must understand and change the php code that renders that button. I'll report the result when the experiment finish. Wish me good luck!
- wakeless 13y ago
  It's worth mentioning that that's exactly why Patrick doesn't normally submit this stuff himself. Infact, I've seen him apologies/ask people not to just re-post stuff here and just subscribe to his newsletter/blog/etc. It's also worth noting that you don't HAVE to click through to something that's on the kalzumeus domain, if you subscribe to the newsletter or don't want to. Personally, I think anything that generates a reasonable number of good comments is great (not necessarily this.) That's why HN is great.
- davidw 13y ago
  Flagging is for stuff that's inappropriate for this site, like political articles, the latest injustice, politics, news stories, conspiracy theories and so on. For stuff like this that you may not happen to like, just don't vote for it, and ignore it. I tend not to vote up patio11's stuff these days even though it's good, just because it gets so much attention. But it is 100% on topic for this site and does not deserve to be flagged. How much time has patio11 spent giving free advice and information here? So what if he also sells some stuff? That's what businesses do, sheez!

119 points by dangrossman 13y ago

> It was a preventable accident. Was it really preventable? How should we know when to make a quick decision and when to A/B test? [Clarification: it seems to me that it's better to decisively get things done] Considering that the most widely known A/B tests are based on 41 shades of blue and pixel-perfection, I'm not sure that this is as obvious as the article claims.

patio11 13y ago
the most widely known A/B tests are based on 41 shades of blue and pixel-perfection This is not true among people who actually A/B test for a living. (The 41 shades of blue thing is a test cherry picked to suggest that testing is not material. The only reason the world knows about that test, as opposed to others conducted by Google/MSN, is because someone who believed they didn't fit in a culture of testing called out that test as the reductio ad absurdum of that culture.) Without saying exactly what it was that the client didn't know at the time, suffice it to say that if you got five A/B testing practitioners in a room and asked them for the top five things to try on that client's site, every last one would have listed the problematic area as something to test. I mean, it wasn't the H1 on the front page, but it could have been. This is similar to "How do we make our pages load faster?" Are there large amounts of subjectivity and risk involved here? Yes. Trying to outguess your favorite SQL query optimizer sometimes feels like reading chicken entrails. But, if you're not using gzip yet, then you should turn on gzip, because gzip always wins.
- ronaldx 13y ago
  I accept that you are an expert and that there is value in this expertise, but I remain unconvinced that it's so obvious where to execute A/B tests (in contrast to page load time, where bottlenecks can be measurably identified).

patio11 13y ago

If you have any questions, feel free to ask.

pisarzp 13y ago
Very often your current users get used to your design and hate it when you change something. Is there a way to do A/B testing on your current users, or should you try it only on new ones?
- patio11 13y ago
  You can, theoretically, only A/B test things against new users. Most A/B testing tools do not support this behavior out of the box. The one time I made a change which was drastic enough to consider doing that, I just put a revert button on the interface (on both sides) and got ready to tell people how to use it. It turns out nobody asked. In general, though, many of the things which are most valuable to test border on imperceptible for long-term users of the site. For example, do you use Dropbox? (Picking a well-known example locally.) Can you identify the H1 on dropbox.com? Can you identify the button copy on dropbox.com? Most seasoned users can't, and won't notice changes to these elements, yet they're strongly influential on free trial signups. Can you freehand sketch the Dropbox credit card form? How about identifying the button copy on it? These have substantial impact on purchases at the margins. People only see them once (well, typical case for software), and nobody remembers them for more than a few minutes. Think back to your first run experience with Dropbox. How many steps did it have? What were their names? I'm going to bet you that the onboarding process for Dropbox has had more dedicated optimization effort than everything else the company does combined, but the median number of exposures per user is one.
  - ampersandy 13y ago
    Great insight! Though I'd expand on your bet and claim that at almost every major site, a majority of testing is channeled into onboarding. It's certainly the case at Twitter and Facebook that a lot of work goes into optimizing a user's first steps.
13y ago

noelwelsh 13y ago

I'm interested to see the sample size calculations. Those numbers don't jibe with any calculations I've ever done.

throwawayg99 13y ago
Nor me, to collect a 1% change: > power.prop.test(p1=0.1, p2=0.11, power=0.8,sig.level=0.05) Two-sample comparison of proportions power calculation n = 14750.79 p1 = 0.1 p2 = 0.11 sig.level = 0.05 power = 0.8 alternative = two.sided NOTE: n is number in *each* group Edit: so that's 30,000 samples. To do the same with a 5% change in the same area is still 1300 samples (alpha=0.8, p=0.05)
- noelwelsh 13y ago
  I normally consider relative minimum discernable effect. Your 5% absolute change is a 50% increase is the base rate. I also typically go for a higher power (e.g. 0.9). Under these conditions 60K samples is more typical. Sample size calculator here: http://www.evanmiller.org/ab-testing/sample-size.html http://www.evanmiller.org/ab-testing/sample-size.html You'll need to change the defaults to match the above to get the figures I mention.
  - throwawayg99 13y ago
    You're right. That's a bit more reasonable, so back to my 10% change in base rate (1% abs.) but with a 90% power: > power.prop.test(p1=0.1, p2=0.11, power=0.9,sig.level=0.05) Two-sample comparison of proportions power calculation n = 19746.62 p1 = 0.1 p2 = 0.11 sig.level = 0.05 power = 0.9 alternative = two.sided NOTE: n is number in *each* group Requires about 40,000 samples per test. I would strongly recommend anyone serious about doing this look in to MAB testing, as A-B testing is way too expensive for reasonable scale testing (unless you have a strong a priori hypothesis to test).
- deleted 13y ago
  [deleted]

GhotiFish 13y ago

One thing that unnerves me about A/B testing is that you can't test for evilness. How much of this glossary[1] do you think is intentionally designed? I doubt that all of these changes were intentional, it's just that those changes showed objective results. A/B testing can't test for morality, and you may very well be implying something with your B design that you didn't mean to imply, which none the less rates higher in your test. In a business setting, it becomes awfully hard to argue morality against objective numbers. It's hard to do that anyway. Many business operate with a profit first motive. So once a dark pattern is in, how are you going to get it out? Don't get me wrong, A/B testing is a tool, and like all good tools it can be used for good or evil. It just worries me that A/B testing, despite good intentions, can lead to evil results. I don't see anything on this page about how to avoid evil results. 1. http://darkpatterns.org/ http://darkpatterns.org/

wikwocket 13y ago
I can see how you don't want to be in a situation where you are holding in your hand evidence that Shady Marketing Tactic X is proven to convert better. But I think the solution is simple: Don't A/B test shady features. If something makes you uncomfortable or seems like a dark pattern, don't test it, because you would never want it on your site anyway. I don't think this is a failing of A/B testing. That someone can get numbers that show "evil" tactics can make money is not materially different from someone coding "evil" HTML in Notepad and getting results. As you said, it is just a tool.
- GhotiFish 13y ago
  The problem is you might not be aware you're making a shady feature. As an example, if you are making an ad for your app fooer. You A/B test some banners on an ad distribution network. You find that a simpler ad with a "download now" hyperlink graphic proves effective. No big deal right? People are going to be seeing these banners on places like news sites right? It's not tricking people. It's also pretty reasonable right? People are used to clicking on hyperlinks, it overcomes the resistance to click on images. A reasonable move for a banner ad to make. Except the ad distribution network serves content sites like softpedia as well. You saw the spike because you emulated the download button right under the mirror list, and people were confused. They downloaded your application under the pretext of getting something else. Or what about if I want a "Hey you should sign up for notifications for other job offerings in this field" message box? I'd love to have one of those in my application. I A/B test to see that this message box drastically increases signups. Except it was because people thought it was a paywall. The road to hell is paved with good intentions. Consider this case about the Weebly blogging service: http://minimaxir.com/2013/05/overly-attached-startup/ http://minimaxir.com/2013/05/overly-attached-startup/ They were A/B testing messaging for first week signups. This hyper aggressive and insane route was actually considered. Do you think they're black hat? I don't think so, but what if this had become standard practice?

noelwelsh 13y ago

I hope no-one minds me touting a free course I'm offering on bandit algorithms: http://noelwelsh.com/data/2013/08/05/why-you-should-know-about-bandit-algorithms/ http://noelwelsh.com/data/2013/08/05/why-you-should-know-abo... I was literally working on the post when Patrick's email arrived. It's more focused on the algorithms and will, I think, be a good compliment to what Patrick is offering.

patio11 13y ago
I upvoted you from zero, in the interest of HN remaining a supportive and congenial place for people who are doing stuff to move the industry's state of play forward.
- noelwelsh 13y ago
  Thanks! I appreciate your support, especially given you don't agree with me on the benefits of bandit testing.
badclient 13y ago
I signed up as someone who has a deep need for this. We are trying to run through 4 months of cold calling data to identify the ideal customer profile / call frequency etc. that has the highest odds of converting into a sale. I vaguely remember learning some of it in my Marketing Research class but have since forgotten. From your post it sounded like a free online course but it seems like the free course will be presented in person? I wish you had a more formal online offering.
- noelwelsh 13y ago
  The free course is online. Let me give the backstory, which I hope will clarify things: I'm giving a talk at Strata London (a "Big Data" conference) in a few months. I won't be able to go into details for the talk, so I want to create supplementary material. I seems sharing this material online will be good for others (and for me). I don't want to plonk it all on the web because 1) I don't want to make all my material available till after the talk and 2) I want to build a mailing list of people interested in this kind of stuff. I may build the material into a more formal course if there is sufficient interest.

rsync 13y ago

I know exactly why we[1] don't A/B test. It is because I don't want to serve the people that it would be effective on. If you're marginally buying / not buying my product based on color scheme or (buzz)word order or some other piece of puff, we're all better served if you just moved along. [1] You know who we are.

baddox 13y ago
Is your desire to only serve certain people based in the long run on profit motive (i.e. you think your fewer customers will be more valuable and stick around longer), or is it just out of principle even if it means less profit in the long run?
- rsync 13y ago
  It is done out of principle.
runako 13y ago
Just a thought -- this POV implies that there is a single way to explain every piece of information on your site that is optimal for each person who might be a customer. (The corollary is that all your customers are the same.) A person doesn't have to be swayed by a buzzword, color scheme, or fluff to respond to different presentation of facts. You know this is true if you've ever had to explain anything to anyone. Some examples of how different presentations of your content might affect an individual: - English: first language? - Country/region of origin? - What did they read before coming to your site? A lot of things influence human behavior, and A/B testing isn't just to tease out the fluffy bits you can use to sell to the gullible.
sethev 13y ago
How do you know your current design is excluding the right people? The people who are buying your product now might just be reacting to your current color scheme (or lack thereof) or word order.
- pc86 13y ago

shadeless 13y ago

The bit about 37signals reminded me of their timelapse video, "evolution of a homepage"[1], which shows just how many things they try before sticking to something (albeit shortly). [1] http://vimeo.com/29088090 http://vimeo.com/29088090

thenomad 13y ago

All very good stuff, but this line stood out in particular: "Video makes Waterfall software development look like bugs-in-your-teeth speed, though" As someone who is coming to the close of a 4-year work stint on a short film, I can tell you, you ain't wrong. Indeed, it has gotten to the point that I've started a computer game design project as light relief.

viggity 13y ago

what the hell is S/J testing? never heard of it and apparently neither has the googles.

dangrossman 13y ago
Steve Jobs.

qeorge 13y ago

If you're looking for things to A/B test, please check out my post: https://news.ycombinator.com/item?id=6163397 https://news.ycombinator.com/item?id=6163397 It details 19 A/B test ideas that have worked for us in the past. I hope HN finds it useful!

Silhouette 13y ago

I've now started flagging these kinds of submissions on HN, for the simple reason that they are little more than ads for something vaguely specified that may or may not even exist yet. There is absolutely no directly actionable content in this article that I can see, nor any other original ideas that anyone following HN doesn't see several times a week on the front page alone. I post this only so I can note that this is not intended as a criticism of Patrick, just a view that the hero worship and endless reposting of A/B evangelism has far overstayed its welcome IMHO. If people want to read the fluff as well, then as we get told every five minutes they can sign up for Patrick's newsletter. Please, for the love of all that is holy, spare those of us who have chosen not to do so the endless join-my-mailing-list spam that is starting to dominate HN, and save the HN submissions for substantial content.

gus_massa 13y ago
This email is more infomercial than the emails from the past weeks. I reread the last two emails and I think they are more interesting, but this got much more upvotes here. Anyway, this email convinced me to try to run an A/B test. I have a button that I don't like and I got negative feedback about that button in the past. So I will (probably) try to optimize the microcopy, but I must understand and change the php code that renders that button. I'll report the result when the experiment finish. Wish me good luck!
wakeless 13y ago
It's worth mentioning that that's exactly why Patrick doesn't normally submit this stuff himself. Infact, I've seen him apologies/ask people not to just re-post stuff here and just subscribe to his newsletter/blog/etc. It's also worth noting that you don't HAVE to click through to something that's on the kalzumeus domain, if you subscribe to the newsletter or don't want to. Personally, I think anything that generates a reasonable number of good comments is great (not necessarily this.) That's why HN is great.
davidw 13y ago
Flagging is for stuff that's inappropriate for this site, like political articles, the latest injustice, politics, news stories, conspiracy theories and so on. For stuff like this that you may not happen to like, just don't vote for it, and ignore it. I tend not to vote up patio11's stuff these days even though it's good, just because it gets so much attention. But it is 100% on topic for this site and does not deserve to be flagged. How much time has patio11 spent giving free advice and information here? So what if he also sells some stuff? That's what businesses do, sheez!