3 ms·
> My guess would be that the typeface on your site is so small (you're running that one now still?) that the lighter font might have made it harder to read. We
by gwern 13y ago
> My guess would be that the typeface on your site is so small (you're running that one now still?) that the lighter font might have made it harder to read.
Well, except that the bigger fonts are performing noticeably worse, and it's a pretty linear relationship. You can see it in the current simplified-model estimate of -0.002, but it jumps out when plotted: http://i.imgur.com/SKw3ScN.png http://i.imgur.com/SKw3ScN.png The 100% group performs best, and it just gets worse, even though you wouldn't think, say, 105% would be much different from 100%.
> Couldn't it be that changing all four variables at once will make for a big boost in conversion, but changing one at a time runs into the limitations of the others? For example, increasing font-size by itself will lead to taller paragraphs and more black exposed.
Yes, that's entirely possible. Actually, it's probable that the fitness landscape looks like that. There's no reason to expect features to be independent and design so easily optimizable.
The problem is, it's hard to explore a high-dimensional landscape. Factorial designs, where you test every possible combination and estimate interactions, let you do this, but things can increase, well, factorially. 2x2 isn't too bad, but let's take just your 3 variables which can be tweaked by CSS: if I use just 4 differing shades of black, 5 font sizes (like the one I'm running now), and 5 differing line lengths (something like `max-width:`), then that gives me 4x5x5=100 different variants to run, and each one will need something like 1000 page views to start to become accurate, so it would take quite a while to accumulate 100,000 page views. (To put this in perspective, getting to #1 on the HN front page is worth ~20,000 page views.)
And that's just with 3 changes each with relatively few possible choices. You can see the problem. If you are really worried about interactions and decide to do a full factorial design to explore them, you can quickly get into infeasible sample sizes.
- ianstormtaylor 13y agoRight, which is why I don't recommend designing by A/B testing single-variable changes in the first place. Instead of assuming that your site does better with extremely long lines of tiny type, even thought that goes against pretty much every typographer's recommendation, I'd assume that you started at a small local maximum if the results really do suggest you are best optimized. _ / \ / \ you? / \ _ / \ __/ \_______/ \_______________ Another thing that came to mind: read time could actually be working against you, how do you know people are leaving because they can't read the type vs. they are leaving because it was so easy to read they finished earlier ;)
- pygy_ 13y agoHis definition of conversion is unusual (compared to what we usually see on HN). His articles are long, and take more than 40 seconds to read.
- gwern 13y agoRight. It's impossible to genuinely read any page on my site in <40 seconds, so I can treat an increase in >40seconders as an increase in reading time.
- gwern 13y ago> Instead of assuming that your site does better with extremely long lines of tiny type, even thought that goes against pretty much every typographer's recommendation It's not really an assumption though, I've already looked at long lines with the `max-width` variable, and the very wide/long versions performed pretty well; see http://www.gwern.net/AB%20testing#max-width http://www.gwern.net/AB%20testing#max-width & http://www.gwern.net/AB%20testing#max-width-redux http://www.gwern.net/AB%20testing#max-width-redux
- ianstormtaylor 13y agoThat was a single-variable test, right? I just talked about that...