3 ms·

Hmm interesting. My guess would be that the typeface on your site is so small (you're running that one now still?) that the lighter font might have made it hard

Hmm interesting. My guess would be that the typeface on your site is so small (you're running that one now still?) that the lighter font might have made it harder to read. It would be interesting to see if these single-variable changes actually keep you in a local maximum. For example, imagine that: - pure black fonts are actually bad for conversion, and conversion gets worse the more pure black that is exposed (ie. the size of the type). - small font sizes are also bad for conversion. - long line lengths are also bad for conversion. - tall paragraphs (in number of lines) are also bad for conversion. Now I'd imagine (purely on design instincts) that the best possible reading experience (which I'm assuming also means more reading time) would be to have off-black-colored type, at a size around 16px, with a line length of 70-80 characters, and an average paragraph height of around 4-5. Couldn't it be that changing all four variables at once will make for a big boost in conversion, but changing one at a time runs into the limitations of the others? For example, increasing font-size by itself will lead to taller paragraphs and more black exposed.

gwern 13y ago
> My guess would be that the typeface on your site is so small (you're running that one now still?) that the lighter font might have made it harder to read. Well, except that the bigger fonts are performing noticeably worse, and it's a pretty linear relationship. You can see it in the current simplified-model estimate of -0.002, but it jumps out when plotted: http://i.imgur.com/SKw3ScN.png http://i.imgur.com/SKw3ScN.png The 100% group performs best, and it just gets worse, even though you wouldn't think, say, 105% would be much different from 100%. > Couldn't it be that changing all four variables at once will make for a big boost in conversion, but changing one at a time runs into the limitations of the others? For example, increasing font-size by itself will lead to taller paragraphs and more black exposed. Yes, that's entirely possible. Actually, it's probable that the fitness landscape looks like that. There's no reason to expect features to be independent and design so easily optimizable. The problem is, it's hard to explore a high-dimensional landscape. Factorial designs, where you test every possible combination and estimate interactions, let you do this, but things can increase, well, factorially. 2x2 isn't too bad, but let's take just your 3 variables which can be tweaked by CSS: if I use just 4 differing shades of black, 5 font sizes (like the one I'm running now), and 5 differing line lengths (something like `max-width:`), then that gives me 4x5x5=100 different variants to run, and each one will need something like 1000 page views to start to become accurate, so it would take quite a while to accumulate 100,000 page views. (To put this in perspective, getting to #1 on the HN front page is worth ~20,000 page views.) And that's just with 3 changes each with relatively few possible choices. You can see the problem. If you are really worried about interactions and decide to do a full factorial design to explore them, you can quickly get into infeasible sample sizes.
- ianstormtaylor 13y ago
  Right, which is why I don't recommend designing by A/B testing single-variable changes in the first place. Instead of assuming that your site does better with extremely long lines of tiny type, even thought that goes against pretty much every typographer's recommendation, I'd assume that you started at a small local maximum if the results really do suggest you are best optimized. _ / \ / \ you? / \ _ / \ __/ \_______/ \_______________ Another thing that came to mind: read time could actually be working against you, how do you know people are leaving because they can't read the type vs. they are leaving because it was so easy to read they finished earlier ;)
  - pygy_ 13y ago
    His definition of conversion is unusual (compared to what we usually see on HN). His articles are long, and take more than 40 seconds to read.
    - gwern 13y ago
      Right. It's impossible to genuinely read any page on my site in <40 seconds, so I can treat an increase in >40seconders as an increase in reading time.
  - gwern 13y ago
    > Instead of assuming that your site does better with extremely long lines of tiny type, even thought that goes against pretty much every typographer's recommendation It's not really an assumption though, I've already looked at long lines with the `max-width` variable, and the very wide/long versions performed pretty well; see http://www.gwern.net/AB%20testing#max-width http://www.gwern.net/AB%20testing#max-width & http://www.gwern.net/AB%20testing#max-width-redux http://www.gwern.net/AB%20testing#max-width-redux
    - ianstormtaylor 13y ago
      That was a single-variable test, right? I just talked about that...
pantalaimon 13y ago
The lines are also pretty crammed, they could need some more padding. I easily find myself slipping a line while reading. And if gwern feels adventurous (and I guess he'd like testing such kind of things), he may try something like [1] to color the text with a gradient, to allow faster reading speeds. [1] http://www.beelinereader.com/ http://www.beelinereader.com/
- gwern 13y ago
  > The lines are also pretty crammed, they could need some more padding. As in line height? Already tried that: http://www.gwern.net/AB%20testing#line-height http://www.gwern.net/AB%20testing#line-height > [1] http://www.beelinereader.com/ http://www.beelinereader.com/ I emailed them when they came out for a JS version which I could use. I guess they are still working on it because I haven't heard anything lately.
  - pantalaimon 13y ago
    I think single-value changes won't make a big difference (as you've proven yourself), but did you look into something like [1]? But I have to admit, I hardly ever read your articles from top to bottom, but just skim through them, looking for paragraphs I find interesting. Your current design works well for that, most readability enhancing matters usually help reading the entire thing (like a story or an article in a magazine), but at the cost of the ability to quickly glance over it. [1] http://www.readability.com/read?url=http://www.gwern.net/AB%20testing http://www.readability.com/read?url=http://www.gwern.net/AB%...
    - gwern 13y ago
      > I think single-value changes won't make a big difference (as you've proven yourself) But it can still help. A local optima is still higher than not being in the local optima. I haven't looked at Readability a lot. For a while, it wasn't displaying gwern.net pages at all... and it's still messing up a lot. Apparently the Readability view starts halfway down the page! /hits the report-bug button yet again.

gwern 13y ago

> My guess would be that the typeface on your site is so small (you're running that one now still?) that the lighter font might have made it harder to read. Well, except that the bigger fonts are performing noticeably worse, and it's a pretty linear relationship. You can see it in the current simplified-model estimate of -0.002, but it jumps out when plotted: http://i.imgur.com/SKw3ScN.png http://i.imgur.com/SKw3ScN.png The 100% group performs best, and it just gets worse, even though you wouldn't think, say, 105% would be much different from 100%. > Couldn't it be that changing all four variables at once will make for a big boost in conversion, but changing one at a time runs into the limitations of the others? For example, increasing font-size by itself will lead to taller paragraphs and more black exposed. Yes, that's entirely possible. Actually, it's probable that the fitness landscape looks like that. There's no reason to expect features to be independent and design so easily optimizable. The problem is, it's hard to explore a high-dimensional landscape. Factorial designs, where you test every possible combination and estimate interactions, let you do this, but things can increase, well, factorially. 2x2 isn't too bad, but let's take just your 3 variables which can be tweaked by CSS: if I use just 4 differing shades of black, 5 font sizes (like the one I'm running now), and 5 differing line lengths (something like `max-width:`), then that gives me 4x5x5=100 different variants to run, and each one will need something like 1000 page views to start to become accurate, so it would take quite a while to accumulate 100,000 page views. (To put this in perspective, getting to #1 on the HN front page is worth ~20,000 page views.) And that's just with 3 changes each with relatively few possible choices. You can see the problem. If you are really worried about interactions and decide to do a full factorial design to explore them, you can quickly get into infeasible sample sizes.

ianstormtaylor 13y ago
Right, which is why I don't recommend designing by A/B testing single-variable changes in the first place. Instead of assuming that your site does better with extremely long lines of tiny type, even thought that goes against pretty much every typographer's recommendation, I'd assume that you started at a small local maximum if the results really do suggest you are best optimized. _ / \ / \ you? / \ _ / \ __/ \_______/ \_______________ Another thing that came to mind: read time could actually be working against you, how do you know people are leaving because they can't read the type vs. they are leaving because it was so easy to read they finished earlier ;)
- pygy_ 13y ago
  His definition of conversion is unusual (compared to what we usually see on HN). His articles are long, and take more than 40 seconds to read.
  - gwern 13y ago
    Right. It's impossible to genuinely read any page on my site in <40 seconds, so I can treat an increase in >40seconders as an increase in reading time.
- gwern 13y ago
  > Instead of assuming that your site does better with extremely long lines of tiny type, even thought that goes against pretty much every typographer's recommendation It's not really an assumption though, I've already looked at long lines with the `max-width` variable, and the very wide/long versions performed pretty well; see http://www.gwern.net/AB%20testing#max-width http://www.gwern.net/AB%20testing#max-width & http://www.gwern.net/AB%20testing#max-width-redux http://www.gwern.net/AB%20testing#max-width-redux
  - ianstormtaylor 13y ago
    That was a single-variable test, right? I just talked about that...

pantalaimon 13y ago

The lines are also pretty crammed, they could need some more padding. I easily find myself slipping a line while reading. And if gwern feels adventurous (and I guess he'd like testing such kind of things), he may try something like [1] to color the text with a gradient, to allow faster reading speeds. [1] http://www.beelinereader.com/ http://www.beelinereader.com/

gwern 13y ago
> The lines are also pretty crammed, they could need some more padding. As in line height? Already tried that: http://www.gwern.net/AB%20testing#line-height http://www.gwern.net/AB%20testing#line-height > [1] http://www.beelinereader.com/ http://www.beelinereader.com/ I emailed them when they came out for a JS version which I could use. I guess they are still working on it because I haven't heard anything lately.
- pantalaimon 13y ago
  I think single-value changes won't make a big difference (as you've proven yourself), but did you look into something like [1]? But I have to admit, I hardly ever read your articles from top to bottom, but just skim through them, looking for paragraphs I find interesting. Your current design works well for that, most readability enhancing matters usually help reading the entire thing (like a story or an article in a magazine), but at the cost of the ability to quickly glance over it. [1] http://www.readability.com/read?url=http://www.gwern.net/AB%20testing http://www.readability.com/read?url=http://www.gwern.net/AB%...
  - gwern 13y ago
    > I think single-value changes won't make a big difference (as you've proven yourself) But it can still help. A local optima is still higher than not being in the local optima. I haven't looked at Readability a lot. For a while, it wasn't displaying gwern.net pages at all... and it's still messing up a lot. Apparently the Readability view starts halfway down the page! /hits the report-bug button yet again.