6 ms·

Lesson from Madlibs Signup Fad: Do Your Own Tests

81 points by soundsop 17y ago

patio11 17y ago
Crikey you guys are fast on the submit buttons. I hit the post button, try to answer one customer email before HNing it, and it is already here.
- soundsop 17y ago
  I was expecting the submission to be an upvote. Looks like my RSS feed updated while you were answering that email.
jrockway 17y ago
Doing your own tests is good, but I think the username/password is too simple to benefit from the madlibs style. I also thought the wording was unusual, "I'd like to use this word ...... as my password." It could be that the strange wording was what caused the drop-off.
- andymism 17y ago
  I thought the username and password was a good choice for this test. There really wasn't that much of a difference in phrasing between Patrick's form and the Huffduffer form. The most likely explanation for the drop off is the difference in the audience these sites are made for. As far as design decisions like these go, the data doesn't say anything on an absolute scale about the merits of one form versus the other, but it does tell us which is more effective for a particular audience in a particular context.
tkiley 17y ago
The madlibs form has a "sign in as guest" link, while the original sign-in form does not. The article doesn't specify whether a guest sign-in is counted as a conversion or not.
- patio11 17y ago
  P.S. If you have good eyes you’ll spot the other A/B test ongoing on this page. I’m using the traditional way of mitigating cross-test interaction… ignoring the possibility of it Both forms have a 50/50 chance of having the guest link on them, and guests don't count as trial conversions (because I have several months of stats that suggests they give me money infrequently and they cause more support issues than all trialers and paying customers put together). The thrust of that A/B test is to see whether aggregate sales go down if the guests option is taken away. If they don't, I will drop it.
  - nitrogen 17y ago
    I noticed something in sprites2.png referenced in your A/Bingo page: the bottom row of pixels on the third blue background image is duplicated from the second blue background image.
arnorhs 17y ago
Your version of the madlibs form also looks much worse than your usual signup form. I don't know how much that affects user signup percentage, but personally I'd have an easier time trusting your standard form because if it's design, not because of the "style"
- patio11 17y ago
  Can all designers exit the room for a moment? I am skeptical that "looks pretty" drives conversions. I think customers pretty much don't care, based on "ugly" versions winning in many, many of my tests. Remember: the customer doesn't get to see the pretty alternative laid out side by side with the ugly alternative. Example: form submit buttons in your Internet browser of choice look fairly pretty, right? (Or at least comfortingly familiar.) I would not describe http://www.bingocardcreator.com/images/bcc.net/next-step.png http://www.bingocardcreator.com/images/bcc.net/next-step.png as pretty -- it is what I could throw together in ten minutes. But it clocks "pretty" buttons with the same text on it. You're welcome to try it yourself if you think pixel-perfect alignment will make a difference. Please post results.
  - pedalpete 17y ago
    It isn't a matter of looking 'pretty', it is how does the brain visually understand it as simple or not. I agree that there designers go overboard with 'it doesn't have the right padding', etc. But you can't ignore HUGE test like red makes people hungry more (McDonalds) etc. That sort of design does translate, and the mad-libs form just looks confusing.
  - mechanical_fish 17y ago
    I would press that button just to make it go away. ;) Not that it matters -- the data matter -- but I can certainly rationalize why such a button would be more effective than a standard form submit button: My peripheral vision has seen a hundred thousand form submit buttons by now. I'm not quite as blind to them as I am to standard-sized non-animated web ads, but it's close. And it seems to me that the problem with being a designer is that it's physically impossible to design a web form in less time than the average user will spend looking at it. You will necessarily spend orders of magnitude more time looking at your forms than a visitor will: Visitors don't look at forms for fun. They click through them fast, or they click away from them even faster. If I had to stare at that button for an hour, as opposed to simply finding it and clicking it as fast as I could, I'd design it differently. Which might very well screw it up.
rauljara 17y ago
Running tests is definitely good advice. But execution definitely matters in these things. Just looking at the two forms tested side by side, the original looks more well spaced out and balanced. The madlibs has the two text fields actually touching. Compare that to the original madlibs form (http://www.lukew.com/ff/entry.asp?1007 http://www.lukew.com/ff/entry.asp?1007) and you'll see that the original madlibs form just looks a lot better.
- deleted 17y ago
  [deleted]
ramchip 17y ago
Isn't it a bit early to call it a fad? I only saw one website try this (the one you gave the link to). Perhaps I missed some, but still...
mechanical_fish 17y ago
While everyone is piling on with the untested hypotheses, here's mine: The madlibs form is just too wordy. Whereas the username/password field pair is not only less wordy, it's an idiom: I've seen those fields, in that arrangement and that order, thousands of times over the last decade. I barely even have to read the captions at all. Not that this hypothesis, or any of the others, matters. The point of this article isn't to design a better form. It's to show us how to design a better form.
Batsu 17y ago
As an intently web savvy group, we want this simple "fix" to work for us. You provided us with reason to believe this (albeit simple) change will not provide us with results. Please, don't do this.
- patio11 17y ago
  If I were to play pop psychologist, I think we have some fairly simple fixes that we want to work because they're cool/sexy/etc, and some simple fixes which we don't want to work because they're "stupid" or sound marketing-y. For example, in the original madlibs test, where seven things changed, we want to attribute the change to the madlib element (which was sexy and creative) rather than to e.g. the text on the conversion button changing. I've gotten 20% lifts out of "stupid little things" like that before, and seen good writeups of the same from other people. But e.g. designers do not want to hear that the thing they spend 98% of their time on is essentially meaningless and the elements of the page which they scrutinize least carefully -- like microcopy or button text or (in one memorable test) which way the stock photo model is facing dominate the measurable results. This is like how doctors do not want to hear that the difference in clinical outcomes due to professional experience is totally dwarfed by the difference in clinical outcomes due to following checklists. (On the plus side, designers don't kill anybody for their professional pride.) In general, I think the point of A/B testing is that you should ruthlessly soulcrush what you want with what the data actually tells you. I respect HN folks enormously, but I'm not hearing "I tested it and it worked" -- I'm hearing "I've got a great theory on why your data does not conform to reality. Reality is, of course, what I predict it to be."
paraschopra 17y ago
Yeah, it fantastically proves the point: do your own A/B tests. I am guessing the implementation can create a huge difference. Maybe you can ask for help from some of the designers here on HN and see if a beautiful implementation makes a difference. Plus your two checkboxes can also go into the "malibs" form.
dpcan 17y ago
Well, he did it all wrong. When you are done reading the Madlibs, it should read like a sentence. What he did was surround regular text fields with strange sentences. When you are done filling it out, you can't just read it like a regular sentence. It should have been: I would like to make bingo cards! Please save them for me under the name [ Your Name ], and and my email address is [ Email address] in case you need to contact me. But please [do / don't] contact me twice with hints, and [do / don't] send me a newsletter. See, when you are done, you can read it like a sentence. All he did was replace "Username" with longer wording. It didn't seem "fun" the way you did it, just odd and off-putting.
- micrypt 17y ago
  True. Made a few adjustments and I think it looks a bit better already: http://twitpic.com/15t9cc/full http://twitpic.com/15t9cc/full
  - Herring 17y ago
    A bit, yeah. But the pixels are still really off. I don't even know why, I just see they're wrong.
    - alttab 17y ago
      I think its possible its the wide difference in font. Because the madlib text looks very different (as opposed to bold input labels) it looks frail and slapped together. Larger fonts, bolder text, and more space could probably help it more visually.
maxklein 17y ago
Seeing as we have two totally different results from different people, my conclusion is that even though the madlibs style can improve your conversion rate, it's too easy to get it wrong and actually reduce your conversion rate. The majority of websites have used the standard sign up form, and it's known and recongized by most users. It's risky to to go with something like this that could have a strong negative impact if you do it wrong.
ams6110 17y ago
Just as a personal observation, I much prefer the traditional style. All these "I want..." and "Give me..." style dialogs make me want to vomit.

81 points by soundsop 17y ago

Crikey you guys are fast on the submit buttons. I hit the post button, try to answer one customer email before HNing it, and it is already here.

soundsop 17y ago
I was expecting the submission to be an upvote. Looks like my RSS feed updated while you were answering that email.

jrockway 17y ago

Doing your own tests is good, but I think the username/password is too simple to benefit from the madlibs style. I also thought the wording was unusual, "I'd like to use this word ...... as my password." It could be that the strange wording was what caused the drop-off.

andymism 17y ago
I thought the username and password was a good choice for this test. There really wasn't that much of a difference in phrasing between Patrick's form and the Huffduffer form. The most likely explanation for the drop off is the difference in the audience these sites are made for. As far as design decisions like these go, the data doesn't say anything on an absolute scale about the merits of one form versus the other, but it does tell us which is more effective for a particular audience in a particular context.

tkiley 17y ago

The madlibs form has a "sign in as guest" link, while the original sign-in form does not. The article doesn't specify whether a guest sign-in is counted as a conversion or not.

patio11 17y ago
P.S. If you have good eyes you’ll spot the other A/B test ongoing on this page. I’m using the traditional way of mitigating cross-test interaction… ignoring the possibility of it Both forms have a 50/50 chance of having the guest link on them, and guests don't count as trial conversions (because I have several months of stats that suggests they give me money infrequently and they cause more support issues than all trialers and paying customers put together). The thrust of that A/B test is to see whether aggregate sales go down if the guests option is taken away. If they don't, I will drop it.
- nitrogen 17y ago
  I noticed something in sprites2.png referenced in your A/Bingo page: the bottom row of pixels on the third blue background image is duplicated from the second blue background image.

arnorhs 17y ago

Your version of the madlibs form also looks much worse than your usual signup form. I don't know how much that affects user signup percentage, but personally I'd have an easier time trusting your standard form because if it's design, not because of the "style"

patio11 17y ago
Can all designers exit the room for a moment? I am skeptical that "looks pretty" drives conversions. I think customers pretty much don't care, based on "ugly" versions winning in many, many of my tests. Remember: the customer doesn't get to see the pretty alternative laid out side by side with the ugly alternative. Example: form submit buttons in your Internet browser of choice look fairly pretty, right? (Or at least comfortingly familiar.) I would not describe http://www.bingocardcreator.com/images/bcc.net/next-step.png http://www.bingocardcreator.com/images/bcc.net/next-step.png as pretty -- it is what I could throw together in ten minutes. But it clocks "pretty" buttons with the same text on it. You're welcome to try it yourself if you think pixel-perfect alignment will make a difference. Please post results.
- pedalpete 17y ago
  It isn't a matter of looking 'pretty', it is how does the brain visually understand it as simple or not. I agree that there designers go overboard with 'it doesn't have the right padding', etc. But you can't ignore HUGE test like red makes people hungry more (McDonalds) etc. That sort of design does translate, and the mad-libs form just looks confusing.
- mechanical_fish 17y ago
  I would press that button just to make it go away. ;) Not that it matters -- the data matter -- but I can certainly rationalize why such a button would be more effective than a standard form submit button: My peripheral vision has seen a hundred thousand form submit buttons by now. I'm not quite as blind to them as I am to standard-sized non-animated web ads, but it's close. And it seems to me that the problem with being a designer is that it's physically impossible to design a web form in less time than the average user will spend looking at it. You will necessarily spend orders of magnitude more time looking at your forms than a visitor will: Visitors don't look at forms for fun. They click through them fast, or they click away from them even faster. If I had to stare at that button for an hour, as opposed to simply finding it and clicking it as fast as I could, I'd design it differently. Which might very well screw it up.

rauljara 17y ago

Running tests is definitely good advice. But execution definitely matters in these things. Just looking at the two forms tested side by side, the original looks more well spaced out and balanced. The madlibs has the two text fields actually touching. Compare that to the original madlibs form (http://www.lukew.com/ff/entry.asp?1007 http://www.lukew.com/ff/entry.asp?1007) and you'll see that the original madlibs form just looks a lot better.

deleted 17y ago
[deleted]

ramchip 17y ago

Isn't it a bit early to call it a fad? I only saw one website try this (the one you gave the link to). Perhaps I missed some, but still...

mechanical_fish 17y ago

While everyone is piling on with the untested hypotheses, here's mine: The madlibs form is just too wordy. Whereas the username/password field pair is not only less wordy, it's an idiom: I've seen those fields, in that arrangement and that order, thousands of times over the last decade. I barely even have to read the captions at all. Not that this hypothesis, or any of the others, matters. The point of this article isn't to design a better form. It's to show us how to design a better form.

Batsu 17y ago

As an intently web savvy group, we want this simple "fix" to work for us. You provided us with reason to believe this (albeit simple) change will not provide us with results. Please, don't do this.

patio11 17y ago
If I were to play pop psychologist, I think we have some fairly simple fixes that we want to work because they're cool/sexy/etc, and some simple fixes which we don't want to work because they're "stupid" or sound marketing-y. For example, in the original madlibs test, where seven things changed, we want to attribute the change to the madlib element (which was sexy and creative) rather than to e.g. the text on the conversion button changing. I've gotten 20% lifts out of "stupid little things" like that before, and seen good writeups of the same from other people. But e.g. designers do not want to hear that the thing they spend 98% of their time on is essentially meaningless and the elements of the page which they scrutinize least carefully -- like microcopy or button text or (in one memorable test) which way the stock photo model is facing dominate the measurable results. This is like how doctors do not want to hear that the difference in clinical outcomes due to professional experience is totally dwarfed by the difference in clinical outcomes due to following checklists. (On the plus side, designers don't kill anybody for their professional pride.) In general, I think the point of A/B testing is that you should ruthlessly soulcrush what you want with what the data actually tells you. I respect HN folks enormously, but I'm not hearing "I tested it and it worked" -- I'm hearing "I've got a great theory on why your data does not conform to reality. Reality is, of course, what I predict it to be."

paraschopra 17y ago

Yeah, it fantastically proves the point: do your own A/B tests. I am guessing the implementation can create a huge difference. Maybe you can ask for help from some of the designers here on HN and see if a beautiful implementation makes a difference. Plus your two checkboxes can also go into the "malibs" form.

dpcan 17y ago

Well, he did it all wrong. When you are done reading the Madlibs, it should read like a sentence. What he did was surround regular text fields with strange sentences. When you are done filling it out, you can't just read it like a regular sentence. It should have been: I would like to make bingo cards! Please save them for me under the name [ Your Name ], and and my email address is [ Email address] in case you need to contact me. But please [do / don't] contact me twice with hints, and [do / don't] send me a newsletter. See, when you are done, you can read it like a sentence. All he did was replace "Username" with longer wording. It didn't seem "fun" the way you did it, just odd and off-putting.

micrypt 17y ago
True. Made a few adjustments and I think it looks a bit better already: http://twitpic.com/15t9cc/full http://twitpic.com/15t9cc/full
- Herring 17y ago
  A bit, yeah. But the pixels are still really off. I don't even know why, I just see they're wrong.
  - alttab 17y ago
    I think its possible its the wide difference in font. Because the madlib text looks very different (as opposed to bold input labels) it looks frail and slapped together. Larger fonts, bolder text, and more space could probably help it more visually.

maxklein 17y ago

Seeing as we have two totally different results from different people, my conclusion is that even though the madlibs style can improve your conversion rate, it's too easy to get it wrong and actually reduce your conversion rate. The majority of websites have used the standard sign up form, and it's known and recongized by most users. It's risky to to go with something like this that could have a strong negative impact if you do it wrong.

ams6110 17y ago

Just as a personal observation, I much prefer the traditional style. All these "I want..." and "Give me..." style dialogs make me want to vomit.