6 ms·

I blame this on A/B tests which ignore externalities. Sure, maybe the 3 metrics you were aiming for were improved, but your brand is fucked. I think brands like

I blame this on A/B tests which ignore externalities. Sure, maybe the 3 metrics you were aiming for were improved, but your brand is fucked. I think brands like Sears/Kenmore/Craftsman were A/Bed to death. So many little 'optimizations' (read: cost cutting that looked-good-on-paper) that eventually destroyed what the brand was about in the first place.

jdmichal 10y ago
Not to mention that, for abstract judgements like "product quality", these can also compound in non-linear fashion. Three A/B tests could show only minor effects on perception of product quality, but then if you put the original A next to the final B, you'll get much bigger differences.
baddox 10y ago
In the case of Fandango showing an interstitial ad when attempting to check out, I can't imagine that wouldn't hurt conversion, and I can't imagine that conversion wouldn't be one the key metrics they would test with. I would guess it's more just a lack of care. Someone said "we need our website to make more money" and someone else said "okay, I hear that ads are for making more money."
- ssharp 10y ago
  It could be that the ads are making more money than they're losing from conversion decrease.
  - dspig 10y ago
    yes, they are already targeting someone who will buy stuff from them.
- patmcguire 10y ago
  It's probable that there is no grand strategy, but it might make sense if their model finds you unlikely to convert and the conversion price is very low. It's not necessarily flat per movie, they may be losing money on that one as part of a package deal.
- sparky_z 10y ago
  Well, presumably they're reselling these tickets on behalf of the theaters which get the bulk of the revenue. It's possible Fandango takes no margin on the tickets sales and just makes money on the ads.
  - greglindahl 10y ago
    That's a classic problem for understanding these things: If you don't know how the money flows, you have no idea how to second-guess what's best.
  - Spooky23 10y ago
    They take a fixed fee per ticket. It's a good model. Like most ticketing operations, they've built a niche monopoly. If you want to buy tickets online, they are usually the only game in town. If the obnoxious ads annoy you, their attitude is basically "fuck you, drive to the mall and miss your movie".
    - asdfasdfa11112 10y ago
      This is called a "moat". Build it for your company, it's a good thing for an entrepreneur. To be significantly successful, you need to stop thinking this way. Start thinking about what benefits you & your company. It may seem like contempt for the consumer, and perhaps it is. But it's also possible the consumer is irrational and you can benefit in the short and long term. I work in marketing, and have seen scant evidence for the value of a "brand". It's mostly branding people making an argument that comic sans devalues our brand, whereas the data clearly shows it doesn't. The "long term harm" argument is indistinguishable from "i know better than you and I want it this way." It may be true, but it's unprovable.
      - _eht 10y ago
        This actually makes a lot of sense.
      - Angostura 10y ago
        > I work in marketing, and have seen scant evidence for the value of a "brand" To what do you attribute the success of, say Coca Cola, rather than generic cola? Price and a good distribution network? Surely brand must contribute a little.
- taeric 10y ago
  The cynic in me:. It is a great natural rate limiter to keep from too many customers seeing sold out while progressing directly through the process.
SCHiM 10y ago
Heh, measuring average and unbiased user opinions on the internet is probably the second most uncertain thing in this universe after actual uncertainty in quantum systems. Both opt-in and opt-out ways of doing surveys on your website introduce bias. The opt-out (read: in your face pop-up) is more likely to make people angry which makes them more likely to leave negative reviews. The opt-in version will only be found by people having a relatively extreme opinion about your site, good or bad. The difference being in the quantity of the data. The first will probably be answered by more people, since the people who'd have completed the survey in the second case are probably more or less a subset of the people who would complete the survey in the first case. All of this makes it no surprise that most websites that want to have any kind of feedback from their users go for the opt-out strategy. More is better if less is not in any way more :)
deleted 10y ago
[deleted]
inlined 10y ago
I think this is related to my fundamental problem with A/B testing: when people A/B test individual variables they'll take a greedy optimization that gets stuck in local minima. The other problem is that you only understand the change in metrics you measure and things like "satisfaction" are often lost.
- bad_user 10y ago
  > greedy optimization that gets stuck in local minima I think you wanted to say local maxima, but yes I agree.
  - deleted 10y ago
    [deleted]
  - TeMPOraL 10y ago
    It's customary to state the optimization problem such that you're minimizing a function - hence "local minima". If you want to maximize something, you simply negate the function.
pryelluw 10y ago
Good point. I see this almost on a daily basis with clients. They miss the forrest for the trees. Its dumb to optimize for email capture when people end up hating your brand as a result of it. I remove all pop ups and simply add clear calls to action in different places in the page. They are static components that are part of the layout. The copy is then adjusted to drive invite people to subscribe. I get higher quality conversions. Its not the quantity but the quality of the leads. People forget that.
edblarney 10y ago
Sears was not killed by any tweaks or optimizations - it was a brand that was past it's time, culture changed, and it could not find a new way forward.
- Cerium 10y ago
  I think it didn't adapt to the times. It was the Amazon of a past generation, with its own very high quality version of AmazonBasics. They didn't make the transition to a new reality though.
- spc476 10y ago
  Sears was the Amazon of its day (around the early 1900s) where you could order almost anything via mail (including housed!). I guess the felt that products via the mail was so 19th century.

jdmichal 10y ago

Not to mention that, for abstract judgements like "product quality", these can also compound in non-linear fashion. Three A/B tests could show only minor effects on perception of product quality, but then if you put the original A next to the final B, you'll get much bigger differences.

baddox 10y ago

In the case of Fandango showing an interstitial ad when attempting to check out, I can't imagine that wouldn't hurt conversion, and I can't imagine that conversion wouldn't be one the key metrics they would test with. I would guess it's more just a lack of care. Someone said "we need our website to make more money" and someone else said "okay, I hear that ads are for making more money."

ssharp 10y ago
It could be that the ads are making more money than they're losing from conversion decrease.
- dspig 10y ago
  yes, they are already targeting someone who will buy stuff from them.
patmcguire 10y ago
It's probable that there is no grand strategy, but it might make sense if their model finds you unlikely to convert and the conversion price is very low. It's not necessarily flat per movie, they may be losing money on that one as part of a package deal.
sparky_z 10y ago
Well, presumably they're reselling these tickets on behalf of the theaters which get the bulk of the revenue. It's possible Fandango takes no margin on the tickets sales and just makes money on the ads.
- greglindahl 10y ago
  That's a classic problem for understanding these things: If you don't know how the money flows, you have no idea how to second-guess what's best.
- Spooky23 10y ago
  They take a fixed fee per ticket. It's a good model. Like most ticketing operations, they've built a niche monopoly. If you want to buy tickets online, they are usually the only game in town. If the obnoxious ads annoy you, their attitude is basically "fuck you, drive to the mall and miss your movie".
  - asdfasdfa11112 10y ago
    This is called a "moat". Build it for your company, it's a good thing for an entrepreneur. To be significantly successful, you need to stop thinking this way. Start thinking about what benefits you & your company. It may seem like contempt for the consumer, and perhaps it is. But it's also possible the consumer is irrational and you can benefit in the short and long term. I work in marketing, and have seen scant evidence for the value of a "brand". It's mostly branding people making an argument that comic sans devalues our brand, whereas the data clearly shows it doesn't. The "long term harm" argument is indistinguishable from "i know better than you and I want it this way." It may be true, but it's unprovable.
    - _eht 10y ago
      This actually makes a lot of sense.
    - Angostura 10y ago
      > I work in marketing, and have seen scant evidence for the value of a "brand" To what do you attribute the success of, say Coca Cola, rather than generic cola? Price and a good distribution network? Surely brand must contribute a little.
taeric 10y ago
The cynic in me:. It is a great natural rate limiter to keep from too many customers seeing sold out while progressing directly through the process.

SCHiM 10y ago

Heh, measuring average and unbiased user opinions on the internet is probably the second most uncertain thing in this universe after actual uncertainty in quantum systems. Both opt-in and opt-out ways of doing surveys on your website introduce bias. The opt-out (read: in your face pop-up) is more likely to make people angry which makes them more likely to leave negative reviews. The opt-in version will only be found by people having a relatively extreme opinion about your site, good or bad. The difference being in the quantity of the data. The first will probably be answered by more people, since the people who'd have completed the survey in the second case are probably more or less a subset of the people who would complete the survey in the first case. All of this makes it no surprise that most websites that want to have any kind of feedback from their users go for the opt-out strategy. More is better if less is not in any way more :)

deleted 10y ago

[deleted]

inlined 10y ago

I think this is related to my fundamental problem with A/B testing: when people A/B test individual variables they'll take a greedy optimization that gets stuck in local minima. The other problem is that you only understand the change in metrics you measure and things like "satisfaction" are often lost.

bad_user 10y ago
> greedy optimization that gets stuck in local minima I think you wanted to say local maxima, but yes I agree.
- deleted 10y ago
  [deleted]
- TeMPOraL 10y ago
  It's customary to state the optimization problem such that you're minimizing a function - hence "local minima". If you want to maximize something, you simply negate the function.

pryelluw 10y ago

Good point. I see this almost on a daily basis with clients. They miss the forrest for the trees. Its dumb to optimize for email capture when people end up hating your brand as a result of it. I remove all pop ups and simply add clear calls to action in different places in the page. They are static components that are part of the layout. The copy is then adjusted to drive invite people to subscribe. I get higher quality conversions. Its not the quantity but the quality of the leads. People forget that.

edblarney 10y ago

Sears was not killed by any tweaks or optimizations - it was a brand that was past it's time, culture changed, and it could not find a new way forward.

Cerium 10y ago
I think it didn't adapt to the times. It was the Amazon of a past generation, with its own very high quality version of AmazonBasics. They didn't make the transition to a new reality though.
spc476 10y ago
Sears was the Amazon of its day (around the early 1900s) where you could order almost anything via mail (including housed!). I guess the felt that products via the mail was so 19th century.