4 ms·

> "We're blind." ... "Everything must be tested." And your solution is to do LESS TESTING? We don't know what we're doing, so let's cut back on the amount of d

> "We're blind." ... "Everything must be tested." And your solution is to do LESS TESTING? We don't know what we're doing, so let's cut back on the amount of data we can use to inform our decisions? > "We are slaves to the numbers. We don’t operate around innovation. We only optimize." I don't see why numbers should ever stop you from innovating. The difference between "innovating" and "optimizing" is just a difference of scale. You can make a huge change to your layout or site function and look at the numbers it the same way you'd look at a font and color change. The quote above seems to say that people shouldn't make decisions based on numbers, and that's absurd for a company like Facebook. What should be the basis of their decisions then? Management's gut reaction? Whoever feels the strongest about a change wins? Customer surveys and user metrics matter - both are often numbers. The real issue here isn't that Facebook uses numbers too much. If they made the wrong choice, it's because they put too much emphasis on the wrong numbers.

exodust 13y ago
The point is about design being held back or distorted by numbers - which are often a false prophesy when it comes to UX improvements. Nobody is suggesting that numbers don't matter, but if you can't decide on a border thickness of 1 or 2 pixels without numbers to tell you which to go with, you've lost your mojo. And that matters in the long term. When the aim is to increase length of time on the site or in the app, the numbers will "tell you" to make the interface more complicated, to hide the sign-out button (common practice), to hide the exit button (like Dropbox did on the desktop application), to force dependencies between unrelated services (Google does this a lot). When the aim is to increase clicks on Ads, the numbers will tell you to make ads look like normal posts. The numbers will tell you to autoplay videos on tabloid news article pages. But all these "optimizations" are not unlike speed cameras placed at the bottom of the hill where lots of people can't help bumping over the speed limit for a couple of seconds. Good for revenue, bad vibes in every other way.
- ignostic 13y ago
  > "design being held back or distorted by numbers - which are often a false prophesy when it comes to UX improvements." > "the numbers will "tell you" to make the interface more complicated, to hide the sign-out button" Oh yes, I definitely understand that we misinterpret what the numbers mean. My point is that this isn't the numbers' fault - it's our fault for not understanding what we're measuring and what the numbers mean. Often additional tests can even help resolve the "what should we do" question. We run tests all the time, but when we see something like, "pageviews are up x%" that's just a starting place. We begin to ask why and continue to develop a hypothesis and proof for it. It might be that users are confused so they start clicking around, or it could be because we've grabbed their attention and enticed them to learn more. Without the numbers we wouldn't even know where to start. If Dustin's point is just to think critically about test numbers that seems to go without saying.
  - exodust 13y ago
    Fair enough. I agree the interpretation of the data we collect leaves a lot to be desired. Where I work, I've seen good products killed off because of low traffic, but the low traffic was due to people not finding the product due to poor navigation - which in turn was made poor because of marketing requirements to "drive traffic" to their latest short term gimmick. It's a complex web.
phillmv 13y ago
I think the problem is that we don't seem to have hypothesis' that guide our interpretation of the numbers. In the absence of a guiding framework on how to interpret your data, you're bound to min-max individual criteria; we'll never know if we're just in a local maxima, however. I'm not saying this is an easy problem, mind you. It's easy to wax poetic about having your culture guide decisions when you can point at a graph and say "yep, we're making $xx,000 less per hour".

exodust 13y ago

The point is about design being held back or distorted by numbers - which are often a false prophesy when it comes to UX improvements. Nobody is suggesting that numbers don't matter, but if you can't decide on a border thickness of 1 or 2 pixels without numbers to tell you which to go with, you've lost your mojo. And that matters in the long term. When the aim is to increase length of time on the site or in the app, the numbers will "tell you" to make the interface more complicated, to hide the sign-out button (common practice), to hide the exit button (like Dropbox did on the desktop application), to force dependencies between unrelated services (Google does this a lot). When the aim is to increase clicks on Ads, the numbers will tell you to make ads look like normal posts. The numbers will tell you to autoplay videos on tabloid news article pages. But all these "optimizations" are not unlike speed cameras placed at the bottom of the hill where lots of people can't help bumping over the speed limit for a couple of seconds. Good for revenue, bad vibes in every other way.

ignostic 13y ago
> "design being held back or distorted by numbers - which are often a false prophesy when it comes to UX improvements." > "the numbers will "tell you" to make the interface more complicated, to hide the sign-out button" Oh yes, I definitely understand that we misinterpret what the numbers mean. My point is that this isn't the numbers' fault - it's our fault for not understanding what we're measuring and what the numbers mean. Often additional tests can even help resolve the "what should we do" question. We run tests all the time, but when we see something like, "pageviews are up x%" that's just a starting place. We begin to ask why and continue to develop a hypothesis and proof for it. It might be that users are confused so they start clicking around, or it could be because we've grabbed their attention and enticed them to learn more. Without the numbers we wouldn't even know where to start. If Dustin's point is just to think critically about test numbers that seems to go without saying.
- exodust 13y ago
  Fair enough. I agree the interpretation of the data we collect leaves a lot to be desired. Where I work, I've seen good products killed off because of low traffic, but the low traffic was due to people not finding the product due to poor navigation - which in turn was made poor because of marketing requirements to "drive traffic" to their latest short term gimmick. It's a complex web.

phillmv 13y ago

I think the problem is that we don't seem to have hypothesis' that guide our interpretation of the numbers. In the absence of a guiding framework on how to interpret your data, you're bound to min-max individual criteria; we'll never know if we're just in a local maxima, however. I'm not saying this is an easy problem, mind you. It's easy to wax poetic about having your culture guide decisions when you can point at a graph and say "yep, we're making $xx,000 less per hour".