5 ms·

I don't get it - it is a nice sentiment, but the crux of the argument seems to "go with your gut." The reason web analytics has been so popular is that you can

by gfodor 3y ago

I don't get it - it is a nice sentiment, but the crux of the argument seems to "go with your gut." The reason web analytics has been so popular is that you can actually improve your product significantly without as much subjectivity. Before web analytics, going with your gut was the only option. It turns out, threading events through a funnel that show you lost 90% of your users because your button doesn't show up on certain screen sizes is a meaningful capability. The question from there is how far do you go with it, but I don't think it makes any sense to both reject it and go on to propose such a fuzzy, underdefined alternative as "you'll gain intuition about if you're making connections." Having built a number of apps, I don't know how to gain such intution once the userbase becomes too large to interact with directly enough to grok it. It's very easy to be fooled by a vocal minority of people who complain about your app sucking when the majority are getting value from it. Without data and analytics, it's easy to be pushed around by your most vocal users, because your gut is saying you should listen to them. Sometimes you should, sometimes you shouldn't, but the difference between the two can often be determined by looking at the data to get a bigger, less biased picture of what is actually going on with your app.

awavering 3y ago
Absolutely, and a good reminder that your web analytics is only one source of data you can use when making decisions. Sites and apps generate so much interesting quantitative and qualitative information that is important to compare and validate assumptions.
mighmi 3y ago
I've personally never been in a situation where analytics drove change. Even mere A/B testing seems like a ghost in (most?) organizations, where management or designers decide to force through their personal preferences. That said, I would love to join such an organization. Any suggestions on where to find data driven jobs?
- dstroot 3y ago
  > Any suggestions where to find data driven jobs? My experiences suggest a large measure of caution. Many organizations describe themselves as data-driven and many actually do try. However humans are humans and many will be “data driven” when it aligns with what they want to do, otherwise it’s off for “more data” or a “new perspective”. In other words sometimes data driven means use data when it supports my already staked out position. I hate to be so cynical but I can’t help it.
  - exolymph 3y ago
    One might argue that your cynicism is (drum roll please) data driven.
- tensor 3y ago
  It may be less used in driving website design, but I assure you it is very very widely used in marketing to determine which campaigns are working.
- dustymcp 3y ago
  Amazon?
  - fmajid 3y ago
    The second half of this article explains Amazon's process around metrics, well worth a read: https://commoncog.com/goodharts-law-not-useful/ https://commoncog.com/goodharts-law-not-useful/
- nine_k 3y ago
  At my previous job, A/B testing was widely used to check which of the small UX tweaks works better.
  - exodust 3y ago
    The problem there is many uncontrollable factors contributing to the data. If engagement is found to be better with A on the day, does that mean B will never be better? It doesn't. A/B testing is often conducted unscientifically or with insufficient sample size and timeframe. Sometimes the cards fall where they fall, and your small UX tweak wasn't involved. It's tempting to conclude A/B testing delivered a valid result. As an experiment, if you deliberately made A and B identical, then A/B test as normal, you will still get a winner. They won't be equal.
    - nine_k 3y ago
      This is fair; valid experiments need to run for weeks, not hours, to account for cyclic variations, should not be confounded by some events (like holidays, elections, etc) that severely change relevant user activity, etc, and need to bring large enough amounts of data points for the results to be able to be statistically significant.
    - gfodor 3y ago
      Well yeah you have to actually do it right.
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
marcosdumay 3y ago
> Before web analytics, going with your gut was the only option. The analytics people insistent denial that usability research ever existed seems to have reached a new level.
- gfodor 3y ago
  What I wrote wasn't an insistent denial, try steelmanning next time. Extrapolating the magnitude of problems from user research is hard, since you're still sampling a distribution.
  - hedora 3y ago
    If the already-released version of your product has so many problems that you have to use computational techniques to extrapolate their magnitude, then you have already lost.
    - gfodor 3y ago
      You can find problems like “oops, the Android they use in India on phone model X has chrome that blocks the screen every week so they can’t get passed the login flow” It’s about better sampling of the actual distribution. User testing has huge selection bias.
  - marcosdumay 3y ago
    What you wrote was just denial. The insistent part comes from every time those discussions going this same way. Anyway, and FYI, one of the most robust findings in user research is that the magnitude of problems isn't really something important to know.
barnabee 3y ago
> improve your product significantly without as much subjectivity It's fake objectivity that often leads to a race to the boottom. Between "engagement" as a terrible proxy for user happiness/success and the idea that it's more important and better if nobody can fail to use your app's "UX" than it is if it to enables a smaller number of people who take the time to learn to do really great things, all wrapped up in the VC driven dream of hypermegablitzscaling, all a reliance on analytics does is guarantee mediocrity. I want software created by people who care enough not to look at the data.
- nine_k 3y ago
  Analytics data are like emotions: a valuable source of information which should not be used as a direct controlling signal. It should be first thought through.
- gfodor 3y ago
  I didn’t say it was objective. I said it allows you to be less subjective. I have no idea how anyone could think otherwise unless you’ve never had, for example, analytics show you your visual design or interactive flow is literally broken on specific but rare device/platform/browser/etc combos. (Just one example.) As usual, it’s possible to be stupid and abuse tools, but what I wrote presumes you’re not doing something stupid. If you think it’s impossible to not be stupid when it comes to this tool, this is false.
hedora 3y ago
You could hire a decent QA department. Then you'd run the app on all screen sizes before each git push and notice that the button doesn't always show up, or isn't always clickable.
- gfodor 3y ago
  The point is that there is a swath of failure modes that only become apparent when you actually have instrumentation point at them first. QA can’t fish these out.
pcmaffey 3y ago
The pendulum has swung too far. We quantify everything now, chasing numbers to the point of losing our sense of quality and craftsmanship. There's a huge difference between "going with your gut" and the ability to assess, qualify, and reason about something. To understand how and why something works or doesn't. Whether from first principles or experience. Numbers provide a happy path for consensus. If the "why" of your project is to please as many people as possible, then web analytics probably works great. I'm not arguing they're categorically useless, only trying to surface the unspoken costs I see bear out in our industry from relying on them so heavily.
- gfodor 3y ago
  The original post proposes they be abandoned. So I’m arguing against that.
1vuio0pswjnm7 3y ago
"It's very easy to be fooled by a vocal minority of people who complain about your app sucking when the majority are getting value from it." Does this apply to "feature requests" by a minority whereby the majority then has to accept those "features".
- djbusby 3y ago
  Yes! Early stage businesses can get bogged down in "next feature folly" - keep adding and adding as each new (bespoke?) client comes on. Then the product has features baked in for narrow cases and turns off the larger part of the bell curve. My line in the sand is: never build a feature on the first request. Ideate, research, measure others opinion of the same features. One should have a tight beta group for this.
wackget 3y ago
> show you lost 90% of your users because your button doesn't show up on certain screen sizes How does analytics do that? A mobile usability scanner might pick that up, but analytics?

by gfodor 3y ago

awavering 3y ago

Absolutely, and a good reminder that your web analytics is only one source of data you can use when making decisions. Sites and apps generate so much interesting quantitative and qualitative information that is important to compare and validate assumptions.

mighmi 3y ago

I've personally never been in a situation where analytics drove change. Even mere A/B testing seems like a ghost in (most?) organizations, where management or designers decide to force through their personal preferences. That said, I would love to join such an organization. Any suggestions on where to find data driven jobs?

dstroot 3y ago
> Any suggestions where to find data driven jobs? My experiences suggest a large measure of caution. Many organizations describe themselves as data-driven and many actually do try. However humans are humans and many will be “data driven” when it aligns with what they want to do, otherwise it’s off for “more data” or a “new perspective”. In other words sometimes data driven means use data when it supports my already staked out position. I hate to be so cynical but I can’t help it.
- exolymph 3y ago
  One might argue that your cynicism is (drum roll please) data driven.
tensor 3y ago
It may be less used in driving website design, but I assure you it is very very widely used in marketing to determine which campaigns are working.
dustymcp 3y ago
Amazon?
- fmajid 3y ago
  The second half of this article explains Amazon's process around metrics, well worth a read: https://commoncog.com/goodharts-law-not-useful/ https://commoncog.com/goodharts-law-not-useful/
nine_k 3y ago
At my previous job, A/B testing was widely used to check which of the small UX tweaks works better.
- exodust 3y ago
  The problem there is many uncontrollable factors contributing to the data. If engagement is found to be better with A on the day, does that mean B will never be better? It doesn't. A/B testing is often conducted unscientifically or with insufficient sample size and timeframe. Sometimes the cards fall where they fall, and your small UX tweak wasn't involved. It's tempting to conclude A/B testing delivered a valid result. As an experiment, if you deliberately made A and B identical, then A/B test as normal, you will still get a winner. They won't be equal.
  - nine_k 3y ago
    This is fair; valid experiments need to run for weeks, not hours, to account for cyclic variations, should not be confounded by some events (like holidays, elections, etc) that severely change relevant user activity, etc, and need to bring large enough amounts of data points for the results to be able to be statistically significant.
  - gfodor 3y ago
    Well yeah you have to actually do it right.

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]

marcosdumay 3y ago

> Before web analytics, going with your gut was the only option. The analytics people insistent denial that usability research ever existed seems to have reached a new level.

gfodor 3y ago
What I wrote wasn't an insistent denial, try steelmanning next time. Extrapolating the magnitude of problems from user research is hard, since you're still sampling a distribution.
- hedora 3y ago
  If the already-released version of your product has so many problems that you have to use computational techniques to extrapolate their magnitude, then you have already lost.
  - gfodor 3y ago
    You can find problems like “oops, the Android they use in India on phone model X has chrome that blocks the screen every week so they can’t get passed the login flow” It’s about better sampling of the actual distribution. User testing has huge selection bias.
- marcosdumay 3y ago
  What you wrote was just denial. The insistent part comes from every time those discussions going this same way. Anyway, and FYI, one of the most robust findings in user research is that the magnitude of problems isn't really something important to know.

barnabee 3y ago

> improve your product significantly without as much subjectivity It's fake objectivity that often leads to a race to the boottom. Between "engagement" as a terrible proxy for user happiness/success and the idea that it's more important and better if nobody can fail to use your app's "UX" than it is if it to enables a smaller number of people who take the time to learn to do really great things, all wrapped up in the VC driven dream of hypermegablitzscaling, all a reliance on analytics does is guarantee mediocrity. I want software created by people who care enough not to look at the data.

nine_k 3y ago
Analytics data are like emotions: a valuable source of information which should not be used as a direct controlling signal. It should be first thought through.
gfodor 3y ago
I didn’t say it was objective. I said it allows you to be less subjective. I have no idea how anyone could think otherwise unless you’ve never had, for example, analytics show you your visual design or interactive flow is literally broken on specific but rare device/platform/browser/etc combos. (Just one example.) As usual, it’s possible to be stupid and abuse tools, but what I wrote presumes you’re not doing something stupid. If you think it’s impossible to not be stupid when it comes to this tool, this is false.

hedora 3y ago

You could hire a decent QA department. Then you'd run the app on all screen sizes before each git push and notice that the button doesn't always show up, or isn't always clickable.

gfodor 3y ago
The point is that there is a swath of failure modes that only become apparent when you actually have instrumentation point at them first. QA can’t fish these out.

pcmaffey 3y ago

The pendulum has swung too far. We quantify everything now, chasing numbers to the point of losing our sense of quality and craftsmanship. There's a huge difference between "going with your gut" and the ability to assess, qualify, and reason about something. To understand how and why something works or doesn't. Whether from first principles or experience. Numbers provide a happy path for consensus. If the "why" of your project is to please as many people as possible, then web analytics probably works great. I'm not arguing they're categorically useless, only trying to surface the unspoken costs I see bear out in our industry from relying on them so heavily.

gfodor 3y ago
The original post proposes they be abandoned. So I’m arguing against that.

1vuio0pswjnm7 3y ago

"It's very easy to be fooled by a vocal minority of people who complain about your app sucking when the majority are getting value from it." Does this apply to "feature requests" by a minority whereby the majority then has to accept those "features".

djbusby 3y ago
Yes! Early stage businesses can get bogged down in "next feature folly" - keep adding and adding as each new (bespoke?) client comes on. Then the product has features baked in for narrow cases and turns off the larger part of the bell curve. My line in the sand is: never build a feature on the first request. Ideate, research, measure others opinion of the same features. One should have a tight beta group for this.

wackget 3y ago

> show you lost 90% of your users because your button doesn't show up on certain screen sizes How does analytics do that? A mobile usability scanner might pick that up, but analytics?