3 ms·

>> I think that over-reliance on analytics has had a similar effect I lost track of how many times the directors would tell us, "Since the analytics say A, we

>> I think that over-reliance on analytics has had a similar effect I lost track of how many times the directors would tell us, "Since the analytics say A, we should do B." then the research person says, "Sure Jim, great idea, lets get a few rounds of useability research and confirm it first." Then the always predictable thing happens: Users never align with your analytics. Seeing data and seeing someone struggling to do something basic with your interface is totally different. When directors see these videos, it really makes them see how important research is and not just relying on data to make decisions. Right now, anything the research team wants, they usually get - its had that profound of an impact on our leadership team.

zahlman 22d ago
> Users never align with your analytics. Any good theories for how this happens (i.e. why the data fails to capture the struggle)?
- skydhash 22d ago
  Not GP, but I can humbly share mine. I think it's because a lot of analytics are data points oriented instead of being workflow oriented. So you can see that feature A is not being used a lot, but it's very important in a particular flow. Feature B may be used a lot, but it can be only important for a particular class of users while very detrimental mentally for another class. People use software for a needs, but rarely I've seen a need being highlighted when interpreting analytics data.
- layer8 22d ago
  Analytics don’t capture the “why” of what they measure, on the user side. Indicators may move in the seemingly right direction for the wrong reasons. Analytics typically can’t tell you what the user wanted to achieve. Knowing the “why” gives a better basis to decide on what changes to try, or to realize what’s actually wrong with the user-facing side of things. Another reason is that they tend to measure an average where in reality there is no average user.
- zmgsabst 22d ago
  Interpreting data is hard. Eg, if you have a bunch of airplanes returning from a war, you might be tempted to armor the areas where they were hit. But you want to armor planes where the holes arent because those are the critical areas. If you can read literal bullet holes backwards, hundred dimensional preference vectors say something, but we generally have no clue.
- zbentley 22d ago
  I think the answer to this is roughly ... the content of an applied statistics degree, or a research methodology practicum. There are so, so many ways that quantitative measurement of human phenomena can mislead, from measurement error to conflation between trends/sample sizes/effect sizes, to p-hacking, to good old bias (in interpretation or in deciding what tools to use to smooth or normalize data), and thousands more.
asdfman123 22d ago
My problem is I can skip all those steps and 8 separate meetings by applying a little common sense to software design. Unfortunately, orgs often treat devs like this as indulgent or wasting time. They need it to be done the inefficient way because that's the only way they have visibility and control.
- cm11 22d ago
  Yes. And in most of those meetings where the researcher invalidated the leaders, the leaders have left the meeting saying (and thinking? I’m not sure) they were validated. So because of the leaders we have we should skip the meetings and research in both cases.

zahlman 22d ago

> Users never align with your analytics. Any good theories for how this happens (i.e. why the data fails to capture the struggle)?

skydhash 22d ago
Not GP, but I can humbly share mine. I think it's because a lot of analytics are data points oriented instead of being workflow oriented. So you can see that feature A is not being used a lot, but it's very important in a particular flow. Feature B may be used a lot, but it can be only important for a particular class of users while very detrimental mentally for another class. People use software for a needs, but rarely I've seen a need being highlighted when interpreting analytics data.
layer8 22d ago
Analytics don’t capture the “why” of what they measure, on the user side. Indicators may move in the seemingly right direction for the wrong reasons. Analytics typically can’t tell you what the user wanted to achieve. Knowing the “why” gives a better basis to decide on what changes to try, or to realize what’s actually wrong with the user-facing side of things. Another reason is that they tend to measure an average where in reality there is no average user.
zmgsabst 22d ago
Interpreting data is hard. Eg, if you have a bunch of airplanes returning from a war, you might be tempted to armor the areas where they were hit. But you want to armor planes where the holes arent because those are the critical areas. If you can read literal bullet holes backwards, hundred dimensional preference vectors say something, but we generally have no clue.
zbentley 22d ago
I think the answer to this is roughly ... the content of an applied statistics degree, or a research methodology practicum. There are so, so many ways that quantitative measurement of human phenomena can mislead, from measurement error to conflation between trends/sample sizes/effect sizes, to p-hacking, to good old bias (in interpretation or in deciding what tools to use to smooth or normalize data), and thousands more.

asdfman123 22d ago

My problem is I can skip all those steps and 8 separate meetings by applying a little common sense to software design. Unfortunately, orgs often treat devs like this as indulgent or wasting time. They need it to be done the inefficient way because that's the only way they have visibility and control.

cm11 22d ago
Yes. And in most of those meetings where the researcher invalidated the leaders, the leaders have left the meeting saying (and thinking? I’m not sure) they were validated. So because of the leaders we have we should skip the meetings and research in both cases.