4 ms·

Absolutely. I think that over-reliance on analytics has had a similar effect; compared to relying primarily on proper usability research and in-person user stud

Absolutely. I think that over-reliance on analytics has had a similar effect; compared to relying primarily on proper usability research and in-person user studies, doing that creates a detachment and distance that further removes the project from reality. How great the impact is depends on how management uses the data. If they use it add resolution to broad strokes from studies and the like, there may be no negative impact at all, but in my experience it's much, much more common for management to read analytics like tea leaves and interpret it in whatever way best fits the individual's/team's biases/agendas.

anonymars 16d ago
Simple example: feature X is rarely used (thus we should get rid of it) Hold on, why is it rarely used? Is it hard to use? Is it hard to find? Was it poorly named? Does it work right? Does it get me 80% of the way there? Does it get me 20% of the way there? Do I rarely need it but when I do need it it's a huge time saver? Am I hesitant to depend on it because I fear it will be taken away in a future update?
- cosmic_cheese 16d ago
  And following this, I'm sure we've all seen the pattern of manufacturing the data required to justify the removal of features. Bury feature → usage numbers tank → oh hey look nobody uses this any more, guess we can safely cut it!
- mjr00 16d ago
  > Simple example: feature X is rarely used (thus we should get rid of it) Oh man, I remember a very specific example of this: many years ago now, Google Chrome pushed an update that got rid of the option on the menu bar for "Close Tabs to the Right". I remember looking into the Google issue tracker where people were complaining, and some PM provided a "data-driven" justification: when people opened the context menu, they only clicked on the "Close Tabs to the Right" option 1-2% of the time. It's a great example of why data without context can give you the wrong answer. Of course the option is used relatively very rarely; you only need to clear out your tabs every once in a while, compared to creating new ones or managing tab groups! But it's still an essential task. It's like saying filing your taxes isn't important because you only need to do it 0.2% days of the year.
  - zahlman 16d ago
    For what it's worth, I definitely don't organize my tabs in a way where "close to the right" would be helpful in cleaning them up.
    - skydhash 16d ago
      It's useful when starting a thread of research of research as new tabs will be opened to the right (Ctrl+Clicking a link). So after it's done, close to the right can be quite useful.
    - fragmede 16d ago
      I do. I open a search page, middle click a bunch of links, then want to close all of the ones to the right because I found one that told me what I needed and don't need the rest of them.
      - anonymars 16d ago
        Probably a perfect example of why it's so useful for developers to talk to users Not only to understand the usage of features, but also use cases perhaps to design even better features
    - bluefirebrand 16d ago
      I do all the time because the feature exists. I would treat it like a sliding line in my tabs bar. When I'm done with a tab for now it goes to the right, and eventually everything to the right gets closed once I haven't used it in a while
  - ronjakoi 16d ago
    I don't like that they have that data in the first place. But then again I don't use Chrome.
- Terr_ 16d ago
  Conversely: Feature X is often used... Because to drive adoption someone made it mandatory for a certain workflow and it nags everyone else so they "engage" with it just enough to make it go away. (Looking at you, "AI" buttons...)
burningChrome 16d ago
>> I think that over-reliance on analytics has had a similar effect I lost track of how many times the directors would tell us, "Since the analytics say A, we should do B." then the research person says, "Sure Jim, great idea, lets get a few rounds of useability research and confirm it first." Then the always predictable thing happens: Users never align with your analytics. Seeing data and seeing someone struggling to do something basic with your interface is totally different. When directors see these videos, it really makes them see how important research is and not just relying on data to make decisions. Right now, anything the research team wants, they usually get - its had that profound of an impact on our leadership team.
- zahlman 16d ago
  > Users never align with your analytics. Any good theories for how this happens (i.e. why the data fails to capture the struggle)?
  - skydhash 16d ago
    Not GP, but I can humbly share mine. I think it's because a lot of analytics are data points oriented instead of being workflow oriented. So you can see that feature A is not being used a lot, but it's very important in a particular flow. Feature B may be used a lot, but it can be only important for a particular class of users while very detrimental mentally for another class. People use software for a needs, but rarely I've seen a need being highlighted when interpreting analytics data.
  - layer8 16d ago
    Analytics don’t capture the “why” of what they measure, on the user side. Indicators may move in the seemingly right direction for the wrong reasons. Analytics typically can’t tell you what the user wanted to achieve. Knowing the “why” gives a better basis to decide on what changes to try, or to realize what’s actually wrong with the user-facing side of things. Another reason is that they tend to measure an average where in reality there is no average user.
  - zmgsabst 16d ago
zer00eyz 16d ago
> I think that over-reliance on analytics has had a similar effect; The over reliance on analytics combine with a lack of metrics and proper accounting. If you are renting all your infrastructure knowing how people use your app, and what the COSTS of that are is kind of a big deal. If your high dollar client is your lowest margin one, thats a problem that is technical and financial as well as a product insight. And that infrastructure your renting, it stopped making sense for a lot of orgs to do that almost a decade ago, but here we are where everything is in the cloud because capacity planning is a lost art and was a great throttle on the insanity that the post is describing.

anonymars 16d ago

Simple example: feature X is rarely used (thus we should get rid of it) Hold on, why is it rarely used? Is it hard to use? Is it hard to find? Was it poorly named? Does it work right? Does it get me 80% of the way there? Does it get me 20% of the way there? Do I rarely need it but when I do need it it's a huge time saver? Am I hesitant to depend on it because I fear it will be taken away in a future update?

cosmic_cheese 16d ago
And following this, I'm sure we've all seen the pattern of manufacturing the data required to justify the removal of features. Bury feature → usage numbers tank → oh hey look nobody uses this any more, guess we can safely cut it!
mjr00 16d ago
> Simple example: feature X is rarely used (thus we should get rid of it) Oh man, I remember a very specific example of this: many years ago now, Google Chrome pushed an update that got rid of the option on the menu bar for "Close Tabs to the Right". I remember looking into the Google issue tracker where people were complaining, and some PM provided a "data-driven" justification: when people opened the context menu, they only clicked on the "Close Tabs to the Right" option 1-2% of the time. It's a great example of why data without context can give you the wrong answer. Of course the option is used relatively very rarely; you only need to clear out your tabs every once in a while, compared to creating new ones or managing tab groups! But it's still an essential task. It's like saying filing your taxes isn't important because you only need to do it 0.2% days of the year.
- zahlman 16d ago
  For what it's worth, I definitely don't organize my tabs in a way where "close to the right" would be helpful in cleaning them up.
  - skydhash 16d ago
    It's useful when starting a thread of research of research as new tabs will be opened to the right (Ctrl+Clicking a link). So after it's done, close to the right can be quite useful.
  - fragmede 16d ago
    I do. I open a search page, middle click a bunch of links, then want to close all of the ones to the right because I found one that told me what I needed and don't need the rest of them.
    - anonymars 16d ago
      Probably a perfect example of why it's so useful for developers to talk to users Not only to understand the usage of features, but also use cases perhaps to design even better features
  - bluefirebrand 16d ago
    I do all the time because the feature exists. I would treat it like a sliding line in my tabs bar. When I'm done with a tab for now it goes to the right, and eventually everything to the right gets closed once I haven't used it in a while
- ronjakoi 16d ago
  I don't like that they have that data in the first place. But then again I don't use Chrome.
Terr_ 16d ago
Conversely: Feature X is often used... Because to drive adoption someone made it mandatory for a certain workflow and it nags everyone else so they "engage" with it just enough to make it go away. (Looking at you, "AI" buttons...)

burningChrome 16d ago

>> I think that over-reliance on analytics has had a similar effect I lost track of how many times the directors would tell us, "Since the analytics say A, we should do B." then the research person says, "Sure Jim, great idea, lets get a few rounds of useability research and confirm it first." Then the always predictable thing happens: Users never align with your analytics. Seeing data and seeing someone struggling to do something basic with your interface is totally different. When directors see these videos, it really makes them see how important research is and not just relying on data to make decisions. Right now, anything the research team wants, they usually get - its had that profound of an impact on our leadership team.

zahlman 16d ago
> Users never align with your analytics. Any good theories for how this happens (i.e. why the data fails to capture the struggle)?
- skydhash 16d ago
  Not GP, but I can humbly share mine. I think it's because a lot of analytics are data points oriented instead of being workflow oriented. So you can see that feature A is not being used a lot, but it's very important in a particular flow. Feature B may be used a lot, but it can be only important for a particular class of users while very detrimental mentally for another class. People use software for a needs, but rarely I've seen a need being highlighted when interpreting analytics data.
- layer8 16d ago
  Analytics don’t capture the “why” of what they measure, on the user side. Indicators may move in the seemingly right direction for the wrong reasons. Analytics typically can’t tell you what the user wanted to achieve. Knowing the “why” gives a better basis to decide on what changes to try, or to realize what’s actually wrong with the user-facing side of things. Another reason is that they tend to measure an average where in reality there is no average user.
- zmgsabst 16d ago

zer00eyz 16d ago

> I think that over-reliance on analytics has had a similar effect; The over reliance on analytics combine with a lack of metrics and proper accounting. If you are renting all your infrastructure knowing how people use your app, and what the COSTS of that are is kind of a big deal. If your high dollar client is your lowest margin one, thats a problem that is technical and financial as well as a product insight. And that infrastructure your renting, it stopped making sense for a lot of orgs to do that almost a decade ago, but here we are where everything is in the cloud because capacity planning is a lost art and was a great throttle on the insanity that the post is describing.