3 ms·
Not really. There's lots of other ways to learn about how your product is used, and without them telemetry will lead to some bad decisions. I've personally had
by indymike 5y ago
Not really. There's lots of other ways to learn about how your product is used, and without them telemetry will lead to some bad decisions. I've personally had to clean up the mess when telemetry driven decisions go sideways so many times:
* No one seems to use American express, so let's remove it from payment options. Turns out, the site had a bug that made sure 100% of AMEX transactions would fail. By fixing, we picked up 6% in sales, and lost a UX associate who was (and probably still is) arguing to drop AMEX because analytics.
* Since very few people actually click the apply button on the job board, let's remove the apply feature. The idea was we could send candidate profiles to employers based on views instead of apply clicks. Had to revert this one.
* Same company: let's take job posts out of our mobile job search app because most interactions end on a job post (and a click on the apply button). Best ticket the next day, "I can't find any jobs in this job search app?"
* No one uses the UI customizer (dashboard colors, fonts, and preferences). 100% of the tickets the next day were from people who were PISSED BEYOND BELIEF that we removed their favorite feater. Turns out the customizer basically was invisible to our app analytics. Had to revert this and re-do two sprints of work because the other new features wouldn't work with the customizer.
In 100% of the cases, talking to actual users would have been beyond wise before using analytics data to make decisions.
- jonpurdy 5y agoTotally agreed. Just because a feature is used by 2% of users doesn't mean it should be killed. That feature (plus others) could be critical to the workflow of those users. It seems the telemetry-driven product development in the past decade has made significantly worse apps than in previous decades. There are definitely a higher quantity of apps on a wider variety of platforms, but it seems harder to find apps that are powerful enough to allow a wide variety of users to do all the things they need.
- mixmastamyk 5y agoThe long tail of features is where a system achieves its status as "polished." Is this why software never gets polished any more? (And no, all features don't need to be visible at once.)
- jmull 5y agoTelemetry is just data. It can help you make good decisions, but can't prevent you from making bad ones. The general lesson in your examples isn't that telemetry isn't useful, but that telemetry gives an incomplete picture.
- majewsky 5y agoTelemetry is part of a toxic pattern: It gives people a sense of control, which makes them feel more confident in their decisions, even though that confidence is usually not actually backed up by anything meaningful.
- indymike 5y agoTelemetry isn't the problem. Making assumptions about what the telemetry means is where most of us go wrong.
- pfranz 5y agoI'm gathering you're not against telemetry, just against over relying on it? Would the AMEX issue have been found without telemetry drawing attention to it?