5 ms·
I’m convinced 90%+ of telemetry is simply to have usage metrics to show managers, rather than actually being used to improve products by identifying inefficienc
by transcriptase 4y ago
I’m convinced 90%+ of telemetry is simply to have usage metrics to show managers, rather than actually being used to improve products by identifying inefficiencies and bugs. And it’s a malevolent spectre if you’re the 1% of users who actually use a niche feature in something, since developers will use that knowledge to take it away.
Despite a 100-fold increase in hardware power, and languages/frameworks that handle 80% of the workload a developer 15 years had to think about, applications these days have fewer settings, open and run slower, and improve far less rapidly than anything written in C/C++/Java from 1990-2005 despite phoning home with novels worth of telemetry every few minutes.
Monitoring network traffic and DNS requests through a firewall and pihole, it’s downright hilarious that it’s the worst products that top the charts in both, even when being online is not a necessary component of the software. Meanwhile the software that just works, and works well, might check for an update once a month, and the changelog generally reveals new useful features and mentions performance increases.
- gcatalfamo 4y ago> And it’s a malevolent spectre if you’re the 1% of users who actually use a niche feature in something, since developers will use that knowledge to take it away. Yes, but also no. If my resources are not infinite, knowing that an "expensive-to-maintain" feature is used by just 1% of users, it is an incredibly important piece of information when I'm gonna have to decide where/how to recalibrate my (team's) effort. Product managing is also this: I have to know what works and what doesn't. Maybe I will use that knowledge to understand how to increase the 1% to 5% with a specific business case. Maybe it won't make sense and I will have to decide if I can keep it or not. I agree that vanity telemetry is bad for users, but killing ALL telemetry is bad for everybody.
- transcriptase 4y agoI don’t disagree with anything you’ve said, and this is only tangentially related, but I wonder how much the long-tail of usage is impacted by the likely correlation of power-users/niche features, and people who firewall/pihole telemetry. I also wonder how much of a chicken-egg effect there is on the trend of developers hiding niche/power-user features that they would prefer to sunset deep into settings menus and then that setting seeing negligible use.
- gcatalfamo 4y agoThat is usually information you already know from other sources (e.g. is the 1% feature a feature that is pivotal to getting new users?) (e.g. is the 1% feature a feature that is key for a specific audience within my total addressable market?) But you have know from somewhere that the 1% is, in fact, 1%...makes sense?
- hedora 4y agoOk, but if we focus on results, why has software usability and functionality declined with the rise of telemetry. I personally think it is because program managers rely on telemetry instead of stakeholder research and industrial design, either because they don’t know better, or because management thinks it will cut costs. Besides, telemetry is a feature that 0% of users use, so it should be the first to be cut.
- dividedbyzero 4y agoSubjectively speaking, while software I use these days has a lot less settings, I also tend to need to change settings a lot less than back in the day, defaults tend to be just right a lot more often. So at least for me it seems to be working, at least somewhat.
- JadeNB 4y ago> Subjectively speaking, while software I use these days has a lot less settings, I also tend to need to change settings a lot less than back in the day, defaults tend to be just right a lot more often. So at least for me it seems to be working, at least somewhat. But there's a trade-off there—as, ideally speaking, most software has become easier for most people to use mostly satisfactorily right out of the box, it becomes all the harder to adapt if you're one of the people for whom it doesn't work mostly satisfactorily, since there's no real culture of design around the idea that a user might want to use software in the way that they, rather than the designer, intended. Even if it's fair to write off those niche users, there are still the most users for whom the software works mostly satisfactorily, but can't be modified to be perfect, again because the culture of user customisability just isn't there.
- JohnFen 4y agoRight. There used to be a design philosophy of making the defaults fit the majority of users, but to expose plenty of settings so that more advanced users, including those who have grown into the software, can still do what they need to do, and can work around design decisions that are bad for them. That seemed to be a reasonable compromise to me.
- BiteCode_dev 4y agoUnfortunately that's the conclusion that most managers will draw, without asking more questions. But knowing that only 1% of users use this feature is not enough to make that decision: - maybe 1% of your users only use it, but they are the ones that pay the most money - or it's the most prolific users so it generates half of your content - or it's a niche feature used by power users, but those users are the ones bringing other users, introducing you to companies and doing free support - or it needs to be used by only 1% of the users by design, and it's enough to be useful (E.G: bug report form) - or your report it wrong, it's one percent of all accounts, but 10% of actually active users etc. Most telemetry will not tell you that of course.
- soiler 4y agoWell, tracking users by cookies or fingerprinting or whatever can tell you that by combining data from different sources. Now we've looped back around to being in favor of the most intrusive methods.
- BiteCode_dev 4y agoThat's my point, if you want really useful information, you need identifiable users, and for that, you need consent. So ask for it. All data will be biased and incomplete anyway, so at least acquire it in an ethical manner.
- gcatalfamo 4y agoAs I said that is a "piece of information", not the whole story. Making an axing decision with ONLY that information would be outright idiotic. Telemetry has to be designed as well, to piece as much as you can, and then you complete the full picture with other sources. Maybe not everybody does this, but I don't think that this practice should be "shocking" for anybody.
- katbyte 4y agoyea you sound like someone who is competent and will actually thing things through and make a rational decision. Many people are not and will latch on to the information "only 1% of people use it!" and refuse to listen to why that may not matter"
- djKianoosh 4y agothe example I like to always use is when EA kept abandoning Madden (especially online franchise mode and gameplay) for like 10+ years straight (while making money hand over fist on MUT). One time there was this new mode, for like 1-2 years, of 3v3 online. it was probably the most fun mode madden has ever had. it just needed a few tweaks. sensible stuff. But it would've been amazing! but no. because it was difficult to get to in the menu system, fewer people used it and few people knew about it. so of course telemetry is going to say few people used it. but it had a TON of potential! a product owner made the wrong decision to cut that feature, and guaranteed they used misleading telemetry to justify it.
- tomrod 4y agoPay users for research. They are not your labrats. Is this really even a question?
- HWR_14 4y agoThe problem isn't that you have a feature used by 1% of the population. It's that you have like 60 features each used by 1% of the population, but with not much overlap. So all the 1% features get axed and suddenly you have only 50% of the users.
- TJSomething 4y agoIf you axe 60 features and there are 20 features left that are used by that 50 percent, then that might be a legitimate tradeoff from a business perspective.
- anonymousab 4y agoAh yes, the Firefox effect.
- gcatalfamo 4y agoAs I said that is a "piece of information", not the whole story. Making an axing decision with ONLY that information would be outright idiotic.
- slackdog 4y ago> So all the 1% features get axed and suddenly you have only 50% of the users. In practice things rarely get that far, thanks mostly due to the complete farce that is "data driven design". Such an approach to design would naturally lead to the axing of every unpopular feature if it were followed mechanistically, but in reality "data driven design" actually refers to the practice of mining to support whatever you (the designer/programmer/manager/etc) have already decided to do, and the data is ignored for anything else. It actually works like this: I form a subjective judgement against some feature which I dislike (maybe because it's a pain to maintain or just because it doesn't suit my preferences for any reason.) Then, I look in the telemetry and start making charts of things until I can find some combination of the data that seems to support my prejudice against the feature. I bury all the presentations of the data which failed to support or even refuted me, I pretend I never saw them. Then I write up my report claiming that the feature should be removed for objective scientific reasons, and my fancy charts and graphs prove it! It's totally not just my subjective prejudices driving the design, no Siree... Besides the privacy/consent concerns, this is why telemetry is bad for users. It devalues dialogue between the users and developers and replaces it with pseudo-scientific arguments based on nothing but the developer's preferences. That's what it's really for, an excuse to never talk with the users.
- ClumsyPilot 4y ago> If my resources are not infinite, knowing that an "expensive-to-maintain" feature is used by just 1% of users Only 1% of mobile phone users need 911 regularly. Lets remove that feature.0
- gcatalfamo 4y ago> Only 1% of mobile phone users need 911 regularly. Lets remove that feature.0 If this is the type of PM you are accustomed to, fire them. Because that's a dumb argument.
- dogleash 4y agoYou know that a lot of legislative and regulatory effort goes into making sure 911 works on as many phones as it does right? > If this is the type of PM you are accustomed to, fire them. People's expectations about how projects treat 911 are born from those hard requirements. If it didn't have those requirements, you would find that every PM is the kind of PM that would let the service suffer. Maybe not scrap 911 altogether, but certainly deliver a less robust and reliable 911 service than they do now.
- klooney 4y agoOr if you're Google, promote them (see pixel 911 issues)
- ClumsyPilot 4y agoUnfortunately there are many PM who are basically that type.
- ryandrake 4y ago> Product managing is also this: I have to know what works and what doesn't. You can't use a blunt usage metric to tell whether a feature works or not. It only tells you that the feature is popular for some reason. That reason might be that it works, but often the reason people use a feature a lot is it's shoved down their throats. It's a vicious cycle. Say you are Product Manager for Feature X. Your bonus/promotion/career progression depends on some vanity usage metric for feature X. The more monthly actives or new users for feature X, the better it is for you. So, naturally, you're going to do whatever you can to funnel users at that feature. Stick it prominently on the main screen. Highlight it in flashing red. Spam the user with notifications begging them to use it. Make competing features low-contrast gray on gray. Whatever you can get away with. Then the next monthly metrics report is in: The feature is now being used by +15% of users since last month! Incredible! Your bonus buys you a Tesla. Management's conclusion is: "This feature really works and users must really like this feature." So, the product focuses more on developing that feature. Now scale the above out to every feature in the app, each with their own competitive and motivated Product Manager. This is how we get apps that are all-telemetry dumpster fires and with more and more flashy features fighting each other to grab your attention and fewer and fewer useful 1%-features. I've personally witnessed meetings with multiple furious product owners literally shouting at each other about where their feature's button gets placed on the home screen because that placement made the difference between 5% feature usage and 25% feature usage. Where was the end user in all these discussions? Not present. End users are to be milked for metrics gains.
- gcatalfamo 4y ago> You can't use a blunt usage metric to tell whether a feature works or not. It only tells you that the feature is popular for some reason. That reason might be that it works, but often the reason people use a feature a lot is it's shoved down their throats. I never said that was the only source of information, but indeed a useful one. Thinking a good PM only uses that one bit of info means you have known really bad PMs in your work.
- ryandrake 4y agoThe product managers know this, but the higher-ups (who control the money flow) tend to get tunnel-vision on a single metric, and that metric is often going to be the one most understandable to a layman. So that's the one that gets juiced.
- JohnFen 4y ago> killing ALL telemetry is bad for everybody. I don't think that a good case has been made for this. Telemetry does bring benefits -- it's cheaper than the old way, and captures useful things that couldn't be captured without it. But it also comes with some costs that are far from insignificant, especially for users. I don't see any real reason to think that on the whole, telemetry has been beneficial for users. I think a stronger case can be made that it's beneficial for software makers.
- Dalewyn 4y ago>killing ALL telemetry is bad for everybody. If you want my data you can fucking pay me for it. Otherwise, GTFO. I'm here to use your product, not give you data that has nothing to do with your product without my permission.
- gcatalfamo 4y agobut telemetry HAS to do with my product, not you.
- Dalewyn 4y agoYour needs are none of my concern. If you want something from me, pay up.
- JohnFen 4y agoYou cannot separate the two. Telemetry is primarily about the user. What features do they use? What is their pattern of usage? Etc. So it's about the user as well as about the software. If it weren't, the only telemetry that would be of any actual value to you would be crash reports.
- Am4TIfIsER0ppos 4y agoHow much work is it for firefox to maintain their text encoding conversion? How much work is it to maintain the menu entry that lets users control it?
- lakomen 4y ago> I agree that vanity telemetry is bad for users, but killing ALL telemetry is bad for everybody It's good for my privacy
- newaccount74 4y agoI think that 90% of telemetry is useless (eg. how often a feature is used says nothing about how important it is), but telemetry related to errors and crashes is extremely important. My experience with shipping software is that if 50 people encounter a bug, maybe 1 of them will click the "feedback" button to report it. If you don't have a feedback button, it's less than that. If you don't have a technical audience, it's even less again. So if you want to ship high quality software, then automatic error reporting is essential.
- madeofpalk 4y ago> how often a feature is used says nothing about how important it is Exactly. But that doesn't mean it's not important. If you think something is important, but it's not being used, it's something to look into.
- Gh0stRAT 4y agoOn a related note, exploiting security vulnerabilities is often non-deterministic. There might be thousands of errors or process crashes before the exploit is successful, and if at least some of those get reported to your telemetry servers it becomes a goldmine for discovering zero-days which are actively being exploited. Microsoft has people who hunt for evidence of zero-days in their telemetry, for example.
- Shared404 4y agoI think opt out error reporting that prompts the user to send a report and lets them review it first is very worthwhile.
- JohnFen 4y agoI would be much more inclined to click a "report bug" button if software didn't put itself into an adversarial position by engaging in shady telemetry tactics. But, especially these days, you pretty much have to regard all software as being a bit hostile.
- Dalewyn 4y agoI don't bother to report bugs because I know I'm just speaking to /dev/null. If you want to ship high quality software, listen to your customers and make it clear their words really matter. No, telemetry does not count; that only reinforces I'm speaking to /dev/null.
- _boffin_ 4y agoYou’re thinking too simply. I think you’re correct in that the telemetry is to show managers, but once it’s in their central log aggregator, there’s another team that then uses that and other aggregated logs from other teams to make business decisions
- than3 4y agoMetadata is useful in building profiles on people. Master Data Management pipelines and systems make this processing quite easy. Third party companies like Accurint will pay for data. Very rarely is it just how much time are you spending in various features.