3 ms·

> https://support.microsoft.com/en-us/help/4468236/diagnostics-feedback-and-privacy-in-windows-10-microsoft-privacy https://support.microsoft.com/en-us/help/446

> https://support.microsoft.com/en-us/help/4468236/diagnostics-feedback-and-privacy-in-windows-10-microsoft-privacy https://support.microsoft.com/en-us/help/4468236/diagnostics... Thanks for sharing the link. However, it doesn’t change my point of view about Windows telemetry. As I read that page, I see a lot of BS justifying why the only way to improve the product is to have telemetry. The reason why I call it BS is because they use phrases like “we hear your feedback to improve the product”... that’s lie. I work in UX, and I’ve been involved in user research many times. The data that you get from event logs is related to measuring adoption and engagement. You get better UX feedback from traditional research methods. Telemetry is good for fast market analysis, rather than UX... but that doesn’t sound good when you try to justify it to end users. Take for example the design of the Office ribbon (there is an excellent video explaining the process). After analyzing the data from usage logs (this was in the pre-telemetry era), and doing interviews they concluded that the main issue was that many features were hard to find (is interesting to note that the conclusion came from user testing, not event logs). The solution: a UI showing all the operations (the ribbon). And because they found that paste was the most used action, they put a big paste button first (this came from event analysis). After the success of the ribbon, MS adopted it in many Windows UIs (even if they don’t have the same issues as Office). So as you can see the main product design decisions had nothing to do with usage data analysis. The same applies to other big product design decisions (the back and forward with the Metro UI, Cortana... etc). The use of telemetry is marginal to many product design decisions, so there is no reason to force it to every user... other than doing market research.

modernerd 6y ago
Sure, telemetry won't inform every product decision and user testing is still valuable. It does provide insights you won't get from user testing and direct observation, though, as you point out with the usage logs that informed the ribbon design. > The use of telemetry is marginal to many product design decisions, so there is no reason to force it to every user If the benefits were purely limited to the design phase, I can see your point for many teams, particularly small ones with niche products where they're less likely to get enough data for it to have much meaning. But telemetry also has increased value over user testing at different stages of the product lifecycle, like when analyzing what features or products to deprecate or replace. It also helps for peripheral things like working out what languages localization and documentation efforts should be prioritized for. This year I was asked what five languages a product we make should be translated to. They wanted to start with locales that have the biggest impact and expand to more later. The team who approached me made a list based on support requests and another product in our sector. But our install data showed usage was completely different. We even had two locales that hadn't made the top-10 shortlist. Their guess about the second-most popular language after English was also wrong. By using telemetry we made sure work and funding was focussed where it helps the most users. Telemetry can definitely be abused or underused, but being on the other side and doing data-driven design and development over the course of a product lifecycle has shown that it has real value.

modernerd 6y ago

Sure, telemetry won't inform every product decision and user testing is still valuable. It does provide insights you won't get from user testing and direct observation, though, as you point out with the usage logs that informed the ribbon design. > The use of telemetry is marginal to many product design decisions, so there is no reason to force it to every user If the benefits were purely limited to the design phase, I can see your point for many teams, particularly small ones with niche products where they're less likely to get enough data for it to have much meaning. But telemetry also has increased value over user testing at different stages of the product lifecycle, like when analyzing what features or products to deprecate or replace. It also helps for peripheral things like working out what languages localization and documentation efforts should be prioritized for. This year I was asked what five languages a product we make should be translated to. They wanted to start with locales that have the biggest impact and expand to more later. The team who approached me made a list based on support requests and another product in our sector. But our install data showed usage was completely different. We even had two locales that hadn't made the top-10 shortlist. Their guess about the second-most popular language after English was also wrong. By using telemetry we made sure work and funding was focussed where it helps the most users. Telemetry can definitely be abused or underused, but being on the other side and doing data-driven design and development over the course of a product lifecycle has shown that it has real value.