7 ms·

I started at MS during Vista and I've been involved (sometimes tangentially) with Windows ever since. This is all my opinion, but It's been very interesting se

I started at MS during Vista and I've been involved (sometimes tangentially) with Windows ever since. This is all my opinion, but It's been very interesting seeing the decision making process change over time. If I had to summarize the change, I'd say that it's evolved from an expertise-based system to a data based system. The reason why eight people were present at every planning meeting is because their expert opinion was the primary tool used in decision making. In addition to poor decisions, this had two very negative outcomes: 1) reputation was fiercely fought for. Individuals feared that if they were ever incorrect, the damage to their reputation would limit their ability to impact future decisions and eventually lead to career death. Whether this actually happened or not is irrelevant; the fear itself caused overt caution and consensus seeking. 2) In the absence of data, an eloquent negotiator is often able to obtain their desired outcome, no matter how sub-optimal that outcome might be. Nowadays, I see data used a lot more, hence the telemetry. A while ago, in response to criticism of Windows telemetry habits, I wrote: "Telemetry is, by now, a fundamental part of the engineering process. Products that don't incorporate it are going to be clobbered by products that do. Microsoft didn't start this paradigm, but I think they had to incorporate it in order to stay competitive." I understand that telemetry is still a delicate issue, but the expertise based decision making of yester-year truly sucked for many, and I don't see a viable alternative.

KiwiCoder 9y ago
Doesn't the focus on data simply shift the problems you describe to telemetry decisions?
- bradford 9y ago
  Yes, and, I might be misunderstanding your question, but I view that as a good thing. I don't ever want to spend my time in a meeting witnessing people bicker over where to position a dialog button, when you could just run an A/B test and let data guide the decision process.
  - andrewfong 9y ago
    Yes, but now you get to bicker about how to structure the A/B test and whether the results are statistically significant.
    - zyamada 9y ago
      "Based on the data collected, most users seem to prefer..." is better than, "Based on my training and personal experiences, I think users would prefer..." It is a lot easier to come to a common agreement on something as objective as statistical significance when compared to personal feelings and opinions.
      - chrischen 9y ago
        Subjectviely interpreted objective data...
  - deleted 9y ago
    [deleted]
giobox 9y ago
I don't work at Microsoft, but you've eloquently described issues I deal with at my own workplace. I think your outcome 1) is almost inevitable in many large software orgs as ego/reputation fills the decision making vacuum left by the absence of 'good' data/telemetry.
sven44 9y ago
Telemetry isn't magic. It might take a few contentious issues off the table but for the ambiguous long term stuff, it still boils down to someone showing some balls and putting their neck on the line. Lumia is a good example where the numbers overruled basic self confidence.
- Lorkki 9y ago
  Can you elaborate? I remember hearing mainly good things about the Lumia UI. The problem was that it appeared in the middle of an already heated competition between iOS and Android and didn't put much else on the table.
bo1024 9y ago
Very interesting and thanks for the post. Does Windows 10 actually conduct ongoing A/B tests and use telemetry to evaluate them? (for example the interface used by two people on two different computers might look different, or something) I'm intrigued but still very skeptical about the effectiveness of the data-based approach to UI design. That seems to be an area where local, hillclimbing optimization can lead to a mess of conflicting decisions across the OS.
- Analemma_ 9y ago
  Yes. The blog for the Windows Insider Program makes frequent references to beta features that were shipped in different forms to different users and then a decision made based on the telemetry from those flights.
cm2187 9y ago
Microsoft introduced a high level of non-opt-out telemetry. That's what is controversial more than the telemetry itself.
- sz4kerto 9y ago
  On desktops, yes. Other companies have been moving everything to the web, where telemetry is easier and more thorough.
  - chipperyman573 9y ago
    The problem is that Web exists in a sandbox. They can collect data about my usage of their website, full stop (large sites like Facebook and Google get around this, but that's not really telemetry). With desktop there's no way of knowing what's collected with "telemetry"
    - deleted 9y ago
      [deleted]
    - codedokode 9y ago
      And that is the problem of using OSes with outdated security model from mainframe era.
      - michaelmrose 9y ago
        To some large degree you need to be able to trust the people that provide software to you or you can't function.
        rapind 9y ago
        If it's open source you don't have to trust them as much. I function just fine with a mostly open stack.
      - Const-me 9y ago
andrewf 9y ago
What is the telemetry optimizing? In the web world, most A/B testing is focused on increasing engagement. Half of the time people are trying to build a better Skinner box. Is this software actually better for users? I would be encouraged if MS was, for instance, optimizing for a user being able to complete the same tasks in less time.
- adwilson 9y ago
  Cannot speak for all the teams in Windows, but my team has used it successfully to improve the battery life of machines through A/B tests. We're able to (on insiders builds) try out postponing various operations until the machine is plugged in to see how it effects the user experience and power draw. It helped us tune a few operations in ways I wouldn't have anticipated to improve battery life without degrading the experience One of the other comments mentioned this but the blog will also highlight some of the more user visible tests that are run. Generally around optimizing UI flows for usability
rbanffy 9y ago
> "Telemetry is, by now, a fundamental part of the engineering process" This is how we get faster horses instead of cars.
guelo 9y ago
I don't see how telemetry would have fixed the organizational problems described in the article. You still have to agree on a handful of features to implement for the AB tests. If cross-team communication across the kernel, shell, and menu teams is bureaucratic and slow and it takes months for dependent code changes to trickle through the repository tree you're still going to end up with the lowest common denominator feature implementations.
codedokode 9y ago
> Telemetry is, by now, a fundamental part of the engineering process. Products that don't incorporate it are going to be clobbered by products that do. I am not sure this is correct. Telemetry can show what users do but it won't show why they do it. Users don't click the button: maybe they don't need it but maybe they don't understand what it does. Or maybe there are too many buttons and users get lost. Also making a decision based on telemetry is making a desicion based on opinion of non-experts. There is a quote by Ford “If I had asked people what they wanted, they would have said faster horses.”. Can you make something innovative this way? For example if we take a code editor like Sublime Text which in my opinion has very good UI, and it has no toolbar - can you use telemetry as an argument for removing a toolbar? No, because if there is a toolbar, there will be users who would use it without realising that they could do better without it. You need an expert to make better UI, not telemetry.
- swsieber 9y ago
  I think it would be inaccurate to say you can't learn things from telemetry. You can easily run A/B tests, or even in the absence of proper A/B tests, you can see if a newly added feature is actually being used or not. The other thing one is the timing of action - I bet you could learn a lot from that, especially if you take a nice group of people, watch and quiz them, line up your learnings with the telemetry, and then use the telemetry of your general user base to infer how your user base is responding. Standard flame-war prevention disclaimer: I'm not saying we should add telemetry to everything. I'm just saying it seems like it could be another useful tool in product creation.
- shawnz 9y ago
  Where does the parent claim that telemetry is the only factor you should be basing your decisions on?
- deleted 9y ago
  [deleted]
- deleted 9y ago
  [deleted]
wrs 9y ago
Putting on my asbestos suit... I started the telemetry project (aka SQM or CEIP) at Microsoft when I was on the MSN Explorer team. (Then Office got excited about it, then SQL Server and Windows and nearly everyone else.) The original inspiration was being in a meeting much like the one described, where people were arguing about whether we should optimize the UI for 640x480, 800x600, or 1024x768 screens. The "argument" consisted almost entirely of anecdotes. I thought "Why don't we just measure how big the screens actually are, and then I won't have to attend any more meetings like this?" It turned out a lot of other people felt that way.
- taneq 9y ago
  There's a right way and a wrong way to do telemetry. The right way: When the Windows 7 RC was released, you guys made it available for free, with telemetry, as the final stage of real-world testing. It was up-front, it was open, it was optional. I was happy to join in, and bought Windows 7 when it came out, and it good. The wrong way: When Windows 10 was released, a whole lot of nebulously defined, mandatory telemetry was baked into the commercial release. The fact that it was "free" is irrelevant - this was a clear transition from "you own the software" to "MS, via the software, owns and monetizes you." Then the "Get Windows 10" push started and for over a year we had a pitched battle between MS trying to force users to upgrade, and users who didn't want Windows 10 with all its invasive marketing, telemetry (weird how that slider keeps changing itself back from "minimal" to "full", huh?), and loss of control of their systems. I've disagreed with MS's customer relations in the past but never before have I felt that they were actively hostile. I will never use Windows 10.
  - spookthesunset 9y ago
    Every website you and everybody know you visits is full of telemetry used to improve the website. Why should "real" desktop software be any different?
    - flukus 9y ago
      Real desktop software has access to a lot of private and often sensitive information, my browser doesn't share that with any website.
deleted 9y ago
[deleted]

KiwiCoder 9y ago

Doesn't the focus on data simply shift the problems you describe to telemetry decisions?

bradford 9y ago
Yes, and, I might be misunderstanding your question, but I view that as a good thing. I don't ever want to spend my time in a meeting witnessing people bicker over where to position a dialog button, when you could just run an A/B test and let data guide the decision process.
- andrewfong 9y ago
  Yes, but now you get to bicker about how to structure the A/B test and whether the results are statistically significant.
  - zyamada 9y ago
    "Based on the data collected, most users seem to prefer..." is better than, "Based on my training and personal experiences, I think users would prefer..." It is a lot easier to come to a common agreement on something as objective as statistical significance when compared to personal feelings and opinions.
    - chrischen 9y ago
      Subjectviely interpreted objective data...
- deleted 9y ago
  [deleted]

giobox 9y ago

I don't work at Microsoft, but you've eloquently described issues I deal with at my own workplace. I think your outcome 1) is almost inevitable in many large software orgs as ego/reputation fills the decision making vacuum left by the absence of 'good' data/telemetry.

sven44 9y ago

Telemetry isn't magic. It might take a few contentious issues off the table but for the ambiguous long term stuff, it still boils down to someone showing some balls and putting their neck on the line. Lumia is a good example where the numbers overruled basic self confidence.

Lorkki 9y ago
Can you elaborate? I remember hearing mainly good things about the Lumia UI. The problem was that it appeared in the middle of an already heated competition between iOS and Android and didn't put much else on the table.

bo1024 9y ago

Very interesting and thanks for the post. Does Windows 10 actually conduct ongoing A/B tests and use telemetry to evaluate them? (for example the interface used by two people on two different computers might look different, or something) I'm intrigued but still very skeptical about the effectiveness of the data-based approach to UI design. That seems to be an area where local, hillclimbing optimization can lead to a mess of conflicting decisions across the OS.

Analemma_ 9y ago
Yes. The blog for the Windows Insider Program makes frequent references to beta features that were shipped in different forms to different users and then a decision made based on the telemetry from those flights.

cm2187 9y ago

Microsoft introduced a high level of non-opt-out telemetry. That's what is controversial more than the telemetry itself.

sz4kerto 9y ago
On desktops, yes. Other companies have been moving everything to the web, where telemetry is easier and more thorough.
- chipperyman573 9y ago
  The problem is that Web exists in a sandbox. They can collect data about my usage of their website, full stop (large sites like Facebook and Google get around this, but that's not really telemetry). With desktop there's no way of knowing what's collected with "telemetry"
  - deleted 9y ago
    [deleted]
  - codedokode 9y ago
    And that is the problem of using OSes with outdated security model from mainframe era.
    - michaelmrose 9y ago
      To some large degree you need to be able to trust the people that provide software to you or you can't function.
      - rapind 9y ago
        If it's open source you don't have to trust them as much. I function just fine with a mostly open stack.
    - Const-me 9y ago

andrewf 9y ago

What is the telemetry optimizing? In the web world, most A/B testing is focused on increasing engagement. Half of the time people are trying to build a better Skinner box. Is this software actually better for users? I would be encouraged if MS was, for instance, optimizing for a user being able to complete the same tasks in less time.

adwilson 9y ago
Cannot speak for all the teams in Windows, but my team has used it successfully to improve the battery life of machines through A/B tests. We're able to (on insiders builds) try out postponing various operations until the machine is plugged in to see how it effects the user experience and power draw. It helped us tune a few operations in ways I wouldn't have anticipated to improve battery life without degrading the experience One of the other comments mentioned this but the blog will also highlight some of the more user visible tests that are run. Generally around optimizing UI flows for usability

rbanffy 9y ago

> "Telemetry is, by now, a fundamental part of the engineering process" This is how we get faster horses instead of cars.

guelo 9y ago

I don't see how telemetry would have fixed the organizational problems described in the article. You still have to agree on a handful of features to implement for the AB tests. If cross-team communication across the kernel, shell, and menu teams is bureaucratic and slow and it takes months for dependent code changes to trickle through the repository tree you're still going to end up with the lowest common denominator feature implementations.

codedokode 9y ago

> Telemetry is, by now, a fundamental part of the engineering process. Products that don't incorporate it are going to be clobbered by products that do. I am not sure this is correct. Telemetry can show what users do but it won't show why they do it. Users don't click the button: maybe they don't need it but maybe they don't understand what it does. Or maybe there are too many buttons and users get lost. Also making a decision based on telemetry is making a desicion based on opinion of non-experts. There is a quote by Ford “If I had asked people what they wanted, they would have said faster horses.”. Can you make something innovative this way? For example if we take a code editor like Sublime Text which in my opinion has very good UI, and it has no toolbar - can you use telemetry as an argument for removing a toolbar? No, because if there is a toolbar, there will be users who would use it without realising that they could do better without it. You need an expert to make better UI, not telemetry.

swsieber 9y ago
I think it would be inaccurate to say you can't learn things from telemetry. You can easily run A/B tests, or even in the absence of proper A/B tests, you can see if a newly added feature is actually being used or not. The other thing one is the timing of action - I bet you could learn a lot from that, especially if you take a nice group of people, watch and quiz them, line up your learnings with the telemetry, and then use the telemetry of your general user base to infer how your user base is responding. Standard flame-war prevention disclaimer: I'm not saying we should add telemetry to everything. I'm just saying it seems like it could be another useful tool in product creation.
shawnz 9y ago
Where does the parent claim that telemetry is the only factor you should be basing your decisions on?
deleted 9y ago
[deleted]
deleted 9y ago
[deleted]

wrs 9y ago

Putting on my asbestos suit... I started the telemetry project (aka SQM or CEIP) at Microsoft when I was on the MSN Explorer team. (Then Office got excited about it, then SQL Server and Windows and nearly everyone else.) The original inspiration was being in a meeting much like the one described, where people were arguing about whether we should optimize the UI for 640x480, 800x600, or 1024x768 screens. The "argument" consisted almost entirely of anecdotes. I thought "Why don't we just measure how big the screens actually are, and then I won't have to attend any more meetings like this?" It turned out a lot of other people felt that way.

taneq 9y ago
There's a right way and a wrong way to do telemetry. The right way: When the Windows 7 RC was released, you guys made it available for free, with telemetry, as the final stage of real-world testing. It was up-front, it was open, it was optional. I was happy to join in, and bought Windows 7 when it came out, and it good. The wrong way: When Windows 10 was released, a whole lot of nebulously defined, mandatory telemetry was baked into the commercial release. The fact that it was "free" is irrelevant - this was a clear transition from "you own the software" to "MS, via the software, owns and monetizes you." Then the "Get Windows 10" push started and for over a year we had a pitched battle between MS trying to force users to upgrade, and users who didn't want Windows 10 with all its invasive marketing, telemetry (weird how that slider keeps changing itself back from "minimal" to "full", huh?), and loss of control of their systems. I've disagreed with MS's customer relations in the past but never before have I felt that they were actively hostile. I will never use Windows 10.
- spookthesunset 9y ago
  Every website you and everybody know you visits is full of telemetry used to improve the website. Why should "real" desktop software be any different?
  - flukus 9y ago
    Real desktop software has access to a lot of private and often sensitive information, my browser doesn't share that with any website.

deleted 9y ago

[deleted]