8 ms·

Having worked on user data before, I think the issue is that for complex software, it’s hard to get value/insight from this data. Imagine if you are Microsoft a

Having worked on user data before, I think the issue is that for complex software, it’s hard to get value/insight from this data. Imagine if you are Microsoft and had every action every Excel user makes with office365. Where would you even start to be able to get value out of this data?

ska 7y ago
> Where would you even start to be able to get value out of this data? It's an interesting question, but I think there are a few obvious easy places to get value (UX on common operations, most common error modes/corrections, etc.) In my experience the problem is more that devs apply logging or telemetry where it is easy rather than where it is insightful. So if you didn't design the system up front to capture user behavior with some granularity you end up with a bunch of questionable proxies for it, which are hard/impossible to interpret.
- panopticon 7y ago
  I've also observed the problem where analysts or product managers cherry pick the data that validates their assumptions. A "funny" example I ran into was product wanted to ship a notification that "test better" (i.e., had a higher interaction rate) in A/B testing. Some developers pointed out that the notification was likely seeing higher click-throughs because it was incredibly vague and users clicked through to figure out what it was about, and this would likely result in user frustration. Well, this insight was ignored and the notification change was shipped. Our notification opt-out rate spiked a few weeks later.
draugadrotten 7y ago
Well, mcDonalds draws maps of where their workers put their feet, and then designs the kitchens to minimize the number of steps the workers needs to take to complete their jobs. One way to use Excel telemetry could surely be to minimize the number of cognitive steps workers needs to take to complete the tasks they are trying to do. Another could be to use telemetry to see at which point a smooth flow suddenly takes a longer time - and perhaps split an action into two actions which are faster (in time) to accomplish I could go on all day.
- james_s_tayler 7y ago
  Can we get telemetry that reports on cognitive steps for developers working on code and minimize for that? I know there is cyclomatic complexity. Mostly you can look at a method and tell "this is a clusterfuck" without needing to break out a specialized tool. I still feel like a lot of devs don't appreciate the simple philosophy of "don't make me think."
- pbhjpbhj 7y ago
  Isn't that what Ribbons are supposed to do, raise the visibility of oft-used functions, respecting context.
- jaclaz 7y ago
  >Well, mcDonalds draws maps of where their workers put their feet, and then designs the kitchens to minimize the number of steps the workers needs to take to complete their jobs. In a completely different field (architecture, public spaces, parks) it has become common (and I find it a particularly smart approach) to create a new garden/public space/park with no "paths" and open it to the public. And put some benches "here and there" semi-randomly. Soon the grass will be worn out where people actually walks (i.e. people that needs to cross the park because it needs to go from point A to point B), so there you make the actual paved paths and the same goes for benches, some will have a path in the grass leading to them and some will look like never used, etc. But when it comes to Excel (or similar software) there is a difference. In the case of McDonalds personnel are instructed on exactly what they have to do and how exactly do it, i.e. their footsteps are their own optimization of an exactly specified chore. In the case of the people on public parks there is no need for instructions, as people actually needs to go through the park to get from point A to point B and knows where they love to seat down. In the case of Excel (or similar software) the issue is IMHO that most often than not the user doesn't actually know what he/she is doing, in the sense that user may (and often does) take a "wrong" or exceedingly convoluted path to get to the actual wanted result (when/if he/she actually gets to the result AND the result is actually correct). I often deal with people that (most probably in good faith) bluffed their way in "familiarity with Office" on their resumè and never actually learned to work in Excel, the spreadsheet they make are "terrible", with material (and sometimes also conceptual) mistakes that are all fine if you are using Excel to - say - keep your monthly budget under control but that cause (big) issues if you are using it at work to manage a project expenses/production/forecasts, a bid offer or similar. Feedback/telemetry from these users would "dumbify" the product and probably even make it much worse. >I could go on all day. I could too, I have seen things ...
hinkley 7y ago
I propose you start by studying a small batch of users, documenting their behaviors, develop a set of theories, then look at whether the larger pool of data agrees with or disproves your theory. Try to come up with theories just from data and you end up with the Texas Sharpshooter scenario. We are complaining about this sort of behavior showing up in the scientific community, let's learn from that mistake.
- munk-a 7y ago
  I agree, I think doing a deep dive interview with a small set of users is better than just collecting telemetry. Ask them questions, carry on conversations, notice when they become frustrated trying to do something. All the users could be using the right-click context menu to copy text not out of preference but instead ignorance - telemetry can tell you what's wrong but you miss out on most of the chance to tell why.
bgdnyxbjx 7y ago
Parent is clearly a PM in Excel looking for ideas
tempguy9999 7y ago
Install an accelerometer in the keyboard. When heavy impacts are detected from someone smashing their face into it repeatedly, pop up a dialog box asking them if there is something wrong and would they like assistance? Extensive feedback will be forthcoming. A bit more helpfully, MS were asked repeatedly not to fuck with the menus but they replaced it with the ribbon. Ditto that terrible new interface that was nothing like windows 7. They could have allowed the choice of new interface but they did not. MS got plenty of info and chose to discard it. MS is now in a position where it doesn't need to care.
- airstrike 7y ago
  I, for one, love the ribbon...
  - tempguy9999 7y ago
    Good for you. But so what? (Edit: my point is not about the ribbon)
    - airstrike 7y ago
      I'm just saying it's not necessarily true that the ribbon is worse than old school menus.
      - tempguy9999 7y ago
        And I'd agree with you. So what do you think my point was?
        airstrike 7y ago
        Your point seemed to be that MS chose not to care. I think they cared but made an informed decision to go down a path that many said was wrong. If anything, I'd say kudos to them for not living in an echo chamber.
esotericn 7y ago
The very first thing I would do is look for repetitive behaviours and crack out Clippy v2.0. You know when someone is like, writing 1 then 2 then 3 then 4 then ... or whatever? Yeah, Clippy 2.0. "Did you know this hotkey called X exists"? Multiply that by N users - bam you've saved the human race thousands of hours. I'll take my Nobel now.
- buerkle 7y ago
  IntelliJ has a plugin for that. https://plugins.jetbrains.com/plugin/9792-key-promoter-x/ https://plugins.jetbrains.com/plugin/9792-key-promoter-x/
- scarejunba 7y ago
  IntelliJ IDEA has a plugin that will prompt you with the keys for commands you click. Pretty neat. Not exactly what you're saying (no compound actions). Still neat.
rzzzt 7y ago
I read about Office 2000's dynamically hidden menu items feature (with the expand button added to the bottom of each sub-menu) somewhere, that the items that _are_ shown initially were determined by collected usage data. So it's probably aggregation that makes the most sense, not digging into each individual user's behavior.
airstrike 7y ago
I'd say just invite me over for a chat. I've probably spent 10,000 hours in Excel and can tell you more about its shortcomings than a PB of telemetry ever will
gxx 7y ago
I worked on the design of the first several versions of Excel. For Excel 3.0 we were intensely interested in user behavior. We often compared alternative prototypes of proposed features in the usability lab. We sometimes would invite develpers to watch users strugging with what they thought was great idea... Also, long before the Internet we asked users permission to send them a special version of Excel (called the Instrumented Version) that would intelligently log actions, but not data to a floppy disk. Users would send the disks back to us for analysis. We used the analysis to understand how users really did things vs what we thought the did. Sometimes we got some surprises. We used this to prioritize features where users were having difficulty that needed special attention. It also indicated frequenty performed actions that would benefit from, for example, being put on the toolbar. I think our user focus was an important reason why Excel has been so enduring. (Although Office 2003 screwed it up to some extent - I was no longer there.) Back then at Microsoft the applications group tended to take the lead in UI developement and many of the things we designed made their way into Windows. Some were even picked up by the Mac! Although Steve Jobs would never have admitted it. It grieves me to see the shoddy state of software design these days. There is no craftsmanship. Visual design take precedence over real usability. Maybe this is because with fast developemnt cycles and the ability to instantly change a web based design, it's easer to throw crap at the wall and keep iterating until it sort of works. These days user interaction designers have endless great user data on which to craft wonderful UI but it rarely happens. If they happen to hit on something good, usually another team takes over and changes it, often for the worse! (End Rant)
- galaxyLogic 7y ago
  Interesting to hear. One thing I never understood about Excel is why the cells which have formulas don't look different than cells which contain leaf-data. That I think would make it a lot more "user-friendly". 2ndly why not allow users name their columns, why do they have to always be "A", "B", "C", .. ?
  - MaulingMonkey 7y ago
    > One thing I never understood about Excel is why the cells which have formulas don't look different than cells which contain leaf-data. There are some styles actually meant for this, but you have to apply them manually. > 2ndly why not allow users name their columns, why do they have to always be "A", "B", "C", .. ? Named columns do exist actually! Not of the excel sheet itself - but in tables you can mark within that sheet. You can use the names in equations and everything. You can also name individual cells as well. Joel Spolsky has a whole talk where he goes into this and some of the other stuff Excel has that you might not realize it has: https://www.youtube.com/watch?v=0nbkaYsR94c https://www.youtube.com/watch?v=0nbkaYsR94c
asdff 7y ago
You can do so much with this data, I'm surprised if it's not already collected. You can stratify your users based on what kind of commands they do, how their mouse or cursor is moved around the screen, and what functions they are getting out of excel, and see if you can improve their experiences by lowering unneeded friction and building new features.
pbhjpbhj 7y ago
>Where would you even start to be able to get value out of this data? I'd start with the most undo-ne operations. Which specific actions are most often immediately undone by users (that could lead to addition of functions, changing UI, adding visibility of the effects of an action, all sorts). Then probably look at most searched terms in both Excel help and online, segmented by user experience if possible (to allow improvements for noobs and advanced users, whilst aiming not to be deleterious to either). I'm not in software development though, curious what answers y'all have? FWIW as a relatively novice LibreOffice user I probably most often undo auto-fills where a reference variable hasn't altered how I hoped. (I'd maybe fix that with a tailored tooltip - probably actually implemented as an info box - fired on long-hover of the auto-fill handle).
bonestamp2 7y ago
> Where would you even start to be able to get value out of this data? Much like excel for certain professions, I work on a piece of software that is used all day by people in another profession. One thing we do is look at which features are used most (and least). We make it easier to find and use the most popular features and we make rarely used features less prominent (and in some rare cases we might remove them altogether). In our most recent redesign, we elevated some of the most commonly used information to the point where the user doesn't even have to click anything to gain some high level insight (previous versions needed at least 1-2 clicks to get to that same info). We have about 30 power users who we work with continuously. Some of them are nearby, and we often go look over their shoulder and pick their brains. Some of them are in other regions where we get valuable global feedback that we wouldn't otherwise get from those regions. They get early access to new features and they get to help shape the product that they rely on for their livelihood and help us decide on what we should be working on next. It's win/win and I would recommend forming this kind of relationship with a subset of your user base if you can.
- jaclaz 7y ago
  >One thing we do is look at which features are used most (and least). We make it easier to find and use the most popular features and we make rarely used features less prominent (and in some rare cases we might remove them altogether). Well, with all due respect, this is not about making a "better" product, it is about making a product "easier to use" and possibly even "more popular" (which not necessarily are the same thing). >We have about 30 power users who we work with continuously. Some of them are nearby, and we often go look over their shoulder and pick their brains. Some of them are in other regions where we get valuable global feedback that we wouldn't otherwise get from those regions. They get early access to new features and they get to help shape the product that they rely on for their livelihood and help us decide on what we should be working on next. It's win/win and I would recommend forming this kind of relationship with a subset of your user base if you can. On the other hand this can be a much more productive approach, the program will "evolve" following the lead of these "power users", until it will become "almost perfect" (and more complex, and less popular because it has so many functions ...) In other words, both approaches have their merits (and downsides), and IMHO you have a good process in having both, but ultimately it is the "common sense" applied to the choices you make for your product that may make a difference.
tinus_hn 7y ago
Microsoft actually tracks this with their telemetry, they say they want to prioritize work to the features that are used most.

ska 7y ago

> Where would you even start to be able to get value out of this data? It's an interesting question, but I think there are a few obvious easy places to get value (UX on common operations, most common error modes/corrections, etc.) In my experience the problem is more that devs apply logging or telemetry where it is easy rather than where it is insightful. So if you didn't design the system up front to capture user behavior with some granularity you end up with a bunch of questionable proxies for it, which are hard/impossible to interpret.

panopticon 7y ago
I've also observed the problem where analysts or product managers cherry pick the data that validates their assumptions. A "funny" example I ran into was product wanted to ship a notification that "test better" (i.e., had a higher interaction rate) in A/B testing. Some developers pointed out that the notification was likely seeing higher click-throughs because it was incredibly vague and users clicked through to figure out what it was about, and this would likely result in user frustration. Well, this insight was ignored and the notification change was shipped. Our notification opt-out rate spiked a few weeks later.

draugadrotten 7y ago

Well, mcDonalds draws maps of where their workers put their feet, and then designs the kitchens to minimize the number of steps the workers needs to take to complete their jobs. One way to use Excel telemetry could surely be to minimize the number of cognitive steps workers needs to take to complete the tasks they are trying to do. Another could be to use telemetry to see at which point a smooth flow suddenly takes a longer time - and perhaps split an action into two actions which are faster (in time) to accomplish I could go on all day.

james_s_tayler 7y ago
Can we get telemetry that reports on cognitive steps for developers working on code and minimize for that? I know there is cyclomatic complexity. Mostly you can look at a method and tell "this is a clusterfuck" without needing to break out a specialized tool. I still feel like a lot of devs don't appreciate the simple philosophy of "don't make me think."
pbhjpbhj 7y ago
Isn't that what Ribbons are supposed to do, raise the visibility of oft-used functions, respecting context.
jaclaz 7y ago
>Well, mcDonalds draws maps of where their workers put their feet, and then designs the kitchens to minimize the number of steps the workers needs to take to complete their jobs. In a completely different field (architecture, public spaces, parks) it has become common (and I find it a particularly smart approach) to create a new garden/public space/park with no "paths" and open it to the public. And put some benches "here and there" semi-randomly. Soon the grass will be worn out where people actually walks (i.e. people that needs to cross the park because it needs to go from point A to point B), so there you make the actual paved paths and the same goes for benches, some will have a path in the grass leading to them and some will look like never used, etc. But when it comes to Excel (or similar software) there is a difference. In the case of McDonalds personnel are instructed on exactly what they have to do and how exactly do it, i.e. their footsteps are their own optimization of an exactly specified chore. In the case of the people on public parks there is no need for instructions, as people actually needs to go through the park to get from point A to point B and knows where they love to seat down. In the case of Excel (or similar software) the issue is IMHO that most often than not the user doesn't actually know what he/she is doing, in the sense that user may (and often does) take a "wrong" or exceedingly convoluted path to get to the actual wanted result (when/if he/she actually gets to the result AND the result is actually correct). I often deal with people that (most probably in good faith) bluffed their way in "familiarity with Office" on their resumè and never actually learned to work in Excel, the spreadsheet they make are "terrible", with material (and sometimes also conceptual) mistakes that are all fine if you are using Excel to - say - keep your monthly budget under control but that cause (big) issues if you are using it at work to manage a project expenses/production/forecasts, a bid offer or similar. Feedback/telemetry from these users would "dumbify" the product and probably even make it much worse. >I could go on all day. I could too, I have seen things ...

hinkley 7y ago

I propose you start by studying a small batch of users, documenting their behaviors, develop a set of theories, then look at whether the larger pool of data agrees with or disproves your theory. Try to come up with theories just from data and you end up with the Texas Sharpshooter scenario. We are complaining about this sort of behavior showing up in the scientific community, let's learn from that mistake.

munk-a 7y ago
I agree, I think doing a deep dive interview with a small set of users is better than just collecting telemetry. Ask them questions, carry on conversations, notice when they become frustrated trying to do something. All the users could be using the right-click context menu to copy text not out of preference but instead ignorance - telemetry can tell you what's wrong but you miss out on most of the chance to tell why.

bgdnyxbjx 7y ago

Parent is clearly a PM in Excel looking for ideas

tempguy9999 7y ago

Install an accelerometer in the keyboard. When heavy impacts are detected from someone smashing their face into it repeatedly, pop up a dialog box asking them if there is something wrong and would they like assistance? Extensive feedback will be forthcoming. A bit more helpfully, MS were asked repeatedly not to fuck with the menus but they replaced it with the ribbon. Ditto that terrible new interface that was nothing like windows 7. They could have allowed the choice of new interface but they did not. MS got plenty of info and chose to discard it. MS is now in a position where it doesn't need to care.

airstrike 7y ago
I, for one, love the ribbon...
- tempguy9999 7y ago
  Good for you. But so what? (Edit: my point is not about the ribbon)
  - airstrike 7y ago
    I'm just saying it's not necessarily true that the ribbon is worse than old school menus.
    - tempguy9999 7y ago
      And I'd agree with you. So what do you think my point was?
      - airstrike 7y ago
        Your point seemed to be that MS chose not to care. I think they cared but made an informed decision to go down a path that many said was wrong. If anything, I'd say kudos to them for not living in an echo chamber.

esotericn 7y ago

The very first thing I would do is look for repetitive behaviours and crack out Clippy v2.0. You know when someone is like, writing 1 then 2 then 3 then 4 then ... or whatever? Yeah, Clippy 2.0. "Did you know this hotkey called X exists"? Multiply that by N users - bam you've saved the human race thousands of hours. I'll take my Nobel now.

buerkle 7y ago
IntelliJ has a plugin for that. https://plugins.jetbrains.com/plugin/9792-key-promoter-x/ https://plugins.jetbrains.com/plugin/9792-key-promoter-x/
scarejunba 7y ago
IntelliJ IDEA has a plugin that will prompt you with the keys for commands you click. Pretty neat. Not exactly what you're saying (no compound actions). Still neat.

rzzzt 7y ago

I read about Office 2000's dynamically hidden menu items feature (with the expand button added to the bottom of each sub-menu) somewhere, that the items that _are_ shown initially were determined by collected usage data. So it's probably aggregation that makes the most sense, not digging into each individual user's behavior.

airstrike 7y ago

I'd say just invite me over for a chat. I've probably spent 10,000 hours in Excel and can tell you more about its shortcomings than a PB of telemetry ever will

gxx 7y ago

I worked on the design of the first several versions of Excel. For Excel 3.0 we were intensely interested in user behavior. We often compared alternative prototypes of proposed features in the usability lab. We sometimes would invite develpers to watch users strugging with what they thought was great idea... Also, long before the Internet we asked users permission to send them a special version of Excel (called the Instrumented Version) that would intelligently log actions, but not data to a floppy disk. Users would send the disks back to us for analysis. We used the analysis to understand how users really did things vs what we thought the did. Sometimes we got some surprises. We used this to prioritize features where users were having difficulty that needed special attention. It also indicated frequenty performed actions that would benefit from, for example, being put on the toolbar. I think our user focus was an important reason why Excel has been so enduring. (Although Office 2003 screwed it up to some extent - I was no longer there.) Back then at Microsoft the applications group tended to take the lead in UI developement and many of the things we designed made their way into Windows. Some were even picked up by the Mac! Although Steve Jobs would never have admitted it. It grieves me to see the shoddy state of software design these days. There is no craftsmanship. Visual design take precedence over real usability. Maybe this is because with fast developemnt cycles and the ability to instantly change a web based design, it's easer to throw crap at the wall and keep iterating until it sort of works. These days user interaction designers have endless great user data on which to craft wonderful UI but it rarely happens. If they happen to hit on something good, usually another team takes over and changes it, often for the worse! (End Rant)

galaxyLogic 7y ago
Interesting to hear. One thing I never understood about Excel is why the cells which have formulas don't look different than cells which contain leaf-data. That I think would make it a lot more "user-friendly". 2ndly why not allow users name their columns, why do they have to always be "A", "B", "C", .. ?
- MaulingMonkey 7y ago
  > One thing I never understood about Excel is why the cells which have formulas don't look different than cells which contain leaf-data. There are some styles actually meant for this, but you have to apply them manually. > 2ndly why not allow users name their columns, why do they have to always be "A", "B", "C", .. ? Named columns do exist actually! Not of the excel sheet itself - but in tables you can mark within that sheet. You can use the names in equations and everything. You can also name individual cells as well. Joel Spolsky has a whole talk where he goes into this and some of the other stuff Excel has that you might not realize it has: https://www.youtube.com/watch?v=0nbkaYsR94c https://www.youtube.com/watch?v=0nbkaYsR94c

asdff 7y ago

You can do so much with this data, I'm surprised if it's not already collected. You can stratify your users based on what kind of commands they do, how their mouse or cursor is moved around the screen, and what functions they are getting out of excel, and see if you can improve their experiences by lowering unneeded friction and building new features.

pbhjpbhj 7y ago

>Where would you even start to be able to get value out of this data? I'd start with the most undo-ne operations. Which specific actions are most often immediately undone by users (that could lead to addition of functions, changing UI, adding visibility of the effects of an action, all sorts). Then probably look at most searched terms in both Excel help and online, segmented by user experience if possible (to allow improvements for noobs and advanced users, whilst aiming not to be deleterious to either). I'm not in software development though, curious what answers y'all have? FWIW as a relatively novice LibreOffice user I probably most often undo auto-fills where a reference variable hasn't altered how I hoped. (I'd maybe fix that with a tailored tooltip - probably actually implemented as an info box - fired on long-hover of the auto-fill handle).

bonestamp2 7y ago

> Where would you even start to be able to get value out of this data? Much like excel for certain professions, I work on a piece of software that is used all day by people in another profession. One thing we do is look at which features are used most (and least). We make it easier to find and use the most popular features and we make rarely used features less prominent (and in some rare cases we might remove them altogether). In our most recent redesign, we elevated some of the most commonly used information to the point where the user doesn't even have to click anything to gain some high level insight (previous versions needed at least 1-2 clicks to get to that same info). We have about 30 power users who we work with continuously. Some of them are nearby, and we often go look over their shoulder and pick their brains. Some of them are in other regions where we get valuable global feedback that we wouldn't otherwise get from those regions. They get early access to new features and they get to help shape the product that they rely on for their livelihood and help us decide on what we should be working on next. It's win/win and I would recommend forming this kind of relationship with a subset of your user base if you can.

jaclaz 7y ago
>One thing we do is look at which features are used most (and least). We make it easier to find and use the most popular features and we make rarely used features less prominent (and in some rare cases we might remove them altogether). Well, with all due respect, this is not about making a "better" product, it is about making a product "easier to use" and possibly even "more popular" (which not necessarily are the same thing). >We have about 30 power users who we work with continuously. Some of them are nearby, and we often go look over their shoulder and pick their brains. Some of them are in other regions where we get valuable global feedback that we wouldn't otherwise get from those regions. They get early access to new features and they get to help shape the product that they rely on for their livelihood and help us decide on what we should be working on next. It's win/win and I would recommend forming this kind of relationship with a subset of your user base if you can. On the other hand this can be a much more productive approach, the program will "evolve" following the lead of these "power users", until it will become "almost perfect" (and more complex, and less popular because it has so many functions ...) In other words, both approaches have their merits (and downsides), and IMHO you have a good process in having both, but ultimately it is the "common sense" applied to the choices you make for your product that may make a difference.

tinus_hn 7y ago

Microsoft actually tracks this with their telemetry, they say they want to prioritize work to the features that are used most.