12 ms·

I used to believe that it was not necessary until I started building my own startup. If you dont have analytics you are flying blind. You don't know what your u

by Sytten 5mo ago

I used to believe that it was not necessary until I started building my own startup. If you dont have analytics you are flying blind. You don't know what your users actually care about and how to optimize a successful user journey. The difference between what people tell you when asked directly and how they actually use your software is actually shocking.

EGG_CREAM 5mo ago
[dead]
sidkshatriya 5mo ago
> If you dont have analytics you are flying blind If you have too much emphasis on (invasive) analytics you might end up flying empty i.e. without customers.
throwaway27448 5mo ago
I'm pretty ok with the github cli tool team flying blind. The tool isn't exactly a necessary part of any workflow. You don't need telemetry to glean that
- merlindru 5mo ago
  that's akin to saying "i do not need their product therefore i don't care"... so what's your point? someone may have made it part of their workflow!
  - deleted 5mo ago
    [deleted]
  - throwaway27448 5mo ago
    True. Some people shouldn't use git if their workflow doesn't beg it.
embedding-shape 5mo ago
You're only flying blind if you make decisions not looking and thinking. Analytics isn't the only way to figure out "what your users actually care about", you can also try the old school way, commonly referred to as "Talking with people", then after taking notes, you think about it, maybe discuss with others. Don't take what people say at face value, but think about it together with your knowledge and experience, and you'll make even better product decisions than the people who are only making "data driven decisions" all the time.
- Sytten 5mo ago
  We do both and they yield different learnings. They are complementary. We also have an issue tracking board with upvotes. I would say to your point that you can't improve what you don't measure.
  - bfivyvysj 5mo ago
    I would say to your point that you can't not spy on me while also spying on me. Maybe just don't?
    - lukevp 5mo ago
      If I was running a physical business and I wrote down each person’s name and credit card number and the exact time and order they placed, that would be pretty invasive and “spying”. If I write down how many units I sold of each item per day, and the volume of transactions by credit card vs cash, it’s anonymized and I don’t think this would generally be considered “spying”, just normal business metrics. How’s the latter much different than anonymized product analytics?
      - lamasery 5mo ago
        Watching me use my computer in my house or office is spying. Aggregating request statistics server-side unless you're only generating those requests to spy on what I'm doing on my computer is more like the not-spying you're talking about.
ubercore 5mo ago
It makes me think, what `gh` features don't generate some activity in the github API that could as easily guide feature development without adding extra telemetry?
- larusso 5mo ago
  Yeah. Unless they plan to move more local git operations in the tool and blur the line between git and gh.
nkrisc 5mo ago
> The difference between what people tell you when asked directly and how they actually use your software is actually shocking. And the difference between what they do and what they want is equally shocking. If what they want isn’t in your app, they can’t do it and it won’t show up in your data. Quantitative data doesn’t tell you what your users want or care about. It tells you only what they are doing. You can get similar data without spying on your users. I don’t necessarily think all data gathering is equivalent to spying, but if it’s not entirely opt-in, I think it is effectively spying no matter what you’re collecting, varying only along a dimension of invasiveness.
- DrScientist 5mo ago
  > If what they want isn’t in your app, they can’t do it and it won’t show up in your data. Excellent point. > but if it’s not entirely opt-in, I think it is effectively spying no matter what you’re collecting, varying only along a dimension of invasiveness. Every web page visit is logged on the http server, and that's been the default since the mid 1990's. Is that spying?
  - nkrisc 5mo ago
    In principle, yes, I believe it is a form of spying. Not particularly invasive nor harmful, but spying nonetheless. Logging every page visited is not a technical requirement of serving the requested resource.
    - vlovich123 5mo ago
      > Logging every page visited is not a technical requirement of serving the requested resource. How will you know which page is having problems being served or is having performance problems?
      - nkrisc 5mo ago
        You won’t, but that’s not what was asked. Logging the requested resource is not a technical requirement of serving that resource.
kodablah 5mo ago
> If you dont have analytics you are flying blind More like flying based on your knowledge as a pilot and not by the whims of your passengers. For many CLIs and developer tooling, principled decisions need to reign. Accepting the unquantifiability of usage in a principled product is often difficult for those that are not the target demographic, but for developer tools specifically (be they programming languages, CLIs, APIs, SDKs, etc), cohesion and common sense are usually enough. It also seems real hard for product teams to accept the value of the status quo with these existing, heavily used tools.
- mckn1ght 5mo ago
  Actually it's more like flying in the clouds with no instruments which can lead to spatial disorientation when you exit the cloud cover and realize you're nosediving towards the earth. https://en.wikipedia.org/wiki/Spatial_disorientation https://en.wikipedia.org/wiki/Spatial_disorientation Flying based on the whims of your passengers would be user testing/interviewing, which is a complementary, and IMO necessary, strategy alongside analytics.
jubilanti 5mo ago
Wow, it really is sad how literally unthinkable it is to you and so much of the industry that you could actually talk to your users and customers like human beings instead of just data points. And you know what happens when you reach out to talk to your customers like human beings instead of spying on them like animals? They like you more and they raise issues that your telemetry would never even think to measure. It's called user research and client relationship management.
- deleted 5mo ago
  [deleted]
- vlovich123 5mo ago
  I think you’re overlooking that they were talking about stated and revealed preferences, a well known economic challenge where what people say is important to them and what shows up in the data is a gap. Of course you talk to users and do relationship management. That doesn’t negate the need to understand revealed preferences. In the OSS world this is not a huge deal. You get some community that’s underserved by the product (ie software package) and they fork, modify, or build something else. If it turned out to be valuable, then you get the old solution complemented or replaced. In the business world this is an existential threat to the business - you want to make sure your users aren’t better served by a competitor who’s focusing on your blindspot.
- skeuomorphism 5mo ago
  Marketing came to the conclusion that people dont know what they actually want. They decided to lump in engineers and programmers as well, since they started abusing their goodwill.
- kalleboo 5mo ago
  Apple and Microsoft reached their peak usability when they employed teams of people to literally sit and watch what users did in real life (and listen to them narrating what they want to do), take notes, and ask followup questions. Everything went to crap in the metric-based era that followed.
- alexchantavy 5mo ago
pc86 5mo ago
You can "optimize a successful user journey" by making the software easy to use, making it load so fast people are surprised by it, and talking to your customers. Telemetry doesn't help you do any of that, but it does help you squeeze more money out of them, or find out where you can pop an interstitial ad to goose your ad revenue, and what features you can move up a tier level to increase revenue without providing any additional value.
a012 5mo ago
You have all info you need on server side, I don’t believe that you’re totally blind without client tracking
ryandrake 5mo ago
This got me thinking: Are there prominent examples of open source projects that 1. collect telemetry, 2. without a way to opt-out (or obfuscating / making it difficult to opt-out)? This practice seems to be specific to corporate software development. Why is it that startups and commercial software developers seem to be the only ones obsessed with telemetry? Why do they need it to "optimize user journeys" but open source projects do just fine while flying blind?
- theplatman 5mo ago
  open source projects are usually creating something for themselves so it's much easier to know what to build when you are the user whereas, commercial software has a disconnect between who are the users and developers are
e12e 5mo ago
I think there's room for a distinction between "not using metrics" and "not using data". Unthinkingly leaning on metrics is likely to help you build a faster, stronger horse, while at the same time avoiding building a car, a bus or a tractor.
yoyohello13 5mo ago
The totality of Microsoft's products is proof that this is false. If telemetry and analytics actually mattered for usability, every product Microsoft puts out wouldn't be good instead of garbage.
- SchemaLoad 5mo ago
  There are far too many factors to assign the quality of microsoft's products to telemetry. Having the data doesn't mean you will act on it. And doesn't mean microsofts interests are aligned with the users.
ctoth 5mo ago
> If you dont have analytics you are flying blind. We... we are talking about a CLI tool. A CLI tool that directly uses the API. A tool which already identifies itself with a User-Agent[0]. A tool which obviously knows who is using it. What information are you gathering by running telemetry on my machine that couldn't.. just. be. a. database. query? Reading the justification the main thing they seem to want to know is if gh is being driven by a human or an agent... Which, F off with your creepy nonsense. Please don't just use generic "but ma analytics!" when this obviously doesn't apply here? [0]: https://github.com/cli/cli/blob/3ad29588b8bf9f2390be652f46ee4ef75b875b03/api/http_client.go#L54 https://github.com/cli/cli/blob/3ad29588b8bf9f2390be652f46ee...
renegade-otter 5mo ago
I agree with you in that regard. That said, knowing that this is Microsoft, the data will be used to extract value from the customers, not provide them with one.
tomrod 5mo ago
Teams that do this need to just dogfood internally. Once you start collecting telemetry on external users defaulted to opt-in you're not a good faith actor in the ecosystem.
goosejuice 5mo ago
You could, I don't know, do user interviews with the various customer segments that use your product.
Apylon777 5mo ago
How did GitHub ever survive without this telemetry? Was it a web application buried in obscurity?
lynndotpy 5mo ago
Game developers benefit tremendously from streams where they get to see peoples webcams _and_ screens as they use their software. This would be _absolutely insane_ telemetry to request from a user for any other piece of software, but it would be fantastically useful in identifying where people get frustrated and why. That said, I do not trust Microsoft with any telemetry, I am not invested in helping them improve their product, and I am happy not to rely on the GitHub CLI.
- account42 5mo ago
  You can get this level of observation by paying testers (both professions as well as existing users who provide informed consent). You don't get to just spy on random users, yes.
deleted 5mo ago
[deleted]
codedokode 5mo ago
Analytics is wrong. I never click any ads, but they keep showing it. I avoid registering or enter fake emails, but they keep showing full screen popups asking for email. I always reject cookies but they still ask me to accept them. And youtube keeps pushing those vertical videos for alternately gifted kids despite me never watching them. What's the point of this garbage analytics. It seems that their only goal is to annoy people.
- sagarm 5mo ago
  All of those are affected by analytics. Ad slots will be filled whether or not you click. If you never click, you'll tend to match with either very low quality ads or ads that pay per impression (display ads). Email registration is highly valuable for a business, so analytics won't be used to decide whether to show the modal but rather test different versions of it. Cookies are too valuable to not push on users, because without them only the previously mentioned low quality ads can be shown. High quality and display ads match on interest or demographic labels. The business decision to keep vertical videos is highly likely to be affected by analytics, and of course the choice of which videos to show is based on recommendation models trained on interaction logs. The priority isn't making your experience better, though that is often an incidental result -- it's driving the business.
attentive 5mo ago
It's not like they don't own API's that those cli's are hitting. They have all the stats they need.
chillfox 5mo ago
Be very careful with that. Analytics driven development easily leads to bad outcomes. 1. Important, but less frequently used feature gets moved to a hidden spot leading to even less usage leading to eventual removal. 2. Poorly functioning features not getting the improvement they need because few use them due to how poorly they function. I have seen these patterns a lot in software where decisions are based on analytics, and I usually stop using that sofware when I find a replacement.
qwertox 5mo ago
They could well use the data from their own developers, couldn't that be enough?
account42 5mo ago
Bats are mostly blind and manage to fly just fine.

by Sytten 5mo ago

EGG_CREAM 5mo ago

[dead]

sidkshatriya 5mo ago

> If you dont have analytics you are flying blind If you have too much emphasis on (invasive) analytics you might end up flying empty i.e. without customers.

throwaway27448 5mo ago

I'm pretty ok with the github cli tool team flying blind. The tool isn't exactly a necessary part of any workflow. You don't need telemetry to glean that

merlindru 5mo ago
that's akin to saying "i do not need their product therefore i don't care"... so what's your point? someone may have made it part of their workflow!
- deleted 5mo ago
  [deleted]
- throwaway27448 5mo ago
  True. Some people shouldn't use git if their workflow doesn't beg it.

embedding-shape 5mo ago

You're only flying blind if you make decisions not looking and thinking. Analytics isn't the only way to figure out "what your users actually care about", you can also try the old school way, commonly referred to as "Talking with people", then after taking notes, you think about it, maybe discuss with others. Don't take what people say at face value, but think about it together with your knowledge and experience, and you'll make even better product decisions than the people who are only making "data driven decisions" all the time.

Sytten 5mo ago
We do both and they yield different learnings. They are complementary. We also have an issue tracking board with upvotes. I would say to your point that you can't improve what you don't measure.
- bfivyvysj 5mo ago
  I would say to your point that you can't not spy on me while also spying on me. Maybe just don't?
  - lukevp 5mo ago
    If I was running a physical business and I wrote down each person’s name and credit card number and the exact time and order they placed, that would be pretty invasive and “spying”. If I write down how many units I sold of each item per day, and the volume of transactions by credit card vs cash, it’s anonymized and I don’t think this would generally be considered “spying”, just normal business metrics. How’s the latter much different than anonymized product analytics?
    - lamasery 5mo ago
      Watching me use my computer in my house or office is spying. Aggregating request statistics server-side unless you're only generating those requests to spy on what I'm doing on my computer is more like the not-spying you're talking about.

ubercore 5mo ago

It makes me think, what `gh` features don't generate some activity in the github API that could as easily guide feature development without adding extra telemetry?

larusso 5mo ago
Yeah. Unless they plan to move more local git operations in the tool and blur the line between git and gh.

nkrisc 5mo ago

> The difference between what people tell you when asked directly and how they actually use your software is actually shocking. And the difference between what they do and what they want is equally shocking. If what they want isn’t in your app, they can’t do it and it won’t show up in your data. Quantitative data doesn’t tell you what your users want or care about. It tells you only what they are doing. You can get similar data without spying on your users. I don’t necessarily think all data gathering is equivalent to spying, but if it’s not entirely opt-in, I think it is effectively spying no matter what you’re collecting, varying only along a dimension of invasiveness.

DrScientist 5mo ago
> If what they want isn’t in your app, they can’t do it and it won’t show up in your data. Excellent point. > but if it’s not entirely opt-in, I think it is effectively spying no matter what you’re collecting, varying only along a dimension of invasiveness. Every web page visit is logged on the http server, and that's been the default since the mid 1990's. Is that spying?
- nkrisc 5mo ago
  In principle, yes, I believe it is a form of spying. Not particularly invasive nor harmful, but spying nonetheless. Logging every page visited is not a technical requirement of serving the requested resource.
  - vlovich123 5mo ago
    > Logging every page visited is not a technical requirement of serving the requested resource. How will you know which page is having problems being served or is having performance problems?
    - nkrisc 5mo ago
      You won’t, but that’s not what was asked. Logging the requested resource is not a technical requirement of serving that resource.

kodablah 5mo ago

> If you dont have analytics you are flying blind More like flying based on your knowledge as a pilot and not by the whims of your passengers. For many CLIs and developer tooling, principled decisions need to reign. Accepting the unquantifiability of usage in a principled product is often difficult for those that are not the target demographic, but for developer tools specifically (be they programming languages, CLIs, APIs, SDKs, etc), cohesion and common sense are usually enough. It also seems real hard for product teams to accept the value of the status quo with these existing, heavily used tools.

mckn1ght 5mo ago
Actually it's more like flying in the clouds with no instruments which can lead to spatial disorientation when you exit the cloud cover and realize you're nosediving towards the earth. https://en.wikipedia.org/wiki/Spatial_disorientation https://en.wikipedia.org/wiki/Spatial_disorientation Flying based on the whims of your passengers would be user testing/interviewing, which is a complementary, and IMO necessary, strategy alongside analytics.

jubilanti 5mo ago

Wow, it really is sad how literally unthinkable it is to you and so much of the industry that you could actually talk to your users and customers like human beings instead of just data points. And you know what happens when you reach out to talk to your customers like human beings instead of spying on them like animals? They like you more and they raise issues that your telemetry would never even think to measure. It's called user research and client relationship management.

deleted 5mo ago
[deleted]
vlovich123 5mo ago
I think you’re overlooking that they were talking about stated and revealed preferences, a well known economic challenge where what people say is important to them and what shows up in the data is a gap. Of course you talk to users and do relationship management. That doesn’t negate the need to understand revealed preferences. In the OSS world this is not a huge deal. You get some community that’s underserved by the product (ie software package) and they fork, modify, or build something else. If it turned out to be valuable, then you get the old solution complemented or replaced. In the business world this is an existential threat to the business - you want to make sure your users aren’t better served by a competitor who’s focusing on your blindspot.
skeuomorphism 5mo ago
Marketing came to the conclusion that people dont know what they actually want. They decided to lump in engineers and programmers as well, since they started abusing their goodwill.
kalleboo 5mo ago
Apple and Microsoft reached their peak usability when they employed teams of people to literally sit and watch what users did in real life (and listen to them narrating what they want to do), take notes, and ask followup questions. Everything went to crap in the metric-based era that followed.
alexchantavy 5mo ago

pc86 5mo ago

You can "optimize a successful user journey" by making the software easy to use, making it load so fast people are surprised by it, and talking to your customers. Telemetry doesn't help you do any of that, but it does help you squeeze more money out of them, or find out where you can pop an interstitial ad to goose your ad revenue, and what features you can move up a tier level to increase revenue without providing any additional value.

a012 5mo ago

You have all info you need on server side, I don’t believe that you’re totally blind without client tracking

ryandrake 5mo ago

This got me thinking: Are there prominent examples of open source projects that 1. collect telemetry, 2. without a way to opt-out (or obfuscating / making it difficult to opt-out)? This practice seems to be specific to corporate software development. Why is it that startups and commercial software developers seem to be the only ones obsessed with telemetry? Why do they need it to "optimize user journeys" but open source projects do just fine while flying blind?

theplatman 5mo ago
open source projects are usually creating something for themselves so it's much easier to know what to build when you are the user whereas, commercial software has a disconnect between who are the users and developers are

e12e 5mo ago

I think there's room for a distinction between "not using metrics" and "not using data". Unthinkingly leaning on metrics is likely to help you build a faster, stronger horse, while at the same time avoiding building a car, a bus or a tractor.

yoyohello13 5mo ago

The totality of Microsoft's products is proof that this is false. If telemetry and analytics actually mattered for usability, every product Microsoft puts out wouldn't be good instead of garbage.

SchemaLoad 5mo ago
There are far too many factors to assign the quality of microsoft's products to telemetry. Having the data doesn't mean you will act on it. And doesn't mean microsofts interests are aligned with the users.

ctoth 5mo ago

> If you dont have analytics you are flying blind. We... we are talking about a CLI tool. A CLI tool that directly uses the API. A tool which already identifies itself with a User-Agent[0]. A tool which obviously knows who is using it. What information are you gathering by running telemetry on my machine that couldn't.. just. be. a. database. query? Reading the justification the main thing they seem to want to know is if gh is being driven by a human or an agent... Which, F off with your creepy nonsense. Please don't just use generic "but ma analytics!" when this obviously doesn't apply here? [0]: https://github.com/cli/cli/blob/3ad29588b8bf9f2390be652f46ee4ef75b875b03/api/http_client.go#L54 https://github.com/cli/cli/blob/3ad29588b8bf9f2390be652f46ee...

renegade-otter 5mo ago

I agree with you in that regard. That said, knowing that this is Microsoft, the data will be used to extract value from the customers, not provide them with one.

tomrod 5mo ago

Teams that do this need to just dogfood internally. Once you start collecting telemetry on external users defaulted to opt-in you're not a good faith actor in the ecosystem.

goosejuice 5mo ago

You could, I don't know, do user interviews with the various customer segments that use your product.

Apylon777 5mo ago

How did GitHub ever survive without this telemetry? Was it a web application buried in obscurity?

lynndotpy 5mo ago

Game developers benefit tremendously from streams where they get to see peoples webcams _and_ screens as they use their software. This would be _absolutely insane_ telemetry to request from a user for any other piece of software, but it would be fantastically useful in identifying where people get frustrated and why. That said, I do not trust Microsoft with any telemetry, I am not invested in helping them improve their product, and I am happy not to rely on the GitHub CLI.

account42 5mo ago
You can get this level of observation by paying testers (both professions as well as existing users who provide informed consent). You don't get to just spy on random users, yes.

deleted 5mo ago

[deleted]

codedokode 5mo ago

Analytics is wrong. I never click any ads, but they keep showing it. I avoid registering or enter fake emails, but they keep showing full screen popups asking for email. I always reject cookies but they still ask me to accept them. And youtube keeps pushing those vertical videos for alternately gifted kids despite me never watching them. What's the point of this garbage analytics. It seems that their only goal is to annoy people.

sagarm 5mo ago
All of those are affected by analytics. Ad slots will be filled whether or not you click. If you never click, you'll tend to match with either very low quality ads or ads that pay per impression (display ads). Email registration is highly valuable for a business, so analytics won't be used to decide whether to show the modal but rather test different versions of it. Cookies are too valuable to not push on users, because without them only the previously mentioned low quality ads can be shown. High quality and display ads match on interest or demographic labels. The business decision to keep vertical videos is highly likely to be affected by analytics, and of course the choice of which videos to show is based on recommendation models trained on interaction logs. The priority isn't making your experience better, though that is often an incidental result -- it's driving the business.

attentive 5mo ago

It's not like they don't own API's that those cli's are hitting. They have all the stats they need.

chillfox 5mo ago

Be very careful with that. Analytics driven development easily leads to bad outcomes. 1. Important, but less frequently used feature gets moved to a hidden spot leading to even less usage leading to eventual removal. 2. Poorly functioning features not getting the improvement they need because few use them due to how poorly they function. I have seen these patterns a lot in software where decisions are based on analytics, and I usually stop using that sofware when I find a replacement.

qwertox 5mo ago

They could well use the data from their own developers, couldn't that be enough?

account42 5mo ago

Bats are mostly blind and manage to fly just fine.