9 ms·

Telemetry required? Ask users first

97 points by trelane 4y ago

franga2000 4y ago
Making telemetry opt-in would not have fixed this issue, the author is simply using this example to make an unrelated point about telemetry. The real solution here is to remove all optional code from the critical path - this not only includes telemetry, but also things like in-app tutorials, automatic updates, crash reporting... If some code is not implementing a core function of the application, it should be allowed to crash and the application continue working. This is a core principle of good software engineering and it needs to be talked about more.
- jon-wood 4y ago
  Absolutely this. The iOS mobile client of a system of I was responsible for once spontaneously stopped working, dumping users back to the login screen. The Android client was fine, and we couldn't see any issues in the backend logs. After quite a bit of digging into the problem we eventually found that the iOS app was using a single error handler for all HTTP requests, including the one being made to query Status Page for any active incidents so they could be shown to users - our Status Page account had been erroneously disabled, which meant the requests were returning a 401, which the app interpreted as the user needing to re-authenticate. I'm not sure I've ever been embarrassed than having to report upstream that the cause of an incident was our solution for flagging incidents to customers.
  - gwbas1c 4y ago
    It's more common than you think. The Liftmaster MyQ app for my garage door openers does the same thing: Any kind of error and it knocks you back to the sign in screen.
  - chrismorgan 4y ago
    > our Status Page account had been erroneously disabled, which meant the requests were returning a 401 Part of the bug here would seem to be this returning a 400-class status code: it should pretty obviously be a 500-. The most suitable standard code is 503 Service Unavailable.
    - d_watt 4y ago
      If we're going to pedant about status codes, it seems like 401 is correct. 500 is a "server error" range, 503 would imply the service is down due to server side issues. 401:"not authorized" seems exactly correct for status page saying "I know who you are, and what you want, but you can't have it because you chose to disable your account."
      - chrismorgan 4y ago
        I am assuming account-specific endpoints, which on reflection isn’t guaranteed, but is I think reasonable to expect. On a shared endpoint, 401 would be reasonable, but on a dedicated endpoint I maintain it’s unreasonable and incorrect. 401 is only permitted if an Authorization header is missing or insufficient, and the server “MUST send a WWW-Authenticate header field containing at least one challenge applicable to the target resource” (https://www.rfc-editor.org/rfc/rfc9110#name-401-unauthorized https://www.rfc-editor.org/rfc/rfc9110#name-401-unauthorized). Presuming an account-specific status page endpoint, this cannot be satisfied; nothing the client can send will work, therefore it’s far more reasonably treated as a server error than a client error: the server has been instructed not to service these requests. “Service Unavailable” is clearly suitable. (Refusing the connection would also be reasonable, and have certain technical advantages for machine use.)
colesantiago 4y ago
> Ask Users First The answer is always "No" to telemetry in every case.
- tomrod 4y ago
  Indeed, as it should be. You should always ask the user when you collect their data, for each transaction. If your software then is annoying as it performs, that's on you as the developer. Fix your processes. Your test harness should cover edge cases. Invest in that, versus the externalized cost of collecting telemetry. Your users are not your labrats. Quit pushing your testing costs on to them.
- FinnKuhn 4y ago
  I don't think this is always true. When I get an error message or some sort of error while using a program on my private PC I don't mind sending the error code and telemetry data to the devs, because it hopefully helps them to find and hopefully fix errors in the software I am using and therefore is a win-win situation in my opinion at least.
  - tomrod 4y ago
    It is always true that you should ask. In this case, you have no qualms living with opt in, versus opt out. In this case, you can live with the minority rule [0] with no impact to your function. [0] https://medium.com/incerto/the-most-intolerant-wins-the-dictatorship-of-the-small-minority-3f1f83ce4e15 https://medium.com/incerto/the-most-intolerant-wins-the-dict...
    - tialaramex 4y ago
      Amos (Rust's fasterthanlime) has pointed out that opt-in causes a problem which the "Eh, they can always opt in if they care" crowd don't spot until it's too late. The telemetry is collected at time T, and your decision that you care must come before time T if it's to have any effect on the outcomes. After the telemetry is collected, and decisions are taken based on that telemetry, it's too late to opt in. This reflect a common problem with humans that what they don't like is regret, but what we offer them as a fix is informed consent -- it turns out what they wanted to be informed about was really whether they would in future regret it, which we cannot know. Now, as a democracy we "fix" this by just deciding that even if you refuse to participate in the process we don't get to claim that your interests don't matter. That feels like a reasonable price to pay for very important decisions like "Should we build a freeway across this land occupied by the people from this weird religious sect?" but it feels way out of proportion for something like, "Should I add more animated Emoji reactions to version X, or should I work on enabling the text-to-speech feature in group chats as well as one-to-one ?" Laws like GDPR can help organisations stick to collecting only what they actually needed to know and not just "Everything because why not" - GDPR is why not. Did you just collect the full name of every user? You'd better have a damn good explanation of why you did that. Whether user has <4GB, 4GB-8GB or 8GB+ of RAM? Very easy to justify.
transcriptase 4y ago
I’m convinced 90%+ of telemetry is simply to have usage metrics to show managers, rather than actually being used to improve products by identifying inefficiencies and bugs. And it’s a malevolent spectre if you’re the 1% of users who actually use a niche feature in something, since developers will use that knowledge to take it away. Despite a 100-fold increase in hardware power, and languages/frameworks that handle 80% of the workload a developer 15 years had to think about, applications these days have fewer settings, open and run slower, and improve far less rapidly than anything written in C/C++/Java from 1990-2005 despite phoning home with novels worth of telemetry every few minutes. Monitoring network traffic and DNS requests through a firewall and pihole, it’s downright hilarious that it’s the worst products that top the charts in both, even when being online is not a necessary component of the software. Meanwhile the software that just works, and works well, might check for an update once a month, and the changelog generally reveals new useful features and mentions performance increases.
- gcatalfamo 4y ago
  > And it’s a malevolent spectre if you’re the 1% of users who actually use a niche feature in something, since developers will use that knowledge to take it away. Yes, but also no. If my resources are not infinite, knowing that an "expensive-to-maintain" feature is used by just 1% of users, it is an incredibly important piece of information when I'm gonna have to decide where/how to recalibrate my (team's) effort. Product managing is also this: I have to know what works and what doesn't. Maybe I will use that knowledge to understand how to increase the 1% to 5% with a specific business case. Maybe it won't make sense and I will have to decide if I can keep it or not. I agree that vanity telemetry is bad for users, but killing ALL telemetry is bad for everybody.
  - transcriptase 4y ago
    I don’t disagree with anything you’ve said, and this is only tangentially related, but I wonder how much the long-tail of usage is impacted by the likely correlation of power-users/niche features, and people who firewall/pihole telemetry. I also wonder how much of a chicken-egg effect there is on the trend of developers hiding niche/power-user features that they would prefer to sunset deep into settings menus and then that setting seeing negligible use.
lumb63 4y ago
I recently opened Minecraft for the first time in several years to smack a few blocks. Their “telemetry settings” were my favorite. Options? “All” or “minimal”. Unsurprisingly, the game still ran after I disconnected from the Internet to disable all telemetry.
- hedora 4y ago
  My favorite is when my young kid old yelled that there was an error screen about a minecraft profanity filter, but “fixed it” before I could read it. I foolishly set up my microsoft account, github, and linkedin using the same email address, so now it is just a matter of time before Microsoft crosses the telemetry streams, and I get insta-cancelled for naming something “kil the pooppy crepers!” Yes, Microsoft also created a social media profile for a kindergartner without any sort of notification or parental consent screen. I don’t see why they get a free pass to ignore all the “protect the children” laws.
BiteCode_dev 4y ago
How does that work in Europe where all tracking must be opt-in thanks to GPRD?
- deleted 4y ago
  [deleted]
- okamiueru 4y ago
  GDPR deals with user identifiable data or that can be tied to the identity of a user through partially anonymous data. It should be fine to collect "telemetry" as long as the data itself isn't personal and that it also cannot be tied in any way to a particular user. At least, this is how I've interpreted the GDPR rules. On the other hand if you tie telemetry to a user ID, fingerprint, unique device identifier, etc, then all the data is personal, and since it's not obvious and necessary, you also cannot argue its a legitimate interest (note that most GDPR popups arguing legitimate interest are absolutely not valid).
  - freedude 4y ago
    My motto is, All data gathered from my device is user identifiable until proven otherwise.
nlnn 4y ago
Wish there was a good solution for this sometimes. I maintain some small open source projects (which I wrote for my own benefit), and judging from GitHub issues and stars, get some use. But I have absolutely no idea who's using the software, or which subsets of features they care about. I'd happily put work in to improve things users cared about, but have no idea how to get that info (I don't know who they are so can't ask), or using telemetry (which I'm loathe to add due to its perception).
- bombcar 4y ago
  If your telemetry is human readable (like a dialog that says “can I send details with the features you’ve used” where it actually shows what is sent, you should get SOME response. Those kinds of telemetries I love.
  - JohnFen 4y ago
    I know that I'm jaded (and I would argue not without reason), but it's hard for me to trust that software is fully and correctly disclosing what data it's sending. What I've seen some software (including some enterprise software) do -- and what I do with my own products -- is generate a human-readable text file that the user can email to the company. Then the user can be 100% certain of what data is and is not being reported. It also gives the user the opportunity to redact any data they are not comfortable sharing.
- freediver 4y ago
  Just ask users for feedback. We run a zero telemetry web browser and have one of the busiesies feedback boards [1] I ever seen for a browser. Never in a single moment I wish I had more data, and actual user feedback is so much valauble than any telemetry. [1] https://orionfeedback.org https://orionfeedback.org
  - Gys 4y ago
    The Orion feedback website looks very useful. Is it open source otherwise available?
ultra_nick 4y ago
I'd rather have my bugs fixed 1000x faster with opt-out telemetry. Most people don't change the default settings, so you'd get significantly less crash reports with opt-in telemetry.
guilhas 4y ago
> Ask Users First I think that is wrong, it increases awareness of something that I would not care otherwise, and should not by default There should be more trust. Telemetry should be auditable, and companies should be more transparent with how they're using it and storing for how long Users should be able to see everything being sent Allow to reduce telemetry levels, all, erros, settings... Allow users to reset user identifiers, reset at every installation, reset periodically Use a trustable third party telemetry service
- JohnFen 4y ago
  > There should be more trust. This is, really, the heart of the problem. There is very little trust. And the software industry, as a whole, is the reason. The amount of abuse we as an industry have engaged in, and continue to engage in, is remarkable. Repairing that trust is something that will take years of good behavior. We as an industry haven't even begun the first steps down that path, though. > Use a trustable third party telemetry service Does one exist? And if so, how would you or I -- let alone an ordinary user -- know it was trustable?
- freedude 4y ago
  "There should be more trust." Nope. Trust is always earned, never given. Why? People, Corporations, NGO, Governments are all broken in some way and are basically as evil as you allow them to treat you.

97 points by trelane 4y ago

franga2000 4y ago

Making telemetry opt-in would not have fixed this issue, the author is simply using this example to make an unrelated point about telemetry. The real solution here is to remove all optional code from the critical path - this not only includes telemetry, but also things like in-app tutorials, automatic updates, crash reporting... If some code is not implementing a core function of the application, it should be allowed to crash and the application continue working. This is a core principle of good software engineering and it needs to be talked about more.

jon-wood 4y ago
Absolutely this. The iOS mobile client of a system of I was responsible for once spontaneously stopped working, dumping users back to the login screen. The Android client was fine, and we couldn't see any issues in the backend logs. After quite a bit of digging into the problem we eventually found that the iOS app was using a single error handler for all HTTP requests, including the one being made to query Status Page for any active incidents so they could be shown to users - our Status Page account had been erroneously disabled, which meant the requests were returning a 401, which the app interpreted as the user needing to re-authenticate. I'm not sure I've ever been embarrassed than having to report upstream that the cause of an incident was our solution for flagging incidents to customers.
- gwbas1c 4y ago
  It's more common than you think. The Liftmaster MyQ app for my garage door openers does the same thing: Any kind of error and it knocks you back to the sign in screen.
- chrismorgan 4y ago
  > our Status Page account had been erroneously disabled, which meant the requests were returning a 401 Part of the bug here would seem to be this returning a 400-class status code: it should pretty obviously be a 500-. The most suitable standard code is 503 Service Unavailable.
  - d_watt 4y ago
    If we're going to pedant about status codes, it seems like 401 is correct. 500 is a "server error" range, 503 would imply the service is down due to server side issues. 401:"not authorized" seems exactly correct for status page saying "I know who you are, and what you want, but you can't have it because you chose to disable your account."
    - chrismorgan 4y ago
      I am assuming account-specific endpoints, which on reflection isn’t guaranteed, but is I think reasonable to expect. On a shared endpoint, 401 would be reasonable, but on a dedicated endpoint I maintain it’s unreasonable and incorrect. 401 is only permitted if an Authorization header is missing or insufficient, and the server “MUST send a WWW-Authenticate header field containing at least one challenge applicable to the target resource” (https://www.rfc-editor.org/rfc/rfc9110#name-401-unauthorized https://www.rfc-editor.org/rfc/rfc9110#name-401-unauthorized). Presuming an account-specific status page endpoint, this cannot be satisfied; nothing the client can send will work, therefore it’s far more reasonably treated as a server error than a client error: the server has been instructed not to service these requests. “Service Unavailable” is clearly suitable. (Refusing the connection would also be reasonable, and have certain technical advantages for machine use.)

colesantiago 4y ago

> Ask Users First The answer is always "No" to telemetry in every case.

tomrod 4y ago
Indeed, as it should be. You should always ask the user when you collect their data, for each transaction. If your software then is annoying as it performs, that's on you as the developer. Fix your processes. Your test harness should cover edge cases. Invest in that, versus the externalized cost of collecting telemetry. Your users are not your labrats. Quit pushing your testing costs on to them.
FinnKuhn 4y ago
I don't think this is always true. When I get an error message or some sort of error while using a program on my private PC I don't mind sending the error code and telemetry data to the devs, because it hopefully helps them to find and hopefully fix errors in the software I am using and therefore is a win-win situation in my opinion at least.
- tomrod 4y ago
  It is always true that you should ask. In this case, you have no qualms living with opt in, versus opt out. In this case, you can live with the minority rule [0] with no impact to your function. [0] https://medium.com/incerto/the-most-intolerant-wins-the-dictatorship-of-the-small-minority-3f1f83ce4e15 https://medium.com/incerto/the-most-intolerant-wins-the-dict...
  - tialaramex 4y ago
    Amos (Rust's fasterthanlime) has pointed out that opt-in causes a problem which the "Eh, they can always opt in if they care" crowd don't spot until it's too late. The telemetry is collected at time T, and your decision that you care must come before time T if it's to have any effect on the outcomes. After the telemetry is collected, and decisions are taken based on that telemetry, it's too late to opt in. This reflect a common problem with humans that what they don't like is regret, but what we offer them as a fix is informed consent -- it turns out what they wanted to be informed about was really whether they would in future regret it, which we cannot know. Now, as a democracy we "fix" this by just deciding that even if you refuse to participate in the process we don't get to claim that your interests don't matter. That feels like a reasonable price to pay for very important decisions like "Should we build a freeway across this land occupied by the people from this weird religious sect?" but it feels way out of proportion for something like, "Should I add more animated Emoji reactions to version X, or should I work on enabling the text-to-speech feature in group chats as well as one-to-one ?" Laws like GDPR can help organisations stick to collecting only what they actually needed to know and not just "Everything because why not" - GDPR is why not. Did you just collect the full name of every user? You'd better have a damn good explanation of why you did that. Whether user has <4GB, 4GB-8GB or 8GB+ of RAM? Very easy to justify.

transcriptase 4y ago

I’m convinced 90%+ of telemetry is simply to have usage metrics to show managers, rather than actually being used to improve products by identifying inefficiencies and bugs. And it’s a malevolent spectre if you’re the 1% of users who actually use a niche feature in something, since developers will use that knowledge to take it away. Despite a 100-fold increase in hardware power, and languages/frameworks that handle 80% of the workload a developer 15 years had to think about, applications these days have fewer settings, open and run slower, and improve far less rapidly than anything written in C/C++/Java from 1990-2005 despite phoning home with novels worth of telemetry every few minutes. Monitoring network traffic and DNS requests through a firewall and pihole, it’s downright hilarious that it’s the worst products that top the charts in both, even when being online is not a necessary component of the software. Meanwhile the software that just works, and works well, might check for an update once a month, and the changelog generally reveals new useful features and mentions performance increases.

gcatalfamo 4y ago
> And it’s a malevolent spectre if you’re the 1% of users who actually use a niche feature in something, since developers will use that knowledge to take it away. Yes, but also no. If my resources are not infinite, knowing that an "expensive-to-maintain" feature is used by just 1% of users, it is an incredibly important piece of information when I'm gonna have to decide where/how to recalibrate my (team's) effort. Product managing is also this: I have to know what works and what doesn't. Maybe I will use that knowledge to understand how to increase the 1% to 5% with a specific business case. Maybe it won't make sense and I will have to decide if I can keep it or not. I agree that vanity telemetry is bad for users, but killing ALL telemetry is bad for everybody.
- transcriptase 4y ago
  I don’t disagree with anything you’ve said, and this is only tangentially related, but I wonder how much the long-tail of usage is impacted by the likely correlation of power-users/niche features, and people who firewall/pihole telemetry. I also wonder how much of a chicken-egg effect there is on the trend of developers hiding niche/power-user features that they would prefer to sunset deep into settings menus and then that setting seeing negligible use.

lumb63 4y ago

I recently opened Minecraft for the first time in several years to smack a few blocks. Their “telemetry settings” were my favorite. Options? “All” or “minimal”. Unsurprisingly, the game still ran after I disconnected from the Internet to disable all telemetry.

hedora 4y ago
My favorite is when my young kid old yelled that there was an error screen about a minecraft profanity filter, but “fixed it” before I could read it. I foolishly set up my microsoft account, github, and linkedin using the same email address, so now it is just a matter of time before Microsoft crosses the telemetry streams, and I get insta-cancelled for naming something “kil the pooppy crepers!” Yes, Microsoft also created a social media profile for a kindergartner without any sort of notification or parental consent screen. I don’t see why they get a free pass to ignore all the “protect the children” laws.

BiteCode_dev 4y ago

How does that work in Europe where all tracking must be opt-in thanks to GPRD?

deleted 4y ago
[deleted]
okamiueru 4y ago
GDPR deals with user identifiable data or that can be tied to the identity of a user through partially anonymous data. It should be fine to collect "telemetry" as long as the data itself isn't personal and that it also cannot be tied in any way to a particular user. At least, this is how I've interpreted the GDPR rules. On the other hand if you tie telemetry to a user ID, fingerprint, unique device identifier, etc, then all the data is personal, and since it's not obvious and necessary, you also cannot argue its a legitimate interest (note that most GDPR popups arguing legitimate interest are absolutely not valid).
- freedude 4y ago
  My motto is, All data gathered from my device is user identifiable until proven otherwise.

nlnn 4y ago

Wish there was a good solution for this sometimes. I maintain some small open source projects (which I wrote for my own benefit), and judging from GitHub issues and stars, get some use. But I have absolutely no idea who's using the software, or which subsets of features they care about. I'd happily put work in to improve things users cared about, but have no idea how to get that info (I don't know who they are so can't ask), or using telemetry (which I'm loathe to add due to its perception).

bombcar 4y ago
If your telemetry is human readable (like a dialog that says “can I send details with the features you’ve used” where it actually shows what is sent, you should get SOME response. Those kinds of telemetries I love.
- JohnFen 4y ago
  I know that I'm jaded (and I would argue not without reason), but it's hard for me to trust that software is fully and correctly disclosing what data it's sending. What I've seen some software (including some enterprise software) do -- and what I do with my own products -- is generate a human-readable text file that the user can email to the company. Then the user can be 100% certain of what data is and is not being reported. It also gives the user the opportunity to redact any data they are not comfortable sharing.
freediver 4y ago
Just ask users for feedback. We run a zero telemetry web browser and have one of the busiesies feedback boards [1] I ever seen for a browser. Never in a single moment I wish I had more data, and actual user feedback is so much valauble than any telemetry. [1] https://orionfeedback.org https://orionfeedback.org
- Gys 4y ago
  The Orion feedback website looks very useful. Is it open source otherwise available?

ultra_nick 4y ago

I'd rather have my bugs fixed 1000x faster with opt-out telemetry. Most people don't change the default settings, so you'd get significantly less crash reports with opt-in telemetry.

guilhas 4y ago

> Ask Users First I think that is wrong, it increases awareness of something that I would not care otherwise, and should not by default There should be more trust. Telemetry should be auditable, and companies should be more transparent with how they're using it and storing for how long Users should be able to see everything being sent Allow to reduce telemetry levels, all, erros, settings... Allow users to reset user identifiers, reset at every installation, reset periodically Use a trustable third party telemetry service

JohnFen 4y ago
> There should be more trust. This is, really, the heart of the problem. There is very little trust. And the software industry, as a whole, is the reason. The amount of abuse we as an industry have engaged in, and continue to engage in, is remarkable. Repairing that trust is something that will take years of good behavior. We as an industry haven't even begun the first steps down that path, though. > Use a trustable third party telemetry service Does one exist? And if so, how would you or I -- let alone an ordinary user -- know it was trustable?
freedude 4y ago
"There should be more trust." Nope. Trust is always earned, never given. Why? People, Corporations, NGO, Governments are all broken in some way and are basically as evil as you allow them to treat you.