9 ms·

No telemetry in the Rust compiler: metrics without betraying user privacy

91 points by estebank 3y ago

Thaxll 3y ago
Go tried the same, and chose to let the user opt-in. https://github.com/golang/go/discussions/58409 https://github.com/golang/go/discussions/58409 The privacy design for Go was actually sound. https://research.swtch.com/telemetry-design https://research.swtch.com/telemetry-design
thih9 3y ago
I’m not clear on the details. E.g.: > Without user action the data would never go anywhere. What is the user action? I wish this was worded in a more specific way. Like: without user’s opt in the data would never go anywhere. Or: without user’s explicit consent (type “yes” to share data) the data would not go anywhere. Is either correct here?
- estebank 3y ago
  I'm not against having some kind of service that users can install and enable for telemetry using these metrics, but metrics would be useful even if we never do anything like that and they only ever exist on the users' machine: "Thank you for your ticket! Can you check your metrics logs if x, y and z are present?"
- remexre 3y ago
  It sounds like "without the user using a separate not-installed-by-default program, they wouldn't get uploaded anywhere" to me?
- thaunatos 3y ago
  Later in the post: > I could picture a future where the Compiler Team realizes we have an important question about the behavior of the software, that could be answered by analyzing these metrics, and have a VSCode or cargo plugin that analyzes these metrics locally, and if relevant information is found in these logs an appropriate summary is presented to the user for them to file a ticket.
- wolrah 3y ago
  > What is the user action? The concept described is that the compiler would generate these metrics locally and store them for a limited period of time, but there would be no built in method for those metrics to leave the system they were generated on. The user would then have that data available for their own use and could choose to manually share it or independently install a service to automatically share it per their own needs and desires. So basically collecting the metrics but with no phone-home method built in. The user action to provide consent for access to the data is actually sharing the data.
Thoeu388 3y ago
[flagged]
Groxx 3y ago
Editing this to make it clear: the article proposes local collection that could later be submitted. This is the right path! Lots of manual submission processes can use this to improve data and streamline stuff, and keeping it local solves nearly all privacy concerns. But nothing manual will ever get good submission rates unless it's zero effort, because that effort stands directly in the path of people's actual goals when they hit an error. It's the worst possible time to ask for something. --- >People saw ICEs on nightly and didn’t report them. Maybe they thought they were already fixed, maybe they didn’t know how to report them. Whatever the reason, nightly and beta failed to be an effective canary. Unless you make it trivial, you're asking for a lot from people. I can't speak to Rust in particular[1], but it's VERY common for "please report this issue! [link]" to be met with a login wall, long list of requirements and metadata requested, asks for detailed lists of what you did and how it's relevant and what you expected and... yeah, of course people don't usually report these issues. They rightfully realize it's asking for a major time investment both right now (presumably they were trying to do a thing and would prefer to continue doing the thing rather than learning your reporting system) and in the future (questions for further info on bug reports are common). Make it as easy as the telemetry you want to add, and it sometimes actually happens. i.e. no login, as near to zero effort as is possible, gather all information for them, anonymize it for them, etc. If you can do it automatically, you have trivially proven that you can also do it manually, so do that. Give people a pre-filled link with a one-click submit and an optional "contact me at [email]" field. --- [1]: https://github.com/rust-lang/rust/issues/93701 https://github.com/rust-lang/rust/issues/93701 implies they do this too. They made or already had a github account, have run multiple commands to gather info, filled in a templated text field by hand, and then they're asked to make a minimal reproduction and check if it still exists in nightly? Is it any surprise at all that people do not usually do this?
- estebank 3y ago
  I do want to make this as easy as possible. As part of that effort I am experimenting[1] with making filing an ICE report be as easy as "open ticket and attach this file". We do require a GitHub login because that is where we keep our issue tracker, and given the user base size I would have to expect that any alternative service would require some kind of login to cut down on spam. [1]: https://github.com/rust-lang/rust/pull/114128 https://github.com/rust-lang/rust/pull/114128
zer8k 3y ago
I don't like even the whisper of anything that resembles telemetry. Can someone explain the counter argument? A compiler is as low level as it goes. There is an assumed level of competence with someone using a compiler. Why is it better to collect data than just have the user cat some logs somewhere and post them? After the recent debacle(s) with the rust foundation I am almost certain a "local only" telemetry will morph into VS Code's full system colonoscopy. Please do not do this. Find another way. Compilers were bug fixed without telemetry just fine for 30 years. The Rust Foundation are by and large bad actors in an otherwise okay ecosystem. Don't let them oracle-ize it.
- estebank 3y ago
  > Why is it better to collect data than just have the user cat some logs somewhere and post them? Those logs are collected data. I'm arguing for those logs to exist. > After the recent debacle(s) with the rust foundation This has nothing to do with the foundation. I have nothing to do with the foundation. > I am almost certain a "local only" telemetry "Local-only telemetry" is an oxymoron.
  - zer8k 3y ago
    The foundation controls Rust in the same way Oracle controls Java. This link is not tenuous. Are you asserting that the compiler team would never be able to be compelled to act against the interests of its users? Furthermore, that there's no way this could happen? > Those logs are collected data. I'm arguing for those logs to exist. Concrete example: I run into an error. I post a minimal example to a bug system. Is it not possible for the compiler team to run that example and pull their own special data using an instrumented compiler? How will these logs improve it?
    - estebank 3y ago
      > The foundation controls Rust in the same way Oracle controls Java. That is not quite correct. The Rust Foundation is an entity that manages the trademark, relationships with other legal entities, and handles money to pay for running costs. The technical part of the house is the somewhat amorphous "Rust Project", which you can check here https://www.rust-lang.org/governance https://www.rust-lang.org/governance. > Concrete example: I run into an error. I post a minimal example to a bug system. Is it not possible for the compiler team to run that example and pull their own special data using an instrumented compiler? How will these logs improve it? Transient errors exist, particularly when incremental compilation comes into play. In other cases the bug/problem only manifests in a closed source project and the reporter cannot produce a minimal reproducer. There are platforms that the people triaging and fixing bugs do not have access to and might have diverging behavior.
weinzierl 3y ago
"I’m convinced that rustc should never have the capability to make network connections." [..] "[..]user information should never leave their machine in an automated manner" Sounds reasonable. The only wish I have is that we should always have the option to download a version without the metrics part as well. So tele never and metry with an easy, clear, reliable and sustainable opt-out.
- richardwhiuk 3y ago
  Out of curiosity, what's your justification for wanting opt out for local logging/metrics, to the extent you need a separate build?
  - weinzierl 3y ago
    Primarily because I don't want this to be a slippery slope where the metry of today becomes the telemetry of tomorrow. As long as these features are easy to rip out and not deeply intermingled with the rest I'm good and willing to contribute metrics even. Also, no code is free, it needs to be written and maintained and the end-user needs to download, store and run it and it brings an additional risk to break things. I just wish there was to possibility to opt out of that, even if I probably wouldn't. Finally, if this metrics code is intermingled with the rest, I can see this as an additional burden for a potential certification of the compiler.
    - richardwhiuk 3y ago
      Logs and metrics have a horrible tendency to be a cross cutting concern, sadly - you normally need to add instrumentation throughout the codebase to get good metrics.
- tareqak 3y ago
  I am thinking out loud here: what would be the comparable word here? topometry? https://en.m.wiktionary.org/wiki/topo- https://en.m.wiktionary.org/wiki/topo-
- jjav 3y ago
optimalsolver 3y ago
[flagged]
- tsion 3y ago
  You should try reading articles before commenting. Scratch that, you should try reading the headline before commenting. You should at least read the first two words of the headline. How far must the bar be lowered?
agilob 3y ago
I'm kind of a privacy freak too, but I actually don't would not and do not mind telemetry in OSS that are transparent about it. I send metrics about archlinux, plasma, firefox or syncthing: check this out https://data.syncthing.net/ https://data.syncthing.net/ I wouldn't mind transparent and honest telemetry in other tools like rustc or even vim. I don't see losing anything by sending my screen resolution and version of Xorg or Wayland. How is my privacy betrayed by this? I don't see what am I losing here, how am I betrayed, but I understand that these metrics will help make the tools better, give data to intelligent developers what to work on, what features are used more frequently. I think the word telemetry is associated with bad vibes thanks to Microsoft and Google abusing telemetry, having it opt-out by default or giving users no control over it. For opensource tools we should use another word like "Automated User Feedback", or something else without the baggage.
- yjftsjthsd-h 3y ago
  > having it opt-in by default or giving users no control over it. I think you want "opt-out"? > For opensource tools we should use another word like "Automated User Feedback", or something else without the baggage. No, because some projects will immediately re-poison the well by using "Automated User Feedback" as a transparent euphemism for the same bad practices (and in fact, if I read that I would immediately assume it was nothing but a euphemism meant to mislead me). Might as well just say "opt-in telemetry" or maybe "opt-in telemetry with extra steps to make it less identifiable" if that's what we mean.
- mike_hock 3y ago
  No, we should not invent a euphemism to dodge the bad associations with the real thing. There's no such thing as "being transparent" about telemetry in OSS because I don't feel like reading a README for every single command that I apt install to see if it has anti-features.
- amiga386 3y ago
  > we should use another word like "Automated User Feedback", or something else without the baggage. Edward Bernays, propagandist, famously coined the term "Public Relations" to give propaganda a new name without the baggage the Germans gave it. https://youtu.be/DnPmg0R1M04?t=486 https://youtu.be/DnPmg0R1M04?t=486 Telemetry, and the "baggage" associated with it, is the right word. We could be _more_ pejorative and call it "spying" or "tattling", but "telemetry" will do.
LightFog 3y ago
Telemetry has a place for product managers but it is a crutch when used for quality - it is heavily lagging and noisy, introducing it has a pull to the ‘right’ in quality gating. This isn’t a webservice with uptime and recovery requirements - it seems better to optimise processes around catching issues before public release.
- workingjubilee 3y ago
  The answer to that is simple: We do. We try to catch things before public release. We run UNSPEAKABLE amounts of regression tests. We ask people to test unstable things. Almost no one actually does test the unstable things. There's too much friction. Almost no one actually gives us feedback on them. There's too much friction. Things land on stable, people FINALLY actually mass-use them, the stable thing turns out to be busted, and THEN people overcome the friction to complain because they (rightly) fear it is going to be unfixable. Sometimes it is. Most of these have been very minor so far, and only affected things easily rolled back, but there will come a day when it is not so minor and not so easy to roll back. And nothing we can do in simulation or stress-testing on our end can replace contact with actual programs. We're not making a typical "product" here. You can't think of it in typical product terms. We're making a programming language here. A contract, effectively, though not a legal one, as it is itself a system of expressing contracts. We cannot simply roll back things that are part of the language's contract in our stable releases if something is flawed, unless the flaw is SO dire as to result in one of the handful of situations that allows us to break it anyways.
  - LightFog 3y ago
    Thanks for the extra context - and just in case my comment came across as very negative or blanket anti-telemetry, I think you are building an awesome tool and the sensitive approach to actually handling metrics seems totally reasonable. Hopefully they end up being useful here - I’ve just seen them not move the needle much on release quality relative to the various nuisances in handling them, although for sure in a different domain.
kibwen 3y ago
For context, the author (estebank) is the current force behind the Rust compiler's excellent error messages. To reiterate the lede, so it doesn't get buried: "I’m convinced that rustc should never have the capability to make network connections. [...] When it comes to Rust, users’ machines are (and should be) entirely under the users’ control." (And I don't believe this post is a response or a rebuttal to anything in particular, other than possibly being in the news because of Go. Telemetry was considered for the Rust compiler pre-1.0 and rejected for privacy reasons, and AFAIK has never seriously been pursued since.)
- SkyMarshal 3y ago
  Tbf he didn't bury the lede, it's right in the second paragraph: > I believe that no compiler needs to call home. But I also believe that we have a viable alternative if we reject the `tele` part of telemetry. Local metrics that are never sent anywhere can still give us valuable insight about how the Rust compiler is behaving.
wrs 3y ago
The paradigm of local logging plus the ability to analyze locally for known questions/concerns and give the user a one-click way to easily, voluntarily report to the maintainers makes lots of sense. You do lose the ability to learn about unknown concerns, but seems like a good 80% solution. This is the same as “sorry your program crashed, would you like to report it?” but for a wider variety of issues/concerns than crashing.
david2ndaccount 3y ago
I doubt this will be well received, mostly due to the author’s problematic framing of the problem. This is not telemetry, it is just additional local logging. I don’t understand why it’s being framed as telemetry. Just say rustc is (optionally) logging more by default and the compiler people are writing tools that can analyze those logs if you choose to upload them. Separately it sounds like rustc needs to do a better job collecting crash data that can be easily attached to a bug ticket.
- estebank 3y ago
  > This is not telemetry, it is just additional local logging. I agree. > I don’t understand why it’s being framed as telemetry. Just say rustc is (optionally) logging more by default and the compiler people are writing tools that can analyze those logs if you choose to upload them. Any kind of metrics mechanism can look like the start of telemetry. I wanted to preempt that misconception by explicitly stating the difference, before any conversation or implementation occurs. > Separately it sounds like rustc needs to do a better job collecting crash data that can be easily attached to a bug ticket. I agree.
nixpulvis 3y ago
I stopped after the first paragraph (don’t have the headspace to really unload my thoughts here) but, This could be a great idea. Local metrics in the know, then shared naturally between people. Sounds almost like how things got done before the internet. Imagine that! I look forward to reading the rest of this and hopefully agreeing with at least some of it ;) It can be a bad look when you start collecting too much information, makes it seem like something is wrong. But I also 100% see the issue. Having watched a few scenes die, it can be really sad. Programming languages are especially tough, because they are so near to permanent it’s tempting to actually need to hold on to them.
zamalek 3y ago
There's also a great resource on how people are using rust already: crates.io.
- estebank 3y ago
  That resource tells us what the code people use looks like, but it doesn't tell us how rustc behaves with it. Crater lets us tell if we've introduced a regression on features people use, but it doesn't tell us for example that compiling tokio got slower or faster, or that there's a high prevalence of code with large types, so we should focus on optimizing those. It also doesn't tell us how rustc behaves on code that is being modified, which will be incomplete/broken/weird. That kind of code is never committed to a repo.
  - zamalek 3y ago
    > It also doesn't tell us how rustc behaves on code that is being modified, which will be incomplete/broken/weird. Hmmm, good point. It also wouldn't tell you what someone did before giving up because they would never commit code. Still, it's a useful data point. You could, say, time the build for the top 100 crates once a week, or gather metrics on language features that they use.
miki123211 3y ago
Unlike Go, many Rust installs happen not through a package manager, but through Rustup. That's a perfect avenue to add useful opt-in telemetry, while making the privacy fundamentalists happy. If somebody is installing Rust interactively through Rustup, ask them whether they want telemetry or not, if they're installing from anywhere else, assume they don't. It feels like the perfect compromise for Rust, there's no way to have telemetry on without specifically agreeing to it, but there's a large enough percentage of users who would see that prompt to still make the limited opt-in data useful.
telios 3y ago
Something tangentially related, but something I'm unaware of - if I do run into an ICE on nightly, what is the current process for reporting that? I've run into one or two before (and I've generally assumed it was me doing something dumb with the environment, and I couldn't replicate it), but I wouldn't mind trying to report them in the future.
- estebank 3y ago
  You would report it on our issue tracker. Here is an example of what that would look like: https://github.com/rust-lang/rust/issues/114324 https://github.com/rust-lang/rust/issues/114324 Even if you can't reproduce it, the backtrace can be enough to figure out the problem, or at least let's us know how often the issue has happened.

91 points by estebank 3y ago

Go tried the same, and chose to let the user opt-in. https://github.com/golang/go/discussions/58409 https://github.com/golang/go/discussions/58409 The privacy design for Go was actually sound. https://research.swtch.com/telemetry-design https://research.swtch.com/telemetry-design

thih9 3y ago

I’m not clear on the details. E.g.: > Without user action the data would never go anywhere. What is the user action? I wish this was worded in a more specific way. Like: without user’s opt in the data would never go anywhere. Or: without user’s explicit consent (type “yes” to share data) the data would not go anywhere. Is either correct here?

estebank 3y ago
I'm not against having some kind of service that users can install and enable for telemetry using these metrics, but metrics would be useful even if we never do anything like that and they only ever exist on the users' machine: "Thank you for your ticket! Can you check your metrics logs if x, y and z are present?"
remexre 3y ago
It sounds like "without the user using a separate not-installed-by-default program, they wouldn't get uploaded anywhere" to me?
thaunatos 3y ago
Later in the post: > I could picture a future where the Compiler Team realizes we have an important question about the behavior of the software, that could be answered by analyzing these metrics, and have a VSCode or cargo plugin that analyzes these metrics locally, and if relevant information is found in these logs an appropriate summary is presented to the user for them to file a ticket.
wolrah 3y ago
> What is the user action? The concept described is that the compiler would generate these metrics locally and store them for a limited period of time, but there would be no built in method for those metrics to leave the system they were generated on. The user would then have that data available for their own use and could choose to manually share it or independently install a service to automatically share it per their own needs and desires. So basically collecting the metrics but with no phone-home method built in. The user action to provide consent for access to the data is actually sharing the data.

Thoeu388 3y ago

[flagged]

Groxx 3y ago

Editing this to make it clear: the article proposes local collection that could later be submitted. This is the right path! Lots of manual submission processes can use this to improve data and streamline stuff, and keeping it local solves nearly all privacy concerns. But nothing manual will ever get good submission rates unless it's zero effort, because that effort stands directly in the path of people's actual goals when they hit an error. It's the worst possible time to ask for something. --- >People saw ICEs on nightly and didn’t report them. Maybe they thought they were already fixed, maybe they didn’t know how to report them. Whatever the reason, nightly and beta failed to be an effective canary. Unless you make it trivial, you're asking for a lot from people. I can't speak to Rust in particular[1], but it's VERY common for "please report this issue! [link]" to be met with a login wall, long list of requirements and metadata requested, asks for detailed lists of what you did and how it's relevant and what you expected and... yeah, of course people don't usually report these issues. They rightfully realize it's asking for a major time investment both right now (presumably they were trying to do a thing and would prefer to continue doing the thing rather than learning your reporting system) and in the future (questions for further info on bug reports are common). Make it as easy as the telemetry you want to add, and it sometimes actually happens. i.e. no login, as near to zero effort as is possible, gather all information for them, anonymize it for them, etc. If you can do it automatically, you have trivially proven that you can also do it manually, so do that. Give people a pre-filled link with a one-click submit and an optional "contact me at [email]" field. --- [1]: https://github.com/rust-lang/rust/issues/93701 https://github.com/rust-lang/rust/issues/93701 implies they do this too. They made or already had a github account, have run multiple commands to gather info, filled in a templated text field by hand, and then they're asked to make a minimal reproduction and check if it still exists in nightly? Is it any surprise at all that people do not usually do this?

estebank 3y ago
I do want to make this as easy as possible. As part of that effort I am experimenting[1] with making filing an ICE report be as easy as "open ticket and attach this file". We do require a GitHub login because that is where we keep our issue tracker, and given the user base size I would have to expect that any alternative service would require some kind of login to cut down on spam. [1]: https://github.com/rust-lang/rust/pull/114128 https://github.com/rust-lang/rust/pull/114128

zer8k 3y ago

I don't like even the whisper of anything that resembles telemetry. Can someone explain the counter argument? A compiler is as low level as it goes. There is an assumed level of competence with someone using a compiler. Why is it better to collect data than just have the user cat some logs somewhere and post them? After the recent debacle(s) with the rust foundation I am almost certain a "local only" telemetry will morph into VS Code's full system colonoscopy. Please do not do this. Find another way. Compilers were bug fixed without telemetry just fine for 30 years. The Rust Foundation are by and large bad actors in an otherwise okay ecosystem. Don't let them oracle-ize it.

estebank 3y ago
> Why is it better to collect data than just have the user cat some logs somewhere and post them? Those logs are collected data. I'm arguing for those logs to exist. > After the recent debacle(s) with the rust foundation This has nothing to do with the foundation. I have nothing to do with the foundation. > I am almost certain a "local only" telemetry "Local-only telemetry" is an oxymoron.
- zer8k 3y ago
  The foundation controls Rust in the same way Oracle controls Java. This link is not tenuous. Are you asserting that the compiler team would never be able to be compelled to act against the interests of its users? Furthermore, that there's no way this could happen? > Those logs are collected data. I'm arguing for those logs to exist. Concrete example: I run into an error. I post a minimal example to a bug system. Is it not possible for the compiler team to run that example and pull their own special data using an instrumented compiler? How will these logs improve it?
  - estebank 3y ago
    > The foundation controls Rust in the same way Oracle controls Java. That is not quite correct. The Rust Foundation is an entity that manages the trademark, relationships with other legal entities, and handles money to pay for running costs. The technical part of the house is the somewhat amorphous "Rust Project", which you can check here https://www.rust-lang.org/governance https://www.rust-lang.org/governance. > Concrete example: I run into an error. I post a minimal example to a bug system. Is it not possible for the compiler team to run that example and pull their own special data using an instrumented compiler? How will these logs improve it? Transient errors exist, particularly when incremental compilation comes into play. In other cases the bug/problem only manifests in a closed source project and the reporter cannot produce a minimal reproducer. There are platforms that the people triaging and fixing bugs do not have access to and might have diverging behavior.

weinzierl 3y ago

"I’m convinced that rustc should never have the capability to make network connections." [..] "[..]user information should never leave their machine in an automated manner" Sounds reasonable. The only wish I have is that we should always have the option to download a version without the metrics part as well. So tele never and metry with an easy, clear, reliable and sustainable opt-out.

richardwhiuk 3y ago
Out of curiosity, what's your justification for wanting opt out for local logging/metrics, to the extent you need a separate build?
- weinzierl 3y ago
  Primarily because I don't want this to be a slippery slope where the metry of today becomes the telemetry of tomorrow. As long as these features are easy to rip out and not deeply intermingled with the rest I'm good and willing to contribute metrics even. Also, no code is free, it needs to be written and maintained and the end-user needs to download, store and run it and it brings an additional risk to break things. I just wish there was to possibility to opt out of that, even if I probably wouldn't. Finally, if this metrics code is intermingled with the rest, I can see this as an additional burden for a potential certification of the compiler.
  - richardwhiuk 3y ago
    Logs and metrics have a horrible tendency to be a cross cutting concern, sadly - you normally need to add instrumentation throughout the codebase to get good metrics.
tareqak 3y ago
I am thinking out loud here: what would be the comparable word here? topometry? https://en.m.wiktionary.org/wiki/topo- https://en.m.wiktionary.org/wiki/topo-
jjav 3y ago

optimalsolver 3y ago

[flagged]

tsion 3y ago
You should try reading articles before commenting. Scratch that, you should try reading the headline before commenting. You should at least read the first two words of the headline. How far must the bar be lowered?

agilob 3y ago

I'm kind of a privacy freak too, but I actually don't would not and do not mind telemetry in OSS that are transparent about it. I send metrics about archlinux, plasma, firefox or syncthing: check this out https://data.syncthing.net/ https://data.syncthing.net/ I wouldn't mind transparent and honest telemetry in other tools like rustc or even vim. I don't see losing anything by sending my screen resolution and version of Xorg or Wayland. How is my privacy betrayed by this? I don't see what am I losing here, how am I betrayed, but I understand that these metrics will help make the tools better, give data to intelligent developers what to work on, what features are used more frequently. I think the word telemetry is associated with bad vibes thanks to Microsoft and Google abusing telemetry, having it opt-out by default or giving users no control over it. For opensource tools we should use another word like "Automated User Feedback", or something else without the baggage.

yjftsjthsd-h 3y ago
> having it opt-in by default or giving users no control over it. I think you want "opt-out"? > For opensource tools we should use another word like "Automated User Feedback", or something else without the baggage. No, because some projects will immediately re-poison the well by using "Automated User Feedback" as a transparent euphemism for the same bad practices (and in fact, if I read that I would immediately assume it was nothing but a euphemism meant to mislead me). Might as well just say "opt-in telemetry" or maybe "opt-in telemetry with extra steps to make it less identifiable" if that's what we mean.
mike_hock 3y ago
No, we should not invent a euphemism to dodge the bad associations with the real thing. There's no such thing as "being transparent" about telemetry in OSS because I don't feel like reading a README for every single command that I apt install to see if it has anti-features.
amiga386 3y ago
> we should use another word like "Automated User Feedback", or something else without the baggage. Edward Bernays, propagandist, famously coined the term "Public Relations" to give propaganda a new name without the baggage the Germans gave it. https://youtu.be/DnPmg0R1M04?t=486 https://youtu.be/DnPmg0R1M04?t=486 Telemetry, and the "baggage" associated with it, is the right word. We could be _more_ pejorative and call it "spying" or "tattling", but "telemetry" will do.

LightFog 3y ago

Telemetry has a place for product managers but it is a crutch when used for quality - it is heavily lagging and noisy, introducing it has a pull to the ‘right’ in quality gating. This isn’t a webservice with uptime and recovery requirements - it seems better to optimise processes around catching issues before public release.

workingjubilee 3y ago
The answer to that is simple: We do. We try to catch things before public release. We run UNSPEAKABLE amounts of regression tests. We ask people to test unstable things. Almost no one actually does test the unstable things. There's too much friction. Almost no one actually gives us feedback on them. There's too much friction. Things land on stable, people FINALLY actually mass-use them, the stable thing turns out to be busted, and THEN people overcome the friction to complain because they (rightly) fear it is going to be unfixable. Sometimes it is. Most of these have been very minor so far, and only affected things easily rolled back, but there will come a day when it is not so minor and not so easy to roll back. And nothing we can do in simulation or stress-testing on our end can replace contact with actual programs. We're not making a typical "product" here. You can't think of it in typical product terms. We're making a programming language here. A contract, effectively, though not a legal one, as it is itself a system of expressing contracts. We cannot simply roll back things that are part of the language's contract in our stable releases if something is flawed, unless the flaw is SO dire as to result in one of the handful of situations that allows us to break it anyways.
- LightFog 3y ago
  Thanks for the extra context - and just in case my comment came across as very negative or blanket anti-telemetry, I think you are building an awesome tool and the sensitive approach to actually handling metrics seems totally reasonable. Hopefully they end up being useful here - I’ve just seen them not move the needle much on release quality relative to the various nuisances in handling them, although for sure in a different domain.

kibwen 3y ago

For context, the author (estebank) is the current force behind the Rust compiler's excellent error messages. To reiterate the lede, so it doesn't get buried: "I’m convinced that rustc should never have the capability to make network connections. [...] When it comes to Rust, users’ machines are (and should be) entirely under the users’ control." (And I don't believe this post is a response or a rebuttal to anything in particular, other than possibly being in the news because of Go. Telemetry was considered for the Rust compiler pre-1.0 and rejected for privacy reasons, and AFAIK has never seriously been pursued since.)

SkyMarshal 3y ago
Tbf he didn't bury the lede, it's right in the second paragraph: > I believe that no compiler needs to call home. But I also believe that we have a viable alternative if we reject the `tele` part of telemetry. Local metrics that are never sent anywhere can still give us valuable insight about how the Rust compiler is behaving.

wrs 3y ago

The paradigm of local logging plus the ability to analyze locally for known questions/concerns and give the user a one-click way to easily, voluntarily report to the maintainers makes lots of sense. You do lose the ability to learn about unknown concerns, but seems like a good 80% solution. This is the same as “sorry your program crashed, would you like to report it?” but for a wider variety of issues/concerns than crashing.

david2ndaccount 3y ago

I doubt this will be well received, mostly due to the author’s problematic framing of the problem. This is not telemetry, it is just additional local logging. I don’t understand why it’s being framed as telemetry. Just say rustc is (optionally) logging more by default and the compiler people are writing tools that can analyze those logs if you choose to upload them. Separately it sounds like rustc needs to do a better job collecting crash data that can be easily attached to a bug ticket.

estebank 3y ago
> This is not telemetry, it is just additional local logging. I agree. > I don’t understand why it’s being framed as telemetry. Just say rustc is (optionally) logging more by default and the compiler people are writing tools that can analyze those logs if you choose to upload them. Any kind of metrics mechanism can look like the start of telemetry. I wanted to preempt that misconception by explicitly stating the difference, before any conversation or implementation occurs. > Separately it sounds like rustc needs to do a better job collecting crash data that can be easily attached to a bug ticket. I agree.

nixpulvis 3y ago

I stopped after the first paragraph (don’t have the headspace to really unload my thoughts here) but, This could be a great idea. Local metrics in the know, then shared naturally between people. Sounds almost like how things got done before the internet. Imagine that! I look forward to reading the rest of this and hopefully agreeing with at least some of it ;) It can be a bad look when you start collecting too much information, makes it seem like something is wrong. But I also 100% see the issue. Having watched a few scenes die, it can be really sad. Programming languages are especially tough, because they are so near to permanent it’s tempting to actually need to hold on to them.

zamalek 3y ago

There's also a great resource on how people are using rust already: crates.io.

estebank 3y ago
That resource tells us what the code people use looks like, but it doesn't tell us how rustc behaves with it. Crater lets us tell if we've introduced a regression on features people use, but it doesn't tell us for example that compiling tokio got slower or faster, or that there's a high prevalence of code with large types, so we should focus on optimizing those. It also doesn't tell us how rustc behaves on code that is being modified, which will be incomplete/broken/weird. That kind of code is never committed to a repo.
- zamalek 3y ago
  > It also doesn't tell us how rustc behaves on code that is being modified, which will be incomplete/broken/weird. Hmmm, good point. It also wouldn't tell you what someone did before giving up because they would never commit code. Still, it's a useful data point. You could, say, time the build for the top 100 crates once a week, or gather metrics on language features that they use.

miki123211 3y ago

Unlike Go, many Rust installs happen not through a package manager, but through Rustup. That's a perfect avenue to add useful opt-in telemetry, while making the privacy fundamentalists happy. If somebody is installing Rust interactively through Rustup, ask them whether they want telemetry or not, if they're installing from anywhere else, assume they don't. It feels like the perfect compromise for Rust, there's no way to have telemetry on without specifically agreeing to it, but there's a large enough percentage of users who would see that prompt to still make the limited opt-in data useful.

telios 3y ago

Something tangentially related, but something I'm unaware of - if I do run into an ICE on nightly, what is the current process for reporting that? I've run into one or two before (and I've generally assumed it was me doing something dumb with the environment, and I couldn't replicate it), but I wouldn't mind trying to report them in the future.

estebank 3y ago
You would report it on our issue tracker. Here is an example of what that would look like: https://github.com/rust-lang/rust/issues/114324 https://github.com/rust-lang/rust/issues/114324 Even if you can't reproduce it, the backtrace can be enough to figure out the problem, or at least let's us know how often the issue has happened.