54 ms·

Transparent telemetry for open-source projects

248 points by trulyrandom 4y ago

colesantiago 4y ago
And there it is. The real intentions of Google and the the Go Programming Language. Google really can’t help themselves, to stick telemetry in anything.
champtar 4y ago
This will collect 0 telemetry from CI builds, so some data will need to be taken with a big grain of salt. I don't have data to prove it, but I would bet most cross-compile happen in CI and not on the dev laptop.
ysopex 4y ago
The only way I'd consider this, is if the telemetry my app generated is available to me or can be rerouted to another target. If so, please add this ASAP. Otherwise, I'll stick with my own observability stack.
Aaronontheweb 4y ago
How transparent is Scarf's product adoption metrics for OSS projects? https://about.scarf.sh/ https://about.scarf.sh/ I follow them on Twitter but haven't looked much into it other than reading their documentation, which makes me think that most of their telemetry is done at the point of the package distribution system: https://about.scarf.sh/package-sdks https://about.scarf.sh/package-sdks
userbinator 4y ago
Nope, nope, and more nope. You're not moving the Overton Window any more on me. In fact it seems there's a clear correlation between the quality of software and how much spyware there is embedded in it. It's often merely another way to justify unpopular changes with "but the data says so". IMHO if you want to collect any information, it should never be anything but opt-in, a conscious decision.
- geodel 4y ago
  Well, its not a humble opinion but a very strong one which is fine since you want certain thing in certain way and nothing else will do.
- mananaysiempre 4y ago
  > IMHO if you want to collect any information, it should never be anything but opt-in, a conscious decision. Serious (general) question: How do you do that given a non-technical user population? Debian’s opt-in popcon kind of manages to get a little bit of data from a fairly technical one, but nowhere near enough to estimate a low usage frequency, and it’s the only opt-in program I’m aware of that gets anything usable at all. Given that I’m unwilling to implement an opt-out system, I don’t really see a workable approach here at all.
  - JohnFen 4y ago
    What I hear you saying here is that people don't do what you want if you give them the choice, so you lean towards not giving them the choice rather than respecting their wishes. Is my interpretation correct?
    - mananaysiempre 4y ago
      >> I’m unwilling to implement an opt-out system > [Y]ou lean towards not giving them the choice rather than respecting their wishes. > Is my interpretation correct? I don’t think it is, no :) Rather, I’m not sure how to sell, to put it crassly, users on a choice when properly investigating or even being confronted with that choice would delay them seeing the dancing bunnies[1], but that would also, if I have any say about it, improve the bunnies in the future. Does that mean there’s a shade of “I know better” in my problem statement? Of course it does, if I didn’t know better than the average user I’d have no business designing such choices. I don’t think there’s anything wrong about that, better than the average at an activity few practice is not a terribly high bar. Not giving the users a choice or manipulating them into making the one I think is right would absolutely be wrong, though. Basically, how do I make the user think, how do I give them the appropriate data to do so, and how do I deal with the obvious contradiction of that goal with principles of good design[2]? The potential benefits to the software and (thereby) the users are too much to give up without even asking those questions. (See nearby comment for extended discussion.) [1] https://blog.codinghorror.com/the-dancing-bunnies-problem/ https://blog.codinghorror.com/the-dancing-bunnies-problem/ [2] https://sensible.com/dont-make-me-think/ https://sensible.com/dont-make-me-think/
kardianos 4y ago
This is well done. It only exposes counters, and rather then pushing data up, the telemetry server must know the names of what it can ask for. No wildcards.
- deleted 4y ago
  [deleted]
deleted 4y ago
[deleted]
autoexec 4y ago
> Although the report would not include any identifiers, the TCP connection uploading the report would expose the system’s public IP address to the server if a proxy is not being used. This IP address would not be associated with the uploaded reports in any way. Any fully transparent data collection is going to have to include IP addresses and timestamps. Even if the IP isn't being used for debugging, the software still phones home and the IP is still being collected and logged when it otherwise wouldn't be. Either when uploading the report or when downloading the “collection configuration”. Honestly, assuming full transparency, I'm not opposed to the concept. I question how much telemetry is actually necessary, but I'm certain there will be times when it's nice to have. It'd also be interesting to see how it would go when for once people can see exactly what is collected, when, and from where. I'm not sure that Google is the best place to showcase such a concept though. I'm sure there are a lot of people who have no problem with handing more data over to Google, but Google has abused the public's good will for the sake of data collection many times, and it's sure to put off some of the people who aren't already completely disgusted by the idea of their favorite open source projects collecting telemetry.
- mananaysiempre 4y ago
  > Any fully transparent data collection is going to have to include IP addresses and timestamps. Even if the IP isn't being used for debugging, the software still phones home and the IP is still being collected and logged when it otherwise wouldn't be. Either when uploading the report or when downloading the “collection configuration”. How do you verifiably not collect users’ IP addresses when receiving data from them? The verifiable part is the problem, of course you can (and should) just not log the addresses, but then the users can only trust you (and hope you or your uplink haven’t received any legal orders to the contrary). The only approach I can think of would be a Tor hidden service, but while it would technically work, as far as not exposing your users to scrutiny it actually sounds worse.
  - rsc 4y ago
    The only option is to have a proxy sit in the middle between the uploader and the server. You mentioned Tor but it doesn't have to be Tor, just some proxy most users would trust not to collude with the server and that doesn't itself derive benefit from seeing the IP addresses. If there were a different entity that could be relied upon to run servers doing this and were highly trusted by users, I'd be interested to use it. Failing that, the usual answer for an enterprise or company is to run their own HTTP proxy. The design explicitly supports that.
4ad 4y ago
This is just part 1, but all articles in the series have been published: https://research.swtch.com/telemetry https://research.swtch.com/telemetry
deleted 4y ago
[deleted]
schmichael 4y ago
> the vast majority of projects, even large ones that would benefit, stay away from telemetry. Nomad is one of these projects. We support a dizzying array of platforms (32bit Intel Linux?!). We have no idea how popular our Consul service mesh integration is. Are bug reports a sign of use or just failed experiments? Is anyone running on macOS in production or just ephemeral dev agents? Surveys about this are just asking humans to do something computers can do better. Obviously privacy and consent are paramount concerns, but not only are they solvable, in open source they’re fully auditable (and a fork could fairly easily maintain a patch that removes it outright). I think open source largely rejecting telemetry puts it at a huge disadvantage to proprietary and SaaS software where it is the norm. I’m very excited to see someone as thoughtful and well reasoned as Russ Cox to be trying to move the status quo forward.
- ergonaught 4y ago
  Everything involves tradeoffs. The times "we" (previous companies) tried to implement telemetry in open source non-SaaS products (as distinct from "projects"), we either got huge blowback or users/customers simply blocked it at the firewall (and security teams at major enterprises were unwilling to open holes anyway). The only workable solution I found was integrating this in a value-add way, so that something in the service/experience/etc was better for the user/customer as a result of enabling telemetry, without the dark pattern of making things intentionally awful/worse without it. We simply never got enough data to matter otherwise. But, again, that was products and not projects.
  - GordonS 4y ago
    > The only workable solution I found was integrating this in a value-add way, so that something in the service/experience/etc was better for the user/customer as a result of enabling telemetry, without the dark pattern of making things intentionally awful/worse without it. This sounds like a great concept, but I'm struggling to come up with concrete examples - how did you approach it?
- smoldesu 4y ago
  On the contrary, I'd argue that the tracing visibility you're looking at isn't inherently a software trait at all. It's a deployment feature, which is something you address at-cost when building a product, but almost never when building FOSS software. It's not that people in FOSS don't see that upsides to it, it's that those upsides are insignificant relative to the cost of sustained market research. It's easier to just... make stuff, and have companies plaster over the gaps when their interests align. Look at GNOME, which recently pushed for it's users to contribute telemetry: https://linuxiac.com/gnome-survey-results/ https://linuxiac.com/gnome-survey-results/ Nothing wrong with what they've done here, but we already had most of these metrics. Nothing was really learned, and it took Red Hat and a few thousand users to get here. For smaller-scale projects, imagine how much smaller the returns would be.
mordae 4y ago
I dunno. It sure makes sense to me to collect telemetry from free software installations, but I feel that having every platform or even piece of software to do it on its own with opt-out will inevitably lead to people being overwhelmed and angry. I would, personally, prefer a single non-profit service that would list publicly what is being collected and publish the results as open data for anyone to use. Applications (at least on Linux) would not submit their reports directly, but would use a local relay service that could be turned off completely or that could filter what reports to send to the server and what to /dev/null. Distributions and other software stores would then make it mandatory for software to use this relay and either patch out any other telemetry from their packages or straight out forbid those that would not comply.
- gen220 4y ago
  I think the issue of telemetry is fundamentally a human issue of incentives and trust. The system you describe is wise because it recognizes this and attempts to address it. The difficulty with telemetry is that even if we design the perfect, privacy-preserving system to begin with, once the pattern of having a network port open is established, there's nothing to prevent us (humans) from changing our policies about what we're allowed to push/pull over that port. In real-world analogues for these kinds of thorny policy problems, we have centralized arbiters to solve these problems. That might be a fruitful course of research for people interested in this problem to explore. Unfortunately, even though this problem has software as its medium, it is a problem that cannot be solved by clever software alone, despite any appearances to the contrary.
  - JohnFen 4y ago
    > The difficulty with telemetry is that even if we design the perfect, privacy-preserving system The other difficult is what you mentioned: trust. Even if a piece of software really does telemetry in a perfect, privacy-preserving way -- as a user, I have to take the developer's word for that in the end. That's a hard hurdle to pass, because that trust has been violated so much in the past that nobody gets the benefit of the doubt anymore. > Unfortunately, even though this problem has software as its medium, it is a problem that cannot be solved by clever software alone I agree entirely. At the heart of it, this is not a technological problem. It's a human one.
bioemerl 4y ago
Honestly, this may be unpopular with hacker news, but just add your own telemetry. If people don't like it they can turn it off, and telemetry is essential for a good product. Do let people turn it off though please.
- groestl 4y ago
  > If people don't like it they can turn it off If I, perchance, encounter software I use phoning home without my explicit permission it's done on my systems. Period.
  - bioemerl 4y ago
    That is fine, but in this case telemetry trades you (and other more hardline users) as a user for all the extra users you gain from instant crash reports, quick feedback, and generally better productivity. I would never personally make that trade-off, and would always put (disableable) telemetry in my projects.
  - Spivak 4y ago
    Well according to our telemetry 0% of users turn it off so it seems pretty popular. But more realistically what you gain in privacy you give up in having your voice heard by the devs. The decisions about the future of the product/project will be driven by the data, specifically the data from the kind of people who leave telemetry on.
    - JohnFen 4y ago
      If 0% of your users disable it, that kind of screams there's something wrong with your opt-out mechanism. Is it broken? Hidden? Difficult to do? I mean, with any group of people, there will always be a percentage that will disable it. If the telemetry is popular, that percentage might be very small, but it would be non-zero.
      - jwilk 4y ago
        I think you missed the joke.
taveras 4y ago
I wish there was a standard way of disabling telemetry across software dependencies. While I leave it turned on for personal projects, several projects at work require disabling it. I have spent hours auditing through transitive dependencies to turn it off. It should not be this painful.
- slimsag 4y ago
  No reason they couldn't all aim to respect a `TELEMETRY=false` env var, akin to the web's 'do not track' request.
  - ComodoHacker 4y ago
    Does anyone really respect DNT though?
- userbinator 4y ago
  A firewall is probably your best bet. Don't allow network traffic originating from anything other than a short whitelist.
hommelix 4y ago
Telemetry in open source exists for a long time. Debian has the popcon package that can be installed and reports weekly usage of the software packages. The telemetry data are published in the open. The Debian popcon FAQ could be used as guideline for other telemetry needs. https://popcon.debian.org/ https://popcon.debian.org/
- ptx 4y ago
  It does sound quite similar. But in addition to the crucial difference of opt-in vs. opt-out there's also an interesting contrast in how it's framed. Debian talks about what you, the user, can do: help out, participate and vote. If you choose to do so. The Go team talks about what the developers and their software will do to the user's machine, but the user is completely passive in their description. This is also reflected in the term "telemetry" itself: the software is not a tool in the user's hands but rather a remote-controlled probe in the user's habitat that pokes at the user to elicit interesting responses.
- agwa 4y ago
  popcon should not be used as an example of how to do telemetry, as it is far worse for privacy than the Go proposal: 1. Sends names of private packages to the server, and publishes them. 2. Sends a unique identifier (a UUID stored in /etc/popularity-contest.conf) to the server, which is stored. 3. Doesn't use sampling, so if you use popcon you will be submitting a report once a week (Go's telemetry would average just one report a year). 4. Submits over plaintext protocols by default. popcon may be opt-in (in the sense that the prompt during installation has "No" selected by default) but the prompt doesn't disclose the large privacy risks. People are not appreciating the thought that has gone into the Go proposal to minimize the collection of private data, either intentionally or by accident, such as the client-enforced requirement that the the names of counters be published in a tamper-proof log so anyone can verify that, for example, no private package names are being disclosed. Everyone is focusing on opt-in vs out-out, but to me these other details are far more important.
- msla 4y ago
  > Debian has the popcon package that can be installed and reports weekly usage of the software packages. Right. That's fully opt-in, to the point the package isn't even installed by default, which is the only moral way to do this.
slimsag 4y ago
I've been a pretty strong advocate of the idea that analytics should always be minimal, 100% anonymous, aggregated, and open to the public - otherwise it’s spying. This is how we do analytics on our websites today[0][1], and how we plan to do it in games we release in the future. Maybe one day I will start a dedicated FOSS service that people can use for exactly this with some trusted reputation/transparency/auditability to it. I think what Russ has described here is decent and well-reasoned. I also think that Go being a product (it is, whether you like that word or not) makes it more fair to desire analytics of this form. I think it being opt-out is reasonable (after all, if it is not, they will make decisions using data that does not come from the vast majority of users, may as well not have analytics at all then.) But I am afraid of this becoming pervasive not just in products (like CLI tools), but also in libraries, imagine every Go/npm package you use wants to ping the network because the authors want to know 'is this popular? can we deprecate XYZ method?' etc. If transparent telemetry in the form Russ and I have been viewing it becomes a more common thing, it won't be a surprise if more library authors begin to try to adopt something like this and it becomes a pervasive problem IMHO. [0] https://hexops.com/privacy https://hexops.com/privacy [1] https://machengine.org https://machengine.org
- rsc 4y ago
  I am concerned about run-time telemetry in libraries as well. It might make sense for language ecosystems to offer more data about library usage gathered at build time eventually, as a different system than the one I'm posting about today. I think when you get to that level of detail you probably need to start thinking hard about differential privacy and probably cryptographic solutions like ESA or Prio. I don't think we know enough to design the library solution yet.
  - JohnFen 4y ago
    Telemetry embedded in libraries is simply abusive, in my opinion. At the very least, the decision about whether or not to include telemetry should be made by the application developers, not the toolmakers.
    - 4y ago
infogulch 4y ago
This is a good plan, very simple and clear, and I like the list of system properties at the end. The solution is pretty tailored for the Go toolchain, which is a good strategy that has worked for them in the past. A more general purpose metrics tool I'm watching closely is Divvi Up https://divviup.org/ https://divviup.org/, a research project by ISRG, the same org that runs LetsEncrypt. The basic idea is to divide up each metric into two parts and publish each part to separate collection servers (one run by you and the other by divviup). Then the servers separately aggregate their half and combine the results, the idea being that each half is useless on it's own but when combined it's still useful. I wouldn't suggest it for this application, but for the majority of typical apps it would be a vast improvement to privacy compared to the status quo.
teraflop 4y ago
I am all for transparency and limited intrusiveness of telemetry. But in practical terms, the problem with this approach -- if I'm understanding it correctly -- is that it has no way to detect and reject outliers, and therefore the data can't be validated in any way. It only makes sense if all your clients are 100% trustworthy. Let's say you want to know whether to keep supporting ARMv5, and your data says 10% of users are using it. There's no way to tell whether that's accurate, or if you have 0.01% of die-hard users who modified their telemetry code to report 1000x as frequently as they're supposed to. Even if you suspect this is happening (and you might not), there's no way to identify the culprit and filter out their data without tracking personal identifiers such as IP addresses. So even if most of the time the telemetry data is valid, over time it will trend toward uselessness, because it can be endlessly second-guessed unless it confirms a decision you wanted to make anyway.
bee_rider 4y ago
I don’t really see why a classic community-driven open source project would care about what non-contributing users are doing with the software. In that case, helpful users come with built-in telemetry (pull requests). But I guess this could be helpful corporatized read-only repo projects, or other groups that aren’t sure if they are building a community or a customer base.
- notpushkin 4y ago
  Because pull requests aren't the only reason you might do a community-driven open source project. Perhaps you're just altruistic, or want to populatize some technology etc.
JohnFen 4y ago
> When you hear the word telemetry, if you’re like me, you may have a visceral negative reaction to a mental image of intrusive, detailed traces of your every keystroke and mouse click headed back to the developers of the software you’re using. But that's not my only objection to telemetry. Equally important to me is that so many bad decisions are justified based on telemetry. It's very easy to misunderstand the data, because telemetry leaves out so much, but developers often treat it as if it's giving a complete picture. As an example, I have seen developers drop really important functionality on the basis that it is rarely used. While that was true, it was also true that when those rare times happen, that functionality was absolutely critical to have.
- userbinator 4y ago
  Or they use the data as an accelerant: move rarely used features to places where they're even less discoverable, making them even less used, and then remove them altogether. The justification then becomes a self-fulfilling prophecy.
Arnavion 4y ago
I haven't worked with golang in some time. How do golang devs generally obtain the compiler? If you're getting it from distro repos, it should be straightforward to convince the distro package maintainer to disable the telemetry / patch it out. Or is it a nvm/pyenv/rustup situation where you prefer to use bespoke toolchain managers to download upstream's compilers?
- aliyeysides 4y ago
  It depends on which compiler you want to use, but precompiled binaries include the gc compiler by default. If you want to compile from source yourself, you can use gc or gccgo assuming you already have the go toolchain, otherwise you would need to bootstrap from an existing binary.
- badrequest 4y ago
  I mainly get it straight from golang.org, but this will be able to be disabled via environment variable just like the modules proxy stuff was. https://research.swtch.com/telemetry-design#opt-out https://research.swtch.com/telemetry-design#opt-out
  - sidewndr46 4y ago
    Running the command in that just shows me a message of "go: unknown go command variable GOTELEMETRY"
    - 4ad 4y ago
      Because this proposal has not yet been implemented.
      - sidewndr46 4y ago
        So is there a way to disable this ahead of time? Or do I have to install the version with telemetry first?
        4y ago
zzzeek 4y ago
Was hoping for a big highly designed webpage with "enter your github URL here". but alas (it did say "transparent", like a service people opt into that could relate installations to github URLs)
caniszczyk 4y ago
We need solutions in this space for open source projects, I've been monitoring https://divviup.org https://divviup.org as an option too!
alyandon 4y ago
Oh Google - never stop being you. Not only is it going to be opt-out (because of course it would be coming from Google), I really like the whole "wait a week before sending telemetry" part that just coincidentally has the benefit of sneaking right past people that actively look for suspicious network activity when they've freshly installed something. Am I being uncharitable?
- geodel 4y ago
  [flagged]
- Thaxll 4y ago
  Very popular programming language and IDE have telemetry on by default, VSCode, C#, Java etc ... People act like they discover telemetry in 2023. I don't think it's a big deal, ultimately it's to improve Go and the proposal makes it very easy to disable it ( single env variable ).
  - alyandon 4y ago
    I think they should all be opt-in as well. However, as a developer and pretend sysadmin, I am generally a nice guy about not turning off telemetry on software products with a user facing UI that I use frequently.
  - xh-dude 4y ago
    Totally agree. Telemetry has been around and matured and benefits users. I’m not sure the benefits for Go would be as significant as other software but, really, why not?
    - masklinn 4y ago
      > Telemetry has been around and matured and benefits users. Does it? Telemetry mostly seems used to justify removing features I need on grounds that they’re little used. As an other user noted, if telemetry is your yardstick, the average backup software removed the “restore” feature because that’s barely ever used.
ergonaught 4y ago
On-by-default makes me question whether rsc's judgement has been compromised, which leads me to question continuing to use the language. A strange miss for him.
- nicce 4y ago
  Off-by-default in a scale likely means that there is no telemetry at all. I would not cancel a guy or programming language based on just suggesting that. He has given a lot of though for that if you read the blog posts.
  - cwkoss 4y ago
    If a take a dollar from everyone but it's opt out, that's still theft. If I make it opt in, nobody is going to give me the dollar, but that doesn't make opt out morally justifiable.
    - nicce 4y ago
      You are comparing apples to oranges. Telemetry is a curse word these days, but you should still read his posts.
  - ergonaught 4y ago
    I would expect nothing less of him than to give a topic a great deal of thought and devise a principled and rational solution. I am, however, reminded of a quote from Peter Drucker: “There is nothing quite so useless as doing with great efficiency something that should not be done at all.” I’m not picking nits regarding the overall proposal. I’m questioning the judgement that concluded/rationalized “on by default” is the right thing to do. Also not “cancelling” at the moment, either, but definitely reassessing my future language choices and taking a more critical appraisal of Go’s direction/choices. This isn’t really an isolated incident, and trust has accumulated some dents.
whoopsie 4y ago
Opaque telemetry can also be a barrier to adoption: my users’ IP addresses may legally be PII that I cannot disclose.
bluehazed 4y ago
This is really slimy, Google swung and missed and let Go of the bat here.
kabdib 4y ago
Oh, hell no.
_ph_ 4y ago
If there is any virtue to collecting telemetry, make it opt-in. Any developer convinced of this being useful will gladly enable it. But making it opt-out is just nefarious, because most users will not be aware of it.
- Thaxll 4y ago
  This is naive, no one ever turn telemetry on if it's turned off by default, that's the reason why it's on by default.
  - ocdtrekkie 4y ago
    > no one ever turn telemetry on if it's turned off by default If nobody would voluntarily do it, why do you think it's okay to do it at all? By your very admission, nobody wants this. Because if they did, they'd turn it on!
  - _ph_ 4y ago
    Still, opt-out is just inacceptable. At least with a mechanism which can easily fail, like setting an environment variable. This basically forces you to wrap the go tool in a script which ensures the environment variable to be set. As this seem to cache the results, another option is to fiddle with the cache to report bogus information.
    - Thaxll 4y ago
      You can set env variable for the go toolchain with a command such as: go env -w TELEMETRY=off which will be written to disk and use by the go cli.
      - _ph_ 4y ago
        Where will it be written to and how is it guaranteed to be picked up by any further invocations of the tools?
      - JetSpiegel 4y ago
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
bombela 4y ago
The information and rate of upload as described seem reasonable. Is the fear from most people that it will be a foot in the door? And a way for Google to collect extra overtime? Note: I think Go is a regressive technology. That would have been great in 1970s. Not today. But that's a different topic. My point is that I tend to be biased very negatively against Go. But here I don't see something wrong.
- throw7777 4y ago
  Interested as to why you believe go is regressive, could you expand on that?
  - bombela 4y ago
    You can find plenty of well articulated rant online. The gist is Go ignores all research in programming languages. Is very hard to use properly. It is really hard to produce safe APIs. You are encouraged to pop threads (preemptive user threads) all over the place. And may the god of lost state saves you from the hell of data races. Also what is this slice/vector type thing? Don't forget to check `err`, nope not this one, the other one.
    - 0xjnml 4y ago
      > The gist is Go ignores all research in programming languages. This is an assumption and it's IMHO false. Go creators watched all the research, probably closely, lived with its results and were not happy with what they get. Their experience reminded them that less features in a PL make it arguably less expressive, sure, but in the same time it makes it faster to learn and master, easier to read and debug. Writing tools is simpler if a parser for your language can be hacked over the weekend from scratch, etc. At the end of the day its a matter of personal preferences, but it has, IMO, nothing to do with "ignoring all research in programming languages", quite the opposite.
      - bombela 4y ago
        You start with a dizizlgy large amount states (RAM, storage, network, threads...). The goal of programming languages is to give you a way to reduce the vast number of states as much as possible. Without removing the ones you need to do the work. Each layer of abstraction reducing what is possible, without impeding the work that ultimately needs to be performed. In my humble opinion, you should learn a bit more about, easy vs simple, complicated vs complex.
cube2222 4y ago
Probably related to[0]. To anybody complaining that this should be opt-in: opt-in telemetry doesn't work. The reason for this is that most people don't care, but they don't care either way. They don't disable it when prompted, nor would they enable it manually. The idea of telemetry is being able to prioritize the work that will be most widely useful. For this you need a good and balanced sample of your users. You don't really get any kind of sensible sample if you only do it opt-in. Additionally, this ship has long sailed, everybody does opt-out. What I do think however, is that it should very clearly notify the user of this, and give them an easy way to disable it. Like in OctoSQL[1] (disclaimer: which I'm the author of) which prompts you on first run and shows explicitly how to disable it. All things considered, this is an open source project, so you're free to maintain a fork without telemetry. The Go toolchain also uses the Google-hosted module proxy by default, which really is a bit like telemetry already. [0]: https://news.ycombinator.com/item?id=34707583 https://news.ycombinator.com/item?id=34707583 [1]: https://asciinema.org/a/eWQsyXQKi1fmithyTekAD5fWS https://asciinema.org/a/eWQsyXQKi1fmithyTekAD5fWS
- marginalia_nu 4y ago
  > The idea of telemetry is being able to prioritize the work that will be most widely useful. It does sort of hinge on the highly suspect assumption that usefulness is correlated with use. An obvious counter-example to this is something like a fire-extinguisher, which will in the ideal case just sit on a wall until it's use-by date passes and then it's discarded having never been used; or on the flip side, an incredibly byzantine workflow that could be reduced to something much simpler will appear important and useful. Even without these edge cases, interpreting statistics is really hard. Like people with PhDs who have studied these things for years still get them wrong all the time. What ends up happening more often than not is it's used as a tool to quiet the critics when pushing through unpopular changes.
  - cube2222 4y ago
    All this boils down to "an unskilled engineer will misinterpret data even if they have it". I'll assume the Go team knows what they're doing, based on their track record so far. There's a lot of very simple questions you can answer very reliably, too, like "what proportion of the users are still using a certain compatibility flag".
tgv 4y ago
They could allow public access to that data. That can help more people than just the Go team, and it would add transparency.
- mseepgood 4y ago
  That's literally the plan if you read it.
1vuio0pswjnm7 4y ago
"The system is on by default, but opting out is easy, effective, and persistent."
wrldos 4y ago
Imagine if GNU started adding telemetry to their compiler toolchain... If that sounds fucking stupid, which it does, then so does this.
r2vcap 4y ago
I have set DOTNET_CLI_TELEMETRY_OPTOUT=1 as an environment variable in my .profile file. What should I do for golang?
- guessmyname 4y ago
  > To opt out, users would set GOTELEMETRY=off in their environment or run a simple command like go env -w GOTELEMETRY=off; The first telemetry report is not sent until at least one week after installation, giving ample time to opt out. Opting out stops all collection and reporting: no “opt out” event is sent. It is simply impossible to see systems that install Go and then opt out in the next seven days. Source: https://research.swtch.com/telemetry-intro https://research.swtch.com/telemetry-intro
ddevault 4y ago
This is not okay. The only ethical way to do telemetry is opt-in. If not enough people are opting in, you need to incentivize them to -- most simply by just paying them for their data. After all, telemetry is "valuable", isn't it? But if you can't figure out how to convince people to opt-in, then tough luck, sucks to be you. Opt-in or GTFO, Google. I'll be patching this out of the Alpine package for Go the day it ships.
- gavinhoward 4y ago
  You and I may not agree on a lot, but I sure agree with you on this one.
xyzzy_plugh 4y ago
I'm usually against telemetry but not only is the approach here somewhat reasonable, I think I actually trust Google more than, say, homebrew to not do something egregious with the data. Google is at least as broadly compliant as one can be with various standards (of questionable value, natch) but is also on the hook socially and perhaps legally if they fuck this up.
- bluehazed 4y ago
  [flagged]
candiddevmike 4y ago
Is this even up for debate, or is this post more of a FYI?
- wrldos 4y ago
  No it's not up for debate at all. Much like when Microsoft did this with .Net core, the Github thread is clearly a misguided post by RSC expecting the community to conform or support it. They didn't so now it's a damage control exercise. It will happen. Any corporate controlled project on this scale is prone to this failure mode.
mftb 4y ago
I hope this proposal is defeated and they don't implement this. I don't buy the premise that the benefit is worth the price. I think CLI tools like the ones in the Go Toolchain and their usage patterns are fairly well understood by this point. I'm sick and tired of every piece of software I interact with phoning home. That said, as long as they give me reasonable means to configure the software the way I want, it's probably not a deal-breaker for me. In other words, I will just set the $ENV_VAR_WHATEVER to turn this off, and that's that.
blibble 4y ago
how long until ads?
silisili 4y ago
Very much against this. Sure, it sounds naive enough, and can give reasons why. But I have 3,436 items in /usr/bin. What if -every- one of these started doing their own telemetry, their own envvars, etc? If we have to deal with telemetry, then I'd instead hope that there can exist a single telemetry systemwide interface. Not sure how that would be designed or implemented, but would be better than everyone doing their own bespoke thing. Plus easier for me to disable them all in one go.
- jodrellblank 4y ago
  > "What if -every- one of these started doing their own telemetry, their own envvars, etc?" What bad thing are you suggesting would happen if they did? Your computer and internet connection can't handle four thousand strings or four thousand HTTP POSTS, or four MB more disk space of telemetry libraries? I bet it can. This isn't a technical problem, it's a control and consent problem.
  - Macha 4y ago
    I wouldn't be so sure it's not also a technical problem, the limit of execve can be as low as 128kb which for 4,000 strings gives a maximum of 32 characters per NAME=VALUE environment value
    - jodrellblank 4y ago
      Free disk space can be as low as zero, but we don't blame the tool makers for adding an extra 100Kb or 20MB, we blame the computer owner for not having enough disk space to install the thing they chose. Wrapper scripts for every utility to do UTIL_TELEMETRY_OPT_OUT=1 util ... so they don't need to be set all at once.
      - JohnFen 4y ago
        > we don't blame the tool makers for adding an extra 100Kb or 20MB Honestly, I do. Code bloat is a real thing.
msla 4y ago
The only moral response is to send false data to the servers.
stefanos82 4y ago
From a legal point of view, how companies will react to this, be it default-on or default-off? Some companies are using it for internal use which I'm sure all of us know cases with a number NDA-ed projects from third-parties or outsourced companies that collaborate on the matter. So, who is going to sue whom here when the one party will disable OR has already disabled the telemetry and the other will have it on by default, for whatever reason?
omginternets 4y ago
I see a very frustrating pattern emerging in which $COMPANY asks its users if it can do something, the users say "no", and $COMPANY storms off under the guise that "the discussion is unproductive". I am left with the impression that the decision has already been made, and that we are witnessing a PR strategy to make Google appear reasonable. I think that Mr. Cox, with all the respect I hold for him, is playing the part of the "useful idiot" here.
deathanatos 4y ago
This week one of my tasks is to figure out how to neutralize some telemetry in one of our apps. We had no idea it was there, we do not want to be sending data. Last week, the parent company decided they didn't want to maintain the telemetry server any longer, and got rid of it. Now the tool has generated thousands of log messages that it can't phone home. And so it must be silenced, since it is cluttering up the logs, generating false alerts, etc. Please, no more.
- JohnFen 4y ago
  The existence of telemetry is the main reasons why I avoid using new software anymore. Really, opt-in, opt-out, it doesn't matter. I can't trust that any of those mechanisms actually work, that if I opt out, an update won't reenable it, or that the data collected is actually limited and anonymized.
philosopher1234 4y ago
There's a lot of confusion in these comments about opt-out vs opt-in. The debate isn't settled, but a lot of the issues raised here have been addressed. Reposting Russ' comment: >Longer answer about opt-out generally, copied from mail I sent to golang-dev. > I wrote a little about this at https://research.swtch.com/telemetry-design#opt-out https://research.swtch.com/telemetry-design#opt-out. Just to quote the beginning: “An explicit goal of this design is to build a system that is reasonable to have enabled by default, for two reasons. First, the vast majority of users do not change any default settings. In systems that have collection off by default, opt-in rates tend to be very low, skewing the results toward power users who understand the system well. Second, the existence of an opt-in checkbox is in my opinion too often used as justification for collecting far more data than is necessary. Aiming for an opt-out system with as few reasons as possible to opt out led to this minimal design instead. Also, because the design collects a fixed number of samples, more systems being opted in means collecting less from any given system, reducing the privacy impact to each individual system.” > To elaborate, one of the core things I believe about designing a system like Go is that it needs to ship with the right defaults, rather than require users to reconfigure the defaults to get best practices for using that system. For example, Go ships with use of the Go module mirror (proxy.golang.org) enabled by default, so that users get more reliable builds out of the box. Similarly, Go ships with the use of the checksum database also enabled by default, so that users get verified module downloads out of the box. We know that most users don't want to and probably won't spend time reconfiguring the system: they trust us to set it up right instead. Of course, that implies a responsibility to actually look out for users' best interests, and we take that very seriously. There are important privacy concerns about the module mirror and about the checksum database, despite their clear benefits, so we designed those systems to address as many of those concerns as possible. Among the decisions we made to improve privacy there: (1) GOPROXY can proxy both the module mirror and the checksum database, (2) we published a very clear privacy policy (proxy.golang.org/privacy), (3) we introduced the concept of a tiled transparency log to keep log fetches from exposing a potential tracking signal. > Moving back to telemetry, enabling telemetry does not confer the same kind of direct benefits to users as the module mirror and the checksum database do. Instead the direct benefits it confers fall on other users: (1) allowing your Go installation to participate in the system means other installations participate just a little bit less, thanks to sampling, and (2) allowing your system to send usage information strengthens the signal from others with similar usage. There is still an important indirect benefit: one system opted out won't have much of an impact, but 99% of systems opted out has a huge impact, and that leads to mistakes like the ones I mentioned in the first blog post, which do make Go worse for you. > Like with the module mirror and checksum database, there are good privacy concerns to telemetry despite the clear benefits, so the design of transparent telemetry aims to address as many of those as possible. The bullet list in the GitHub discussion (also at the end of the blog post) enumerates the most important ones. > Most people leave defaults alone or make intuitive guesses about what they want. That's totally reasonable: no one wants to spend half an hour learning the details of each specific setting. But my goal for the system is that if I did spend half an hour explaining how the system worked, then the vast majority of users would agree with the default and see no reason to opt out. Of course, some people will always opt out on general principle, and perhaps there are others who would opt in to some systems but not this one. For those people, my goal is simply to make the opt-out as easy and effective as possible. That's why opting out is just an environment variable (GOTELEMETRY=off) or a single command (go env -w GOTELEMETRY=off), and there's a quiet period of at least a week after installation to give plenty of opportunity to opt out before there's any chance of data being sent. > I expect that this will not change your mind, and that you and a few others will still believe the telemetry should be opt-out. I accept that: I don't expect to convince everyone about this point. But I hope this helps explain how I am thinking about the decision.
gavinhoward 4y ago
@rsc, if you ever see this, your proposal here means that I will never use any software written in Go ever again, if at all possible. What others have said in this thread about telemetry becoming an "accelerant" will happen. Abuse will happen. Data will be put up for sale. IP's will be logged because users can't verify that they're not. The only thing users can verify is what is sent and to whom. And only if they run packet inspection. Most users don't. (Edit: I just realized that users may not even be able to tell who data is sent to because of proxies or the original collector selling the data.) I have no reason to believe your personal motives are anything but pure; however, this capability will not just be in your hands. It will be in the hands of anyone with less-than-pure motives. I applaud your efforts to make telemetry more transparent, but they are destined to fail. When it comes to figuring out how users use software, the only thing to do legwork. Ask your users. Watch them if they'll let you do user studies. Pay non-users to use the software for a user study and put them through all situations, including rare ones. This is the same thing we programmers tell the police to do when the police whine about end-to-end encryption: do old-fashioned legwork. Why should we, as programmers, demand that of police when we give ourselves tools to violate the privacy of users in the exact same way that police want? Yes, that's right, the exact same way. Telemetry is a backdoor on a private conversation between a user and a machine. Just do the work. I'm pretty sure Google has the money to do so. You may respond that this is for Open Source developers to get data on their users. Well, if those developers are hobbyists, they don't have time to crunch data, and they're probably scratching an itch. If they are not hobbyists, they are paid and should do the legwork. There is no excuse for telemetry. Just do the work.
- deleted 4y ago
  [deleted]
- icholy 4y ago
  What world do you live in?
  - Thaxll 4y ago
    All browser have telemetry in them, but he probably used netcat to post that message.
photochemsyn 4y ago
This is perhaps unintentionally amusing: > To be clear, I am only suggesting that the instrumentation be added to the Go command-line tools written and distributed by the Go team, such as the go command, the Go compiler, gopls, and govulncheck. I am not suggesting that instrumentation be added by the Go compiler to all Go programs in the world: that’s clearly inappropriate." Well that dispels any lingering thoughts I might have had about ever using golang for anything (not many to be sure). Someone feels the need to assure everyone that they won't be stuffing telemetry code into every binary their compiler produces? Google just wants all the data about everyone everywhere all the time... https://www.komando.com/security-privacy/ways-google-invades-your-privacy/804545/ https://www.komando.com/security-privacy/ways-google-invades...
creepycrawler 4y ago
If they add "telemetry" my response would not be to set an environment variable, but to uninstall golang. I used it a few years ago, both personally and in a work setting, but I'll do so no more in the future. Just my opinion.
xavdid 4y ago
There's a lot of strong reactions here, which I don't think are generally unfounded. Telemetry has certainly been misused and will continue to be, but it can also be an invaluable tool for product development. For example, we had a CLI with many commands and flags, some of which were costly to maintain. By adding analytics, we were able to see that literally no one was using certain commands, and we could safely remove them without messing up workflows. On each CLI invocation, we collected: - hash of user ID - which command is run - which flags were included - operating system (not version information, just mac/pc/linux) This data wasn't used for marketing, had no identifiable information, and was diasablable (but opt-out). You could also log exactly what was sent to the server, so you could see. We could have collected some of this via occasional surveys, but the data would have been less useful and less accurate. I didn't look into the details of what Go is proposing to collect, but treating all telemetry of any kind as a boogeyman isn't productive; just have to do it the right way.
- mort96 4y ago
  > Telemetry has certainly been misused and will continue to be, but it can also be an invaluable tool for product development. This perfectly exemplifies the whole problem. Google views Go as a product which Google is responsible for. Go is a programming language. Go is critical infrastructure. Go is not a product. Viewing it as such is a fundamental misunderstanding of what Go is to its users. This will practically implement my choice of programming language going forward. My question will no longer be, "is Go the right choice for this?", but rather, "is a Google product the right choice for this?". The answer is often yes to the former question, and no to the latter question.
  - xavdid 4y ago
    I guess that begs the question - if Go was an independent project (totally unattached from google) and had the exact same telemetry plans, how would you feel?
    - mort96 4y ago
      Better, because most independent projects aren't the world's biggest advertising company whose business model entirely revolves around invading people's privacy as much as possible. But that doesn't mean it would be good. The fundamental misunderstanding of Go's place as infrastructure rather than a product remains. And even though I acknowledge that spying on your users provides useful insight into their behavior, that doesn't mean spying on users is good.
2h 4y ago
> That's why opting out is just an environment variable (GOTELEMETRY=off) or a single command (go env -w GOTELEMETRY=off)
godmode2019 4y ago
My new TV wouldn't work unless I agreed to it recoding and uploading those recordings to it's servers which may be temporary stored while they are transcribing the audio to text for more permanent storage. My TV is forcing me into a employment agreement where I generated data to train their models or otherwise 'improve service'. Data is so valuable companies are risking a huge PR backlash. Data collection is the business model and I assume the same ethos will make its away into open source.
devdiary 4y ago
While this articles point out all the right explanation on why telemetry is needed and how it can be made little more transparent by Go toolchain acting as intermediary and publishing the telemetry data publicly, it fails to point out the disadvantages/risks of such system. At the core, the issue is about trust and the user not having any incentive.
chaps 4y ago
The Go team at Google would run a collection server. Each week, with 10% probability (averaging ~5 times per year) the user’s Go installation would download a “collection configuration” to find out which counter values are of interest to the server and at what sample rate. If there's interest to use config files to determine how telemetry is done, why can't similar be done about turning telemetry off? I don't want to deal with environment variables (for a gazillion reasons) and would prefer to just use a config file. Especially when it comes to sending arbitrary information from my system to another arbitrary host. It's so strange to me that the configuration of telemetry has been escalated to uses-configs status while opting out hasn't. Really feels like opting out is an after thought.
- rsc 4y ago
  As designed, the system allows an opt-out _either_ by setting GOTELEMETRY=off in your environment or by running 'go env -w GOTELEMETRY=off' which writes a config file. (Specifically the one reported by 'go env GOENV'.) If you prefer to edit the config file directly, you are of course welcome to do that.
  - chaps 4y ago
    That's fair, but doesn't take away from the deeply, deeply exhausting need to do this sort of niche configuration thing for everything out there, let alone even recognizing that it's needed. Never mind the unduly need to read changelogs and blog posts to make sure things haven't changed since I last used things. Have you considered using a more-generic opt-out environment variable that's not go specific? For example "USER_PREF_NO_TELEMETRY=true" would have the same effect, or would ensure that GOTELEMETRY=off is set. I have no idea if anything like that exists right now in other projects, but if it's not, then go is large enough and embedded across enough systems that it could be a good place to start.
  - simoncion 4y ago
    Sure, and have to remember to do that every single time you mint a new Docker image for use in CI, every single time you spin up a new Cloud workstation, every single time you get a new physical workstation, every single time you have to run a job in a brand new environment, every single time you... etc, etc, etc. Thing that work inside the Googleplex often don't work for the rest of us. Not every company is a multi-billion-dollar behemoth that can afford to spend hundreds of millions of dollars annually to work exclusively on internal developer-tooling.
fomine3 4y ago
This is really well considered way to telemetry. I wish all telemetries are like this.
lispegistus 4y ago
Setting aside the question if on by default telemetry is unethical in general, I personally think it is, my point in this comment is that in the context of open source it is impossible for it to be because: The whole point of open source is the security of the rights and freedoms of the users, and in case of a conflict with the convenience of the developers, the user rights take priority EVERY TIME. If you're not ok with this, you should not write open source software. If nobody opts in to your telemetry scheme if it were the default to choose, too bad, you're just gonna have to live with it and respect user choice no matter how inconvenient or how much better the alternative would be for everyone. If you fail to grasp this very basic thing you will be better served working on proprietary products instead. OSS is not a product you own, it's a shared resource you are in charge of stewarding and the ethical burden is much higher because of that. I checked, Go uses a permissive license, Google is more than welcome to run a proprietary fork with telemetry built in. Keep that out of open source.
deafpolygon 4y ago
This kind of push is only going to make people want to disable telemetry even more. Privacy is sacrosanct and should be accepted as the norm, not something we need to opt-in. Go already has some form of telemetry built-in (by way of a google proxy, I suppose) and adding an official one that is opt-out is just going to make me refuse to ever work with it. Telemetry should always be opt-in, and only opt-in. We have so much issues with telemetry, privacy, and such because the big players and corporations insists opt-out is better (maybe you get more data, but you violate end-users trust as well). Is that really worth it? There is blow-back and distrust in the industry as a result and it's only going to get worse the more you try to push for opt-out telemetry (or just assuming telemetry should be the default).
- rogpeppe1 4y ago
  Have you read the articles? How is this in any way violating privacy?
  - deafpolygon 4y ago
    Because it's impossible to get telemetry from any source without violating some aspect of the users' privacy.
    - rogpeppe1 4y ago
      So you see this as just the same, from a privacy perspective, as the way that the Go tool already dials out to the Go proxy by default? That is, if you're OK with that (I'd assume not, but it is at least existing functionality), you'd see the telemetry proposal as similar? In other words, I think you object to the Go tool already, so this is really no different?
account42 4y ago
Sometimes the post sorting algorithm produces interesting results 53. Transparent telemetry for open-source projects (swtch.com) 224 points by trulyrandom 1 day ago | flag | hide | 265 comments 54. Windows 11: a spyware machine out of users' control (techspot.com) 419 points by jlpcsl 19 hours ago | flag | hide | 292 comments

248 points by trulyrandom 4y ago

And there it is. The real intentions of Google and the the Go Programming Language. Google really can’t help themselves, to stick telemetry in anything.

champtar 4y ago

This will collect 0 telemetry from CI builds, so some data will need to be taken with a big grain of salt. I don't have data to prove it, but I would bet most cross-compile happen in CI and not on the dev laptop.

ysopex 4y ago

The only way I'd consider this, is if the telemetry my app generated is available to me or can be rerouted to another target. If so, please add this ASAP. Otherwise, I'll stick with my own observability stack.

Aaronontheweb 4y ago

How transparent is Scarf's product adoption metrics for OSS projects? https://about.scarf.sh/ https://about.scarf.sh/ I follow them on Twitter but haven't looked much into it other than reading their documentation, which makes me think that most of their telemetry is done at the point of the package distribution system: https://about.scarf.sh/package-sdks https://about.scarf.sh/package-sdks

userbinator 4y ago

Nope, nope, and more nope. You're not moving the Overton Window any more on me. In fact it seems there's a clear correlation between the quality of software and how much spyware there is embedded in it. It's often merely another way to justify unpopular changes with "but the data says so". IMHO if you want to collect any information, it should never be anything but opt-in, a conscious decision.

geodel 4y ago
Well, its not a humble opinion but a very strong one which is fine since you want certain thing in certain way and nothing else will do.
mananaysiempre 4y ago
> IMHO if you want to collect any information, it should never be anything but opt-in, a conscious decision. Serious (general) question: How do you do that given a non-technical user population? Debian’s opt-in popcon kind of manages to get a little bit of data from a fairly technical one, but nowhere near enough to estimate a low usage frequency, and it’s the only opt-in program I’m aware of that gets anything usable at all. Given that I’m unwilling to implement an opt-out system, I don’t really see a workable approach here at all.
- JohnFen 4y ago
  What I hear you saying here is that people don't do what you want if you give them the choice, so you lean towards not giving them the choice rather than respecting their wishes. Is my interpretation correct?
  - mananaysiempre 4y ago
    >> I’m unwilling to implement an opt-out system > [Y]ou lean towards not giving them the choice rather than respecting their wishes. > Is my interpretation correct? I don’t think it is, no :) Rather, I’m not sure how to sell, to put it crassly, users on a choice when properly investigating or even being confronted with that choice would delay them seeing the dancing bunnies[1], but that would also, if I have any say about it, improve the bunnies in the future. Does that mean there’s a shade of “I know better” in my problem statement? Of course it does, if I didn’t know better than the average user I’d have no business designing such choices. I don’t think there’s anything wrong about that, better than the average at an activity few practice is not a terribly high bar. Not giving the users a choice or manipulating them into making the one I think is right would absolutely be wrong, though. Basically, how do I make the user think, how do I give them the appropriate data to do so, and how do I deal with the obvious contradiction of that goal with principles of good design[2]? The potential benefits to the software and (thereby) the users are too much to give up without even asking those questions. (See nearby comment for extended discussion.) [1] https://blog.codinghorror.com/the-dancing-bunnies-problem/ https://blog.codinghorror.com/the-dancing-bunnies-problem/ [2] https://sensible.com/dont-make-me-think/ https://sensible.com/dont-make-me-think/

kardianos 4y ago

This is well done. It only exposes counters, and rather then pushing data up, the telemetry server must know the names of what it can ask for. No wildcards.

deleted 4y ago
[deleted]

deleted 4y ago

[deleted]

autoexec 4y ago

> Although the report would not include any identifiers, the TCP connection uploading the report would expose the system’s public IP address to the server if a proxy is not being used. This IP address would not be associated with the uploaded reports in any way. Any fully transparent data collection is going to have to include IP addresses and timestamps. Even if the IP isn't being used for debugging, the software still phones home and the IP is still being collected and logged when it otherwise wouldn't be. Either when uploading the report or when downloading the “collection configuration”. Honestly, assuming full transparency, I'm not opposed to the concept. I question how much telemetry is actually necessary, but I'm certain there will be times when it's nice to have. It'd also be interesting to see how it would go when for once people can see exactly what is collected, when, and from where. I'm not sure that Google is the best place to showcase such a concept though. I'm sure there are a lot of people who have no problem with handing more data over to Google, but Google has abused the public's good will for the sake of data collection many times, and it's sure to put off some of the people who aren't already completely disgusted by the idea of their favorite open source projects collecting telemetry.

mananaysiempre 4y ago
> Any fully transparent data collection is going to have to include IP addresses and timestamps. Even if the IP isn't being used for debugging, the software still phones home and the IP is still being collected and logged when it otherwise wouldn't be. Either when uploading the report or when downloading the “collection configuration”. How do you verifiably not collect users’ IP addresses when receiving data from them? The verifiable part is the problem, of course you can (and should) just not log the addresses, but then the users can only trust you (and hope you or your uplink haven’t received any legal orders to the contrary). The only approach I can think of would be a Tor hidden service, but while it would technically work, as far as not exposing your users to scrutiny it actually sounds worse.
- rsc 4y ago
  The only option is to have a proxy sit in the middle between the uploader and the server. You mentioned Tor but it doesn't have to be Tor, just some proxy most users would trust not to collude with the server and that doesn't itself derive benefit from seeing the IP addresses. If there were a different entity that could be relied upon to run servers doing this and were highly trusted by users, I'd be interested to use it. Failing that, the usual answer for an enterprise or company is to run their own HTTP proxy. The design explicitly supports that.

4ad 4y ago

This is just part 1, but all articles in the series have been published: https://research.swtch.com/telemetry https://research.swtch.com/telemetry

deleted 4y ago

[deleted]

schmichael 4y ago

> the vast majority of projects, even large ones that would benefit, stay away from telemetry. Nomad is one of these projects. We support a dizzying array of platforms (32bit Intel Linux?!). We have no idea how popular our Consul service mesh integration is. Are bug reports a sign of use or just failed experiments? Is anyone running on macOS in production or just ephemeral dev agents? Surveys about this are just asking humans to do something computers can do better. Obviously privacy and consent are paramount concerns, but not only are they solvable, in open source they’re fully auditable (and a fork could fairly easily maintain a patch that removes it outright). I think open source largely rejecting telemetry puts it at a huge disadvantage to proprietary and SaaS software where it is the norm. I’m very excited to see someone as thoughtful and well reasoned as Russ Cox to be trying to move the status quo forward.

ergonaught 4y ago
Everything involves tradeoffs. The times "we" (previous companies) tried to implement telemetry in open source non-SaaS products (as distinct from "projects"), we either got huge blowback or users/customers simply blocked it at the firewall (and security teams at major enterprises were unwilling to open holes anyway). The only workable solution I found was integrating this in a value-add way, so that something in the service/experience/etc was better for the user/customer as a result of enabling telemetry, without the dark pattern of making things intentionally awful/worse without it. We simply never got enough data to matter otherwise. But, again, that was products and not projects.
- GordonS 4y ago
  > The only workable solution I found was integrating this in a value-add way, so that something in the service/experience/etc was better for the user/customer as a result of enabling telemetry, without the dark pattern of making things intentionally awful/worse without it. This sounds like a great concept, but I'm struggling to come up with concrete examples - how did you approach it?
smoldesu 4y ago
On the contrary, I'd argue that the tracing visibility you're looking at isn't inherently a software trait at all. It's a deployment feature, which is something you address at-cost when building a product, but almost never when building FOSS software. It's not that people in FOSS don't see that upsides to it, it's that those upsides are insignificant relative to the cost of sustained market research. It's easier to just... make stuff, and have companies plaster over the gaps when their interests align. Look at GNOME, which recently pushed for it's users to contribute telemetry: https://linuxiac.com/gnome-survey-results/ https://linuxiac.com/gnome-survey-results/ Nothing wrong with what they've done here, but we already had most of these metrics. Nothing was really learned, and it took Red Hat and a few thousand users to get here. For smaller-scale projects, imagine how much smaller the returns would be.

mordae 4y ago

I dunno. It sure makes sense to me to collect telemetry from free software installations, but I feel that having every platform or even piece of software to do it on its own with opt-out will inevitably lead to people being overwhelmed and angry. I would, personally, prefer a single non-profit service that would list publicly what is being collected and publish the results as open data for anyone to use. Applications (at least on Linux) would not submit their reports directly, but would use a local relay service that could be turned off completely or that could filter what reports to send to the server and what to /dev/null. Distributions and other software stores would then make it mandatory for software to use this relay and either patch out any other telemetry from their packages or straight out forbid those that would not comply.

gen220 4y ago
I think the issue of telemetry is fundamentally a human issue of incentives and trust. The system you describe is wise because it recognizes this and attempts to address it. The difficulty with telemetry is that even if we design the perfect, privacy-preserving system to begin with, once the pattern of having a network port open is established, there's nothing to prevent us (humans) from changing our policies about what we're allowed to push/pull over that port. In real-world analogues for these kinds of thorny policy problems, we have centralized arbiters to solve these problems. That might be a fruitful course of research for people interested in this problem to explore. Unfortunately, even though this problem has software as its medium, it is a problem that cannot be solved by clever software alone, despite any appearances to the contrary.
- JohnFen 4y ago
  > The difficulty with telemetry is that even if we design the perfect, privacy-preserving system The other difficult is what you mentioned: trust. Even if a piece of software really does telemetry in a perfect, privacy-preserving way -- as a user, I have to take the developer's word for that in the end. That's a hard hurdle to pass, because that trust has been violated so much in the past that nobody gets the benefit of the doubt anymore. > Unfortunately, even though this problem has software as its medium, it is a problem that cannot be solved by clever software alone I agree entirely. At the heart of it, this is not a technological problem. It's a human one.

bioemerl 4y ago

Honestly, this may be unpopular with hacker news, but just add your own telemetry. If people don't like it they can turn it off, and telemetry is essential for a good product. Do let people turn it off though please.

groestl 4y ago
> If people don't like it they can turn it off If I, perchance, encounter software I use phoning home without my explicit permission it's done on my systems. Period.
- bioemerl 4y ago
  That is fine, but in this case telemetry trades you (and other more hardline users) as a user for all the extra users you gain from instant crash reports, quick feedback, and generally better productivity. I would never personally make that trade-off, and would always put (disableable) telemetry in my projects.
- Spivak 4y ago
  Well according to our telemetry 0% of users turn it off so it seems pretty popular. But more realistically what you gain in privacy you give up in having your voice heard by the devs. The decisions about the future of the product/project will be driven by the data, specifically the data from the kind of people who leave telemetry on.
  - JohnFen 4y ago
    If 0% of your users disable it, that kind of screams there's something wrong with your opt-out mechanism. Is it broken? Hidden? Difficult to do? I mean, with any group of people, there will always be a percentage that will disable it. If the telemetry is popular, that percentage might be very small, but it would be non-zero.
    - jwilk 4y ago
      I think you missed the joke.

taveras 4y ago

I wish there was a standard way of disabling telemetry across software dependencies. While I leave it turned on for personal projects, several projects at work require disabling it. I have spent hours auditing through transitive dependencies to turn it off. It should not be this painful.

slimsag 4y ago
No reason they couldn't all aim to respect a `TELEMETRY=false` env var, akin to the web's 'do not track' request.
- ComodoHacker 4y ago
  Does anyone really respect DNT though?
userbinator 4y ago
A firewall is probably your best bet. Don't allow network traffic originating from anything other than a short whitelist.

hommelix 4y ago

Telemetry in open source exists for a long time. Debian has the popcon package that can be installed and reports weekly usage of the software packages. The telemetry data are published in the open. The Debian popcon FAQ could be used as guideline for other telemetry needs. https://popcon.debian.org/ https://popcon.debian.org/

ptx 4y ago
It does sound quite similar. But in addition to the crucial difference of opt-in vs. opt-out there's also an interesting contrast in how it's framed. Debian talks about what you, the user, can do: help out, participate and vote. If you choose to do so. The Go team talks about what the developers and their software will do to the user's machine, but the user is completely passive in their description. This is also reflected in the term "telemetry" itself: the software is not a tool in the user's hands but rather a remote-controlled probe in the user's habitat that pokes at the user to elicit interesting responses.
agwa 4y ago
popcon should not be used as an example of how to do telemetry, as it is far worse for privacy than the Go proposal: 1. Sends names of private packages to the server, and publishes them. 2. Sends a unique identifier (a UUID stored in /etc/popularity-contest.conf) to the server, which is stored. 3. Doesn't use sampling, so if you use popcon you will be submitting a report once a week (Go's telemetry would average just one report a year). 4. Submits over plaintext protocols by default. popcon may be opt-in (in the sense that the prompt during installation has "No" selected by default) but the prompt doesn't disclose the large privacy risks. People are not appreciating the thought that has gone into the Go proposal to minimize the collection of private data, either intentionally or by accident, such as the client-enforced requirement that the the names of counters be published in a tamper-proof log so anyone can verify that, for example, no private package names are being disclosed. Everyone is focusing on opt-in vs out-out, but to me these other details are far more important.
msla 4y ago
> Debian has the popcon package that can be installed and reports weekly usage of the software packages. Right. That's fully opt-in, to the point the package isn't even installed by default, which is the only moral way to do this.

slimsag 4y ago

I've been a pretty strong advocate of the idea that analytics should always be minimal, 100% anonymous, aggregated, and open to the public - otherwise it’s spying. This is how we do analytics on our websites today[0][1], and how we plan to do it in games we release in the future. Maybe one day I will start a dedicated FOSS service that people can use for exactly this with some trusted reputation/transparency/auditability to it. I think what Russ has described here is decent and well-reasoned. I also think that Go being a product (it is, whether you like that word or not) makes it more fair to desire analytics of this form. I think it being opt-out is reasonable (after all, if it is not, they will make decisions using data that does not come from the vast majority of users, may as well not have analytics at all then.) But I am afraid of this becoming pervasive not just in products (like CLI tools), but also in libraries, imagine every Go/npm package you use wants to ping the network because the authors want to know 'is this popular? can we deprecate XYZ method?' etc. If transparent telemetry in the form Russ and I have been viewing it becomes a more common thing, it won't be a surprise if more library authors begin to try to adopt something like this and it becomes a pervasive problem IMHO. [0] https://hexops.com/privacy https://hexops.com/privacy [1] https://machengine.org https://machengine.org

rsc 4y ago
I am concerned about run-time telemetry in libraries as well. It might make sense for language ecosystems to offer more data about library usage gathered at build time eventually, as a different system than the one I'm posting about today. I think when you get to that level of detail you probably need to start thinking hard about differential privacy and probably cryptographic solutions like ESA or Prio. I don't think we know enough to design the library solution yet.
- JohnFen 4y ago
  Telemetry embedded in libraries is simply abusive, in my opinion. At the very least, the decision about whether or not to include telemetry should be made by the application developers, not the toolmakers.
  - 4y ago

infogulch 4y ago

This is a good plan, very simple and clear, and I like the list of system properties at the end. The solution is pretty tailored for the Go toolchain, which is a good strategy that has worked for them in the past. A more general purpose metrics tool I'm watching closely is Divvi Up https://divviup.org/ https://divviup.org/, a research project by ISRG, the same org that runs LetsEncrypt. The basic idea is to divide up each metric into two parts and publish each part to separate collection servers (one run by you and the other by divviup). Then the servers separately aggregate their half and combine the results, the idea being that each half is useless on it's own but when combined it's still useful. I wouldn't suggest it for this application, but for the majority of typical apps it would be a vast improvement to privacy compared to the status quo.

teraflop 4y ago

I am all for transparency and limited intrusiveness of telemetry. But in practical terms, the problem with this approach -- if I'm understanding it correctly -- is that it has no way to detect and reject outliers, and therefore the data can't be validated in any way. It only makes sense if all your clients are 100% trustworthy. Let's say you want to know whether to keep supporting ARMv5, and your data says 10% of users are using it. There's no way to tell whether that's accurate, or if you have 0.01% of die-hard users who modified their telemetry code to report 1000x as frequently as they're supposed to. Even if you suspect this is happening (and you might not), there's no way to identify the culprit and filter out their data without tracking personal identifiers such as IP addresses. So even if most of the time the telemetry data is valid, over time it will trend toward uselessness, because it can be endlessly second-guessed unless it confirms a decision you wanted to make anyway.

bee_rider 4y ago

I don’t really see why a classic community-driven open source project would care about what non-contributing users are doing with the software. In that case, helpful users come with built-in telemetry (pull requests). But I guess this could be helpful corporatized read-only repo projects, or other groups that aren’t sure if they are building a community or a customer base.

notpushkin 4y ago
Because pull requests aren't the only reason you might do a community-driven open source project. Perhaps you're just altruistic, or want to populatize some technology etc.

JohnFen 4y ago

> When you hear the word telemetry, if you’re like me, you may have a visceral negative reaction to a mental image of intrusive, detailed traces of your every keystroke and mouse click headed back to the developers of the software you’re using. But that's not my only objection to telemetry. Equally important to me is that so many bad decisions are justified based on telemetry. It's very easy to misunderstand the data, because telemetry leaves out so much, but developers often treat it as if it's giving a complete picture. As an example, I have seen developers drop really important functionality on the basis that it is rarely used. While that was true, it was also true that when those rare times happen, that functionality was absolutely critical to have.

userbinator 4y ago
Or they use the data as an accelerant: move rarely used features to places where they're even less discoverable, making them even less used, and then remove them altogether. The justification then becomes a self-fulfilling prophecy.

Arnavion 4y ago

I haven't worked with golang in some time. How do golang devs generally obtain the compiler? If you're getting it from distro repos, it should be straightforward to convince the distro package maintainer to disable the telemetry / patch it out. Or is it a nvm/pyenv/rustup situation where you prefer to use bespoke toolchain managers to download upstream's compilers?

aliyeysides 4y ago
It depends on which compiler you want to use, but precompiled binaries include the gc compiler by default. If you want to compile from source yourself, you can use gc or gccgo assuming you already have the go toolchain, otherwise you would need to bootstrap from an existing binary.
badrequest 4y ago
I mainly get it straight from golang.org, but this will be able to be disabled via environment variable just like the modules proxy stuff was. https://research.swtch.com/telemetry-design#opt-out https://research.swtch.com/telemetry-design#opt-out
- sidewndr46 4y ago
  Running the command in that just shows me a message of "go: unknown go command variable GOTELEMETRY"
  - 4ad 4y ago
    Because this proposal has not yet been implemented.
    - sidewndr46 4y ago
      So is there a way to disable this ahead of time? Or do I have to install the version with telemetry first?
      - 4y ago

zzzeek 4y ago

Was hoping for a big highly designed webpage with "enter your github URL here". but alas (it did say "transparent", like a service people opt into that could relate installations to github URLs)

caniszczyk 4y ago

We need solutions in this space for open source projects, I've been monitoring https://divviup.org https://divviup.org as an option too!

alyandon 4y ago

Oh Google - never stop being you. Not only is it going to be opt-out (because of course it would be coming from Google), I really like the whole "wait a week before sending telemetry" part that just coincidentally has the benefit of sneaking right past people that actively look for suspicious network activity when they've freshly installed something. Am I being uncharitable?

geodel 4y ago
[flagged]
Thaxll 4y ago
Very popular programming language and IDE have telemetry on by default, VSCode, C#, Java etc ... People act like they discover telemetry in 2023. I don't think it's a big deal, ultimately it's to improve Go and the proposal makes it very easy to disable it ( single env variable ).
- alyandon 4y ago
  I think they should all be opt-in as well. However, as a developer and pretend sysadmin, I am generally a nice guy about not turning off telemetry on software products with a user facing UI that I use frequently.
- xh-dude 4y ago
  Totally agree. Telemetry has been around and matured and benefits users. I’m not sure the benefits for Go would be as significant as other software but, really, why not?
  - masklinn 4y ago
    > Telemetry has been around and matured and benefits users. Does it? Telemetry mostly seems used to justify removing features I need on grounds that they’re little used. As an other user noted, if telemetry is your yardstick, the average backup software removed the “restore” feature because that’s barely ever used.

ergonaught 4y ago

On-by-default makes me question whether rsc's judgement has been compromised, which leads me to question continuing to use the language. A strange miss for him.

nicce 4y ago
Off-by-default in a scale likely means that there is no telemetry at all. I would not cancel a guy or programming language based on just suggesting that. He has given a lot of though for that if you read the blog posts.
- cwkoss 4y ago
  If a take a dollar from everyone but it's opt out, that's still theft. If I make it opt in, nobody is going to give me the dollar, but that doesn't make opt out morally justifiable.
  - nicce 4y ago
    You are comparing apples to oranges. Telemetry is a curse word these days, but you should still read his posts.
- ergonaught 4y ago
  I would expect nothing less of him than to give a topic a great deal of thought and devise a principled and rational solution. I am, however, reminded of a quote from Peter Drucker: “There is nothing quite so useless as doing with great efficiency something that should not be done at all.” I’m not picking nits regarding the overall proposal. I’m questioning the judgement that concluded/rationalized “on by default” is the right thing to do. Also not “cancelling” at the moment, either, but definitely reassessing my future language choices and taking a more critical appraisal of Go’s direction/choices. This isn’t really an isolated incident, and trust has accumulated some dents.

whoopsie 4y ago

Opaque telemetry can also be a barrier to adoption: my users’ IP addresses may legally be PII that I cannot disclose.

bluehazed 4y ago

This is really slimy, Google swung and missed and let Go of the bat here.

kabdib 4y ago

Oh, hell no.

_ph_ 4y ago

If there is any virtue to collecting telemetry, make it opt-in. Any developer convinced of this being useful will gladly enable it. But making it opt-out is just nefarious, because most users will not be aware of it.

Thaxll 4y ago
This is naive, no one ever turn telemetry on if it's turned off by default, that's the reason why it's on by default.
- ocdtrekkie 4y ago
  > no one ever turn telemetry on if it's turned off by default If nobody would voluntarily do it, why do you think it's okay to do it at all? By your very admission, nobody wants this. Because if they did, they'd turn it on!
- _ph_ 4y ago
  Still, opt-out is just inacceptable. At least with a mechanism which can easily fail, like setting an environment variable. This basically forces you to wrap the go tool in a script which ensures the environment variable to be set. As this seem to cache the results, another option is to fiddle with the cache to report bogus information.
  - Thaxll 4y ago
    You can set env variable for the go toolchain with a command such as: go env -w TELEMETRY=off which will be written to disk and use by the go cli.
    - _ph_ 4y ago
      Where will it be written to and how is it guaranteed to be picked up by any further invocations of the tools?
    - JetSpiegel 4y ago

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

bombela 4y ago

The information and rate of upload as described seem reasonable. Is the fear from most people that it will be a foot in the door? And a way for Google to collect extra overtime? Note: I think Go is a regressive technology. That would have been great in 1970s. Not today. But that's a different topic. My point is that I tend to be biased very negatively against Go. But here I don't see something wrong.

throw7777 4y ago
Interested as to why you believe go is regressive, could you expand on that?
- bombela 4y ago
  You can find plenty of well articulated rant online. The gist is Go ignores all research in programming languages. Is very hard to use properly. It is really hard to produce safe APIs. You are encouraged to pop threads (preemptive user threads) all over the place. And may the god of lost state saves you from the hell of data races. Also what is this slice/vector type thing? Don't forget to check `err`, nope not this one, the other one.
  - 0xjnml 4y ago
    > The gist is Go ignores all research in programming languages. This is an assumption and it's IMHO false. Go creators watched all the research, probably closely, lived with its results and were not happy with what they get. Their experience reminded them that less features in a PL make it arguably less expressive, sure, but in the same time it makes it faster to learn and master, easier to read and debug. Writing tools is simpler if a parser for your language can be hacked over the weekend from scratch, etc. At the end of the day its a matter of personal preferences, but it has, IMO, nothing to do with "ignoring all research in programming languages", quite the opposite.
    - bombela 4y ago
      You start with a dizizlgy large amount states (RAM, storage, network, threads...). The goal of programming languages is to give you a way to reduce the vast number of states as much as possible. Without removing the ones you need to do the work. Each layer of abstraction reducing what is possible, without impeding the work that ultimately needs to be performed. In my humble opinion, you should learn a bit more about, easy vs simple, complicated vs complex.

cube2222 4y ago

Probably related to[0]. To anybody complaining that this should be opt-in: opt-in telemetry doesn't work. The reason for this is that most people don't care, but they don't care either way. They don't disable it when prompted, nor would they enable it manually. The idea of telemetry is being able to prioritize the work that will be most widely useful. For this you need a good and balanced sample of your users. You don't really get any kind of sensible sample if you only do it opt-in. Additionally, this ship has long sailed, everybody does opt-out. What I do think however, is that it should very clearly notify the user of this, and give them an easy way to disable it. Like in OctoSQL[1] (disclaimer: which I'm the author of) which prompts you on first run and shows explicitly how to disable it. All things considered, this is an open source project, so you're free to maintain a fork without telemetry. The Go toolchain also uses the Google-hosted module proxy by default, which really is a bit like telemetry already. [0]: https://news.ycombinator.com/item?id=34707583 https://news.ycombinator.com/item?id=34707583 [1]: https://asciinema.org/a/eWQsyXQKi1fmithyTekAD5fWS https://asciinema.org/a/eWQsyXQKi1fmithyTekAD5fWS

marginalia_nu 4y ago
> The idea of telemetry is being able to prioritize the work that will be most widely useful. It does sort of hinge on the highly suspect assumption that usefulness is correlated with use. An obvious counter-example to this is something like a fire-extinguisher, which will in the ideal case just sit on a wall until it's use-by date passes and then it's discarded having never been used; or on the flip side, an incredibly byzantine workflow that could be reduced to something much simpler will appear important and useful. Even without these edge cases, interpreting statistics is really hard. Like people with PhDs who have studied these things for years still get them wrong all the time. What ends up happening more often than not is it's used as a tool to quiet the critics when pushing through unpopular changes.
- cube2222 4y ago
  All this boils down to "an unskilled engineer will misinterpret data even if they have it". I'll assume the Go team knows what they're doing, based on their track record so far. There's a lot of very simple questions you can answer very reliably, too, like "what proportion of the users are still using a certain compatibility flag".

tgv 4y ago

They could allow public access to that data. That can help more people than just the Go team, and it would add transparency.

mseepgood 4y ago
That's literally the plan if you read it.

1vuio0pswjnm7 4y ago

"The system is on by default, but opting out is easy, effective, and persistent."

wrldos 4y ago

Imagine if GNU started adding telemetry to their compiler toolchain... If that sounds fucking stupid, which it does, then so does this.

r2vcap 4y ago

I have set DOTNET_CLI_TELEMETRY_OPTOUT=1 as an environment variable in my .profile file. What should I do for golang?

guessmyname 4y ago
> To opt out, users would set GOTELEMETRY=off in their environment or run a simple command like go env -w GOTELEMETRY=off; The first telemetry report is not sent until at least one week after installation, giving ample time to opt out. Opting out stops all collection and reporting: no “opt out” event is sent. It is simply impossible to see systems that install Go and then opt out in the next seven days. Source: https://research.swtch.com/telemetry-intro https://research.swtch.com/telemetry-intro

ddevault 4y ago

This is not okay. The only ethical way to do telemetry is opt-in. If not enough people are opting in, you need to incentivize them to -- most simply by just paying them for their data. After all, telemetry is "valuable", isn't it? But if you can't figure out how to convince people to opt-in, then tough luck, sucks to be you. Opt-in or GTFO, Google. I'll be patching this out of the Alpine package for Go the day it ships.

gavinhoward 4y ago
You and I may not agree on a lot, but I sure agree with you on this one.

xyzzy_plugh 4y ago

I'm usually against telemetry but not only is the approach here somewhat reasonable, I think I actually trust Google more than, say, homebrew to not do something egregious with the data. Google is at least as broadly compliant as one can be with various standards (of questionable value, natch) but is also on the hook socially and perhaps legally if they fuck this up.

bluehazed 4y ago
[flagged]

candiddevmike 4y ago

Is this even up for debate, or is this post more of a FYI?

wrldos 4y ago
No it's not up for debate at all. Much like when Microsoft did this with .Net core, the Github thread is clearly a misguided post by RSC expecting the community to conform or support it. They didn't so now it's a damage control exercise. It will happen. Any corporate controlled project on this scale is prone to this failure mode.

mftb 4y ago

I hope this proposal is defeated and they don't implement this. I don't buy the premise that the benefit is worth the price. I think CLI tools like the ones in the Go Toolchain and their usage patterns are fairly well understood by this point. I'm sick and tired of every piece of software I interact with phoning home. That said, as long as they give me reasonable means to configure the software the way I want, it's probably not a deal-breaker for me. In other words, I will just set the $ENV_VAR_WHATEVER to turn this off, and that's that.

blibble 4y ago

how long until ads?

silisili 4y ago

Very much against this. Sure, it sounds naive enough, and can give reasons why. But I have 3,436 items in /usr/bin. What if -every- one of these started doing their own telemetry, their own envvars, etc? If we have to deal with telemetry, then I'd instead hope that there can exist a single telemetry systemwide interface. Not sure how that would be designed or implemented, but would be better than everyone doing their own bespoke thing. Plus easier for me to disable them all in one go.

jodrellblank 4y ago
> "What if -every- one of these started doing their own telemetry, their own envvars, etc?" What bad thing are you suggesting would happen if they did? Your computer and internet connection can't handle four thousand strings or four thousand HTTP POSTS, or four MB more disk space of telemetry libraries? I bet it can. This isn't a technical problem, it's a control and consent problem.
- Macha 4y ago
  I wouldn't be so sure it's not also a technical problem, the limit of execve can be as low as 128kb which for 4,000 strings gives a maximum of 32 characters per NAME=VALUE environment value
  - jodrellblank 4y ago
    Free disk space can be as low as zero, but we don't blame the tool makers for adding an extra 100Kb or 20MB, we blame the computer owner for not having enough disk space to install the thing they chose. Wrapper scripts for every utility to do UTIL_TELEMETRY_OPT_OUT=1 util ... so they don't need to be set all at once.
    - JohnFen 4y ago
      > we don't blame the tool makers for adding an extra 100Kb or 20MB Honestly, I do. Code bloat is a real thing.

msla 4y ago

The only moral response is to send false data to the servers.

stefanos82 4y ago

From a legal point of view, how companies will react to this, be it default-on or default-off? Some companies are using it for internal use which I'm sure all of us know cases with a number NDA-ed projects from third-parties or outsourced companies that collaborate on the matter. So, who is going to sue whom here when the one party will disable OR has already disabled the telemetry and the other will have it on by default, for whatever reason?

omginternets 4y ago

I see a very frustrating pattern emerging in which $COMPANY asks its users if it can do something, the users say "no", and $COMPANY storms off under the guise that "the discussion is unproductive". I am left with the impression that the decision has already been made, and that we are witnessing a PR strategy to make Google appear reasonable. I think that Mr. Cox, with all the respect I hold for him, is playing the part of the "useful idiot" here.

deathanatos 4y ago

This week one of my tasks is to figure out how to neutralize some telemetry in one of our apps. We had no idea it was there, we do not want to be sending data. Last week, the parent company decided they didn't want to maintain the telemetry server any longer, and got rid of it. Now the tool has generated thousands of log messages that it can't phone home. And so it must be silenced, since it is cluttering up the logs, generating false alerts, etc. Please, no more.

JohnFen 4y ago
The existence of telemetry is the main reasons why I avoid using new software anymore. Really, opt-in, opt-out, it doesn't matter. I can't trust that any of those mechanisms actually work, that if I opt out, an update won't reenable it, or that the data collected is actually limited and anonymized.

philosopher1234 4y ago

There's a lot of confusion in these comments about opt-out vs opt-in. The debate isn't settled, but a lot of the issues raised here have been addressed. Reposting Russ' comment: >Longer answer about opt-out generally, copied from mail I sent to golang-dev. > I wrote a little about this at https://research.swtch.com/telemetry-design#opt-out https://research.swtch.com/telemetry-design#opt-out. Just to quote the beginning: “An explicit goal of this design is to build a system that is reasonable to have enabled by default, for two reasons. First, the vast majority of users do not change any default settings. In systems that have collection off by default, opt-in rates tend to be very low, skewing the results toward power users who understand the system well. Second, the existence of an opt-in checkbox is in my opinion too often used as justification for collecting far more data than is necessary. Aiming for an opt-out system with as few reasons as possible to opt out led to this minimal design instead. Also, because the design collects a fixed number of samples, more systems being opted in means collecting less from any given system, reducing the privacy impact to each individual system.” > To elaborate, one of the core things I believe about designing a system like Go is that it needs to ship with the right defaults, rather than require users to reconfigure the defaults to get best practices for using that system. For example, Go ships with use of the Go module mirror (proxy.golang.org) enabled by default, so that users get more reliable builds out of the box. Similarly, Go ships with the use of the checksum database also enabled by default, so that users get verified module downloads out of the box. We know that most users don't want to and probably won't spend time reconfiguring the system: they trust us to set it up right instead. Of course, that implies a responsibility to actually look out for users' best interests, and we take that very seriously. There are important privacy concerns about the module mirror and about the checksum database, despite their clear benefits, so we designed those systems to address as many of those concerns as possible. Among the decisions we made to improve privacy there: (1) GOPROXY can proxy both the module mirror and the checksum database, (2) we published a very clear privacy policy (proxy.golang.org/privacy), (3) we introduced the concept of a tiled transparency log to keep log fetches from exposing a potential tracking signal. > Moving back to telemetry, enabling telemetry does not confer the same kind of direct benefits to users as the module mirror and the checksum database do. Instead the direct benefits it confers fall on other users: (1) allowing your Go installation to participate in the system means other installations participate just a little bit less, thanks to sampling, and (2) allowing your system to send usage information strengthens the signal from others with similar usage. There is still an important indirect benefit: one system opted out won't have much of an impact, but 99% of systems opted out has a huge impact, and that leads to mistakes like the ones I mentioned in the first blog post, which do make Go worse for you. > Like with the module mirror and checksum database, there are good privacy concerns to telemetry despite the clear benefits, so the design of transparent telemetry aims to address as many of those as possible. The bullet list in the GitHub discussion (also at the end of the blog post) enumerates the most important ones. > Most people leave defaults alone or make intuitive guesses about what they want. That's totally reasonable: no one wants to spend half an hour learning the details of each specific setting. But my goal for the system is that if I did spend half an hour explaining how the system worked, then the vast majority of users would agree with the default and see no reason to opt out. Of course, some people will always opt out on general principle, and perhaps there are others who would opt in to some systems but not this one. For those people, my goal is simply to make the opt-out as easy and effective as possible. That's why opting out is just an environment variable (GOTELEMETRY=off) or a single command (go env -w GOTELEMETRY=off), and there's a quiet period of at least a week after installation to give plenty of opportunity to opt out before there's any chance of data being sent. > I expect that this will not change your mind, and that you and a few others will still believe the telemetry should be opt-out. I accept that: I don't expect to convince everyone about this point. But I hope this helps explain how I am thinking about the decision.

gavinhoward 4y ago

@rsc, if you ever see this, your proposal here means that I will never use any software written in Go ever again, if at all possible. What others have said in this thread about telemetry becoming an "accelerant" will happen. Abuse will happen. Data will be put up for sale. IP's will be logged because users can't verify that they're not. The only thing users can verify is what is sent and to whom. And only if they run packet inspection. Most users don't. (Edit: I just realized that users may not even be able to tell who data is sent to because of proxies or the original collector selling the data.) I have no reason to believe your personal motives are anything but pure; however, this capability will not just be in your hands. It will be in the hands of anyone with less-than-pure motives. I applaud your efforts to make telemetry more transparent, but they are destined to fail. When it comes to figuring out how users use software, the only thing to do legwork. Ask your users. Watch them if they'll let you do user studies. Pay non-users to use the software for a user study and put them through all situations, including rare ones. This is the same thing we programmers tell the police to do when the police whine about end-to-end encryption: do old-fashioned legwork. Why should we, as programmers, demand that of police when we give ourselves tools to violate the privacy of users in the exact same way that police want? Yes, that's right, the exact same way. Telemetry is a backdoor on a private conversation between a user and a machine. Just do the work. I'm pretty sure Google has the money to do so. You may respond that this is for Open Source developers to get data on their users. Well, if those developers are hobbyists, they don't have time to crunch data, and they're probably scratching an itch. If they are not hobbyists, they are paid and should do the legwork. There is no excuse for telemetry. Just do the work.

deleted 4y ago
[deleted]
icholy 4y ago
What world do you live in?
- Thaxll 4y ago
  All browser have telemetry in them, but he probably used netcat to post that message.

photochemsyn 4y ago

This is perhaps unintentionally amusing: > To be clear, I am only suggesting that the instrumentation be added to the Go command-line tools written and distributed by the Go team, such as the go command, the Go compiler, gopls, and govulncheck. I am not suggesting that instrumentation be added by the Go compiler to all Go programs in the world: that’s clearly inappropriate." Well that dispels any lingering thoughts I might have had about ever using golang for anything (not many to be sure). Someone feels the need to assure everyone that they won't be stuffing telemetry code into every binary their compiler produces? Google just wants all the data about everyone everywhere all the time... https://www.komando.com/security-privacy/ways-google-invades-your-privacy/804545/ https://www.komando.com/security-privacy/ways-google-invades...

creepycrawler 4y ago

If they add "telemetry" my response would not be to set an environment variable, but to uninstall golang. I used it a few years ago, both personally and in a work setting, but I'll do so no more in the future. Just my opinion.

xavdid 4y ago

There's a lot of strong reactions here, which I don't think are generally unfounded. Telemetry has certainly been misused and will continue to be, but it can also be an invaluable tool for product development. For example, we had a CLI with many commands and flags, some of which were costly to maintain. By adding analytics, we were able to see that literally no one was using certain commands, and we could safely remove them without messing up workflows. On each CLI invocation, we collected: - hash of user ID - which command is run - which flags were included - operating system (not version information, just mac/pc/linux) This data wasn't used for marketing, had no identifiable information, and was diasablable (but opt-out). You could also log exactly what was sent to the server, so you could see. We could have collected some of this via occasional surveys, but the data would have been less useful and less accurate. I didn't look into the details of what Go is proposing to collect, but treating all telemetry of any kind as a boogeyman isn't productive; just have to do it the right way.

mort96 4y ago
> Telemetry has certainly been misused and will continue to be, but it can also be an invaluable tool for product development. This perfectly exemplifies the whole problem. Google views Go as a product which Google is responsible for. Go is a programming language. Go is critical infrastructure. Go is not a product. Viewing it as such is a fundamental misunderstanding of what Go is to its users. This will practically implement my choice of programming language going forward. My question will no longer be, "is Go the right choice for this?", but rather, "is a Google product the right choice for this?". The answer is often yes to the former question, and no to the latter question.
- xavdid 4y ago
  I guess that begs the question - if Go was an independent project (totally unattached from google) and had the exact same telemetry plans, how would you feel?
  - mort96 4y ago
    Better, because most independent projects aren't the world's biggest advertising company whose business model entirely revolves around invading people's privacy as much as possible. But that doesn't mean it would be good. The fundamental misunderstanding of Go's place as infrastructure rather than a product remains. And even though I acknowledge that spying on your users provides useful insight into their behavior, that doesn't mean spying on users is good.

2h 4y ago

> That's why opting out is just an environment variable (GOTELEMETRY=off) or a single command (go env -w GOTELEMETRY=off)

godmode2019 4y ago

My new TV wouldn't work unless I agreed to it recoding and uploading those recordings to it's servers which may be temporary stored while they are transcribing the audio to text for more permanent storage. My TV is forcing me into a employment agreement where I generated data to train their models or otherwise 'improve service'. Data is so valuable companies are risking a huge PR backlash. Data collection is the business model and I assume the same ethos will make its away into open source.

devdiary 4y ago

While this articles point out all the right explanation on why telemetry is needed and how it can be made little more transparent by Go toolchain acting as intermediary and publishing the telemetry data publicly, it fails to point out the disadvantages/risks of such system. At the core, the issue is about trust and the user not having any incentive.

chaps 4y ago

The Go team at Google would run a collection server. Each week, with 10% probability (averaging ~5 times per year) the user’s Go installation would download a “collection configuration” to find out which counter values are of interest to the server and at what sample rate. If there's interest to use config files to determine how telemetry is done, why can't similar be done about turning telemetry off? I don't want to deal with environment variables (for a gazillion reasons) and would prefer to just use a config file. Especially when it comes to sending arbitrary information from my system to another arbitrary host. It's so strange to me that the configuration of telemetry has been escalated to uses-configs status while opting out hasn't. Really feels like opting out is an after thought.

rsc 4y ago
As designed, the system allows an opt-out _either_ by setting GOTELEMETRY=off in your environment or by running 'go env -w GOTELEMETRY=off' which writes a config file. (Specifically the one reported by 'go env GOENV'.) If you prefer to edit the config file directly, you are of course welcome to do that.
- chaps 4y ago
  That's fair, but doesn't take away from the deeply, deeply exhausting need to do this sort of niche configuration thing for everything out there, let alone even recognizing that it's needed. Never mind the unduly need to read changelogs and blog posts to make sure things haven't changed since I last used things. Have you considered using a more-generic opt-out environment variable that's not go specific? For example "USER_PREF_NO_TELEMETRY=true" would have the same effect, or would ensure that GOTELEMETRY=off is set. I have no idea if anything like that exists right now in other projects, but if it's not, then go is large enough and embedded across enough systems that it could be a good place to start.
- simoncion 4y ago
  Sure, and have to remember to do that every single time you mint a new Docker image for use in CI, every single time you spin up a new Cloud workstation, every single time you get a new physical workstation, every single time you have to run a job in a brand new environment, every single time you... etc, etc, etc. Thing that work inside the Googleplex often don't work for the rest of us. Not every company is a multi-billion-dollar behemoth that can afford to spend hundreds of millions of dollars annually to work exclusively on internal developer-tooling.

fomine3 4y ago

This is really well considered way to telemetry. I wish all telemetries are like this.

lispegistus 4y ago

Setting aside the question if on by default telemetry is unethical in general, I personally think it is, my point in this comment is that in the context of open source it is impossible for it to be because: The whole point of open source is the security of the rights and freedoms of the users, and in case of a conflict with the convenience of the developers, the user rights take priority EVERY TIME. If you're not ok with this, you should not write open source software. If nobody opts in to your telemetry scheme if it were the default to choose, too bad, you're just gonna have to live with it and respect user choice no matter how inconvenient or how much better the alternative would be for everyone. If you fail to grasp this very basic thing you will be better served working on proprietary products instead. OSS is not a product you own, it's a shared resource you are in charge of stewarding and the ethical burden is much higher because of that. I checked, Go uses a permissive license, Google is more than welcome to run a proprietary fork with telemetry built in. Keep that out of open source.

deafpolygon 4y ago

This kind of push is only going to make people want to disable telemetry even more. Privacy is sacrosanct and should be accepted as the norm, not something we need to opt-in. Go already has some form of telemetry built-in (by way of a google proxy, I suppose) and adding an official one that is opt-out is just going to make me refuse to ever work with it. Telemetry should always be opt-in, and only opt-in. We have so much issues with telemetry, privacy, and such because the big players and corporations insists opt-out is better (maybe you get more data, but you violate end-users trust as well). Is that really worth it? There is blow-back and distrust in the industry as a result and it's only going to get worse the more you try to push for opt-out telemetry (or just assuming telemetry should be the default).

rogpeppe1 4y ago
Have you read the articles? How is this in any way violating privacy?
- deafpolygon 4y ago
  Because it's impossible to get telemetry from any source without violating some aspect of the users' privacy.
  - rogpeppe1 4y ago
    So you see this as just the same, from a privacy perspective, as the way that the Go tool already dials out to the Go proxy by default? That is, if you're OK with that (I'd assume not, but it is at least existing functionality), you'd see the telemetry proposal as similar? In other words, I think you object to the Go tool already, so this is really no different?

account42 4y ago

Sometimes the post sorting algorithm produces interesting results 53. Transparent telemetry for open-source projects (swtch.com) 224 points by trulyrandom 1 day ago | flag | hide | 265 comments 54. Windows 11: a spyware machine out of users' control (techspot.com) 419 points by jlpcsl 19 hours ago | flag | hide | 292 comments