5 ms·

Is the telemetry available to users? I glady opted into Synchthing telemetry after seeing this page: https://data.syncthing.net/ https://data.syncthing.net/ Wh

by papaf 6y ago

Is the telemetry available to users? I glady opted into Synchthing telemetry after seeing this page: https://data.syncthing.net/ https://data.syncthing.net/ When the data is available to the community, just like the source code, its a much easier sell.

KenoFischer 6y ago
The plan is to make aggregate usage data available publicly and potentially share more detailed usage data with individual package authors. The exact format is TBD since it'll depend on the quality of the data that we get (this is not active yet, except on the preview build). The raw logs will be accessible to core developers with a reasonable need to access (e.g. they're working on the infrastructure or running the analytics), but will not be public.
- j88439h84 6y ago
  How about deleting the IP addresses within 48 hours like 1.1.1.1 and 8.8.8.8 do? https://developers.google.com/speed/public-dns/privacy https://developers.google.com/speed/public-dns/privacy
  - staticfloat 6y ago
    We do have a limited retention policy for the package server logs we keep (which include client IP addresses). It's not publicly stated anywhere right now, but one reason why we need to keep IP addresses is for abuse mitigation. We have been hit in the past by users that do things like download large (100MB+) files from our package cache servers multiple times a second for days on end. This is a particularly easy case to catch (since it easily pops to the top of any analysis you'd care to run, across any timespan) but there are more subtle forms that require a longer time window of analysis (e.g. users that download once per hour, all month) that would be lost in the noise without the ability to see what's going on. This comment is not meant to serve as an official policy, just pointing out one of the reasons why we can't delete IP addresses like 1.1.1.1 and 8.8.8.8 do; because the abuse vectors for a server that serves the community large resources is very different from that of a DNS server. Most of the "abuse" we see is not malicious in nature, but is instead users that have some kind of very poorly-configured autoinstaller on a cluster. In the case of a catastrophic issue like the one mentioned above, we null-routed the IP address, reached out to the abuse contact for that IP, and worked with the user to architect a better system. Everyone is happy now, and we can continue to provide a high quality service for the community without breaking the bank.
    - edw 6y ago
      How about hashing IPs? You could still see if someone were on your abuse list if abusers.contains(hashfn(req.addr)).
      - codedokode 6y ago
        Hash of IPv4 address can be easily reverted because there is a limited number of addresses.
      - KenoFischer 6y ago
        Doesn't help for two reasons 1) If the has has enough bits to be useful for blocking, it's trivial to reverse 2) Even if it did make the IPs anonymous, we want to be able to email the NOC at whoever is sending the abusive traffic, so they can go investigate
        j88439h84 6y ago
        > we want to be able to email the NOC at whoever is sending the abusive traffic, so they can go investigate If you block their traffic with HTTP 429 Too Many Requests, they can email you instead.
        KenoFischer 6y ago
        We prefer not to break researchers' workflow because the group next door misconfigured their server. Happens all the time. We only sinkhole IPs if the traffic is malicious or on track to exceed or budget.
  - KenoFischer 6y ago
    I don't work on this particular thing, so I can't say precisely what the planned retention period is. I suspect 48 hours is too short, since people do take weekends ;). It'll probably become clear with experience what retention periods work. DNS servers are in a very unique position of course since they essentially get your browsing history.
- ptx 6y ago
  Since the data is not being made public, presumably it is judged to be sensitive to some extent? So if follows then that users are right to be concerned and would have every reason to not opt in if they were presented with the choice.
  - KenoFischer 6y ago
    What's sensitive or not depends very much on what other information the entity doing the analysis has available. Of course raw log records are more sensitive than aggregate data. For example, if somebody is wiretapping your internet connection, then even if the connection is encrypted raw logs would let them draw conclusions from timing. To some extent you're trusting the Julia project (or at least the people who have access) to not clandestinely be in the wiretapping business, but then again you're already trusting it with arbitrary code execution on your machine, so if it were in that business, you'd have bigger problems ;). In any case that's why it's important to be transparent about what is sent, and for what purpose and who has access, so people can make informed decisions. Ironically, I think people are jumping on the authors of this particular piece of functionality precisely because they tried to be very transparent.
    - ptx 6y ago
      Yes, they are transparent about deciding not to offer the user the choice in a straight-forward upfront way (i.e. opt in) because "the vast majority of users will not opt-in". In other words, deciding that what the users want is not as important as the marketing stats. And, as you say, users trust the developers with access to their systems and data. Deciding unilaterally to sacrifice user privacy to benefit other interests might be seen as a breach of that trust.
      - orf 6y ago
        It’s actually more that “the vast majority of users do not care” and so go with the defaults. An opt-out is the way that makes sense.

by papaf 6y ago

KenoFischer 6y ago

The plan is to make aggregate usage data available publicly and potentially share more detailed usage data with individual package authors. The exact format is TBD since it'll depend on the quality of the data that we get (this is not active yet, except on the preview build). The raw logs will be accessible to core developers with a reasonable need to access (e.g. they're working on the infrastructure or running the analytics), but will not be public.

j88439h84 6y ago
How about deleting the IP addresses within 48 hours like 1.1.1.1 and 8.8.8.8 do? https://developers.google.com/speed/public-dns/privacy https://developers.google.com/speed/public-dns/privacy
- staticfloat 6y ago
  We do have a limited retention policy for the package server logs we keep (which include client IP addresses). It's not publicly stated anywhere right now, but one reason why we need to keep IP addresses is for abuse mitigation. We have been hit in the past by users that do things like download large (100MB+) files from our package cache servers multiple times a second for days on end. This is a particularly easy case to catch (since it easily pops to the top of any analysis you'd care to run, across any timespan) but there are more subtle forms that require a longer time window of analysis (e.g. users that download once per hour, all month) that would be lost in the noise without the ability to see what's going on. This comment is not meant to serve as an official policy, just pointing out one of the reasons why we can't delete IP addresses like 1.1.1.1 and 8.8.8.8 do; because the abuse vectors for a server that serves the community large resources is very different from that of a DNS server. Most of the "abuse" we see is not malicious in nature, but is instead users that have some kind of very poorly-configured autoinstaller on a cluster. In the case of a catastrophic issue like the one mentioned above, we null-routed the IP address, reached out to the abuse contact for that IP, and worked with the user to architect a better system. Everyone is happy now, and we can continue to provide a high quality service for the community without breaking the bank.
  - edw 6y ago
    How about hashing IPs? You could still see if someone were on your abuse list if abusers.contains(hashfn(req.addr)).
    - codedokode 6y ago
      Hash of IPv4 address can be easily reverted because there is a limited number of addresses.
    - KenoFischer 6y ago
      Doesn't help for two reasons 1) If the has has enough bits to be useful for blocking, it's trivial to reverse 2) Even if it did make the IPs anonymous, we want to be able to email the NOC at whoever is sending the abusive traffic, so they can go investigate
      - j88439h84 6y ago
        > we want to be able to email the NOC at whoever is sending the abusive traffic, so they can go investigate If you block their traffic with HTTP 429 Too Many Requests, they can email you instead.
        KenoFischer 6y ago
        We prefer not to break researchers' workflow because the group next door misconfigured their server. Happens all the time. We only sinkhole IPs if the traffic is malicious or on track to exceed or budget.
- KenoFischer 6y ago
  I don't work on this particular thing, so I can't say precisely what the planned retention period is. I suspect 48 hours is too short, since people do take weekends ;). It'll probably become clear with experience what retention periods work. DNS servers are in a very unique position of course since they essentially get your browsing history.
ptx 6y ago
Since the data is not being made public, presumably it is judged to be sensitive to some extent? So if follows then that users are right to be concerned and would have every reason to not opt in if they were presented with the choice.
- KenoFischer 6y ago
  What's sensitive or not depends very much on what other information the entity doing the analysis has available. Of course raw log records are more sensitive than aggregate data. For example, if somebody is wiretapping your internet connection, then even if the connection is encrypted raw logs would let them draw conclusions from timing. To some extent you're trusting the Julia project (or at least the people who have access) to not clandestinely be in the wiretapping business, but then again you're already trusting it with arbitrary code execution on your machine, so if it were in that business, you'd have bigger problems ;). In any case that's why it's important to be transparent about what is sent, and for what purpose and who has access, so people can make informed decisions. Ironically, I think people are jumping on the authors of this particular piece of functionality precisely because they tried to be very transparent.
  - ptx 6y ago
    Yes, they are transparent about deciding not to offer the user the choice in a straight-forward upfront way (i.e. opt in) because "the vast majority of users will not opt-in". In other words, deciding that what the users want is not as important as the marketing stats. And, as you say, users trust the developers with access to their systems and data. Deciding unilaterally to sacrifice user privacy to benefit other interests might be seen as a breach of that trust.
    - orf 6y ago
      It’s actually more that “the vast majority of users do not care” and so go with the defaults. An opt-out is the way that makes sense.