6 ms·

Show HN: Measure downloads and commercial adoption of any file you distribute

41 points by aviaviavi 5y ago

jdorfman 5y ago
Saw this on Twitter yesterday, and it looks interesting. With that said, the one concern my team has is around privacy. The blog post says: "All without ever having access to personally identifiable information or invading the privacy of your users." Can you elaborate on how you go about that?
- aviaviavi 5y ago
  In short, using Scarf does not provide personally identifiable information about who is downloading your artifacts because we don't have that data ourselves. The main way this is achieved is by purging any personally identifiable information from our system, mainly the IP address of a download request. Scarf uses the IP to look up metadata like company affiliation, cloud provider, course grained location, etc, to surface that to you. Once that metadata is looked up, the original IP address is discarded. All information stored long term is fully anonymized.
  - putnambr 5y ago
    This is impressive, but seems like a dark pattern to me a la tracking pixels in emails. An annoying use case I could see this used for is targeted spam. Say a company selling a software tool publishes a PDF of industry insights and then reaches out to everyone who's downloaded it. Or they publish an OCI image, and then try to sell everyone who uses it a support package.
    - aviaviavi 5y ago
      Well, Scarf offers free pixel tracking too so you definitely have the correct model for what we do, though sorry to hear you dislike the approach. Our goal is to help enable OSS developers to financially support their work. Do you think it's still wrong when it's OSS developers trying to sell their services or premium offerings to the companies that already rely on their work? If so - companies are tracking people all the time at a very granular, personally identifiable level. Why should we hold OSS developers to an even higher standard than what we tolerate from large companies?
      - esrh 5y ago
        > Why should we hold OSS developers to an even higher standard than what we tolerate from large companies? The problem here is that i DON'T tolerate this from large companies either. I find the pixel tracking thing outrageous and disable images by default in my email client to avoid it. I understand your argument, I just find it personally strongly disagreeable, and I'm willing to bet poster above did as well.
        aviaviavi 5y ago
        > The problem here is that i DON'T tolerate this from large companies either. I find the pixel tracking thing outrageous and disable images by default in my email client to avoid it. If you are already using OSS today and grabbing that software over the internet, you are tolerating it even if you claim otherwise. If you pull something down from GitHub, Microsoft has all the data that we're talking about here. > I understand your argument, I just find it personally strongly disagreeable Fair! And I understand yours too. I also think your argument is more idealistic than practical for the current state of the ecosystem, especially considering how many parties already have access to this web traffic data. Maintainers having this data too is a very benign additional party to have access to it. Furthermore, it's a concrete way we can all chip in to help OSS maintainers and make their jobs a little bit easier, short of reaching for your credit card (which we should all be doing too).
      - smoldesu 5y ago
        You're going to be walking a thin (and difficult) line if you're trying to find open source developers also interested in introducing involuntary tracking to their software. Open Source software, since it's inception, has been about creating respectful software for the commons. > Do you think it's still wrong when it's OSS developers trying to sell their services or premium offerings to the companies that already rely on their work? No, but I shouldn't have to worry about that as a user. The onus is on corporations to disclose the software that they use in accordance with their respective licenses, the regular user doesn't deserve to suffer for the incompetence of funded organizations. > Why should we hold OSS developers to an even higher standard than what we tolerate from large companies? You don't, they do. That's the point of open source licensing in the first place: defining what you're comfortable with other people using your software for. By choosing an Open Source license, you're assuming one of the most difficult and thankless positions in the world of software. That's how it's intended to be though, because that kind of transparency is imperative when we're distributing free software. You wouldn't poison the rations being donated to the homeless, so why are you comfortable poisoning the CDN of my download? This all seems pretty cut and dried to me. sigh Time to start dropping Scarf URLs in my hosts file...
  - yjftsjthsd-h 5y ago
    That means you can't distinguish between one user downloading many times and many users on one ISP, right?
    - bpodgursky 5y ago
      Do you mean IP, not ISP?
      - yjftsjthsd-h 5y ago
        No, I meant ISP, although IP could work as a special case - if this isn't recording the user's actual IP but just information derived from it (rough location, residential/commercial/datacenter, whatever), I would expect many addresses under that ISP to have the same recorded details. Granted, CGNAT with the same exact public IP would be even more like that, but if you don't record the actual IP then you probably can't deduplicate close "neighbors". (I should add that your sibling comment says they're using browser headers, which probably reduces this issue a lot)
        aviaviavi 5y ago
        Information derived from IP can be much more granular than just ISP level
    - aviaviavi 5y ago
      User agent and other headers can be used to provide more differentiation, but you're correct to point out that limitation (assuming you meant IP not ISP).
aviaviavi 5y ago
Hi HN, a comment to give a little more backstory here: At Scarf, we aim to give open source developers more visibility into how their software is being used. As people with experience distributing binaries and artifacts hosted on platforms like GitHub Releases and S3, a repeated struggle was not having any visibility into downloads. Which versions of the software were being downloaded the most? On which platforms? Where in the world? Which companies were downloading? This year we built Scarf Gateway, which acts as a redirect/analytics layer for any container registry. Supporting other kinds of artifacts was a natural extension, and arbitrary file downloads is perhaps the most general extension we could build! Curious to hear what people think.
- inetknght 5y ago
  > Curious to hear what people think. How would someone opt-out of being tracked that something's been downloaded?
  - aviaviavi 5y ago
    This still needs to be added to our docs. A `dnt=1` query param in a download URL is interpreted as an end-user opt-out. We plan to add more forms of opting out based on user feedback. We want to ensure it's low-friction to opt out of tracking.
    - milkytron 5y ago
      Why not check the Do Not Track or Global Privacy Control header? Having a query param seems like more friction. https://en.wikipedia.org/wiki/Do_Not_Track https://en.wikipedia.org/wiki/Do_Not_Track
      - aviaviavi 5y ago
        Great suggestion, very appreciated. Global privacy control wasn't on my radar but this looks like what we should do. DNT is considered deprecated, at least according to MDN docs.
- 5y ago
_query 5y ago
Highly recommend to try this out if you run an open source project and want to get some insights about usage. Avi gave me a demo last week as we’ve been looking on how to get better analytics for our open source framework IHP https://ihp.digitallyinduced.com/ https://ihp.digitallyinduced.com/ it’s quick to set up and they also provides tools for doing analytics for eg the documentation.
nixwatch510 5y ago
I wonder why GitHub shows number of visits / clones, but not release artifact downloads.
uzername 5y ago
I like this approach slightly better than what scarfjs was doing. I first ran into that with react query a few years ago. It was a chilling effect for me at least and I was glad it changed. The idea of this being an enhanced link shortener is a good metaphor. I think if were ever encountering an image or a download that's not going through a dockerhub or GH link, I might give pause and think about it a bit. I'm not defending enterprise usage of packages, especially somehow if it's core to the business, but I don't need anyone to come knocking either.
- aviaviavi 5y ago
  > I like this approach slightly better than what scarfjs was doing. I first ran into that with react query a few years ago. It was a chilling effect for me at least and I was glad it changed. Glad to hear you like this better, we do too. We built Scarf Gateway in a large part due to the response of the react-query community (and a handful of other projects) to scarf-js. Lots of discussion on GitHub and the Reactiflux discord provided good learnings for us: mainly that mechanisms that phone home, especially at unexpected times, were particularly unpopular. We also heard more acceptance of the idea that the registry/host platforms who already have this information could be sharing it with maintainers. We want to support maintainers with better data in a way that best suits the OSS community and respects privacy. And so we went back to the drawing board, and Scarf Gateway is the result! Still, I understand you may still have remaining hesitations here anyway with Scarf-powered download links. Are there any specific privacy concerns we can mitigate?

41 points by aviaviavi 5y ago

Saw this on Twitter yesterday, and it looks interesting. With that said, the one concern my team has is around privacy. The blog post says: "All without ever having access to personally identifiable information or invading the privacy of your users." Can you elaborate on how you go about that?

aviaviavi 5y ago
In short, using Scarf does not provide personally identifiable information about who is downloading your artifacts because we don't have that data ourselves. The main way this is achieved is by purging any personally identifiable information from our system, mainly the IP address of a download request. Scarf uses the IP to look up metadata like company affiliation, cloud provider, course grained location, etc, to surface that to you. Once that metadata is looked up, the original IP address is discarded. All information stored long term is fully anonymized.
- putnambr 5y ago
  This is impressive, but seems like a dark pattern to me a la tracking pixels in emails. An annoying use case I could see this used for is targeted spam. Say a company selling a software tool publishes a PDF of industry insights and then reaches out to everyone who's downloaded it. Or they publish an OCI image, and then try to sell everyone who uses it a support package.
  - aviaviavi 5y ago
    Well, Scarf offers free pixel tracking too so you definitely have the correct model for what we do, though sorry to hear you dislike the approach. Our goal is to help enable OSS developers to financially support their work. Do you think it's still wrong when it's OSS developers trying to sell their services or premium offerings to the companies that already rely on their work? If so - companies are tracking people all the time at a very granular, personally identifiable level. Why should we hold OSS developers to an even higher standard than what we tolerate from large companies?
    - esrh 5y ago
      > Why should we hold OSS developers to an even higher standard than what we tolerate from large companies? The problem here is that i DON'T tolerate this from large companies either. I find the pixel tracking thing outrageous and disable images by default in my email client to avoid it. I understand your argument, I just find it personally strongly disagreeable, and I'm willing to bet poster above did as well.
      - aviaviavi 5y ago
        > The problem here is that i DON'T tolerate this from large companies either. I find the pixel tracking thing outrageous and disable images by default in my email client to avoid it. If you are already using OSS today and grabbing that software over the internet, you are tolerating it even if you claim otherwise. If you pull something down from GitHub, Microsoft has all the data that we're talking about here. > I understand your argument, I just find it personally strongly disagreeable Fair! And I understand yours too. I also think your argument is more idealistic than practical for the current state of the ecosystem, especially considering how many parties already have access to this web traffic data. Maintainers having this data too is a very benign additional party to have access to it. Furthermore, it's a concrete way we can all chip in to help OSS maintainers and make their jobs a little bit easier, short of reaching for your credit card (which we should all be doing too).
    - smoldesu 5y ago
      You're going to be walking a thin (and difficult) line if you're trying to find open source developers also interested in introducing involuntary tracking to their software. Open Source software, since it's inception, has been about creating respectful software for the commons. > Do you think it's still wrong when it's OSS developers trying to sell their services or premium offerings to the companies that already rely on their work? No, but I shouldn't have to worry about that as a user. The onus is on corporations to disclose the software that they use in accordance with their respective licenses, the regular user doesn't deserve to suffer for the incompetence of funded organizations. > Why should we hold OSS developers to an even higher standard than what we tolerate from large companies? You don't, they do. That's the point of open source licensing in the first place: defining what you're comfortable with other people using your software for. By choosing an Open Source license, you're assuming one of the most difficult and thankless positions in the world of software. That's how it's intended to be though, because that kind of transparency is imperative when we're distributing free software. You wouldn't poison the rations being donated to the homeless, so why are you comfortable poisoning the CDN of my download? This all seems pretty cut and dried to me. sigh Time to start dropping Scarf URLs in my hosts file...
- yjftsjthsd-h 5y ago
  That means you can't distinguish between one user downloading many times and many users on one ISP, right?
  - bpodgursky 5y ago
    Do you mean IP, not ISP?
    - yjftsjthsd-h 5y ago
      No, I meant ISP, although IP could work as a special case - if this isn't recording the user's actual IP but just information derived from it (rough location, residential/commercial/datacenter, whatever), I would expect many addresses under that ISP to have the same recorded details. Granted, CGNAT with the same exact public IP would be even more like that, but if you don't record the actual IP then you probably can't deduplicate close "neighbors". (I should add that your sibling comment says they're using browser headers, which probably reduces this issue a lot)
      - aviaviavi 5y ago
        Information derived from IP can be much more granular than just ISP level
  - aviaviavi 5y ago
    User agent and other headers can be used to provide more differentiation, but you're correct to point out that limitation (assuming you meant IP not ISP).

aviaviavi 5y ago

Hi HN, a comment to give a little more backstory here: At Scarf, we aim to give open source developers more visibility into how their software is being used. As people with experience distributing binaries and artifacts hosted on platforms like GitHub Releases and S3, a repeated struggle was not having any visibility into downloads. Which versions of the software were being downloaded the most? On which platforms? Where in the world? Which companies were downloading? This year we built Scarf Gateway, which acts as a redirect/analytics layer for any container registry. Supporting other kinds of artifacts was a natural extension, and arbitrary file downloads is perhaps the most general extension we could build! Curious to hear what people think.

inetknght 5y ago
> Curious to hear what people think. How would someone opt-out of being tracked that something's been downloaded?
- aviaviavi 5y ago
  This still needs to be added to our docs. A `dnt=1` query param in a download URL is interpreted as an end-user opt-out. We plan to add more forms of opting out based on user feedback. We want to ensure it's low-friction to opt out of tracking.
  - milkytron 5y ago
    Why not check the Do Not Track or Global Privacy Control header? Having a query param seems like more friction. https://en.wikipedia.org/wiki/Do_Not_Track https://en.wikipedia.org/wiki/Do_Not_Track
    - aviaviavi 5y ago
      Great suggestion, very appreciated. Global privacy control wasn't on my radar but this looks like what we should do. DNT is considered deprecated, at least according to MDN docs.
5y ago

_query 5y ago

Highly recommend to try this out if you run an open source project and want to get some insights about usage. Avi gave me a demo last week as we’ve been looking on how to get better analytics for our open source framework IHP https://ihp.digitallyinduced.com/ https://ihp.digitallyinduced.com/ it’s quick to set up and they also provides tools for doing analytics for eg the documentation.

nixwatch510 5y ago

I wonder why GitHub shows number of visits / clones, but not release artifact downloads.

uzername 5y ago

I like this approach slightly better than what scarfjs was doing. I first ran into that with react query a few years ago. It was a chilling effect for me at least and I was glad it changed. The idea of this being an enhanced link shortener is a good metaphor. I think if were ever encountering an image or a download that's not going through a dockerhub or GH link, I might give pause and think about it a bit. I'm not defending enterprise usage of packages, especially somehow if it's core to the business, but I don't need anyone to come knocking either.

aviaviavi 5y ago
> I like this approach slightly better than what scarfjs was doing. I first ran into that with react query a few years ago. It was a chilling effect for me at least and I was glad it changed. Glad to hear you like this better, we do too. We built Scarf Gateway in a large part due to the response of the react-query community (and a handful of other projects) to scarf-js. Lots of discussion on GitHub and the Reactiflux discord provided good learnings for us: mainly that mechanisms that phone home, especially at unexpected times, were particularly unpopular. We also heard more acceptance of the idea that the registry/host platforms who already have this information could be sharing it with maintainers. We want to support maintainers with better data in a way that best suits the OSS community and respects privacy. And so we went back to the drawing board, and Scarf Gateway is the result! Still, I understand you may still have remaining hesitations here anyway with Scarf-powered download links. Are there any specific privacy concerns we can mitigate?