7 ms·

They log too much data about their users. So they should promptly update their policies to a) stop logging so much, b) delete all past logs, and c) sharply lim

They log too much data about their users. So they should promptly update their policies to a) stop logging so much, b) delete all past logs, and c) sharply limit the span of time until deletion of whatever logs they decide they really need to track for internal needs. They should avoid logging, and rapidly rotate logs, to thwart future subpoenas from the total surveillance state.

takeda 3y ago
For the kind of service they are providing I think the logging is appropriate. I mean if DOJ is interested in PyPI logs the only reason I could think of, is if it was used as a supply chain vector into breaking in into other organizations.
- alwayslikethis 3y ago
  or locating the people responsible for DRM breakers like youtube-dl? Keeping all the data makes you susceptible to subpoenas like this, which costs money to comply with. There is no reason to keep any data that isn't necessary to the service.
  - takeda 3y ago
    youtube-dl is an application, there's no benefit for the author to publish it there. I also checked PyPI repo and looking at it I don't think it is published, there are few that look like might be it, but looking at them, they feel like something that would install something nasty on your computer. As a somebody who relies on PyPI for development, I want them to store all kinds of data that would discourage PyPI to be used for any shady purpose. Also perhaps there's a better example than youtube-dl, as they really don't do anything illegal. Yes their github repo was taken by DMCA request but then it was reinstated back. The GitHub spin it that they are standing up for developers, but the DMCA was just frivolous. I guess, still good for them, because they could just take it down and do nothing.
manicennui 3y ago
[flagged]
- deleted 3y ago
  [deleted]
- brigandish 3y ago
  I did so perhaps you could point out what we both missed?
  - bigiain 3y ago
    "As a result we are currently developing new data retention and disclosure policies. These policies will relate to our procedures for future government data requests, how and for what duration we store personally identifiable information such as user access records, and policies that make these explicit for our users and community."
    - brigandish 3y ago
      That's not the same as the stuff zerealshadowban outlined.
dpifke 3y ago
PyPI used to be somewhat decentralized, with multiple independent organizations (in multiple legal jurisdictions) running mirrors. For $REASONS they've decided Fastly should be the only official mirror. This makes it a one-stop shop for malicious governments.
- donaldstufft 3y ago
  This post is pretty misleading. PyPI still fully supports mirrors (though it is becoming increasingly hard to run a full mirror of PyPI, last I looked a full copy of PyPI is about 30TB). The only thing we ever removed was designating any particular mirror as official and an auto discovery protocol that was quite frankly extremely insecure and slow. That worked by giving every single mirror that wanted to be an "official" mirror for auto discovery a subdomain of `pypi.python.org`, labeled {a-z}.pypi.python.org. A client would determine what mirrors were available by querying last.pypi.python.org, which was a CNAME pointing to the last letter that we had assigned, that would tell it how many mirrors there were, then they could work backwards from that letter. So if the CNAME pointed to c.pypi.python.org, the client would know that a, b, and c existed. Immediately you should be able to see a few problems with this: - It is grossly insecure. Subdomains of a domain can set cookies on the parent domain, depending on ~things~ they can also read cookies. - It does not scale past having 26 mirrors. - It does not support removing a mirror, there can be no gaps in the letters. So we needed to remove that auto discovery mechanism, which raised the question of what, if anything, we should replace it with? Well at the time we had only ever made it up to g.pypi.python.org. So there was only 7 total mirrors that ever asked to become an official mirror. To my knowledge we never reused a letter, if a mirror went away we would just point the mirror back at the main PyPI instance. I don't remember exactly, but my email references there being only 4 mirrors left. From my memory at the time, most of those 4 mirrors were regularly hours or days behind PyPI, would regularly go offline, etc. But again, we never stopped anyone from running a mirror, we just removed the auto discovery mechanism and required them get their own domain name. We even linked to a third party site that would index all of the servers and keep track of how "fresh" they were, and other stats (at least until that site went away). Running a mirror of PyPI is a non trivial undertaking, and most people simply don't want to do that. We never had many mirrors of PyPI running, and as it turns out once we improved PyPI most people decided they simply didn't care to use a mirror and preferred to just use PyPI, but still to this day we support anyone to mirror us.
chatmasta 3y ago
PyPi is in a tough spot because they're also getting hit with an onslaught of malicious packages, which got to such a bad point they had to disable signups. How do they mitigate that kind of activity without logging basic metadata like the IP address that published a package? Also, as a user of PyPi, wouldn't you prefer that a malicious package is at least _somewhat_ traceable to an attacker? Of course most would be behind a VPN but it's better than nothing (or maybe it's not, depending on the tradeoff). Note that the blog post doesn't say they handed the entire database over to the feds. They received three warrants scoped to specific packages and returned only the data they had available that was associated with those packages.
- ginto 3y ago
  > How do they mitigate that kind of activity without logging basic metadata like the IP address that published a package An effective mitigation for abuse/spam is to increase its cost. One example is to require payment.
  - duckmysick 3y ago
    Now they need to subpoena both PyPI and the payment processor. It does slow them down but effectively does nothing to "thwart future subpoenas from the total surveillance state".
  - mort96 3y ago
    It would not go over well if PyPI started charging people to host packages.
    - ginto 3y ago
      Charging goes well for many online services. Hosted email, hosted VPS, and hosted SaaS are some that come to mind. Apple Store and Google Play charge to host mobile games.
      - chatmasta 3y ago
        And the feds can subpoena all those services and their payment providers for logs including IP address that made the transaction.
pluto_modadic 3y ago
This is for package management. I want the supply chain to be secure and would rather know when something unusual happens. Not logging that data would be irresponsible on PyPi's part.
hayleox 3y ago
Which data are excessive? Everything described in the post seemed totally reasonable to me.

takeda 3y ago

For the kind of service they are providing I think the logging is appropriate. I mean if DOJ is interested in PyPI logs the only reason I could think of, is if it was used as a supply chain vector into breaking in into other organizations.

alwayslikethis 3y ago
or locating the people responsible for DRM breakers like youtube-dl? Keeping all the data makes you susceptible to subpoenas like this, which costs money to comply with. There is no reason to keep any data that isn't necessary to the service.
- takeda 3y ago
  youtube-dl is an application, there's no benefit for the author to publish it there. I also checked PyPI repo and looking at it I don't think it is published, there are few that look like might be it, but looking at them, they feel like something that would install something nasty on your computer. As a somebody who relies on PyPI for development, I want them to store all kinds of data that would discourage PyPI to be used for any shady purpose. Also perhaps there's a better example than youtube-dl, as they really don't do anything illegal. Yes their github repo was taken by DMCA request but then it was reinstated back. The GitHub spin it that they are standing up for developers, but the DMCA was just frivolous. I guess, still good for them, because they could just take it down and do nothing.

manicennui 3y ago

[flagged]

deleted 3y ago
[deleted]
brigandish 3y ago
I did so perhaps you could point out what we both missed?
- bigiain 3y ago
  "As a result we are currently developing new data retention and disclosure policies. These policies will relate to our procedures for future government data requests, how and for what duration we store personally identifiable information such as user access records, and policies that make these explicit for our users and community."
  - brigandish 3y ago
    That's not the same as the stuff zerealshadowban outlined.

dpifke 3y ago

PyPI used to be somewhat decentralized, with multiple independent organizations (in multiple legal jurisdictions) running mirrors. For $REASONS they've decided Fastly should be the only official mirror. This makes it a one-stop shop for malicious governments.

donaldstufft 3y ago
This post is pretty misleading. PyPI still fully supports mirrors (though it is becoming increasingly hard to run a full mirror of PyPI, last I looked a full copy of PyPI is about 30TB). The only thing we ever removed was designating any particular mirror as official and an auto discovery protocol that was quite frankly extremely insecure and slow. That worked by giving every single mirror that wanted to be an "official" mirror for auto discovery a subdomain of `pypi.python.org`, labeled {a-z}.pypi.python.org. A client would determine what mirrors were available by querying last.pypi.python.org, which was a CNAME pointing to the last letter that we had assigned, that would tell it how many mirrors there were, then they could work backwards from that letter. So if the CNAME pointed to c.pypi.python.org, the client would know that a, b, and c existed. Immediately you should be able to see a few problems with this: - It is grossly insecure. Subdomains of a domain can set cookies on the parent domain, depending on ~things~ they can also read cookies. - It does not scale past having 26 mirrors. - It does not support removing a mirror, there can be no gaps in the letters. So we needed to remove that auto discovery mechanism, which raised the question of what, if anything, we should replace it with? Well at the time we had only ever made it up to g.pypi.python.org. So there was only 7 total mirrors that ever asked to become an official mirror. To my knowledge we never reused a letter, if a mirror went away we would just point the mirror back at the main PyPI instance. I don't remember exactly, but my email references there being only 4 mirrors left. From my memory at the time, most of those 4 mirrors were regularly hours or days behind PyPI, would regularly go offline, etc. But again, we never stopped anyone from running a mirror, we just removed the auto discovery mechanism and required them get their own domain name. We even linked to a third party site that would index all of the servers and keep track of how "fresh" they were, and other stats (at least until that site went away). Running a mirror of PyPI is a non trivial undertaking, and most people simply don't want to do that. We never had many mirrors of PyPI running, and as it turns out once we improved PyPI most people decided they simply didn't care to use a mirror and preferred to just use PyPI, but still to this day we support anyone to mirror us.

chatmasta 3y ago

PyPi is in a tough spot because they're also getting hit with an onslaught of malicious packages, which got to such a bad point they had to disable signups. How do they mitigate that kind of activity without logging basic metadata like the IP address that published a package? Also, as a user of PyPi, wouldn't you prefer that a malicious package is at least _somewhat_ traceable to an attacker? Of course most would be behind a VPN but it's better than nothing (or maybe it's not, depending on the tradeoff). Note that the blog post doesn't say they handed the entire database over to the feds. They received three warrants scoped to specific packages and returned only the data they had available that was associated with those packages.

ginto 3y ago
> How do they mitigate that kind of activity without logging basic metadata like the IP address that published a package An effective mitigation for abuse/spam is to increase its cost. One example is to require payment.
- duckmysick 3y ago
  Now they need to subpoena both PyPI and the payment processor. It does slow them down but effectively does nothing to "thwart future subpoenas from the total surveillance state".
- mort96 3y ago
  It would not go over well if PyPI started charging people to host packages.
  - ginto 3y ago
    Charging goes well for many online services. Hosted email, hosted VPS, and hosted SaaS are some that come to mind. Apple Store and Google Play charge to host mobile games.
    - chatmasta 3y ago
      And the feds can subpoena all those services and their payment providers for logs including IP address that made the transaction.

pluto_modadic 3y ago

This is for package management. I want the supply chain to be secure and would rather know when something unusual happens. Not logging that data would be irresponsible on PyPi's part.

hayleox 3y ago

Which data are excessive? Everything described in the post seemed totally reasonable to me.