34 ms·
PyPI Was Subpoenaed
- snapcaster 3y agoReally weird, anyone have some inside gossip on what this is about?
- Retr0id 3y agoThe most optimistic reason would be that they were investigating a supply-chain attack, or something of that nature.
- commandlinefan 3y agoThis makes me wonder... it's entirely possible that the PyPI people would be enthusiastic about helping to track down offenders, and their users might agree, _if they knew what the offense was_. Instead, they're presented with a typically antagonistic demand for details, so they understandably get defensive on behalf of their users. I wonder if there's not a better, less heavy-handed way to get cooperation with law enforcement when the request is reasonable.
- Retr0id 3y agoPersonally I would rather not set a precedent of handing data over to government agencies just because they ask nicely, even if it seems like it's for a mutually agreeable good cause. That is, I would rather they go through these "formal" channels, even if it seems a bit heavy-handed. Further, whatever they're investigating here is probably "important", for some definition of important, so they likely value the ability to lean on non-disclosure clauses etc.
- bhouston 3y agoI wish it was that but those people would be smart enough to not use their real name when signing up - those doing supply chain attacks are often at least somewhat professional and take precautions. I suspect it was more about going after software that was enabling piracy, those are often created by naive students who are not expecting the power of government to be unleashed on them.
- nibbleshifter 3y ago> those doing supply chain attacks are often at least somewhat professional and take precautions. Not really. The vast majority of supply chain attacks in practice are idiots exploiting namespacing, bitflips, or typos on pypi/npm to drop miners or infostealers. Yes, even the shit tier supply chain attacks count :)
- paulpauper 3y agomaybe to do with web scrapping, auto-posting spam, etc. https://www.developer-tech.com/news/2023/may/22/pypi-suspends-new-projects-and-users-malicious-activity/ https://www.developer-tech.com/news/2023/may/22/pypi-suspend...
- yuvadam 3y agoSubpoenas are from March and April, predating the spamming of the past few weeks.
- richbell 3y agoPyPI has had a pretty consistent spam problem for a while now.
- paulpauper 3y agoit likely shows that it was an ongoing problem
- jamesmurdza 3y agoIt could be related to the large number of malicious or booby-trapped packages that have been uploaded recently to the index.
- foota 3y agoMy guess? A hacking case against someone for typosquatting or malicious packages or something.
- guhcampos 3y agoCould be anything I guess, even legitimate reasons. T Think of the supply-chain attacks going on in the past few years. I'd say investigating these would be a legitimate reason for a subpoena.
- deleted 3y ago[deleted]
- misterpigs 3y agoI love this level of transparency.
- voynich 3y agoYeah, whether necessary or not, it's still nice to have such a level of detail in a transparency report.
- tomjen3 3y ago> We will not be releasing the usernames involved publicly or to the users themselves. Which is the most important part.
- tptacek 3y agoThey're not allowed to release that. Edit I read 'chaps as saying there was an NDA on the subpoena, but apparently there wasn't, so this might just be flatly wrong.
- AnotherGoodName 3y agoThe NDA isn't the only reason you don't risk interference in an ongoing investigation though so regardless the basic point still stands.
- remram 3y agoEven in the absence of NDA, are you allowed to? Counsel has apparently advised them not to. Would it not carry the risk of being complicit to a crime?
- kevin_thibedeau 3y agoDisclosing facts is not a crime.
- remram 3y agoThere are lots of situations in which disclosing facts is indeed a crime. You are answering my specific question with a nice sounding maxim which is obviously not true in general.
- jacquesm 3y ago> We will not be releasing the usernames involved publicly or to the users themselves. Why not to the users themselves? Have they been prohibited from doing so? (TFA does not say afaict)
- shadowgovt 3y agoThat, or they have reason to believe the investigation is legitimate and they would prefer not to hinder it.
- jacquesm 3y agoThey say very explicitly that they do not know what it was about.
- shadowgovt 3y agoThere's a wide gulf between concrete knowledge and belief. I see an ambulance going lights-and-sirens behind me. I don't know they're on their way to or from a hospital, but I pull over because I have reason to believe they are.
- junon 3y agoWeird analogy. An ambulance has a very narrow scope of responsibility. Legal processes have a very wide scope. Clearly this is related to a legal matter and not an immediate medical matter. But the nature of the legal matter could be a very wide variety of things, ranging from lower court civil proceedings up to treason, etc.
- weaksauce 3y agothey have five usernames... that can narrow down what projects they were associated with pretty quickly to infer if there was something nefarious about them. though it could be entirely unrelated to their activity on pypi and be a trawl for leads based on username similarity from some other messageboard or activity that was used for illegality. though, thinking about it more, that seems legally dubious a reason to be able to get a subpoena issued for. ianal
- rolph 3y ago[In March and April 2023, the Python Software Foundation (PSF) received three (3) subpoenas for PyPI user data. All three subpoenas were issued by the United States Department of Justice. The PSF was not provided with context on the legal circumstances surrounding these subpoenas. In total, user data related to five (5) PyPI usernames were requested.] either a small group of users, or one user with multi aliases wrote a nastyapp ?
- cubefox 3y agoApparently no plans to set up a canary.
- skullone 3y agoWhy would they? It's a public repository, nothing confidential or private
- JohnFen 3y agoAccount details are confidential and private.
- jrockway 3y agoIs there any precedent for people not facing legal consequences for failing to update the canary? The subpoena probably says "and also update your warrant canary to say there were no legal requests." Now you're in contempt of court and in jail for 5 years while you wait for your "compelled speech" case to go to the Supreme Court. In general, I think it usually goes poorly when programmers invent clever legal workarounds. The legal system isn't a computer program. It's guys with guns.
- buildbot 3y agoIsn’t the idea that the (US) government can’t (technically) compel you to lie?
- User23 3y agoThe process is the punishment.
- short_sells_poo 3y agoThat may be the case but if the cost of testing it is 5 years in jail while the case works it's way through the courts, few people will be willing to rely on it.
- EatingWithForks 3y ago
- LordShredda 3y agoI'm guessing some poor typosquatter managed to hit a gov agency and is about to get alphabet soup all over him.
- eur0pa 3y agoThat or fairly unlucky bug bounty hunters
- nonrepeating 3y ago“Get alphabet soup all over him” This is my new favorite alternative to “vanned” (or “v&”)
- flyinghamster 3y agoI think I'm gonna snarf that one too. It's just too good.
- techbro92 3y agoThink I’m gonna snarf snarf. Actually I just looked it up and apparently that word means to eat or drink greedily. Not sure why you used it here
- nonethewiser 3y agoWell you certainly snarfed it up
- pjbeam 3y agoAs in eagerly consume into poster's lexicon I think.
- lagniappe 3y agoit means copy http://acme.cat-v.org/readme http://acme.cat-v.org/readme
- 3y ago
- kjkjadksj 3y agoI don’t understand how the information requested is relevant at all for any purpose. Most users of pypi merely download through pip; they arent registering anything. Furthermore, I would think a bad actor who would register would spoof their ip and use burner accounts anyhow.
- buildbot 3y agoCorrelating IP address use to something else happening at the same time? Like a malware author being incredibly dumb and using their home IP to upload PyPy packages, while IDK, using that same IP as a C&C server endpoint.
- shadowgovt 3y agoThey may not even need to have slipped up and direct-connected via their home IP. The FBI has sufficiently compromised subsets of Tor in the past to do correlative attacks on specific targets.
- caturopath 3y ago> Most users of pypi Presumably the 5 users in question were interesting in some way, not just random. > I would think a bad actor who would register would spoof their ip and use burner accounts anyhow Maybe, but they could find that out with the information. If there's a 10% chance each was sloppy or un-paranoid, there's a 40% chance they get at least one piece of real info. The person might not have thought they were doing anything wrong. Some judge might have greenlit this for a piracy case against the five maintainaers of youtube_dl{c} or something silly.
- tomjen3 3y ago>We will not be releasing the usernames involved publicly or to the users themselves. They point out that they are not subject to a gag order.
- buildbot 3y agoYeah this is interesting, because they could in absence of the gag order but choose not too. Unless it’s not a gag order but a specific don’t tell these users anything?
- stjohnswarts 3y agoI doubt any investigators worth their salt would let the people they're investigating know that they are investigating them before they're ready to charge them.
- firstlink 3y ago> associated with the subpoenas received in March and April 2023. Oddly specific wording there. It would seem they have received additional subpoenas outside that timeframe which do have gag orders, and someone slipped up and failed to put the gag orders in these particular subpoenas. Seems like the DOJ may be doing some long-term fishing for, what, software developers? First the DOJ came for the conservatives, and I said, "go get 'em!" because I wasn't a conservative; next the DOJ came for ____?
- bratbag 3y agoIt's interesting how you manage to leap from what is probably a supply chain attack investigation straight into a pErSeCuTeD CoNsErVaTeS conspiracy.
- metiscus 3y agoAcross the ages, government has applied a disproportionate level of scrutiny to groups and people perceived as dissidents, minorities, and anyone else who could potentially be conceived as a threat to institutional power regardless of the magnitude of the threat or if that threat is true or false. Historically this is a bipartisan issue, for decades the FBI vigorously attacked anti-war groups, black civil rights groups, and various left wing groups via COINTELPRO. I guess to summarize, the way I think of it is that the government is coming for anyone they see as a potential threat and it doesn't matter what the politics of that group are.
- burnished 3y agoStep one: have position and power as part of a dominant group Step two: style yourself as an oppressed minority Step three: defend any action, decision, or position as a persecuted martyr
- Zetice 3y agoDumb legal question; what's the difference, if any, between "We've been subpoenaed" and "Someone had a warrant for data"?
- paxys 3y agoWarrant = we (police or other authority) have the right to come and search your property for evidence. Subpoena = the court compels you to hand over the evidence we need.
- woodruffw 3y agoSubpoenas are orders, but they're not necessarily court-issued. Warrants, on the other hand, are court-issued -- the police can't issue warrants on their own in the US.
- deleted 3y ago[deleted]
- rocqua 3y agoA warrant for a things isn't an order to the owner of that thing. It's an order to (and peemission for) officers to go and seize the thing. You get shown the warrant to prove that they have permission, not to order you to comply.
- woodruffw 3y agoYes, I'm aware -- my other comment says that. I realize this comment is a little ambiguous: the order in the warrant case is an order by the court to the court's officers to perform an arrest, seizure, etc. It's not an order for you (the subject of the warrant) to comply.
- woodruffw 3y agoNot a dumb question: a subpoena is an order to provide information or access, while a warrant is a court-issued document authorizing the government (or an agent of the government) to perform an act (e.g., an arrest, or seizure of an item). Subpoenas can be issued by attorneys (including prosecuting attorneys) as part of the investigative and discovery processes.
- ur-whale 3y ago> We will not be releasing the usernames involved publicly or to the users themselves. Emphasis on the last part: or to the users themselves. In other words: unless they actually let the users involved in spite of claiming the opposite, the whole article is complete posturing.
- burnished 3y agoWhat a weird take
- ralmidani 3y agoDoes a "subpoena" mean a judge was involved? The post says the subpoena was issued by the DOJ.
- tptacek 3y agoIt means a court is involved, but not a judge. Edit Even that is technically wrong; some DOJ subpoenas are apparently preauthorized by statute.
- idlewords 3y agoThere's a pretty extensive list of administrative subpoena authority here: https://www.justice.gov/archive/olp/rpt_to_congress.htm https://www.justice.gov/archive/olp/rpt_to_congress.htm tl;dr: Everyone from the Appalachian Regional Commission on down can subpoena you without a court being involved. And of course Congress has inherent subpoena powers.
- tptacek 3y agoOh, this is so cool. Thank you! This is a step towards answering my noodly question earlier in the thread: authorization for NDAs and "gag orders" in subpoenas appears to be controlled by (varying) statutes.
- avgcorrection 3y ago> The privacy of PyPI users is of utmost concern to PSF and the PyPI Administrators, and we are committed to protecting user data from disclosure whenever possible. Don’t lead with this. > In this case, however, PSF determined with the advice of counsel that our only course of action was to provide the requested data. If you’re going to say this. I’m not judging their decision. Maybe not going to prison is a greater concern to them. It’s fine to just say that you thought it was best to comply because [lawyer reasons that you don’t have to disclose to anyone]/counsel. EDIT: Or say “there are bad people out there and we trust the DOJ”. Whatever.
- tptacek 3y agoLighten up. Nobody's going to federal court to stop the DOJ from investigating botnets, carding rings, and ransomware scams, which is what these things are usually about. Nobody's mental model of PyPI was that they had Signal's priorities.
- avgcorrection 3y agoThen all the less reason to roll out the “of utmost importance” boilerplate. So what’s your point? Also I don’t see how being light-hearted has anything to do with this submission, Thomas.
- davidguetta 3y agoIts just they have no choice. And when they do the choose their "utmost priority". Its not that complicated
- mrguyorama 3y agoThen they shouldn't say protecting their users are their top priority, because they have shown it is not. That's called lying. A correct statement would have been "we will comply with lawful LEO requests"
- x0x0 3y ago
- aa_is_op 3y agoBy the number of malicious packages that site has hosted over the past few months, this was only a matter of time. I've lost track of the number of "white hats" that contact us with extortion requests after they used some dependency confusion attack.
- Mystery-Machine 3y agoWhy is it an extortion request and "white hats" if they have successfully found a security issue in your project and reported it to you, without actually exploiting it? Would you rather them not report it to you or even worse, exploit it?
- cjsawyer 3y agoSounds like they’re the ones who implemented the bad code in the first place, as a honeypot. That just extortion with extra steps.
- JadeNB 3y ago> Why is it an extortion request and "white hats" if they have successfully found a security issue in your project and reported it to you, without actually exploiting it? Presumably because there is some demand for compensation before disclosure?
- dahwolf 3y agoPay me or I will harm you is extortion, as simple as that. There's an entire industry now of people that check known vulnerabilities (so they don't invent anything themselves) in software/packages and cross check this against outdated websites, at a very large scale. They have no morals or security ethics, they barely even have knowledge, they just want to make money with the least amount of effort possible. Don't ever pay them a cent. They're just as ruthless as spammers.
- autoexec 3y agoIf several groups of people who "barely even have knowledge" can profit from checking for well known vulnerabilities on websites and reporting them I say more power to them. If there is an entire industry of people doing low effort work which then discovers vulnerabilities on a company's website that company should pay them, and probably fire some people they've already been paying for not putting in even that much effort to secure their own stuff. Who is less ethical? The people reporting vulnerabilities and wanting to be paid for it or the companies who don't bother to invest in even basic security practices putting people's data at risk and allowing scammers and hackers to leverage those insecure systems to hurt others?
- wongarsu 3y ago> We have waited for the string of subpoenas to subside, though we were committed from the beginning to write and publish this post as a matter of transparency, and as allowed by the lack of a non-disclosure order associated with the subpoenas received in March and April 2023. That's suspiciously specific. Sounds to me like they also received some other subpoenas they aren't allowed to talk about.
- VWWHFSfQ 3y agoI think it just sounds like the three subpoenas they received
- wongarsu 3y agoI'm not sure I'd call three subpoenas "a string of subpoenas" even if it's technically correct. But I'm more talking about specifically mentioning that the subpoenas from March and April 2023 don't have a gag order. Why mention those months specifically if in the other months they didn't receive any? The natural thing would have been to end the sentence six words earlier.
- deleted 3y ago[deleted]
- florbo 3y agoIt sounds more like they're addressing the inevitable "why didn't you post as soon as it happened" party.
- Eisenstein 3y agoIt is perfectly clear that you are correct because trying to tell anyone about confidential subpoenas could be illegal.
- hunter2_ 3y agoWhen it requires so much "reading between the lines" that even this community doesn't have a strong consensus on whether this is being (illegally) communicated or not, I think it's plausibly deniable, but IANAL. Contrast with well-known canaries.
- throw_a_grenade 3y agoSo just yesterday PyPI announced they're retiring cryptographic signatures: https://news.ycombinator.com/item?id=36044543 https://news.ycombinator.com/item?id=36044543. It's hard to keep those things separated. I would very much like the code submitted to PyPI be protected end-to-end by cryptographic signatures, when PyPI has either no resources, or no spine to stand up to a government. Any signatures, even PGP, which should be in place until someone provides better mechanism.
- stjohnswarts 3y agoI don't have a problem with this as it was 5 particular users and not "give us all the data for for all your users". They didn't really have much of a choice. I don't think they would have had a choice in any of the 5 eyes countries or their allies
- etaioinshrdlu 3y agoAmazing how upset users here get over the very reasonable response to very normal police work.
- throwaway_13140 3y agoAgreed - how else was the DOJ supposed to do their job? They clearly need the data for an investigation. No need for PyPI to give information about how current users can alter their accounts to thwart future requests.
- sneak 3y agoNormal police work doesn't go fishing for the IP addresses (potentially millions of users) of everyone who downloaded a package. > "IP download logs of any Python Package Index (PyPI) packages uploaded by..." given usernames Do you feel the same way if the cops are receiving the IPs of everyone who downloaded yt-dlp? IP addresses and timestamps resolve to physical locations and oftentimes street addresses.
- buzzscale 3y agoThat doesn't make any sense though. What benefit would DOJ get from getting the IP address of everyone who downloaded ytp-dlp? They aren't the enforcement arm of google's terms of service, which is a civil matter. Even if they were, and the DOJ was going for a dragnet operation to go after tools that could potentially infringe terms of service of big corporations, they would go after every tool and every fork. Not just 1 package. But again, what court would allow such action and why? If I was in the DOJ and was investigating a malicious package uploaded to PyPI, I would ask for the IP's of the downloaders to see if the uploaders dun goofed and downloaded their package shortly after uploading off VPN. Or to find out if any major corporations were impacted by downloading the malicious package and to inform them.
- ranger_danger 3y agoIn the US at least, it has been ruled that an IP address is not sufficient evidence to link activity to any particular person. You could have been hacked for example.
- NelsonMinar 3y agoTotal speculation on my part but PyPI hosts yt-dlp, the unauthorized video downloader. https://pypi.org/project/yt-dlp/ https://pypi.org/project/yt-dlp/
- vore 3y agoI would think the government has bigger fish to fry than to spend time subpoenaing yt-dlp.
- throwaway09223 3y agoGoogle is a pretty big fish themselves. What usually happens is the large corporation lays out a case like "yt-dlp is responsible for billions in damages" and they press the DOJ to investigate and prosecute.
- account42 3y agoAFAIK Google hasn't even taken any civil action against y(ou)t(ube)-dl(p) at all, that was all the music industry.
- dual_dingo 3y agoNot a US cititzen, but "The government" is a wide term and any law enforcement agency would fit this, including the ones that are responsible to deal with things like copyright enforcement - that's exactly the type of fish they exist to fry ...
- vore 3y agoGiven the discussion around how lacking PyPI supply chain security is, how juicy of a target it is for attackers, and how critical infrastructure is probably relying on PyPI, yt-dlp is the last thing on my mind.
- ChuckMcM 3y agoFYI for non US readers ... In the US, subpoenas come from the Justice Department (either state or federal depending on the crime for which evidence is being sought). The court that issued the subpoena is on it, and the person or entity being served, has the right to see why some government agency felt it could aid in the uncovering of a crime that had already been committed. The person or entity then has the opportunity to challenge that in court prior to complying with it. This is sometimes informally called "quashing the subpoena." From my sister-in-law who is a defense attorney, the most common result of challenging a subpoena is to get what it asks for narrowed down to just what is plausibly responsive. In the article, this response: As a result we are currently developing new data retention and disclosure policies. These policies will relate to our procedures for future government data requests, how and for what duration we store personally identifiable information such as user access records, and policies that make these explicit for our users and community. Is good practice for limiting what a subpoena can request (you can't give what you don't have). At Blekko we logged access records in such a way that we could use PII for 48 hours and then it was deleted. The CTO, Greg Lindahl, is a huge privacy advocate and this sort of architecture made it possible to get information to improve our ranking and service without compromising people's privacy. In practice I don't think any agency could go from "we have a suspect" to "issue a subpoena" in 48 hrs so it was a useful way for us to stay out of the crosshairs. The most interesting event was the FBI asking for information on IP addresses that had accessed their honeypot CSAM site. That turned out to be some of the machines in the crawling cluster. Given that the site was outside the crawl "horizon" and didn't rank (very few sites linked to it) it didn't even make it into the cache for rank analysis. But in that case the turn around time was impressive. Of course that is because they were just using their own logs to generate subpoena requests.
- CarbonCycles 3y agoWhat an odd article and release statement. It’s almost as if they’re signaling w-out literally signaling the parties of interest. Surprised the doj didn’t issue any gag orders.
- throwaway_13140 3y agoExactly. I guess the transparency is nice but at what point are you potentially helping someone cover their tracks who may or may not actually deserve that help?
- rossdavidh 3y agoOne gets the impression that this was an artfully crafted way around the specifics of the gag order, to disclose whatever wasn't specifically prohibited by it. IANAL.
- whimsicalism 3y ago> as allowed by the lack of a non-disclosure order associated with the subpoenas received in March and April 2023. Yeah no way they haven't had other subpoenas then.
- sneak 3y ago> "IP download logs of any Python Package Index (PyPI) packages uploaded by..." given usernames This is way overbroad. The fact that a judge granted this is very bad.
- kjs3 3y agoWe get "please provide the logged IP addresses of user X" subpoenas on a weekly if not daily basis. Which law school did you go to so I can tell our corp counsel they've been doing it wrong and stop asking?
- robryk 3y agoNote that GP complains not about the request for IP addresses of user X, but the request for IP addresses of anyone who downloaded content uploaded by user X.
- tw-0981230981 3y agoYou should re-read the quote. This was not a request for the IP addresses of the users in question, but for the everyone that downloaded any packages uploaded by those users.
- kjs3 3y agoYou're right. Sorry that I implied that was the only thing they asked for. But the point still stands that it's very common to be compelled to provide the equivalent of "tell me everything you know about this IP address", and our legal has said we (usually) legally have to provide it.
- duskwuff 3y agoIt's hard to say that it's "overbroad" without knowing the details of the situation. It's not hard at all, on the other hand, to imagine situations where this would be a reasonable request. Probably the most obvious would be if the packages contained material which was illegal to possess or distribute (like CSAM). Another would be if the packages were being used as part of a malware C&C operation -- knowing what IP addresses downloaded the packages would aid in determining the scope of the campaign.
- hajimuz 3y agoDude I thought it’s Chinese Gov. Hey, America!!!
- junon 3y agoGood on the PyPi folks. This is an incredibly well done disclosure, an example to be sure.
- jupp0r 3y ago"9. IP download logs of any Python Package Index (PyPI) packages uploaded by the given usernames" This was the point where I was wondering if this is really about some malicious packages or something more along the lines of copyright infringement software.
- femto113 3y agoThis definitely seems like a significant element of the ask, but for any popular package a list of all the downloaders would be pretty overwhelming in size (and I think of very limited utility). I'm guessing that some versions of some more obscure package(s) were identified as being used in an attack and they're either trying to identify potential attackers or other victims (or both) of that attack. From a 2021 article[1] about packages used to deliver malware "we have alerted PyPI about the existence of the malicious packages which promptly removed them. Based on data from pepy.tech, we estimate the malicious packages were downloaded about 30,000 times." For comparison yt-dlp has tens of millions of total downloads and gets downloaded over 70,000 times every day [2] [1] https://jfrog.com/blog/malicious-pypi-packages-stealing-credit-cards-injecting-code/#products https://jfrog.com/blog/malicious-pypi-packages-stealing-cred... [2] https://pepy.tech/project/yt-dlp https://pepy.tech/project/yt-dlp
- rektide 3y agoIt'd be lovely to see better patterns emerge to aggregate and/or anonymize data. Great respect for the response. Reevaluating data retention is a great move.
- rendx 3y agoWay too much unnecessary data collected and stored as usual. But one of the best transparency reports ever. Thanks PyPI!
- tgbugs 3y agoOne theory that I don't see mentioned yet is that someone used an upload to pypi to exfiltrate data or simply as a way to upload arbitrary data somewhere. In a sense pypi is just a file hosting service, so it could have nothing to do with any actual python projects at all.
- rocqua 3y agoInteresting approach to data exfil. Though it seems predictable that exactly this kind of subpoena would be issued. If you can predict it, you can probably mitigate it. Which means the subpoena would only be useful if the criminals made an opsec mistake. That is generally how most sophisticated criminals get caught, but here it feels like anyone inventive enough to try will probably also be prudent enough not to leave a trail.
- dvt 3y agoMost likely caused by phishing, ransomware, or (unlikely) crypto mining. I'd bet someone from some agency had credentials leaked due to a malicious package. Honestly, PyPI is stuck between a rock and a hard place, but having something like a "verified" badge (where someone's real identity is tied to it) for certain packages would go a long way to ensure some level of security. The problem gets a bit hairier when dealing with dependency chains, however.
- scrum-treats 3y agoIt would be great to see this for VS Code extensions as well.
- BrandoElFollito 3y agoI wonder why such organizations that hold critical data for the community at large do not use an international canary system. Should one of the countries issue an order, the ones outside of its jurisdiction can openly disclose the information. Say if the US forces the US entity to not do something, the French one sees it and can warn all users.
- detaro 3y ago"I've been ordered not to tell the details, but I know you will publish them, so I'm going to tell you the details" is not going to be taken as "obeyed the order" by law enforcement or courts.
- BrandoElFollito 3y agoSorry but I think I do not understand (English is not my first language). Who would be in trouble? In case anything happens with the content of the service, the detail of the changes would be made clear by someone outside the jurisdiction. A typical example is TrueCrypt that, one day, changed their page to say to use something else instead of their product. If the code was shared between several countries, the others could simply publish that this and that was changes out of band, and that it means that the code is now positively unsafe.
- detaro 3y agoWe're not talking code changes here, but purely a data request. For code changes the trail is more obvious and it being noticed is easier to explain - but you are still in a situation that anything that can be read as you publishing it in an indirect way (e.g. by giving details to a connected organization which you know won't keep it private) will be taken as such and get you in trouble. I think it'd be quite hard construct that in a truly "safe" way.
- BrandoElFollito 3y ago> We're not talking code changes here, but purely a data request You are right. My comment was a bit offroad, I could have made that clearer (about how to deal with "data" (code, ...) in international context) > I think it'd be quite hard construct that in a truly "safe" way. For open source code it is easy - everyone sees teh chnages and why they've been promoted. For closed source, having your source at a third party (or synchronized), build from only the identical code (between the two repositories), and enforce a two-eyes kind of code promotion (merge) will make it so that any change in the code that is not vetted by both parties (or multiple parties) will not get built. I gave the example of Truecrypt that was unfortunately US-only and they had to revert to allusions in order to inform that it was tempered with.
- zerealshadowban 3y agoThey log too much data about their users. So they should promptly update their policies to a) stop logging so much, b) delete all past logs, and c) sharply limit the span of time until deletion of whatever logs they decide they really need to track for internal needs. They should avoid logging, and rapidly rotate logs, to thwart future subpoenas from the total surveillance state.
- takeda 3y agoFor the kind of service they are providing I think the logging is appropriate. I mean if DOJ is interested in PyPI logs the only reason I could think of, is if it was used as a supply chain vector into breaking in into other organizations.
- alwayslikethis 3y agoor locating the people responsible for DRM breakers like youtube-dl? Keeping all the data makes you susceptible to subpoenas like this, which costs money to comply with. There is no reason to keep any data that isn't necessary to the service.
- takeda 3y agoyoutube-dl is an application, there's no benefit for the author to publish it there. I also checked PyPI repo and looking at it I don't think it is published, there are few that look like might be it, but looking at them, they feel like something that would install something nasty on your computer. As a somebody who relies on PyPI for development, I want them to store all kinds of data that would discourage PyPI to be used for any shady purpose. Also perhaps there's a better example than youtube-dl, as they really don't do anything illegal. Yes their github repo was taken by DMCA request but then it was reinstated back. The GitHub spin it that they are standing up for developers, but the DMCA was just frivolous. I guess, still good for them, because they could just take it down and do nothing.
- manicennui 3y ago[flagged]
- einpoklum 3y agoHere is what I consider the key section: > The privacy of PyPI users is of utmost concern to PSF and the PyPI Administrators, and we are committed to protecting user data from disclosure whenever possible. In this case, however, PSF determined with the advice of counsel that our only course of action was to provide the requested data. I, as Director of Infrastructure of the Python Software Foundation, fulfilled the requests in consultation with PSF's counsel. The first part of this section contradicts all of the rest. If user data privacy is of "utmost concern", then it is a concern above fulfilling legal obligations under US law. Plus, such supposed obligations must be staunchly fought before even considering whether or not to observe them. So, in fact, user data privacy is a minor concern for the Python Software Foundation, while swift prostration towards the US federal state is what's of utmost concern. Of course, they almost admit it themselves. If we carefully read the second clause, they don't say "we're committed to protecting user data from disclosure", but - the "we're committed... when possible". So, they're saying that if they believe it isn't possible to protect, they have _no_ commitment to try their utmost to protect. i.e. when they see fit, user data protection is _not_ a concern at all. ... ok, ok, it is a public relations concern.
- ChrisMarshallNY 3y agoThat's an excellent transparency report.
- deleted 3y ago[deleted]
- jehb 3y agoSuggestion: Start slipping unique URLs into the "hidden" backend fields of systems where you'd like to know if your data was breached, improperly used, or handed over to a three letter agency. Suddenly getting hits at mydomain.com/[uuid]? At least you know somebody has looked at the data, or at the very least fed it through some processing tool that is extracting and visiting the URLs.
- mmsc 3y agoThis is called a canary and can be used in so many places: https://blog.thinkst.com/2022/09/sensitive-command-token-so-much-offense.html https://blog.thinkst.com/2022/09/sensitive-command-token-so-...
- austinjp 3y agoI'm pretty sure I've seen a SaaS that does this, but I can't remember the name.
- tailspin2019 3y agohttps://canarytokens.org https://canarytokens.org
- krick 3y agoHow do you suggest it should've been used in this case? As a PyPI username (obviously pointless), or what?
- jehb 3y ago
- __MatrixMan__ 3y agoKudos to PyPI for handling this professionally. That said, I think we should be working towards a world where they're unnecessary. As a middle party to what ought to be a developer/developer trust relationship, they're attack surface that threatens depender sometimes and dependee other times. Going peer-to-peer will be less convenient, but worth the investment in the long run.
- tru3_power 3y agoIs this related to that Microsoft disclosure?
- svaha1728 3y agoIf it takes subpoenas to get package management fixed in Python so be it. Can the Rust Crates.io team take over Python management as well?
- b33j0r 3y agoWhy don’t nerds get the same rights? According to US news over the past 3-4 years, you can just ignore subpoenas, then get a contributor job on a cable news network. Bonus points, the more you flout the law as arrogantly as possible ;p
- gjmacd 3y agoI would point to Jim Jordan and all the other Republicans after January 6th who didn't honor a subpoena and toss them in the trash. Nobody in our government honors them, why should we in the private sector? What's going to happen, they going to raid offices and get a bunch of PC's and books?
- dpifke 3y agoBeing reminded that PyPI is a target for law enforcement makes me even more irked that they've removed end-to-end package signing without providing a replacement[0]. PGP signatures—even though rarely used—would allow someone to verify that a signed package was not modified by PyPI after being uploaded by its original author. Without any sort of signing mechanism, we have to trust the U.S. Government to never demand that PyPI insert a backdoor, via a National Security Letter, FISA court order, or other kangaroo court process. Good luck with that. The existing PGP signing mechanism had usability issues and security footguns[1], but was better than nothing. It's a shame they didn't roll out a more usable and secure alternative before removing the existing functionality. [0]: https://news.ycombinator.com/item?id=36044543 https://news.ycombinator.com/item?id=36044543 [1]: https://blog.yossarian.net/2023/05/21/PGP-signatures-on-PyPI-worse-than-useless https://blog.yossarian.net/2023/05/21/PGP-signatures-on-PyPI...
- wokwokwok 3y agoIf you want to start with tinfoil hat theories, think about this: The PGP signatures were removed, nominally because few people used them. ...but the timing of the removals is coincidental, no? "You need to have a backdoor that lets us see who's downloading what packages and let us inject custom code to particular targets" "That's technically impossible because of..." "Here is a court order. Implementation is your problem. You're not allowed to tell anyone you even received a court order." "...well, I guess signed packages have to go then..." (: I don't actually believe that, since PGP signing was frankly, barely used and really there's hardly any meaningful difference between a PGP you can't verify (which was most of them) and not having it; in fact the illusion of security is probably worse than not having it at all. ...but still. As you say. It sucks there's no meaningful replacement for it.
- donaldstufft 3y agoI would resign from PyPI before I ever allowed a backdoor to be installed. I haven't explicitly asked, but I would be very surprised if any of the other PyPI admins felt differently.
- WiSaGaN 3y agoHow difficult is it for the US government to supply chain attack other countries that are using popular Python packages such as PyTorch given their power over these kind of global software infrastructure under US internal law? Are we able to assess it given there are gag orders as well?
- asne11 3y agoI keep seeing people trying to assure other readers that the recipients of these subpoenas have some recourse to appeal. This is not the case if the subpoena is issued by the FISA court, otherwise known as "the court of no rejection."
- dragonwriter 3y ago> I keep seeing people trying to assure other readers that the recipients of these subpoenas have some recourse to appeal. Because they do, in regular federal courts. > This is not the case if the subpoena is issued by the FISA court The Foreign Intelligence Surveillance Court doesn’t (and courts in general don’t) issue subpoenas.
- asne11 3y ago> The Foreign Intelligence Surveillance Court doesn’t (and courts in general don’t) issue subpoenas. This sounds like a matter of technicality, no? Whatever you call it, what they issue has the effect of a subpoena that you can neither appeal nor speak about. My point is not that this is what's going on here, but that the right to appeal is not generally a given.
- dragonwriter 3y ago> This sounds like a matter of technicality, no? No. > Whatever you call it, what they issue has the effect of a subpoena that you can neither appeal nor speak about. No, it doesn’t. FISC doesn’t generally issue orders that are subpoena-like that would be presented to people outside of government. It provides surveillance orders to the FBI under which they conduct surveillance and, should someone later question if maybe that surveillance was a criminal violation of the Foreign Intelligence Surveillance Act—which would also be the FBI—the FBI can show the FBI the paper from the FISC that says its okay and then the FBI will not arrest the FBI for criminal violations of FISA. Or if someone somehow finds out they were surveilled and files civil charges, the paper serves a similar function. Where they do (and this only, IIRC, occured under law that has now expired) issue orders that apply to external parties, those can be challenged directly in the FISC, appealed from the FISC to The Foreign Intelligence Surveillance Court of Review, and, from there, to the US Supreme Court. There is at least one such appeal to FISC-R by Yahoo! that has been published in redacted form. In no case does FISC issue orders that would go to an outside party who would then have no means or forum to challenge them.
- casey2 3y agoHow come when PyPI hosts unwanted malware they get subpoenaed but when Apple or Microsoft or anyone else with a big team of lawyers distributes auto-installing "updates" designed to harm/scam users the DOJ is silent?
- phendrenad2 3y agoAn important aspect here is the fact that subpoenaes only share information one way. You are asked to provide information, and you get zero information about why. There is a simpler way to get information: Contact the person you want information from and make the case that they should give you the information. Using a subpoenae seems like something a government would do when they're doing something shady and don't want you to know about it.
- paradox242 3y agoWhen your platform is the source of malware used in supply chain attacks you gotta expect the Feds are going to come knocking eventually.
- HeavyStorm 3y agoThere were lots of specifics asks from the court. What if Pypi didn't had those data, like IP addresses? I don't remember ever recording those in a longstanding storage.
- morkalork 3y agoIs there a way to be only contactable in public? Like say a blind man standing in a town square, where any subpoena would have to be read out loud to them in earshot of anyone walking by? Is such a thing possible, digitally?
- seventytwo 3y agoAny bets on this being part of a broader investigation into software supply-chain attacks?
- krick 3y agoI'd say nothing but nickname and the list of packages uploaded (which should be public in the first place) should've been stored anyway. It immediately reminded me, that PyPI content is really trash as it is because of all the squatting and pointless unfinished toy-projects, and whatever they are logging clearly doesn't help, but I think that big problem for PyPI is one seemingly minor detail: lack of namespaces (as in packagist.org of pkg.go.dev). It is not a solution for all sorts of malicious behaviour, of course, but it really makes things much easier. It doesn't solve typosquatting and such, but, honestly, neither does the current system, obviously. And at least it allows to keep the actual package names semantic. And which one of countless "*/time" libraries you wanna get you just kinda have to decide separately, using the number of starts on the github as a reference and carefully copy-pastying the id to your requirements.txt The same issue I have with Cargo. I mean, really, isn't it obvious that making users compete for better project names just makes everything shit?
- fijiaarone 3y agoIt’s ok, this government is perfect.
- Liberonostrud 3y agoHow can you pronounce this word?
- diligence_ 3y ago[dead]
- Aeolun 3y agoKind makes me feel like a lot of these services should just _not_ be hosted in the US? If we'd have this hosted in Germany or Sweden, would the government be so casually requesting data from these registries?
- mschuster91 3y ago> PyPI retains records of all changes to projects on the index, and has since 2002-11-01 17:11:36 UTC. Wow, I had zero idea how old Python is.
- dragonwriter 3y agoPython was released in 1991 with 1.0 in 1994, PyPI is newer (the date listed there appears to be when it went online.)
- jwilk 3y agohttps://www.pypa.io/en/latest/history.html#before-2013 https://www.pypa.io/en/latest/history.html#before-2013 says PyPI was launched in 2003.
- dragonwriter 3y agoWikipedia agrees with the activity history PyPI themselves posted in TFA and indicates the index at its original home under python.org went live in November 2002, and PEP 301 was posted with present-tense descriptions of the URL where the index was available on 11/8/2002. But,
- jwilk 3y agoAFAICS PEP 301 linked only to a demo site until April 2003: https://github.com/python/peps/commit/c773b1b2ce0a2817 https://github.com/python/peps/commit/c773b1b2ce0a2817
- dark-star 3y agoStrange that they didn't mention in 3b for how long they record IP addresses and logins. One has to assume that they keep this data indefinitely, even though they don't need it except for such subpoenas
- account42 3y ago> PyPI retains records of all changes to projects on the index, and has since 2002-11-01 17:11:36 UTC. > submitted_from | IP Address > PyPI retains records of critical user events including account creation, emails sent, email address changes, logins, and login failures. See this list for the comprehensive set of events recorded. > ip_address_string | IP Address Retaining IP addresses for over a decade doesn't sound GDPR-compliant.
- bogwog 3y agoIt’s nice that they’re committed to user privacy, and this post really gives me confidence that my privacy will be reasonably protected. …but why is that a goal for PyPi? As a publisher of packages, it’s a nice-to-have, but as an end user it’s kind of scary. I don’t want to use software packages published by anonymous and potentially unaccountable people. That’s probably why they have so many malicious packages. Maybe you live in an oppressive regime who will imprison/murder you for publishing some code; ok, but that’s an outlier, and there are a lot of ways to get around that situation. I just don’t see the benefit of privacy in this situation? Is it just to reduce the administrative overhead of collecting/verifying identity info? I’m genuinely curious to learn about a realistic use case that justifies the risks to all users. I know you can self host your own package index, but very few users have the resources to do that.
- kortex 3y agoI think largely because the prerogative is on the code author to reveal as little or as much about themselves, and the prerogative of library users is to sufficiently vet a package. If folks want to publish code pseudonymously, and folks want to use that code, as long as it's not abusive, what's to stop them? You can achieve basically the same effect with github, gitlab, or even plain self-hosted HTTP packages (pip just uses a convention for listing packages in a dir, any HTTP file host can be a package server), without PyPI. I actually think the larger problem is Python's reliance on imperative code that executes at install time. Yeah you can use pip --download and extract it yourself, but folks rarely do that.
- miga 3y agoExcellent example of how difficult it is to design privacy-friendly service. Any information that can be subpoenae-ed can also be hacked away.
- berkes 3y ago> Any information that can be subpoenae-ed What if you operate from e.g. Iceland, Switzerland or even many EU states? Can a US Gov. agency still subpoena you?
- kgbcia 3y agoProbably someone made a YouTube or Tor package so the state is after them.
- stuaxo 3y agoThe list of things they asked for seems to be basically "everything" - seems like a dragnet approach.