10 ms·
I mirrored all the code from PyPI to GitHub and analysed it
This is a side project I've been working on for the last few months. I built an automated system to continuously mirror all the code on PyPI to a series of Github repositories. Mirroring PyPI code to Github enables:
1. Scanning of all new Python packages for accidentally published credentials
2. A browsable/searchable index of published code with a nice UI
3. Large-scale analysis of all published code to see how the language is evolving
Using this project anyone is able to download the contents of PyPI to their personal machine and analyse every piece of code ever published in a matter of hours.
I hope it enables people to do things with the worlds largest and oldest corpus of Python code that wasn't possible before, and while this is likely totally useless to most people I think that is kind of cool and unique.
- ehPReth 3y agoVery interesting! Just a heads up that your charts on dark mode under Growth have yellow text on a white background which is near-impossible to read for me on hover
- orf 3y agoThanks, fixed!
- tentacleuno 3y agoI'm all for archivism, but wouldn't this get taken down by GitHub? If what your website says is true, you're storing 300GB+ of code on GitHub. I've heard stories from people who've also tested the limits, and they've had emails from GitHub asking them to cease activity.
- orf 3y agoI was in contact with them, and they are apparently OK with having the repositories be split up. There are over 230 of them each under 1.3gb in total size. I’m working on distributing this data without GitHub - git packfiles are a fantastic way of compressing this data, and you can serve those easily enough from a bucket.
- deleted 3y ago[deleted]
- tlocke 3y agoInteresting. In the language feature list, what is 'try star' ?
- orf 3y agoIt's exception groups[1], with the syntax `except*`. 1. https://peps.python.org/pep-0654/ https://peps.python.org/pep-0654/
- Einenlum 3y agoI wondered what it was too. Maybe "exception groups" or "except star" would be clearer for the reader (thanks for your amazing work btw)
- glaucon 3y agoI hadn't heard of it either. In addition to the PEP this link may be useful https://realpython.com/python311-exception-groups/#exception-groups-and-except-in-python-311 https://realpython.com/python311-exception-groups/#exception... . The feature was new in 3.11. BTW RealPython is quite boring about wanting you to create an account, I'm sure they didn't use to be as bad but I can barely read that page in its entirety without getting forced to create an account.
- swyx 3y agohighly recommend clicking the "witness the inevitable future" button as a python oldie coming back into python, i've been surprised by dataclasses. are they basically "backwards compatible better classes"? any strong blogposts or readings that people can share about better typed python/OSS module publishing practices?
- deleted 3y ago[deleted]
- dcastm 3y agoIt’s under Growth > Files for those struggling to find the button
- kzrdude 3y agoDataclasses are orthogonal to typing (IMO), they just use types for their evocative syntax for fields. Dataclasses are nice - they are a pared down version of the attrs library, so a simple way to create data-only or mainly-data records through classes. They are not intended to replace all classes.
- fbdab103 3y agoYou reminded me of this article[0] where the author asks why not dataclasses by default? I am inclined to agree, dataclasses feel Pythonic in that they remove boilerplate with reasonable defaults (sorting, hashing, etc). [0] https://blog.glyph.im/2023/02/data-classification.html https://blog.glyph.im/2023/02/data-classification.html
- acbart 3y agoYes, I really wish this was the reality. I would even be very happy with a new `data` keyword for this purpose. I'm teaching `dataclasses` in my CS1, and it's nice, but frustrating that A) students have to remember the import, and B) all my slides always have to be a bit longer thanks to the `@dataclass` line taking up space. This is a real problem for some of my longer examples...
- 3y ago
- teddyh 3y agoSome people don’t put their code on GitHub, since they object to GitHub’s ToS, especially those pertaining to analysis and use by Copilot and various other uses which Microsoft may see fit to put the code. Will Microsoft see this as a free license to use all of PyPI?
- orf 3y agoThey are within their rights to make that choice, but when you publish a package to PyPI you agree to their terms which gives anyone the right to mirror, distribute and otherwise use the code you’ve published.
- kzrdude 3y agoThe rights you find in the PyPI terms, do they provide everything you need to comply with the Github terms? Ultimately it's tricky to understand what Github really means with their terms (they say User-generated contents a lot.)
- uxp8u61q 3y agoThe terms of PyPI give anyone the right to mirror. Do they give anyone the right to mirror on github, is the question.
- pjc50 3y agoClearly yes. It doesn't say any restrictions. Code which has a "can't be placed on github" license restriction is definitely not open source, regardless of what other terms the license purports to have.
- xigoi 3y agoGitHub does more to the code than just mirror it, so having the right to mirror is not enough. If trying to prevent your software from being used to create proprietary software makes it not open-source, is the ONU GPL not an open-source license?
- deleted 3y ago
- codetrotter 3y ago> https://py-code.org/download https://py-code.org/download This is a perfect job for task-spooler! :D To mirror your pypi data, I sshed into my server and did this: mkdir -p ~/src/github.com/pypi-data cd !$ wget https://raw.githubusercontent.com/pypi-data/data/main/links/repositories.txt xargs -L1 tsp git clone --bare < repositories.txt And then I closed the ssh connection to my server, knowing that my server will proceed to mirror all of those repositories of yours one by one :D
- deleted 3y ago[deleted]
- mrwizrd 3y agoYou have very serendipitous timing, I was needing to solve a similar task just the other day and had a hell of a time getting it worked out. Thanks for taking the time to post it! :) Straightforward, but seeing a "worked example" helps a ton.
- henrydark 3y agoI didn't know about task spooler. Is it better than using xargs with a parallel pool? xargs -L1 -P20 git clone --bare < repositories.txt
- codetrotter 3y agoYes, for me it is better because if you do it your way you have to keep your ssh connection open until all of the git clones have been done, which in this case takes several hours. (Or you could also run your way in tmux or screen.) With task-spooler, it puts all of the commands (in this case, the individual git clone commands for each of the repos) in a queue and it runs the commands independently of my ssh session, so I can quickly add a bunch of jobs like this to the queue and immediately disconnect my ssh session.
- allarm 3y agoI’ve recently worked on something similar that involved cloning multiple repositories and found gnu parallel to be ideal for that task. Parallel gives more control than xargs.
- gustavus 3y ago[flagged]
- galenmarchetti 3y agohttps://pypi.org/policy/terms-of-use/ https://pypi.org/policy/terms-of-use/ Seems like PyPi explicitly allows this behavior: "If I upload Content covered by a royalty-free license included with such Content, giving the PSF the right to copy and redistribute such Content unmodified on PyPI as I have uploaded it, with no further action required by the PSF (an "Included License"), I represent and warrant that the uploaded Content meets all the requirements necessary for free redistribution by the PSF and any mirroring facility, public or private, under the Included License. If I upload Content other than under an Included License, then I grant the PSF and all other users of the web site an irrevocable, worldwide, royalty-free, nonexclusive license to reproduce, distribute, transmit, display, perform, and publish the Content, including in digital form."
- aragilar 3y agoThere's a popular package in a subfield (which I'm not going to name) which disallows redistribution in its license, but is uploaded by its author to PyPI. I wonder how that interacts with PyPI (given that it has a license, so would hit the first clause, but it doesn't allow redistribution, so would seem to be under the second clause, which contradicts the original license to users), and if the DMCA could be validly used to take down the GitHub repositories? It would also be interesting to see how many other packages have such issues.
- deleted 3y ago[deleted]
- orf 3y agoIt's not that I didn't think "it was easy enough to use PyPI do analytics and analysis": it is near impossible for the layman to use PyPI to do analytics and analysis in it's current form. The volume of data is very unweildly, there are numerous quirks with extracting packages and no tooling exists to help you in any way. This means no analysis has been done on the contents of PyPI. In turn this means malicious packages are harder to detect (and for sure still present somewhere in there), it means people publish an absolutely crazy number of credentials to PyPI on a daily basis without ever knowing (+ no simple way to find concrete ways to improve this) and it means there is a lack of exploration on the impacts of language features/changes on the ecosystem. To me the GitHub aspect isn't important or interesting. Would it make any difference if it was distributed from a series of git repositories hosted on S3? It's the git apsect that is interesting, because it lowers the barrier for anyone to access the corpus of already public, already mirrored and already automatically-scanned-by-bad-actors code that is on PyPI. While this project is more "a number of things glued-together" than "a groundbreaking invention", I have to disagree with the triviality aspect. Most problems we deal with can be reduced to 'copying X from one place to another' (sorting?), and the devil is always in the details. > I don't like this, it's this kind of stunt that makes me reluctant to publish my code in general. Isn't this quite circular? People using code you publish publicly makes you reluctant to publish code publicly?
- DrNosferatu 3y agoWhat fraction of the total is TensorFlow related code?
- orf 3y agoThere is a section on the page that tells you (near the bottom): 16% of all uncompressed data in PyPI is from the Tensorflow project.
- geysersam 3y agoWhat the? 16%? Of the Python ecosystem. Why is it so big?
- orf 3y agoYou can go check out the page linked in the post to see? That’s kind of why it exists! Spoiler: the answer is nightly builds, huge binaries and a lot of wheels per release.
- deleted 3y ago[deleted]
- geysersam 3y agoI was asking more on the level of: why do they do nightly builds of huge binaries and wheels when similar frameworks are not?
- glandium 3y agoHave you tried building Tensorflow?
- disgruntledphd2 3y agoI tried on release day. Managed to break my entire Linux system. That was an annoying day. And pip install some tensor flow nonsense has broken my projects many, many times. At this point I try really hard to ever avoid depending on anything in that ecosystem.
- krmboya 3y agoOne the issue of uploaded credentials, is there a chance that a portion contain dummy placeholder values in config files, or are they hardcoded?
- codetrotter 3y agoSome of them might even be canary credentials. For example, here is one service that is able to generate various canary tokens: https://www.canarytokens.org/generate https://www.canarytokens.org/generate
- jwilk 3y agoWhy would anyone upload canary credentials to PyPI?
- codetrotter 3y agoSecurity researchers wanting to investigate what kinds of credentials other people are looking for, for example. Or as another example, someone who wants to research how often people copy paste example config files without replacing things like credentials given in the example. Probably a bunch of other reasons too.
- nhggfu 3y agoi had never heard of this "smart stuff as a service" before. thanks @codetrotter
- orf 3y agoLots of dummy ones in tests for sure. I’m going to follow up with some deeper analysis on this topic in a few weeks, but you’d be surprised.
- chanux 3y agoThanks for the fantastic work! So you found 57 live AWS keys[1]. That's out of 1631 according to your stats (If I didn't misread something). I wonder how many of the top two in the list are live :D Google API Key 4,015 OpenAI API Key 3,531 [1] https://tomforb.es/i-scanned-every-package-on-pypi-and-found-57-live-aws-keys/ https://tomforb.es/i-scanned-every-package-on-pypi-and-found...
- zomglings 3y agoThis is actually amazing, great work! There is a search page where I can look up which repo a specific version of a specific PyPI package is in. But I didn't find that index in the metadata files ("repositories_with_releases.json" seemed the most likely candidate). Is it available as a flat file but I'm just too blind to see it?
- orf 3y agoThe website pulls it directly from the JSON “index” files in each mirror repo at build time, and there isn’t a metadata dataset with the versions yet. You can use the parquet indexes for this I guess, but it could be simpler. The problem is that it’s pretty huge and would need to be an artefact. I’ll have a think.
- albert_e 3y agoHow is Open AI API key the second biggest secret in repo already. Others like AWS and Google Cloud have been around and popular for decades! Also why no Azure alongside AWS and Google?
- orf 3y agoI can’t remember how I set this up right now, but I think it’s unique keys. And someone did manage to accidentally publish a master list of 500 individual keys at one point, which definitely boots the numbers. This analysis will be re-done with some other data soon.
- amethyst 3y agoGithub has automation to detect and mitigate repos with AWS/GCC API keys. They probably don't (yet?) have equivalents for OpenAI or other similar services that aren't as popular.
- judge2020 3y agoGH has the Secret Scanning Partner Program https://docs.github.com/en/code-security/secret-scanning/secret-scanning-partner-program https://docs.github.com/en/code-security/secret-scanning/sec... OpenAI is one of the supported platforms - A list of providers is at https://docs.github.com/en/code-security/secret-scanning/secret-scanning-patterns#supported-secrets https://docs.github.com/en/code-security/secret-scanning/sec...
- progmetaldev 3y agoAzure secrets works differently, and stores the keys outside of your project directory (traditionally). I'm sure someone could find a way to include them (and probably include multiple projects rather than a single project), but they would have to go out of their way to do this.
- deleted 3y ago[deleted]
- fbdab103 3y agoOne potential readability improvement - if the numbers in the tables were all right aligned. More controversial opinion - if you standardized all of the sizes onto either TiB or GiB. Anyway, very cool. I am shocked at how many header files are present. Another idea - how many unique files are there between releases and how many unique files are there total? Take a sha hash of every file, every commit. Calculate how many shas are shared between releases vs novel (ie # files churned per release). Can then also calculate on the global uniqueness over time. Of course, this means calculating billions of sha sums, so it could take forever, unless you had some cute trick to rip the value out of the git repos directly. Maybe you could even beat the odds and find that 1 in a quadrillion hash collision.
- orf 3y agoThe datasets contain the file hashes, so this is definitely something you can do already. https://play.clickhouse.com/play?user=play#U0VMRUNUIHByb2plY3RfbmFtZSwgc3VtKGxpbmVzKSwgc3VtKHNpemUpIEZST00gcHlwaSBXSEVSRSBza2lwX3JlYXNvbiA9ICcnIEFORCBwcm9qZWN0X25hbWUgSUxJS0UgJyVDbGlja0hvdXNlJScgR1JPVVAgQlkgcHJvamVjdF9uYW1lIE9SREVSIEJZIDIgREVTQw== https://play.clickhouse.com/play?user=play#U0VMRUNUIHByb2plY...
- Galanwe 3y ago<sarcasm>That walrus operator was really worth it and has seen a huge adoption</sarcasm>
- zX41ZdbW 3y agoI've uploaded the metadata to ClickHouse Playground for analysis. Example: https://play.clickhouse.com/play?user=play#U0VMRUNUIHByb2plY3RfbmFtZSwgc3VtKGxpbmVzKSwgc3VtKHNpemUpIEZST00gcHlwaSBXSEVSRSBza2lwX3JlYXNvbiA9ICcnIEFORCBwcm9qZWN0X25hbWUgSUxJS0UgJyVDbGlja0hvdXNlJScgR1JPVVAgQlkgcHJvamVjdF9uYW1lIE9SREVSIEJZIDIgREVTQw== https://play.clickhouse.com/play?user=play#U0VMRUNUIHByb2plY... This is a nice dataset for exploration, although small (only one billion records). Going to upload all lines of code...
- zX41ZdbW 3y agoAlthough the same query can be run directly from Parquet files with clickhouse-local: clickhouse-local --query "SELECT project_name, sum(lines), sum(size) FROM '*.parquet' WHERE skip_reason = '' AND project_name ILIKE '%ClickHouse%' GROUP BY project_name ORDER BY 2 DESC" https://clickhouse.com/blog/extracting-converting-querying-local-files-with-sql-clickhouse-local https://clickhouse.com/blog/extracting-converting-querying-l...
- orf 3y agoThanks, that’s really neat. I’ll see if I can embed it somehow onto the site. I was looking into some browser duckdb WASM setup that reads the parquet files via http but I ran out of steam.
- orf 3y agoI’ve embedded it on this page here: https://py-code.org/datasets https://py-code.org/datasets Thanks so much for that!
- KirillPanov 3y agoSorry, can you explain this? > Not all files can be committed to GitHub ... very long lines ... VCS directories Wait so github will reject a push if one of the commits contains a file with very long lines in it?
- FridgeSeal 3y agoMaybe it’s a protection for windows users? Windows has some arbitrary and reasonably small path limit IIRC.
- Thorrez 3y agoThe path length limit only applies to files and directory names, it doesn't limit how long the lines of text can be in the content of a file.
- jwilk 3y agoMaybe they meant very long names?
- orf 3y agoNot quite: the aim is to keep the size of the repositories within a manageable bound. This is achieved by heuristically excluding some text files that won’t compress well based on their content and size before committing. The “very long lines” exclusion is for text files that are very large but contain less than 5 or so lines. If I remember correctly there are some “py-armour” files that are basically just big dumps of base64 encoded python byte code on two or so lines. These are likely unique files and so won’t compress well, which bloats the size of the repositories. If you’re interested you can use the SQL console on the datasets page to take a look at the specific files.
- KirillPanov 3y ago> The “very long lines” exclusion is for text files that are very large but contain less than 5 or so lines. And github will block a push containing a commit which creates one of these files? And git itself does not?
- mianos 3y agoIs there some reason the largest space is used up by .so objects or is this just people going git add *?
- Jarobq18 3y agoApart from being easy to analyze, why is Python so interesting? It's a nice-to-play with language, very useful for researchers, but not practical for enterprise code. There are too many ways to do the same thing, not type safe, and I personally don't know many real python pros, the majority are just using python to play with.
- nine_k 3y agoReal enterprises like YouTube, Instagram, Spotify, and even Reddit would likely politely disagree.
- msm_ 3y agoNot all code is enterprise code. For the last 5 years of my professional career I use Python almost exclusively. >There are too many ways to do the same thing, (...) Fair, but that's a funny statement to make because Python from the start tried to have just one obvious solution for every problem. Maybe that's just what happens with languages over time. >not type safe Nitpick: Python is dynamically typed, but is actually quite type safe as used in practice (i.e. type errors are usually caught in runtime instead of silently doing the wrong thing). YMMV of course. >and I personally don't know many real python pros, the majority are just using python to play with. The beautiful thing about Python is that you don't have to be a pro to use it effectively :). And I think this may be a result of your professional bubble - for example I don't know any Java pros, but I have no doubts there are many.
- bagels 3y agoOne small change that can make any graph easier to understand: Label the units on the axes. I think that the y axis is # of projects, but I'm not totally sure.
- matt3210 3y agoI wonder how much duplication is there. I mean different code that does the same thing.
- orf 3y agoThis would be a really really interesting project to do with this data. You could maybe parse the files and run a similarity function of some kind on class and method bodies, ignoring identifier names.
- ivanjermakov 3y agoIf only function equivalence was decidable[1]. [1]: Rice's theorem https://en.wikipedia.org/wiki/Rice%27s_theorem https://en.wikipedia.org/wiki/Rice%27s_theorem
- orf 3y agoI've been thinking about this today. Function equivalence is one thing, but detecting similar code fragements is another? Like why not assign a character to each keyword: 'if' = 'i', 'while' = 'w', 'true' = 't', etc. Then reduce code to this, removing all whitespace comments and identifiers. so `if True: while True: pass` would become `itwtp`. then the similarity is just an edit distance between two given strings?
- 4sak3n 3y ago"PyPI is growing fast. If this dangerous expansion not stopped, our advanced machine learning models predict that in only 8 years the number of packages will outnumber human beings." This is one of the funniest things I've read all week.
- isoprophlex 3y agoThe WITNESS THIS INEVITABLE FUTURE button that slowly, then steadily started increasing the date on the graphs is executed very well. Truly hysterical, I laughed out loud.
- orf 3y agoThank you, I’m glad you appreciate it. I spent way more time on it than I would like to admit. The original version actually did run a small model trained on the data in the browser to generate the predictions.
- leroman 3y agoMaybe its forward thinking of the ML folk to create big data to train the models that will replace them
- nologic01 3y ago> If this dangerous expansion not stopped, our advanced machine learning models predict that in only 8 years the number of packages will outnumber human beings. Incredibly there seems to be no sign yet of an S-curve type saturation (which would be only normal at this point of the Python hype cycle). Would be interesting to control by repeating the analysis with other languages to see if that exponential growth is Python specific or really the exponential growth of all open source Maybe publish in JOSS?
- pletnes 3y agoPython’s growth is not limited in the number of software developers, as is the case for Java or what have you. It’s used by many other professions, and it just keeps exploding.
- zX41ZdbW 3y agoThis reminded me about https://github.com/cdnjs/cdnjs/ https://github.com/cdnjs/cdnjs/ - every version of every popular JS library in one repository - one of the largest repositories on GitHub by size.