orf

53 ms·

781.
by orf 3y ago
I took the code from their post, the one they highlighted as the issue, string allocations and all, and ran it. It was 1000x as fast as their self-reported runtime. Does this really, really not scream “there is another issue” to you? Do you
782.
by orf 3y ago
> But is the code shitty? It looks like boring normal code to me. I'm not sure what point you're making here. 1. It should never be that stupidly slow 2. That should be obvious to anyone with a napkin and some math 3. Excessi
783.
by orf 3y ago
They didn't debug anything. They just stuck a job queue on it and then rewrote part of it in Rust. That's the point of my comment. Using their code in the post (gist here[1]): $ pv lol.tsv | node lol.js 1.71GiB 0:00:07 [ 2
784.
by orf 3y ago
> having 25 Node.js instances running for three hours to parse 200GB of data seems like an indication that something isn't quite right. That’s about 700kb/s. I’m all for playing with rust, but this post strikes me as engineers
785.
by orf 3y ago
It does, but that’s accounted for in the chart (tensorflow- and tf- prefixes). That will include other projects, but it doesn’t significantly contribute to the overall % as they are dwarfed by the official projects.
786.
by orf 3y ago
CVEs are not what they once (ever?) where. With the proliferation of automated systems involved with vulnerability management and dependency updates, it’s possible for a single GitHub PR close event to create a CVE for an issue that verifia
787.
by orf 3y ago
A simple dark mode would be nice, and so would not clicking flag all the time by accident.
788.
by orf 3y ago
I've been thinking about this today. Function equivalence is one thing, but detecting similar code fragements is another? Like why not assign a character to each keyword: 'if' = 'i', 'while' = 'w&#x27
789.
by orf 3y ago
No, if you check above: "the aim is to keep the size of the repositories within a manageable bound. This is achieved by heuristically excluding some text files that won’t compress well based on their content and size before committing.
790.
by orf 3y ago
I’ve embedded it on this page here: https://py-code.org/datasets Thanks so much for that!
791.
by orf 3y ago
Not quite: the aim is to keep the size of the repositories within a manageable bound. This is achieved by heuristically excluding some text files that won’t compress well based on their content and size before committing. The “very long l
792.
by orf 3y ago
Thank you, I’m glad you appreciate it. I spent way more time on it than I would like to admit. The original version actually did run a small model trained on the data in the browser to generate the predictions.
793.
by orf 3y ago
This would be a really really interesting project to do with this data. You could maybe parse the files and run a similarity function of some kind on class and method bodies, ignoring identifier names.
794.
by orf 3y ago
The datasets contain the file hashes, so this is definitely something you can do already. https://play.clickhouse.com/play?user=play#U0VMRUNUIHByb2plY...
795.
by orf 3y ago
Thanks, that’s really neat. I’ll see if I can embed it somehow onto the site. I was looking into some browser duckdb WASM setup that reads the parquet files via http but I ran out of steam.
796.
by orf 3y ago
Oh no, that’s the first incarnation of this project. It scans the code without committing it. The project that leverages GitHub secret scanning found a lot more.
797.
by orf 3y ago
I can’t remember how I set this up right now, but I think it’s unique keys. And someone did manage to accidentally publish a master list of 500 individual keys at one point, which definitely boots the numbers. This analysis will be re-done
798.
by orf 3y ago
The website pulls it directly from the JSON “index” files in each mirror repo at build time, and there isn’t a metadata dataset with the versions yet. You can use the parquet indexes for this I guess, but it could be simpler. The problem is
799.
by orf 3y ago
Lots of dummy ones in tests for sure. I’m going to follow up with some deeper analysis on this topic in a few weeks, but you’d be surprised.
800.
by orf 3y ago
You can go check out the page linked in the post to see? That’s kind of why it exists! Spoiler: the answer is nightly builds, huge binaries and a lot of wheels per release.
801.
by orf 3y ago
Thanks for that, I can definitely appreciate this perspective. I’d say it’s more akin to uploading photos to a shared public host like imgur rather than Facebook, but regardless I can see how someone’s expectations of who/what would us
802.
by orf 3y ago
There is a section on the page that tells you (near the bottom): 16% of all uncompressed data in PyPI is from the Tensorflow project.
803.
by orf 3y ago
It's exception groups[1], with the syntax `except*`. 1. https://peps.python.org/pep-0654/
804.
by orf 3y ago
Thanks, fixed!
805.
by orf 3y ago
It's not that I didn't think "it was easy enough to use PyPI do analytics and analysis": it is near impossible for the layman to use PyPI to do analytics and analysis in it's current form. The volume of data is ve
806.
by orf 3y ago
They are within their rights to make that choice, but when you publish a package to PyPI you agree to their terms which gives anyone the right to mirror, distribute and otherwise use the code you’ve published.
807.
by orf 3y ago
I was in contact with them, and they are apparently OK with having the repositories be split up. There are over 230 of them each under 1.3gb in total size. I’m working on distributing this data without GitHub - git packfiles are a fantastic
808.
I mirrored all the code from PyPI to GitHub and analysed it(py-code.org)
238 points by orf 3y ago|122 comments
809.
by orf 3y ago
Have you ever been to France or Germany? That’s obviously and clearly very not true.
810.
by orf 3y ago
Just a note, if you are reading this post then click the logo in the top right, you cannot navigate back to the post - it immediately jumps to the homepage again. Safari on iOS.

More ›