15 ms·
I've compared nearly all Rust crates.io crates to contents of their Git repos
- jraph 2y agoGood initiative. Now people need to go through this and do the reviews :-) Next step would be to do reproducible builds (if it's not already the case).
- SubiculumCode 2y agoFirst pass gpt?
- SubiculumCode 2y agoHeavily down voted, which is fair because I didn't really explain what I meant, which was: Would using LLM's to parse the generated diffs, as a first pass, be useful/efficient for spotting and interpreting discrepancies?
- arccy 2y agowhen your goal is to improve security, the unreliability that comes with LLMs is not the answer.
- chipdart 2y agoI don't think this is a relevant take. Your goal is to implement a system to automatically scan countless packages and run a heuristic to determine if a package is suspicious or not. You're complaining about false positives/false negatives while ignoring that packages that not checking packages at all is not an improvement.
- Zambyte 2y agoPersonally I think using LLMs to scan is a good idea, but an honesty negative is a potential false sense of security. I think using LLMs here are useful for finding unintentional security flaws. I don't think it's a great tool to find intentional security flaws a la the xz situation. People might be less inclined to dig into the code directly if it was stamped with a green check mark by a GPT.
- Aeolun 2y agoNot necessarily, it reduces false positives. It just doesn’t do anything for false negatives (arguably makes the problem worse). If you just want to see if there is incidence of valid differences, this seems fine. But I wouldn’t use it as a guarantee.
- seoulmetro 2y agoNeither does grabbing yet another online third party's untested data?
- SubiculumCode 2y agoUsing machine learning, including LLMs, to detect and mitigate malicious code is of interest to a whole lot of smarter people than me, really suggests your flippant rejection of their potential is premature. https://arxiv.org/abs/2405.17238 https://arxiv.org/abs/2405.17238 https://arxiv.org/abs/2404.02056 https://arxiv.org/abs/2404.02056 https://arxiv.org/abs/2404.19715 https://arxiv.org/abs/2404.19715 https://www.sciencedirect.com/science/article/pii/S2666389923001241 https://www.sciencedirect.com/science/article/pii/S266638992...
- pornel 2y agoIt could work for classifying honest/innocent differences. However, LLMs are incredibly naive, so they could be easily fooled by a malicious actor (probably as easy as adding a comment that this is definitely NOT a backdoor).
- SubiculumCode 2y agoLLMs are broadly naive, but when fine tuned on a small domain of expertise/knowledge, this problem is less impactful.
- jonahx 2y agoNot a Rust dev so maybe a dumb question, but is this more involved than just running diffs? If so, what needs to be done?
- johannes1234321 2y agoThe key thing is interpretation of the diff. Is there a difference since they ran some code generator ins the crate contains generated code, not present in the repo or did they add a backdoor?
- thayne 2y agoMost of the diffs are probably innocuous. I suspect the most common diff would be the version line of Cargo.toml, both from CI that automatically updates that line, and people who forgot to update it before making a tag in git.
- mmastrac 2y agoAs someone with a crate that's in the 50MM plus range, this happens all the time. I really should automate this via a GH action.
- mberning 2y agoHow could you rank them for review priority? Use a combination of repo popularity multiplied by amount of significant differences? Where significant differences are determined by excluding non-code files?
- swiftcoder 2y agoLooking through the code, it already ignores the majority of non-code files
- pornel 2y agoI'd use popularity (how many people are using the crate indirectly) divided by trust level in the publisher of the crate. However, publishing a list that basically says "these are the least trustworthy Rust users" would cause quite a stir, so I'm not doing that.
- grahar64 2y agoDeterministic compilation is the best way to let people validate what they are downloading from repositories is what is in the codebases.
- xpe 2y ago> Deterministic compilation is the best way to let people validate what they are downloading from repositories is what is in the codebases. "the best way"? Please make the argument for why. To do it properly, you must steel-man the alternatives (not shoot down straw-men)
- miki123211 2y agoBecause deterministic compilation lets you (or someone else) do this automatically. If you introduce a backdoor into the compilation step, you run a much greater risk of detection. As long as there are multiple machines compiling packages and verifying whether the checksums match, any single backdoored machine will immediately be caught. This is much more important for package managers that do their own builds and ship their own binaries than for those who just ship whatever they got from the developer. Without deterministic compilation, two builds of the exact same code might differ. This makes backdoors very hard to detect unless you have prior suspicion that one is present in a particular program. Deterministic compilation forces people to embed backdoors directly in the source code repository, which creates an audit trail, is very visible in diffs, much easier to catch in reviews and so on. You can still get away with it (see the XZ situation), but it requires far more work.
- xpe 2y agoOk, but you didn't talk about alternatives. Why not just checksum or sign the source code? It is important to remember that crates.io doesn't store binaries.
- xpe 2y agoSee also a related comment in the overall thread saying "crates.io does not host compiled artifacts. If packages on crates.io differ from their Git repository it's because of a custom pre-build step of that particular package, so a deterministic compilation toolchain won't help here." Prove me wrong! I'm open to it. Or be that person who is too lazy to respond with an actual comment, downvotes, and probably assumes they are right.
- corytheboyd 2y agoHow crazy would it be to have a package repository that also builds the artifacts it distributes? You’d need a high barrier to entry to save on costs and time sifting through garbage. Perhaps it’s this high barrier that would prevent such a repository from taking off though. Perhaps this is just a really dumb step on a path leading back to simple checksum validations… though with those, you’re only validating that whatever was uploaded is what you downloaded, it doesn’t ensure that it was built from a known set of source files… hard problems.
- deleted 2y ago[deleted]
- kichimi 2y agoIsn't this gentoo?
- GauntletWizard 2y agoNo, Gentoo does something far from it - it builds everything on the host machine every time, more or less.
- KolmogorovComp 2y agocrates.io already builds the artefacts. But the code-source that is sent to crates.io is not necessarily the same as the one in the public repo linked to the crate.
- progval 2y agoDo you have a source for crates.io building artefacts? I have a couple of crates on it and never saw any sign it tried to compile them, even when they were broken.
- corytheboyd 2y agoAh yeah, I suppose that’s what I really mean, a means of verifying builds link to source that is publicly available. Sounds like the source repository has to be in on it too
- makeworld 2y agoThis is why I like what Go does, where you're downloading from Git directly (optionally proxied through Google, yes)
- jesprenj 2y agoRelated: Proxying can be disabled by setting the environment variable GOPROXY=direct [0]. I put it in my bashrc. [0] https://www.practical-go-lessons.com/chap-18-go-module-proxies#configuration-of-the-go-module-proxy https://www.practical-go-lessons.com/chap-18-go-module-proxi...
- Zambyte 2y agoThe fact that this isn't default is honestly the biggest thing I dislike about Go.
- kibwen 2y agoI'm not sure what is meant by "downloading from Git", I assume you mean downloading from Github. And Github is far less secure than what crates.io does, because crates.io is immutable (once published, uploaders can't change anything without opening a support ticket which will get rejected if they don't have a good reason), whereas Github history is trivially rewriteable. This means that if you rely on "v1.2.3" of a library from crates.io, that's always going to give everyone the same code; conversely, relying on a git tag of "v1.2.3" from a random Github repo could be anything at any point.
- fsmv 2y agoThe goproxy server makes it somewhat immutable for go too. Once they have a version cached they will never delete it. You can only supercede it with a new version and mark the old version as bad.
- estebank 2y agoTBH, "somewhat immutable" is not "immutable". The Go approach aids to limit the effects of an attempted attach where you're misled into building your project with different code than originally intended, but does nothing to guarantee continuity over time of dependencies being available. For that you have to rely on vendoring. The crates.io approach instead defends you both from a dependency changing silently and from it disappearing from one day to the next, without having to deal with vendoring.
- gigatexal 2y agoHas anyone analyzed the data?