14 ms·
Always Review Your Dependencies, AGPL Edition
- fenomas 7y agoNot sure about this bit: > This is quite a bit of work, but is necessary to avoid falling victim to attacks like _event-stream_. Reviewing dependencies is important, but I don't think anything the author mentions would have made a difference with event-stream. The whole issue there was that malicious changes were snuck in via a change of maintainers and a later update to a child dependency, so when people initially adopted it as a dependency there were no red flags in the library to find.
- square_usual 7y agoHe talks about reviewing dependencies when they are updated, as well, which certainly would help if someone snuck in malicious code in a minor version change.
- fenomas 7y agoWell, he talks about reviewing when he upgrades a dependency, but the tricksy thing about event-stream was that people could get the malicious change without having intended to upgrade anything. The only thing that really prevents such issues is version locking of transitive dependencies (which the author doesn't mention, but it could be that his package manager does it by default, or similar..).
- colonwqbang 7y agoI had missed that npm doesn't (didn't?) lock package versions by default. That's really scary.
- fenomas 7y agoIt does today, and I could be crazy but I think it did even before event-stream. But of course some users will have been on old versions of npm, or not checking in their lockfiles, etc. I should add, I was probably wrong to talk about lockfiles preventing such issues. With event-stream the malicious code was hidden deviously enough to evade a pretty rigorous check - if a Node update hadn't deprecated one of the functions used in the payload, I suppose we might still not know about it. In such cases a lockfile is at least a layer of defense, but naturally it only helps if you're lucky enough to have installed the library before it got corrupted..
- brian-armstrong 7y agoIt seems like this is a good opportunity for Github to warn you if a) a PR would merge a new dependency with an incompatible license to yours and b) allow you to filter out projects based on licenses from search results. in general most of us would prefer to avoid being tainted by GPL code and it'd be great to hide it on the site entirely.
- choeger 7y agoThat's the hidden cost of npm, cargo, pip, et. Al. The other one is IMO akin to overweight. Try to modernize a mid-sized project after one or two years and cry when you see the dependency graph. Ceterum censeo go inferior est.
- atoav 7y agoThat is exactly why I always ask myself: Do I really need dependency X? Because that dependency might itself have dependencies and this quickly grows out if hand with different versions etc. It might work now, but will it in the future? How many different versions of the same package do I really need to depend on?
- pbhjpbhj 7y ago>Ceterum censeo go inferior est. AFAICT this means something like "furthermore, I opine that Go[lang] is inadequate". It's some weird reference to an ancient Roman who always went on about destroying Carthage ("Carthago delenda est"). You, choeger, get the award for "weird reference I bothered to chase up". Aside, I wonder if PM Johnson (a Classics scholar) had this in mind when his handlers instructed him to mimic Trump's rhetorical technique of "ignore the question just brainwash the suckers into associating you with this 3-word phrase".
- valvar 7y agoI'm not sure if you are being ironic or not, but Cato the Elder is an extremely famous roman figure. I think most educated people, especially on HN, are familiar with his most famous phrase even if it is not in the most common formulation.
- squiggleblaz 7y agoWell I've heard of him. But I suspect you greatly overestimate the importance of dead Romans to contemporary programmers. It is no doubt possible to be a good, English speaking programmer, without knowing anything much about the Roman empire and its inhabitants, if you come from the east Asia for instance. Even for those from the West, there are cultures which do not rate Ancient Rome as so significant that it is necessary for an educated person to be familiar with Cato the Elder's most famous phrase, whatever it may be.
- mattigames 7y agoIt's always been weird to me how Microsoft invests so much in making JavaScript easy to develop and maintain with TypeScript but does so little to make it safe, the one thing JS needs is a standard library by Microsoft (or similar, e.g, Google) that we can trust, aiming to significantly reduce the number of dubious-origin dependencies of every JS project (on node and the browser)
- kijin 7y agoYeah, remember the left-pad incident? It took until ECMAScript 2017 to make such a simple function part of the standard library. Things are slowly getting better, but JS is still rather lacking compared to "batteries included" languages like PHP and Python.
- tastroder 7y agoThis comes up pretty often, what would you like to see in such a standard library? I consider the libraries included in Node pretty extensive these days.
- mattigames 7y agoNode is a tiny part, the big fish is JavaScript in the browser
- tastroder 7y agoThe person I replied to compared the stdlib to that of PHP and Python, both of which are backend components. Frontend land is getting frequent updates already, with a relatively good process, vendor backing and incubation in browsers, not sure what the complaint is supposed to be there? That it's too slow? Imho that's a good thing when it comes to most language changes.
- mattigames 7y agoYeah is too slow, but is not only that, is that it makes senses for it to exist years on user-land as a library (or multiple ones) before being implemented on any standard.
- sansnomme 7y agoPackage managers need to automatically derive properties of end builds based on licenses. E.g. Eclipse License 2.0 without the presence of another more liberal license means it cannot be used in copyleft software, any dependency that is copyleft is also infectious etc. Of course it won't account for every single legal property but the basic checks should be done. To prevent work duplication a single binary/library written in Go/Rust can take care of this problem and be used across all package managers. It is in the interest of GitHub to sponsor such a project.
- hlieberman 7y agoThis is one of the services that distros offer, for what it's worth. Every package in Debian is checked for copyright compliance against the DFSG, and annotated with the licenses that are controlling for that code.
- 256dpi 7y agoThe licensing of a dependency could be easily determined programmatically (GitHub already built a decent scanner). However, I think that the quality of a dependency is most important and that requires a manual vetting process. A trivial solution would be to create a crowd-sourced dependency vetting platform.
- kijin 7y agoGitHub's license detection algorithm is crap. It tends to get GPLv2 right, but most other licenses are hit-and-miss. And I still can't find a setting where I can manually specify the license when GitHub can't autodetect one or gets it wrong.
- rurounijones 7y agoRuby gems have the license as an explicit field in the gemspec (manifest file). Is this not the case elsewhere? It makes license scanning a doddle.
- 7y ago
- povils 7y agoI think you would end up in a rabbit hole. Do you also review your all GNU/Linux libraries and dependencies? Probably not because you trust them. Thus I think we should be pragmatic and review only libraries which are created by unknown/untrusted creators.
- atoav 7y agoWell I think you should at least do two things: - avoid dependencies where you reasonably can. Less moving parts are usually good - just have a look at the dependencies of your dependencies - this might help you decide which one to trust
- marcus_holmes 7y agoOpen-source projects can and do change maintainers. Adding a dependency means you not only trust the maintainer now, but you also trust all future maintainers of the project. Dependencies are more dangerous (in this sense) because they compile into your application, so they can do anything they like to your customer data. A malicious tool could monitor your keystrokes and phone home, but it won't get installed on your production server. A further problem with npm dependencies is that they get told when they're operating in dev mode and when in production mode. So malicious code can hide itself during dev and test, and then only do the bad thing on the production server.
- progval 7y agoDebian packages are maintained (and quickly vetted) by a rather small group of people (who vet each other). npm packages can be published by anyone without review. On a somewhat related topic: https://drewdevault.com/2019/12/09/Developers-shouldnt-distribute.html https://drewdevault.com/2019/12/09/Developers-shouldnt-distr...
- samat 7y agoI would love to have this guy write my security sensitive software.
- AmericanChopper 7y agoI used to work with a guy that would review every line of our projects node modules. He’d find issues all the time, so there’s no way he wasn’t pay attention. He also managed to be productive with writing his features. Having that guy on the team was amazing, I have no idea what motivated him to do such tedious work, but he seemed to love it...
- atoav 7y agoWith a bit of electronics background this feels like the difference between a hobbyist and a professional. Most programmers feel like hobbyists to me, in their YOLO-approach. Your software might ruin someones day, their life or maybe even end up killing people and more people should take that thing seriously. The way your collegue worked should be the norm.
- TeMPOraL 7y agoI'd think the comparison goes the other way around :). Hobbyists care enough to do things right even if it's not in the short-term interest. Professionals, judging by all the advice I read on-line, are supposed to focus on delivering value - which is usually measured short-term, and not aligned with doing things right. It's that attitude that makes most companies care little to none about security. Bringing in tons of dependencies in order to increase velocity and implement user-facing features faster is what I see advertised as archetype of a professional in this industry.
- atoav 7y ago> I'd think the comparison goes the other way around :) Only in the Software industry. This is however not normal: * Mechanical Engineering: Hobbyist bridge vs. professional Bridge – which one should you be able to trust more to carry you? * Electrical Engineering: Hobbyist wall wart vs. professional wall wart – which one should you be able to trust more not to burn your house down? * Medical Treatment: Hobbyist vs. professional – which one do you trust more with your body? The list goes on. Professional work isn't about cobbling together something that barely works at the edge of complexity you can just about manage. Professional work is building something reliable, robust that you can guarantuee for at least to a certain degree. I know that this isn't how Software Development works right now in most places – that was the whole point of my comment. However it is not normal. It is a bit like with early cars: back then a hobbyist could have easily made a safer car than any professional manufacturer with the right amount of commitment, because there were no real safety standards. Try doing that nowadays. Edit: Good related talk by the legend himself (Ross Anderson): https://media.ccc.de/v/36c3-10924-the_sustainability_of_safety_security_and_privacy https://media.ccc.de/v/36c3-10924-the_sustainability_of_safe...
- Vizarddesky 7y ago> I repeat the above recursively on transitive dependencies as many times as necessary. I also repeat the cursory code review any time I upgrade a dependency. If this guy has to work on a "modern" frontend project, he's gonna review dependencies until the heat death of the universe.
- fhars 7y agoBut doesn‘t that say more about modern leftpaddable frontend frameworks than about the author?
- deleted 7y ago[deleted]
- taneq 7y agoIt sure does. ‘Don’t repeat yourself’ and ‘avoid NIH syndrome’ are noble goals but ‘automatically update myriad libraries from random sources on the internet and then run them’ gives me the heebie jeebies.
- kerkeslager 7y agoOkay, but let's be clear about why we're putting "modern" in quotes. Sure, pulling in half of NPM is a common way to do things currently, but it's also a very painful way to do things, currently. A "modern" dependency tree is going to cause you tons of pain, starting with having to configure your dependency tree and getting worse from there. If you use a few, small, effective dependencies, you can reasonably do a cursory code review with every upgrade, and there are other major benefits. Don't let some drive to be "modern" cause you to use libraries that make things more difficult than using vanilla JS.
- josteink 7y agoWhat this guy does sounds like a machine-doable job. At our company we use whitesource to scan each and every build for these kind of license violations.
- randomsearch 7y ago
- herpderperator 7y agoWhen this webpage loads, it briefly flashes in plaintext before loading CSS - at least in Chrome. This is unusual since it has the CSS stylesheet in the head which I thought would block until it's fully loaded. Does anyone know what's going on with that?
- C14L 7y agoOnly <script> blocks by default, <link rel="stylesheet"> doesn't.
- holdenc 7y agoThis is what happens when you dynamically insert your CSS tag in the head after the DOM loads.
- herpderperator 7y agoIf you view the raw data coming from the web server, the link tag containing the stylesheet is already present in the head. Although what you're saying sounds like it can absolutely cause this type of behaviour, I'm not sure if that's what's happening in this case.
- holdenc 7y agoYou are correct. I suppose it could be an unreviewed NPM dependency causing this ;)
- zuzun 7y agoI guess render-blocking isn't really a thing, the HTML spec only describes script-blocking and is light on the rendering aspects. Now most web pages have styles and scripts and since styles block scripts, you usually perceive styles as render-blocking. But this web page doesn't have any scripts, so the browser reaches DOMContentLoaded right after parsing the HTML and does the first paint while the request for the style is still in flight. At least that's my theory. You could test this by inserting an empty <script> tag into the <body>, which should delay DOMContentLoaded until the style arrives and prevent the flash of unstyled content.
- rubenbe 7y agoThis is one of the items embedded Linux development using Yocto has particulary well covered. It checks every license for every package and you can ask it to verify the checksum of the license file. If the license changes after upgrading a package, you'll get a build error.
- correct_horse 7y agoWhy don't npm-like package managers have settings for licenses in applications (as opposed to libraries)? Settings like "no AGPL" or "no copyleft dependencies" would allow easy vendoring with modifications. This (disabled by default) feature might break some proprietary code, but if it does, that indicates you were not following copyright law prior. Obviously this doesn't solve the general quality problem with dependencies that the author notes, but it fixes some licensing issues.
- arnon 7y agoNode's popular `pm2` itself is AGPL. I don't think they're too fussed about copyleft.
- correct_horse 7y ago> I don't think they're too fussed about copyleft. Yeah, and I'm not either. I have never decided to vendor dependencies, and my only modifications to GPL code were to upstream (so I didn't have a legal obligation to distribute the modified code - the distributors of the code I modified do). I just think automated license compatibility checking could be a good move for the ecosystem. Also to be clear, I was talking about npm-like ecosystems. In my mind this includes npm, poetry and cargo (and probably more).
- oaiey 7y agoI hope on your comment you do not mix up LGPL and GPL.
- baybal2 7y agoNode has tons of AGPL packages which seem to be uses by quite a lot things. I myself saw a pm2 error message in Linkedin's error 500 output
- oaiey 7y agoA GPL or AGPL software should not exist on NPM for libraries. For tools yes, but GPL for libraries is for 99% of the users a non-starter. It just does not belong into a library ecosystem. To be clear: everyone has the free right to license their stuff how they like. I just think NPM as a ecosystem should be opinionated (and it is... Towards MIT. Just not completely).
- cygned 7y agoI try to convince my team that the node.js ecosystem has gotten into a stage where it cannot be used for security/financial applications because the sheer amount of dependencies pose an inherent threat. I advocate for Go because of the tendency to less and easier reviewable dependencies. Nobody except me seems to see a problem there, despite me being able to point out specific security incidents. I am wondering if I am missing something obvious here and would value any opinion.
- lowdose 7y agoTJ Holowaychuk said good bye to node in 2014 and switched to go. So your not the first. https://medium.com/code-adventures/farewell-node-js-4ba9e7f3e52b https://medium.com/code-adventures/farewell-node-js-4ba9e7f3...
- eitland 7y agoThere's also Java and .NET for those who want better IDE support, better dependency management (without recursively downloading hundreds of packages), and a more traditional language and more fully featured language. PS: I came to statically typed languages as an adult, I actually disliked those languages before I had the lightbulb moment, so I think I might even be more qualified than everyone who hasn't managed to enjoy both sides ;-)
- pomhellxml 7y agoWhilst the dependency management story on Node, Python is sub-par, it’s not all roses in Java land either with Maven and POM file XML-hell.
- guipsp 7y agoUse Gradle
- eitland 7y agoActually, again as someone who has some real life experience with a number of different systems, including - pip for Python, - NPM, - Nuget - and Maven (and Ant) Maven is by far my favorite, despite XML. In fact, give how small pom files are and how little you have to deal with them (if you know what you are doing) I find it amazing how many comments I have had to read about "XML-hell" etc.
- oaiey 7y agoI am doing similar analysis within my company. While I usually never dig that deep asking why something is AGPl or GPL, the amount of careless managed dependency trees within the NPM ecosystem is astonishing. It is not only about licenses, but also code quality and vulnerabilities. I came to the absolute same conclusion as the author: focus on platforms where there is a reasonable standard library. That is .NET in my case and Go in his case. These product dependency trees are much (much) cleaner. I ended up with writing software to analyze the dependency trees licenses for our browser based products
- marcan_42 7y ago> The bulk of my trust is consolidated in the Go project, and thanks to their stellar reputation and solid operating procedures, I don't feel a need to review the source code of the Go compiler and standard libraries. Well... I have reviewed the Go runtime, after I ran into a bug in it... And that's one of the reasons I no longer use Go unless forced to. That thing is ugly under the covers. There is seriously quite a bit of insanity and poor design going on, e.g. when they did the ARM port they had to make a ton of changes to data structures/functions to include the link register, because they didn't or couldn't abstract out architecture specific tidbits like that. Also, the whole C interop stuff is nuts with a ridiculous call chain and several stack switches every time you call to or from C code (this is where I found the bug). Not saying people shouldn't use it, but I wouldn't hold the deep guts of Go up as an example of stellar development practices. Maybe all the stdlib stuff on top is prettier, but the runtime sure isn't.
- self 7y ago> That thing is ugly under the covers. Was the code you looked at in Go 1.5 and above? Is it cleaner in 1.4? They used automated tools to convert the C code into Go for 1.5.
- marcan_42 7y agoThey were halfway through the process when I made my contribution, and in fact I had to rewrite it in Go. The Go version was just a straight conversion; being Go didn't make it inherently cleaner.
- zeveb 7y agoI think his point is that it stands to reason that machine-converted C→Go code may be pretty ugly. Maybe it was better when it was clean, Plan-9-style C? Maybe it is better now, after some number of years of cleanup?
- Blackstone4 7y agoSo would you use?
- fmajid 7y agoThe problem is reviewing dependencies is labor-intensive, thus expensive. The economics would justify that being a paid service, like UL/TÜV certification for electronics, but most companies don't actually care enough about security to pay for such a thing.
- pythonist 7y agoAnd also very important to watch for changes of your dependencies, as they can be quite dynamic. A single review too early or too late in the development process may have misleading results. Watching for commit changes is too time consuming, so watching for releases is a bit easier, especially if you use something like https://newreleases.io https://newreleases.io or other similar sites.
- henvic 7y agoSlightly related: I have created this useful tool called vendorlicenses that allows you to check and concatenate licenses on Go programs (to give credit on a CLI "legal" command). It will only work if you are vendoring your packages though (and you should!). https://github.com/henvic/vendorlicenses https://github.com/henvic/vendorlicenses
- saluuu 7y agookay , great advice .
- generationP 7y ago> Aside: this is why I don't like to accept pull requests that move code around. Even if the new code organization is better, it's usually not worth the time it takes to ensure the pull request isn't doing anything extra. This sounds like a solvable problem no one has bothered to solve. We need an analogue of diff highlighting for move-around changes, ideally one that decomposes a changeset into the coarsest block partition such that the changes boil down to a permutation of the blocks. Something similar should be done for merge commits, which at the moment are completely undebuggable.
- robbya 7y agoFor changes that are a pure file rename 'git mv' tracks those pretty cleanly. Anything beyond that... I don't know of any good tooling either.
- saagarjha 7y agogit mv doesn't actually do anything special to track moves, so it can't figure out anything that's non-trivial.
- progval 7y agoMercurial goes a bit further by tracking copies, so if you split a file in two, a diff viewer will show there is no added code (but a lot of deletions). https://stackoverflow.com/a/4156146/539465 https://stackoverflow.com/a/4156146/539465
- baybal2 7y agoNPM ecosystem has quite similar situation. A lot of popular packages being used in complete contravention to their license terms, and care not at all their dependency chains pulling quite restrictive licenses.
- vemv 7y ago> It started off poorly when I noticed some Brown M&M's: despite being a library, it was logging messages to stdout In many language ecosystems, logging to stdout tends to be the right thing to do in a lib as it does not impose any specific choice of logging implementation. One always can recapture stdout and route it through one's logging solution of choice; doing something analog with an extraneous logging library can be harder if not unfeasible. Are things different in Go land?
- robmccoll 7y agoIn what language or environment would recapturing and routing stdout be a desirable solution? Seeing that without a really good explanation would concern me. That's a global hack with fun side effects that can lead to very confusing bugs. Also, you would lose which library is which log if you have to do this for more than one library. In Go, I would assume that a library that needs to log would do one or more of the following a) offer the ability to disable logging b) offer overriding the output that it uses for logging probably with an io.Writer interface or something c) accept a log.Logger or offer an interface that something like log.Logger meets. Ideally it would offer an interface that supports some key-value meta data to be able to have nicely structured logs with formatting independent of the library.
- paulmd 7y agologging to stdout prevents any sort of systematic parsing or handling of stack traces, package names, or error levels, cause people are going to dump whatever shit they personally find most useful onto stdout and everyone's going to use a different format. That is a far far more "analog" solution than using any logging API... or even multiple ad-hoc logging APIs. The idiomatic solution is to have a standardized facade like SLF4J and then behind the scenes the "end user" (final developer writing the application) can choose their own logging backend to drive the facade. The slightly less idiomatic way is that everybody does their own thing and then you use bridge libraries to hijack all of the various logging APIs and redirect them through SLF4J or through your logging backend of choice.
- jbboehr 7y agoAs a library, if you just have to avoid logging frameworks, at least log to `stderr` so you don't break tools that parse output.
- ones_and_zeros 7y agoJohnny come lately but is that true of the AGPL, that merely linking to agpl code infects your code? No modification to the agpl code needed to trigger the clause? What is the point of having a license like that? Trolling?
- paulmd 7y agoYes, that is generally how copyleft (GPL) software works. Linking against it confers a requirement that your source also be distributed under a compatible license. This is why it's called a "viral" license. AGPL was intended to close the SaaS gap in copyleft software. Essentially, prior copyleft provisions don't kick in if you never distribute the software itself, so if you only offer network interaction with the software you have no duty to redistribute. AGPL is intended to restore the freedom of end users to have access to the source of the software they interact with, even if it lives on someone else's computer. As with all copyleft software, it does this by restricting the rights of the developers of the software (to keep their source proprietary). As always, it is the old philosophical split between MIT/BSD style licenses and GPL style licenses. MIT/BSD provide you with complete freedom, including the freedom to use that library and keep your source closed. GPL is intended to prevent that and to build a common base of software on which further things can be built. (there is also Lesser GPL license, which only kicks in for modifications to the library itself. So if you write something that uses a LGPL library, you don't have to give away source for the larger program as a whole, just the LGPL program and any modifications you made to it. This is essentially removing the "viral" copyleft provisions.)
- robmccoll 7y agoOn the licensing front, one thing that has really helped me is adding a script to my build / CI process that scans every package I'm dependent on for a LICENSE / COPYING / etc. file and then runs through a set of dumb fingerprints to match on different licenses for a combined whitelist / blacklist approach. Packages with no recognized license file or license are rejected. Then it spits the licences out into a map[string]string in a file where the Go package maps to the license string which gets served up on its own HTTPS endpoint to keep in compliance with copyright notice requirements. In case it's useful to anyone else, here's an older copy of it: https://gist.github.com/robmccoll/240317eceb73e3f4e29ea662e3ea4063 https://gist.github.com/robmccoll/240317eceb73e3f4e29ea662e3...
- dwheeler 7y agoI completely agree that you should run 1+ license checkers in your CI process, just like you should check anything else in your CI process that you care about. In the CII Best Practices badge project we use both the "license_finder" program (which is an OSS tool) and the FOSSA service ( https://fossa.com/ https://fossa.com/ ). Both examine the dependencies for licensing issues. While something could still slip through, it's much less likely, and more importantly we have a really good case for showing due diligence. I'm not a lawyer, but I do know that courts look very favorably on people who are demonstrably making an effort to meet their legal obligations. You're way less likely to have legal problems that way.
- mcguire 7y ago"I just finished writing a vastly simpler, attestation-free library which is less than one tenth the size (I will open source it soon - watch this space)." ... "Even though I love Rust, I am terrified every time I look at the dependency graph of a typical Rust library: I usually see dozens of transitive dependencies written by Internet randos whom I have zero reason to trust." I have some kind of point to this comment, but I'm trapped in a sudden feeling of professional malaise.