7 ms·

The Case Against Third Party Libraries (2014)

42 points by adrianmacneil 11y ago

nulltype 11y ago
StableLib https://stablelib.com/ https://stablelib.com/ seems to be taking an interesting approach to libraries. Go guarantees backwards compatibility and it's awesome for developers. Why shouldn't libraries have the same guarantees? The standard library is pretty nice, but one of the reasons for that is that it is a well tested well supported backwards compatible library. It would be nice if there were more of those.
- acveilleux 11y ago
  It's pretty much a given that the first few libraries solving problem X will have a sub-optimal API, especially if they aim to be both general (say read and display JPEG) and full-featured (support all the optional bits of the format or work efficiently in low-memory embedded situations, etc.) at the same time. All of that while preserving the feel and idioms of the host language. This is key since it invalidates a lot of prior experience with similar APIs. So until the "right" API coalesces from collective experience with the problem domain and APIs for similar libraries (read and display {TIFF, GIF, PNG, etc.} in $LANGUAGE) backward compatibility is not a feature you want. Wait for the second or third wave of libraries where the problem is solved and understood in a nice idiomatic way for the language and its quirks.
  - nulltype 11y ago
    Perfect is the enemy of good I guess. If I'm building a product now, I want a library that works well now and never breaks when I upgrade to the latest version. If someone gets a great idea for a better library, make it a separate library. So backwards compatibility is definitely a feature I want.
alexcasalboni 11y ago
"Ctrl-C, Ctrl-V"? This is so wrong...
- simoncion 11y ago
  Yeah. If you're concerned about limiting the amount of additional code one needs to understand in order to use part of a library of code, then the fix is relatively simple: make each function in your library depend on as little of the rest of the library as is possible. If the author is legitimately concerned about code bloat from the unused parts of a library, Go's linker needs to get a lot smarter.
- pbreit 11y ago
  Wrong as in unethical or prohibited by common licensing? Or just your opinion of the tactic?
  - alexcasalboni 11y ago
    It's just against every software engineering best practices. You find a library which implements what you want/need? You don't copy&paste 10% of it into your repository just to make sure it won't ever brake. What if it's broken already? They will fix it and your version won't be updated. This logic is not even ok with your own code, why should it be promoted with somebody else's code?
- EliRivers 11y ago
  Exactly. Should be "Meta-W, Ctrl-Y" :)
veeti 11y ago
The Stockholm syndrome shown by Go users in things like this is just incredible.
- pbreit 11y ago
  What do you mean? It sounded sensible to me.
- zeveb 11y ago
  I think I agree. There are a lot of great things about Go, but (currently) library versioning is not one of them. This doesn't contradict much of what the author writes: it really is true that two different versions of one library are actually two different libraries; it really is true that sometimes copy-and-paste is better than generalising (when the cost of being general obscures the meaning of the code itself); it really is true that many libraries should just specify and implement an API, then stop. The issue with libraries in versions in Go could probably be resolved with a more flexible import syntax. It'd be sweet to be able to write 'import foo.invalid/bar@94d0ef312881e694ded9ff38e33a2c07fb398eb8' and know that I'd always either get the exact version that I specified. It'd be even cooler to have some way to specify 'I want the latest version of library X which offers the exact API I'm using'; in principal this would be possible by calculating some deterministic ordering of the used, exposed API and hashing that, but the details would be tricky (as an example, consider a case where Foo() and Bar() still exist, with the same signatures, but in version 2.0 Bar() can only run properly if Baz() has been called previously—there's no way in Go to express that sort of contract to the compiler.
  - deleted 11y ago
    [deleted]
    - TheDong 11y ago
      You'll already have to recompile because go is statically linked. The current state of the art is just vendoring dependencies (see godeps) which means they never get updated ever. Security updates don't happen now, and version pinning by vendoring is a hack, so it would be better to at least have version pinning built in so I can delete a few million lines of (vendored) code from my repository and instead just use a better import path. I assure you that that will be easier than dealing with the hell of vendoring tools and hacks go has now. And yes, the semver way to do it is great and is actually done well in some ecosystems. npm does a really good job of encouraging users to use semver and as a result you can just say "I want the latest 1.1.x release" or "1.x.x". The hardest problem there is to get people to version things. Your next point, that you don't want to recompile all go programs, is where I'll again point out go is statically linked. The authors of the language designed it so that no, you can't just update a go library in one place and build once. You have to build everything that depends on it again. You have to hope there were no breaking changes or you have to backport patches. Go does not handle this and is designed so as to be actively hostile to security. And yeah, the idea that semantic changes are backwards incompatible is a developer call, but you need a programmatic way to express it.... hence semver. In conclusion, having that sort of import-path thing would be awesome not because it solves any of the things you complain about (it doesn't) but because it's a step in the right direction. Any change is good change when your package management is "eh, fuck it" like Go's is now. If you look at all other good package managers (e.g. npm, rubygems, crate, etc) you can specify a repository and specific commit because there are plenty of valid cases for that, even when you have other options. No one is saying Go shouldn't have those other options in the future too here, just that having this option is a step in the right direction.
infinity0 11y ago
> That’s when I realized that a library with 30 versions is actually 30 separate libraries. Should be output to the screen in BLINKING ALL CAPS on every invocation of rubygems and npm.
- brightball 11y ago
  As somebody who spends a lot of time on a 100,000 line of Ruby monolith with upwards of 50 dependencies...I support this.
  - woah 11y ago
    50 dependencies is a pretty small number. My guess is your main issue is the monolith that has piled up. A pragmatic node.js developer would have been able to greatly reduce the size of your monolith by breaking out independent functionality and managing it with npm.
    - brightball 11y ago
      Oh, I didn't build it that way. We're far down the path of extracting microservices at this point but it was a fully entrenched monolith about 4 years ago.
  - grandalf 11y ago
    I am surprised Rubygems has not implemented some kind of semver helper to easily report which libraries have minor updates that can likely be trivially installed... The ideal utility would look at a Gemfile.lock and show changelog information to help the developer evaluate which libraries can be updated immediately, which require some code/api review, and which will require refactoring.
    - yo-code-sucks 11y ago
      It does do this.
  - aikah 11y ago

42 points by adrianmacneil 11y ago

StableLib https://stablelib.com/ https://stablelib.com/ seems to be taking an interesting approach to libraries. Go guarantees backwards compatibility and it's awesome for developers. Why shouldn't libraries have the same guarantees? The standard library is pretty nice, but one of the reasons for that is that it is a well tested well supported backwards compatible library. It would be nice if there were more of those.

acveilleux 11y ago
It's pretty much a given that the first few libraries solving problem X will have a sub-optimal API, especially if they aim to be both general (say read and display JPEG) and full-featured (support all the optional bits of the format or work efficiently in low-memory embedded situations, etc.) at the same time. All of that while preserving the feel and idioms of the host language. This is key since it invalidates a lot of prior experience with similar APIs. So until the "right" API coalesces from collective experience with the problem domain and APIs for similar libraries (read and display {TIFF, GIF, PNG, etc.} in $LANGUAGE) backward compatibility is not a feature you want. Wait for the second or third wave of libraries where the problem is solved and understood in a nice idiomatic way for the language and its quirks.
- nulltype 11y ago
  Perfect is the enemy of good I guess. If I'm building a product now, I want a library that works well now and never breaks when I upgrade to the latest version. If someone gets a great idea for a better library, make it a separate library. So backwards compatibility is definitely a feature I want.

alexcasalboni 11y ago

"Ctrl-C, Ctrl-V"? This is so wrong...

simoncion 11y ago
Yeah. If you're concerned about limiting the amount of additional code one needs to understand in order to use part of a library of code, then the fix is relatively simple: make each function in your library depend on as little of the rest of the library as is possible. If the author is legitimately concerned about code bloat from the unused parts of a library, Go's linker needs to get a lot smarter.
pbreit 11y ago
Wrong as in unethical or prohibited by common licensing? Or just your opinion of the tactic?
- alexcasalboni 11y ago
  It's just against every software engineering best practices. You find a library which implements what you want/need? You don't copy&paste 10% of it into your repository just to make sure it won't ever brake. What if it's broken already? They will fix it and your version won't be updated. This logic is not even ok with your own code, why should it be promoted with somebody else's code?
EliRivers 11y ago
Exactly. Should be "Meta-W, Ctrl-Y" :)

veeti 11y ago

The Stockholm syndrome shown by Go users in things like this is just incredible.

pbreit 11y ago
What do you mean? It sounded sensible to me.
zeveb 11y ago
I think I agree. There are a lot of great things about Go, but (currently) library versioning is not one of them. This doesn't contradict much of what the author writes: it really is true that two different versions of one library are actually two different libraries; it really is true that sometimes copy-and-paste is better than generalising (when the cost of being general obscures the meaning of the code itself); it really is true that many libraries should just specify and implement an API, then stop. The issue with libraries in versions in Go could probably be resolved with a more flexible import syntax. It'd be sweet to be able to write 'import foo.invalid/bar@94d0ef312881e694ded9ff38e33a2c07fb398eb8' and know that I'd always either get the exact version that I specified. It'd be even cooler to have some way to specify 'I want the latest version of library X which offers the exact API I'm using'; in principal this would be possible by calculating some deterministic ordering of the used, exposed API and hashing that, but the details would be tricky (as an example, consider a case where Foo() and Bar() still exist, with the same signatures, but in version 2.0 Bar() can only run properly if Baz() has been called previously—there's no way in Go to express that sort of contract to the compiler.
- deleted 11y ago
  [deleted]
  - TheDong 11y ago
    You'll already have to recompile because go is statically linked. The current state of the art is just vendoring dependencies (see godeps) which means they never get updated ever. Security updates don't happen now, and version pinning by vendoring is a hack, so it would be better to at least have version pinning built in so I can delete a few million lines of (vendored) code from my repository and instead just use a better import path. I assure you that that will be easier than dealing with the hell of vendoring tools and hacks go has now. And yes, the semver way to do it is great and is actually done well in some ecosystems. npm does a really good job of encouraging users to use semver and as a result you can just say "I want the latest 1.1.x release" or "1.x.x". The hardest problem there is to get people to version things. Your next point, that you don't want to recompile all go programs, is where I'll again point out go is statically linked. The authors of the language designed it so that no, you can't just update a go library in one place and build once. You have to build everything that depends on it again. You have to hope there were no breaking changes or you have to backport patches. Go does not handle this and is designed so as to be actively hostile to security. And yeah, the idea that semantic changes are backwards incompatible is a developer call, but you need a programmatic way to express it.... hence semver. In conclusion, having that sort of import-path thing would be awesome not because it solves any of the things you complain about (it doesn't) but because it's a step in the right direction. Any change is good change when your package management is "eh, fuck it" like Go's is now. If you look at all other good package managers (e.g. npm, rubygems, crate, etc) you can specify a repository and specific commit because there are plenty of valid cases for that, even when you have other options. No one is saying Go shouldn't have those other options in the future too here, just that having this option is a step in the right direction.

infinity0 11y ago

> That’s when I realized that a library with 30 versions is actually 30 separate libraries. Should be output to the screen in BLINKING ALL CAPS on every invocation of rubygems and npm.

brightball 11y ago
As somebody who spends a lot of time on a 100,000 line of Ruby monolith with upwards of 50 dependencies...I support this.
- woah 11y ago
  50 dependencies is a pretty small number. My guess is your main issue is the monolith that has piled up. A pragmatic node.js developer would have been able to greatly reduce the size of your monolith by breaking out independent functionality and managing it with npm.
  - brightball 11y ago
    Oh, I didn't build it that way. We're far down the path of extracting microservices at this point but it was a fully entrenched monolith about 4 years ago.
- grandalf 11y ago
  I am surprised Rubygems has not implemented some kind of semver helper to easily report which libraries have minor updates that can likely be trivially installed... The ideal utility would look at a Gemfile.lock and show changelog information to help the developer evaluate which libraries can be updated immediately, which require some code/api review, and which will require refactoring.
  - yo-code-sucks 11y ago
    It does do this.
- aikah 11y ago