5 ms·

My issue is that the default behavior is to install these libraries globally, even when they could probably be installed locally for the thing you’re trying to

My issue is that the default behavior is to install these libraries globally, even when they could probably be installed locally for the thing you’re trying to build. (I picked libwebsockets out of the air, I don’t necessarily have a complaint with that particular library) On my old MacBook Pro I have dozens of node projects that I’ve downloaded over the years just to mess around. If I ever want to clean up, I can delete a project’s folder, and know that everything it installed is now gone (because it would be in its node_modules folder). On my linux desktop if I want to mess around with a cool project that’s written in C/C++ I immediately pull out Docker and start trying to build up an image, since I don’t want to pollute my global library folders with libraries that I might be downloading just to try out this one thing. I can then delete the docker image if I want to clean up. This works, but it feels like a workaround to the main problem of libraries being global.

inimino 7y ago
You seem to be confused about how typical C/C++ projects are built. If you need globally installed dependencies, you're going to have to install them yourself. A typical makefile is not going to download a long list of random junk from the internet and install it your machine without asking. Although you'd be forgiven for thinking so if you're used to npm. If you don't like where the project installs its binaries or libraries, you can configure that before `make` or `make install`. You can install it to your home directory, have a separate directory tree for experiments, or whatever you want, and clean it up whenever it is convenient. You can also install things in usr/local and just ignore them. The system is actually well designed if you learn how to use it. If you need a dependency, use your package manager to install it and you can trivially delete it later... and if you try five different frontends for the same library, you only install one copy of it. Docker is a workaround for a lack of Unix awareness and sysadmin experience.
- spion 7y ago
  The system is not well designed. For example: if I have 3 projects I'm developing locally, each needing a dozen (somewhat overlapping) libxxx-dev dependencies to build, how can I automatically clean up the dependencies that are no longer needed when I delete one of the 3 projects? Distro package managers are focused on providing a good experience for sysadmins while ignoring needs during development time. Unfortunately, docker and the developer-centric improvements in language package managers have come at the expense of worse sysadmin experience. The situation will remain sub-optimal until both groups start working together to address both needs.
  - inimino 7y ago
    If you've got some leftover libraries in /usr/local/lib on your development machine because of an experiment, why is that the end of the world? If you're worried about it, you shouldn't be. The worst thing about Docker is that people think learning Docker is a substitute for basic competence as a sysadmin.
    - spion 7y ago
      I'll put it in context. Let's say I have leftover garbage in usr/local/lib. I clone a new project that I need to patch because it's broken. I try to fix the new project but the fix isn't working. I have no idea why. I spend hours trying to trace down the problem. What was the issue? The project picked up some of the old garbage libraries I had in usr/local/lib, which had a bug that made my fix not work. For developers, global dev library garbage = global mutable state. It interferes with development, creates more sources of potential issues, and takes time to trace and debug. A clean and/or localized environment is essential for the accurate pinpointing of bugs and issues.
      - inimino 7y ago
        > I spend hours trying to trace down the problem. This is the problem. Once you understand library loading, once on your system, and learn, once, how to query the system and see what libraries are being loaded and from where, you won't have to waste hours on this problem again. Until that point, you're operating a system without understanding, and that will always lead to breakage.
        saagarjha 7y ago
        I've installed hundreds of pieces of software from source, and I have a fairly decent idea of how linking and loading works. I still have to spend time figuring out why something doesn't build when it should due to something strange about my system.
        spion 7y ago
        Of course I understand how it works. I just can't constantly keep in mind what garbage I played around with 3 months ago that might be responsible for the issue I'm looking at right now. I don't expect it, and more importantly, I shouldn't have to expect it. It's a basic tenant of development. Eliminate all additional sources of issues and complexity by presenting a clean, well-controlled sandbox in order to effectively diagnose it and fix the real bug. In fact, you should be able to try a newer version of a dependency, an older one, or run 10 variants (containers) each with different versions to e.g. test whether the project you're working on is widely compatible with a flexible range of that dependency. These things are done all the time. The tools are inadequate, not the developers. Engineers on both sides need to stop assuming everyone else is an idiot.
spion 7y ago
The benefit of global libraries is that they are easier to update or patch. If there is a vulnerability in an npm package, you will need to update it in all your node projects (and/or rebuild all your docker images analogously). A hybrid solution where the package manager is aware of all installed versions, can symlink to a globally installed copy and can update all symlinks would probably work. Unfortunately, there is very little interest in fixing the problem on both sides (distro PMs and language PMs) since they both believe their own approach is right and don't care about the other group's problems.
- pas 7y ago
  yarn uses a global cache and symlink if I remember correctly. and npm has an "audit" feature, which is default on, so there's quite a bit of movement in JS land to address the disadvantages of having a myriad small packages.
  - spion 7y ago
    Its not about the size of the packages, its about managing an entire system, for whatever definition of the word "system" In the "olden" days, you would patch the vulnerable dependency and be sure that all services running on the machine have been updated to the new version. (in theory - if everything works great and there is ABI compatibility etc etc). Since the library is installed in a well-known system location, there are only a few paths in the filesystem that you would need to check to make sure it's up to date. This makes machine maintenance easier. Nowadays, to patch a vulnerable dependency you need to potentially go to all individual projects, check whether they depend on it, update the version for that project, make sure to rebuild any containers that... contain it. Rinse, repeat. There is also not a single place where the dependencies live, so you can't just run the same command across multiple machines to do the check - you have to check the directory structure of every individual service. This can be very slow and painful.
    - pas 7y ago
      Agreed. Yet it seems the ecosystem simply accepts this trade-off. For increased productivity, increased memory-management security, etc.
sevensor 7y ago
My experience has been that, for sane libraries, the configuration process supports installing to an arbitrary prefix. In general, I end up making a new prefix for every project. And if doing this turns into dependecy hell, shouldn't that make you think twice about your dependencies? Worst case, you find out you need a different libc, and then you start questioning your life decisions.
Arch-TK 7y ago
It sounds like you just want a good package manager. On my system I have built lots of random projects to "mess around" and to achieve the same effect of cleaning them up I can just make a list of all the packages I explicitly installed to make them work (including packages which I already had) in a simple flat file. I can then feed that file into my package manager to delete only packages which aren't used by anything else. Now, I can hear you say: "this isn't practical, these projects depends on 400 dependencies". The question I would then ask is why has it become commonplace for a "dependency" to sometimes equate to 20 lines of code. For these "dependencies" sure I think maybe some kind of local storage is appropriate, but I think that this is already solved in languages like C using git submodules. I don't personally like this approach myself, I would rather people just flesh out a standard library or other common libraries than to produce a 20 line snippet that everyone includes in their project through submodules. But the fact of the matter is, this is still an option. The other thing I should point out is that your docker image is surely going to take up an order of magnitude more space than just installing a few libraries and keeping them around. This in turn means that you could have just kept an order of magnitude more projects in the same amount of space. And finally, the approach of putting everything in one directory is a big hit in terms of disk space. I don't know much about node or JS but I have to assume that the language is still primarily text based. Keeping around hundreds of dependencies written in text (even with minification but does that happen with node?) is going to take up a lot more space than a few shared objects. So in an ecosystem where everyone has decided that disk space isn't so important, what is the point of easy cleanup? Given the wording you use by "pollute" I assume you literally mean having those files there in the first place. But I don't understand what the problem would be with that. Especially when most projects use dependencies you're most likely already going to have installed. As long as you don't work against your package manager, nothing should ever break as a result of having too many packages installed.

inimino 7y ago

You seem to be confused about how typical C/C++ projects are built. If you need globally installed dependencies, you're going to have to install them yourself. A typical makefile is not going to download a long list of random junk from the internet and install it your machine without asking. Although you'd be forgiven for thinking so if you're used to npm. If you don't like where the project installs its binaries or libraries, you can configure that before `make` or `make install`. You can install it to your home directory, have a separate directory tree for experiments, or whatever you want, and clean it up whenever it is convenient. You can also install things in usr/local and just ignore them. The system is actually well designed if you learn how to use it. If you need a dependency, use your package manager to install it and you can trivially delete it later... and if you try five different frontends for the same library, you only install one copy of it. Docker is a workaround for a lack of Unix awareness and sysadmin experience.

spion 7y ago
The system is not well designed. For example: if I have 3 projects I'm developing locally, each needing a dozen (somewhat overlapping) libxxx-dev dependencies to build, how can I automatically clean up the dependencies that are no longer needed when I delete one of the 3 projects? Distro package managers are focused on providing a good experience for sysadmins while ignoring needs during development time. Unfortunately, docker and the developer-centric improvements in language package managers have come at the expense of worse sysadmin experience. The situation will remain sub-optimal until both groups start working together to address both needs.
- inimino 7y ago
  If you've got some leftover libraries in /usr/local/lib on your development machine because of an experiment, why is that the end of the world? If you're worried about it, you shouldn't be. The worst thing about Docker is that people think learning Docker is a substitute for basic competence as a sysadmin.
  - spion 7y ago
    I'll put it in context. Let's say I have leftover garbage in usr/local/lib. I clone a new project that I need to patch because it's broken. I try to fix the new project but the fix isn't working. I have no idea why. I spend hours trying to trace down the problem. What was the issue? The project picked up some of the old garbage libraries I had in usr/local/lib, which had a bug that made my fix not work. For developers, global dev library garbage = global mutable state. It interferes with development, creates more sources of potential issues, and takes time to trace and debug. A clean and/or localized environment is essential for the accurate pinpointing of bugs and issues.
    - inimino 7y ago
      > I spend hours trying to trace down the problem. This is the problem. Once you understand library loading, once on your system, and learn, once, how to query the system and see what libraries are being loaded and from where, you won't have to waste hours on this problem again. Until that point, you're operating a system without understanding, and that will always lead to breakage.
      - saagarjha 7y ago
        I've installed hundreds of pieces of software from source, and I have a fairly decent idea of how linking and loading works. I still have to spend time figuring out why something doesn't build when it should due to something strange about my system.
      - spion 7y ago
        Of course I understand how it works. I just can't constantly keep in mind what garbage I played around with 3 months ago that might be responsible for the issue I'm looking at right now. I don't expect it, and more importantly, I shouldn't have to expect it. It's a basic tenant of development. Eliminate all additional sources of issues and complexity by presenting a clean, well-controlled sandbox in order to effectively diagnose it and fix the real bug. In fact, you should be able to try a newer version of a dependency, an older one, or run 10 variants (containers) each with different versions to e.g. test whether the project you're working on is widely compatible with a flexible range of that dependency. These things are done all the time. The tools are inadequate, not the developers. Engineers on both sides need to stop assuming everyone else is an idiot.

spion 7y ago

The benefit of global libraries is that they are easier to update or patch. If there is a vulnerability in an npm package, you will need to update it in all your node projects (and/or rebuild all your docker images analogously). A hybrid solution where the package manager is aware of all installed versions, can symlink to a globally installed copy and can update all symlinks would probably work. Unfortunately, there is very little interest in fixing the problem on both sides (distro PMs and language PMs) since they both believe their own approach is right and don't care about the other group's problems.

pas 7y ago
yarn uses a global cache and symlink if I remember correctly. and npm has an "audit" feature, which is default on, so there's quite a bit of movement in JS land to address the disadvantages of having a myriad small packages.
- spion 7y ago
  Its not about the size of the packages, its about managing an entire system, for whatever definition of the word "system" In the "olden" days, you would patch the vulnerable dependency and be sure that all services running on the machine have been updated to the new version. (in theory - if everything works great and there is ABI compatibility etc etc). Since the library is installed in a well-known system location, there are only a few paths in the filesystem that you would need to check to make sure it's up to date. This makes machine maintenance easier. Nowadays, to patch a vulnerable dependency you need to potentially go to all individual projects, check whether they depend on it, update the version for that project, make sure to rebuild any containers that... contain it. Rinse, repeat. There is also not a single place where the dependencies live, so you can't just run the same command across multiple machines to do the check - you have to check the directory structure of every individual service. This can be very slow and painful.
  - pas 7y ago
    Agreed. Yet it seems the ecosystem simply accepts this trade-off. For increased productivity, increased memory-management security, etc.

sevensor 7y ago

My experience has been that, for sane libraries, the configuration process supports installing to an arbitrary prefix. In general, I end up making a new prefix for every project. And if doing this turns into dependecy hell, shouldn't that make you think twice about your dependencies? Worst case, you find out you need a different libc, and then you start questioning your life decisions.

Arch-TK 7y ago

It sounds like you just want a good package manager. On my system I have built lots of random projects to "mess around" and to achieve the same effect of cleaning them up I can just make a list of all the packages I explicitly installed to make them work (including packages which I already had) in a simple flat file. I can then feed that file into my package manager to delete only packages which aren't used by anything else. Now, I can hear you say: "this isn't practical, these projects depends on 400 dependencies". The question I would then ask is why has it become commonplace for a "dependency" to sometimes equate to 20 lines of code. For these "dependencies" sure I think maybe some kind of local storage is appropriate, but I think that this is already solved in languages like C using git submodules. I don't personally like this approach myself, I would rather people just flesh out a standard library or other common libraries than to produce a 20 line snippet that everyone includes in their project through submodules. But the fact of the matter is, this is still an option. The other thing I should point out is that your docker image is surely going to take up an order of magnitude more space than just installing a few libraries and keeping them around. This in turn means that you could have just kept an order of magnitude more projects in the same amount of space. And finally, the approach of putting everything in one directory is a big hit in terms of disk space. I don't know much about node or JS but I have to assume that the language is still primarily text based. Keeping around hundreds of dependencies written in text (even with minification but does that happen with node?) is going to take up a lot more space than a few shared objects. So in an ecosystem where everyone has decided that disk space isn't so important, what is the point of easy cleanup? Given the wording you use by "pollute" I assume you literally mean having those files there in the first place. But I don't understand what the problem would be with that. Especially when most projects use dependencies you're most likely already going to have installed. As long as you don't work against your package manager, nothing should ever break as a result of having too many packages installed.