5 ms·

I have been thinking a lot lately about a possible solution for a small portion of this problem: Microdependencies. I'll explain in more detail in the context o

I have been thinking a lot lately about a possible solution for a small portion of this problem: Microdependencies. I'll explain in more detail in the context of JS, but it applies to other languages as well. Currently package repositories like NPM host 2 types of dependencies: big community packages (frameworks, database drivers, validation libraries, query builders,...) and smaller function-scoped utility packages (left-pad, is-even, math functions,...) My proposal is to create a new package manager for the latter (or adapt existing ones) which handles micro-packages, typically scoped to 1 function, 1 file or a small set of files. The big difference with regular NPM is that these micropackages do not get downloaded into node_modules, but are downloaded inside the src directory and committed to source control. This means that when you add a micropackage to your project, its source code is pulled into the regular src folder and is subject to the same code review as the other code your team writes. When the micropackage is updated the changes are visible during code review as well (instead of just being an opaque version bump in package.json) This process makes small dependencies more visible and has review builtin. It also solves the problem of having to rewrite the same logic in every project in a slightly different way. Edit: I'm not currently working on this due to time constraints but if anyone wants to talk about this, hit me up.

gioele 6y ago
CCAN (C Code Archive Network) http://ccodearchive.net/ http://ccodearchive.net/ follows a similar philosophy. CCAN contains various small, self contained snippet of C code are meant to be be downloaded in a local directory ("vendored") and used directly or with small modifications. The most straightforward way to use CCAN is to 1) `git clone` it in a subdir; 2) create a `config.h`; 3) include whatever `lib.h` you want in your project. Updating these micro dependencies means doing `git pull`. All libraries come with tests and a somewhat standardized API design. What is missing (compared to your aims) is a declarative way to lock a dependency to a specific version.
- bonkabonka 6y ago
  Reminds me of [Bob Stout's Snippets](https://github.com/vonj/snippets.org https://github.com/vonj/snippets.org). I used his collection often back when I were a lad.
manicdee 6y ago
What about git submodules?
- tarkin2 6y ago
  Oh lord no. They’re a pain to manage. The world needs fewer gitsubmodules.
  - manicdee 6y ago
    How do you go about preparing a PR for an upstream if all the code is magically in your own version control system? And then how do you separate site-specific patches from bug fixes and new features for the upstream?
marcus_holmes 6y ago
I think the answer is to not import the dependency at all. If you absolutely can't write a single function, then copy/pasting it from a set of "known good" functions would be better than importing a dependency. Of course, just writing the function would be better. It would probably take less time than discovering the package, working out the interface for it, discovering that it doesn't actually work for your use case, discovering another, better package, etc.
- chrismeller 6y ago
  I agree, but I would take it a step further. If you think it’s easier to find and import a library to test if a number is even than to use the modulus operator in an if clause, should you really be responsible for writing any code at all? I feel like JavaScript in particular has developed this ecosystem of inexperienced programmers that “don’t know what they don’t know”. I don’t want to discourage anyone, but I feel like other languages (particularly more mature ones) don’t have that problem to the same degree. For instance, I feel like the cluster module in Node is far easier for people to do things they shouldn’t with than any of the equivalents in C# or Java or even PHP (so it’s not just compiled vs scripting languages).
- jamie_ca 6y ago
  I think the catch is that "microlibraries" can still contain reasonably large amounts of functionality that's non-trivial to replicate. I'd be happy enough to import single-function dependencies for "urlencode" or "is_valid_email" rather than manually grow out all the edge cases (again).
- mleonhard 6y ago
  Many useful libraries cannot be easily re-created. Consider datetime handling and X.509 parsing.
  - marcus_holmes 6y ago
    I agree, and happily import moment.js for that reason. But that's not what the GP was talking about. This is not a "microdependency". It's something that should have been included in the standard library, except that JS doesn't have one of those.
pjc50 6y ago
> possible solution for a small portion of this problem In CS research, this is known as "software components", and people have been worrying about it since at least the 1970s with very slow progress. Here's Stoustroup writing about it in the 90s, for example: https://www.semanticscholar.org/paper/Language-technical-aspects-of-reuse-Stroustrup/80f17151348d458fef29b2b2a54c90381caeba1d https://www.semanticscholar.org/paper/Language-technical-asp... "Reuse happens only when a variety of social conditions are favorable. However, social conditions, development processes, and design methods alone cannot guarantee success. In the end, working code must be produced" The dream used to be that software could be like electronics: standard elements like resistors would have a small number of well-characterised parameters, and you can just pick them from a catalog. Large components would come from vendors with large datasheets that would also describe their performance, and would be guaranteed a certain period of availability such as ten years. The end result is not quite like that. A lot of components have made it into standard libraries (what python calls "batteries included"). Everything else is available on repositories. However, because everything has to be free (and sometimes Free), that also selects for the absolute minimum maintenance effort and quality. Freedom has many huge advantages, as does not having to get purchase order or "BOM" approval every time you add a dependency to your project, but it does leave people in the "if it breaks you get to keep both pieces" situation.
dna_polymerase 6y ago
Horrible idea, if a package get corrupted by an adversary not only would you have to clean the npm package but also hope everyone who committed the code into a repository gets the news and fixes it. At least with NPM there is the chance that a fixed version gets automatically pulled on the next build. In addition to that, most package managers rely on metadata, like description files, the bloat introduced by these micro-packages is unfathomable. Then there is the obvious answer to all this, use a proper language, learn it and don't pull in stuff like "left-pad" from external sources. It's a one liner. If you can't come up with that yourself get the fuck out of this industry. Seriously.
- mercer 6y ago
  I dunno, there's something to it. I almost /never/ commit code to my repo without reading it, so not only would I read the first version that enters my repo, but I'd read every subsequent file changes before committing. I can definitely imagine a package manager that, in some way, differentiates between the two (in repo or not), whether manually specified or as OP suggests some distinction based on how 'big' the package is. Right now, it feels too dichotomous. Either I use a package that itself relies on a ton of packages, and I won't read all the code changes, or I copy and paste bits of code into my repo and now have to manually update things of any consequence.
mleonhard 6y ago
I really like this idea. When reviewing changes of the imported code, one will need the original change sets with comments. Therefore the system will need a format for representing these diffs. I don't think git is the right answer. After review, the tool could upload a "passed review" signature to the central repository. Folks can see which version has been reviewed and approved by which organizations. This would let small organizations benefit from the review work done by large organizations. For example, a startup founder can feel more confident using a library version that has passed review by several of the FAANG companies.

gioele 6y ago

CCAN (C Code Archive Network) http://ccodearchive.net/ http://ccodearchive.net/ follows a similar philosophy. CCAN contains various small, self contained snippet of C code are meant to be be downloaded in a local directory ("vendored") and used directly or with small modifications. The most straightforward way to use CCAN is to 1) `git clone` it in a subdir; 2) create a `config.h`; 3) include whatever `lib.h` you want in your project. Updating these micro dependencies means doing `git pull`. All libraries come with tests and a somewhat standardized API design. What is missing (compared to your aims) is a declarative way to lock a dependency to a specific version.

bonkabonka 6y ago
Reminds me of [Bob Stout's Snippets](https://github.com/vonj/snippets.org https://github.com/vonj/snippets.org). I used his collection often back when I were a lad.

manicdee 6y ago

What about git submodules?

tarkin2 6y ago
Oh lord no. They’re a pain to manage. The world needs fewer gitsubmodules.
- manicdee 6y ago
  How do you go about preparing a PR for an upstream if all the code is magically in your own version control system? And then how do you separate site-specific patches from bug fixes and new features for the upstream?

marcus_holmes 6y ago

I think the answer is to not import the dependency at all. If you absolutely can't write a single function, then copy/pasting it from a set of "known good" functions would be better than importing a dependency. Of course, just writing the function would be better. It would probably take less time than discovering the package, working out the interface for it, discovering that it doesn't actually work for your use case, discovering another, better package, etc.

chrismeller 6y ago
I agree, but I would take it a step further. If you think it’s easier to find and import a library to test if a number is even than to use the modulus operator in an if clause, should you really be responsible for writing any code at all? I feel like JavaScript in particular has developed this ecosystem of inexperienced programmers that “don’t know what they don’t know”. I don’t want to discourage anyone, but I feel like other languages (particularly more mature ones) don’t have that problem to the same degree. For instance, I feel like the cluster module in Node is far easier for people to do things they shouldn’t with than any of the equivalents in C# or Java or even PHP (so it’s not just compiled vs scripting languages).
jamie_ca 6y ago
I think the catch is that "microlibraries" can still contain reasonably large amounts of functionality that's non-trivial to replicate. I'd be happy enough to import single-function dependencies for "urlencode" or "is_valid_email" rather than manually grow out all the edge cases (again).
mleonhard 6y ago
Many useful libraries cannot be easily re-created. Consider datetime handling and X.509 parsing.
- marcus_holmes 6y ago
  I agree, and happily import moment.js for that reason. But that's not what the GP was talking about. This is not a "microdependency". It's something that should have been included in the standard library, except that JS doesn't have one of those.

pjc50 6y ago

> possible solution for a small portion of this problem In CS research, this is known as "software components", and people have been worrying about it since at least the 1970s with very slow progress. Here's Stoustroup writing about it in the 90s, for example: https://www.semanticscholar.org/paper/Language-technical-aspects-of-reuse-Stroustrup/80f17151348d458fef29b2b2a54c90381caeba1d https://www.semanticscholar.org/paper/Language-technical-asp... "Reuse happens only when a variety of social conditions are favorable. However, social conditions, development processes, and design methods alone cannot guarantee success. In the end, working code must be produced" The dream used to be that software could be like electronics: standard elements like resistors would have a small number of well-characterised parameters, and you can just pick them from a catalog. Large components would come from vendors with large datasheets that would also describe their performance, and would be guaranteed a certain period of availability such as ten years. The end result is not quite like that. A lot of components have made it into standard libraries (what python calls "batteries included"). Everything else is available on repositories. However, because everything has to be free (and sometimes Free), that also selects for the absolute minimum maintenance effort and quality. Freedom has many huge advantages, as does not having to get purchase order or "BOM" approval every time you add a dependency to your project, but it does leave people in the "if it breaks you get to keep both pieces" situation.

dna_polymerase 6y ago

Horrible idea, if a package get corrupted by an adversary not only would you have to clean the npm package but also hope everyone who committed the code into a repository gets the news and fixes it. At least with NPM there is the chance that a fixed version gets automatically pulled on the next build. In addition to that, most package managers rely on metadata, like description files, the bloat introduced by these micro-packages is unfathomable. Then there is the obvious answer to all this, use a proper language, learn it and don't pull in stuff like "left-pad" from external sources. It's a one liner. If you can't come up with that yourself get the fuck out of this industry. Seriously.

mercer 6y ago
I dunno, there's something to it. I almost /never/ commit code to my repo without reading it, so not only would I read the first version that enters my repo, but I'd read every subsequent file changes before committing. I can definitely imagine a package manager that, in some way, differentiates between the two (in repo or not), whether manually specified or as OP suggests some distinction based on how 'big' the package is. Right now, it feels too dichotomous. Either I use a package that itself relies on a ton of packages, and I won't read all the code changes, or I copy and paste bits of code into my repo and now have to manually update things of any consequence.

mleonhard 6y ago

I really like this idea. When reviewing changes of the imported code, one will need the original change sets with comments. Therefore the system will need a format for representing these diffs. I don't think git is the right answer. After review, the tool could upload a "passed review" signature to the central repository. Folks can see which version has been reviewed and approved by which organizations. This would let small organizations benefit from the review work done by large organizations. For example, a startup founder can feel more confident using a library version that has passed review by several of the FAANG companies.