3 ms·

It sounds like your devs are not following best practices with regards to lockfiles and version pinning.

satvikpendem 3y ago
Indeed. If half won't even build anymore, that's insane. It seems like they want to be on the latest version all the time which obviously will have breaking changes.
- gkbrk 3y ago
  > they want to be on the latest version all the time which obviously will have breaking changes Not very common in sane ecosystems.
  - satvikpendem 3y ago
    It's common everywhere that at least one dependency will have some breaking change, even in "same ecosystems."
    - zo1 3y ago
      No, this is highly unique to JS/TS. I've worked in plenty other languages and it's bo where near as bad. Yes they all have some form of this, but it's an order of magnitude smaller. One big contributor is JS's proclivity to have lots of smaller libraries. This increases the chances of one of the included libraries having a breaking change. Pile ontop of that this insane notion of constant version pinning and "semantic versioning" and you have devs that willy nilly change public facing interfaces.
      - satvikpendem 3y ago
        Not really. Python's is worse for example.
        zo1 3y ago
        Sorry, I have to call BS on this. I'm hugely familiar with Python and used it for over 10 years, pretty much always on "latest version" instead of pinning. And had issues with incompatible versions less than 5 times. JS? Pretty much every single project, if any apreciable chunk of code gets ignored for a few months its almost guaranteed that it has breaking changes somewhere.
        satvikpendem 3y ago
        I've been using Python longer than you have and I've always had issues. JS is bad for sure, but it's not "highly unique" to JS as the parent states.
        gkbrk 3y ago
        In Python it's really common to install or depend on packages without specifying or locking versions. When you want to use a new feature that's missing in the version you have, you can upgrade, but in general people expect things to stay in future versions.
      - chatmasta 3y ago
        I agree that many packages in the JS ecosystem don't respect semantic versioning, which is why I pin my dependencies to an exact version. For updates, I rely on automated bots submitting PRs that update a single package at a time and run my test suites. But even if they did respect semver, I'd still use exact versions, because I want full control over when I update my dependencies, and I don't want lockfile updates leaking into unrelated PRs just because a developer ran `yarn install`. Yes, you can use `yarn install --immutable` (and you should in CI/prod), but this will fail if the lockfile changes, which it would if the semver resolved to an updated version. I just want to `yarn install` without changing the lockfile. If you pin to an exact version, and use a lockfile, then you won't ever pull in any updates on `yarn install`, and installing your package in three years will result in downloading the exact same code as it does today. Just make sure you use nvm to pin Node, and corepack to pin your package manager, and you have a fully reproducible build chain as long as the dependencies are not unpublished from npm (which they can't be, after 72 hours of publishing). The only time it makes sense to use a semver range is in the `peerDependencies` of a published library. Even the `dependencies` of a library should use an exact range. Why? Because firstly, if I'm following best practices, I should be bundling my library before publishing it as a package. And secondly, if that bundle depends on another package, that's okay because npm dependency model allows having multiple copies/versions of the same package. But if it's something like Emotion depending on React, where I only want one copy of the library at runtime, then I should use a peer dependency, and here it does make sense to refer to it with a semver range.
        satvikpendem 3y ago
        Exactly, people don't use (or commit to git) lockfiles and then are surprised when dependencies update, which is literally the point of what a lockfile is supposed to prevent.
cosmojg 3y ago
Lockfiles and version pinning are bandaid solutions to be applied in the absence of resources for supporting new versions of libraries, especially given that, unless you maintain your own internal package repositories, you're then dependent on those ancient versions remaining available through your usual channels for the unforeseeable future. On my team, we stick to pulling the latest versions of dependencies, but we run our CI/CD pipeline on every commit and hold onto the logs from the last few successful builds. When it's clear that a dependency upgrade has caused a failure, we check to see if we can afford the work we'd need to do to accommodate the new version, and only if it's nontrivial do we resort to pinning the dependency (but it's usually trivial). Even then, the intention is that this is a temporary workaround until we have the time and people required to make the change. Using lockfiles and version pinning by default opens you up to all sorts of issues. Breaking changes accumulate, and you'll fall further and further behind, increasing the development effort required to get up to date. If you ever want to use some hot, new library that just dropped with a must-have feature for your project, you're highly likely to run into a conflict that will force you to make all those changes you've been procrastinating. Of course, you'll also find yourself stuck with whatever security vulnerabilities exist in your pinned version of the dependency, to be discovered and exploited at some future date. This is a complete nonstarter in my field of work.
- chatmasta 3y ago
  > Using lockfiles and version pinning by default opens you up to all sorts of other issues Agreed this is a tradeoff. That's why you need a rigorous auto-update process. Your approach works too, it's just a question of when you want to detect failures caused by an upgrade Regarding security, you're still vulnerable until you update. In your approach, you couple those updates with ongoing development. But what if nobody's pushed code for a month? You'd probably want automated upgrades, right? So in that case why not pin your dependencies too? > you're then dependent on those ancient versions remaining available through your usual channels for the unforeseeable future It's a safe-ish bet that versions will remain available on npm since it's not possible to unpublish a package from there after 72 hours of publishing it. But you don't need to rely on the registry if you cache your dependencies, and you can take more advanced measures to hold onto that cache. In our codebase we build Docker images from every commit, and for the JS images we store the Yarn cache in a shared Docker layer. This was a bit finicky to get working, but isn't strictly necessary anyway since the final build includes the node_modules too. So we have reproducible builds at the application level, but we also hold onto all versions of Docker images (tagged by commit, and only rebuilt when their source changes), so we have reproducible deployments and - crucially - the ability to rollback to an earlier version without relying on any external dependencies. This was not an easy system to build, but it's more of a general devops problem than one that's limited to any specific packaging system. Our monorepo includes JS, Python, C, Lua, and a bunch of other languages and packages. I will say that the code for making Python properly reproducible was a lot more complicated (we use Pants and some hackery for generating a locked requirements.txt).
jb_hs 3y ago
lockfiles and version pinning is great unless you care about security fixes and maintaining projects over many years. This turns into a slog of maintenance.

Indeed. If half won't even build anymore, that's insane. It seems like they want to be on the latest version all the time which obviously will have breaking changes.

gkbrk 3y ago
> they want to be on the latest version all the time which obviously will have breaking changes Not very common in sane ecosystems.
- satvikpendem 3y ago
  It's common everywhere that at least one dependency will have some breaking change, even in "same ecosystems."
  - zo1 3y ago
    No, this is highly unique to JS/TS. I've worked in plenty other languages and it's bo where near as bad. Yes they all have some form of this, but it's an order of magnitude smaller. One big contributor is JS's proclivity to have lots of smaller libraries. This increases the chances of one of the included libraries having a breaking change. Pile ontop of that this insane notion of constant version pinning and "semantic versioning" and you have devs that willy nilly change public facing interfaces.
    - satvikpendem 3y ago
      Not really. Python's is worse for example.
      - zo1 3y ago
        Sorry, I have to call BS on this. I'm hugely familiar with Python and used it for over 10 years, pretty much always on "latest version" instead of pinning. And had issues with incompatible versions less than 5 times. JS? Pretty much every single project, if any apreciable chunk of code gets ignored for a few months its almost guaranteed that it has breaking changes somewhere.
        satvikpendem 3y ago
        I've been using Python longer than you have and I've always had issues. JS is bad for sure, but it's not "highly unique" to JS as the parent states.
      - gkbrk 3y ago
        In Python it's really common to install or depend on packages without specifying or locking versions. When you want to use a new feature that's missing in the version you have, you can upgrade, but in general people expect things to stay in future versions.
    - chatmasta 3y ago
      I agree that many packages in the JS ecosystem don't respect semantic versioning, which is why I pin my dependencies to an exact version. For updates, I rely on automated bots submitting PRs that update a single package at a time and run my test suites. But even if they did respect semver, I'd still use exact versions, because I want full control over when I update my dependencies, and I don't want lockfile updates leaking into unrelated PRs just because a developer ran `yarn install`. Yes, you can use `yarn install --immutable` (and you should in CI/prod), but this will fail if the lockfile changes, which it would if the semver resolved to an updated version. I just want to `yarn install` without changing the lockfile. If you pin to an exact version, and use a lockfile, then you won't ever pull in any updates on `yarn install`, and installing your package in three years will result in downloading the exact same code as it does today. Just make sure you use nvm to pin Node, and corepack to pin your package manager, and you have a fully reproducible build chain as long as the dependencies are not unpublished from npm (which they can't be, after 72 hours of publishing). The only time it makes sense to use a semver range is in the `peerDependencies` of a published library. Even the `dependencies` of a library should use an exact range. Why? Because firstly, if I'm following best practices, I should be bundling my library before publishing it as a package. And secondly, if that bundle depends on another package, that's okay because npm dependency model allows having multiple copies/versions of the same package. But if it's something like Emotion depending on React, where I only want one copy of the library at runtime, then I should use a peer dependency, and here it does make sense to refer to it with a semver range.
      - satvikpendem 3y ago
        Exactly, people don't use (or commit to git) lockfiles and then are surprised when dependencies update, which is literally the point of what a lockfile is supposed to prevent.

cosmojg 3y ago

Lockfiles and version pinning are bandaid solutions to be applied in the absence of resources for supporting new versions of libraries, especially given that, unless you maintain your own internal package repositories, you're then dependent on those ancient versions remaining available through your usual channels for the unforeseeable future. On my team, we stick to pulling the latest versions of dependencies, but we run our CI/CD pipeline on every commit and hold onto the logs from the last few successful builds. When it's clear that a dependency upgrade has caused a failure, we check to see if we can afford the work we'd need to do to accommodate the new version, and only if it's nontrivial do we resort to pinning the dependency (but it's usually trivial). Even then, the intention is that this is a temporary workaround until we have the time and people required to make the change. Using lockfiles and version pinning by default opens you up to all sorts of issues. Breaking changes accumulate, and you'll fall further and further behind, increasing the development effort required to get up to date. If you ever want to use some hot, new library that just dropped with a must-have feature for your project, you're highly likely to run into a conflict that will force you to make all those changes you've been procrastinating. Of course, you'll also find yourself stuck with whatever security vulnerabilities exist in your pinned version of the dependency, to be discovered and exploited at some future date. This is a complete nonstarter in my field of work.

chatmasta 3y ago
> Using lockfiles and version pinning by default opens you up to all sorts of other issues Agreed this is a tradeoff. That's why you need a rigorous auto-update process. Your approach works too, it's just a question of when you want to detect failures caused by an upgrade Regarding security, you're still vulnerable until you update. In your approach, you couple those updates with ongoing development. But what if nobody's pushed code for a month? You'd probably want automated upgrades, right? So in that case why not pin your dependencies too? > you're then dependent on those ancient versions remaining available through your usual channels for the unforeseeable future It's a safe-ish bet that versions will remain available on npm since it's not possible to unpublish a package from there after 72 hours of publishing it. But you don't need to rely on the registry if you cache your dependencies, and you can take more advanced measures to hold onto that cache. In our codebase we build Docker images from every commit, and for the JS images we store the Yarn cache in a shared Docker layer. This was a bit finicky to get working, but isn't strictly necessary anyway since the final build includes the node_modules too. So we have reproducible builds at the application level, but we also hold onto all versions of Docker images (tagged by commit, and only rebuilt when their source changes), so we have reproducible deployments and - crucially - the ability to rollback to an earlier version without relying on any external dependencies. This was not an easy system to build, but it's more of a general devops problem than one that's limited to any specific packaging system. Our monorepo includes JS, Python, C, Lua, and a bunch of other languages and packages. I will say that the code for making Python properly reproducible was a lot more complicated (we use Pants and some hackery for generating a locked requirements.txt).

jb_hs 3y ago

lockfiles and version pinning is great unless you care about security fixes and maintaining projects over many years. This turns into a slog of maintenance.