4 ms·

I maintain what is probably the most popular Ethereum simulator (Ganache) and it uses the npm ecosystem. Years ago I chose to pin/lock all dependencies, even t

I maintain what is probably the most popular Ethereum simulator (Ganache) and it uses the npm ecosystem. Years ago I chose to pin/lock all dependencies, even transitive dependencies (direct dependencies' dependencies) to much disagreement from the semver purists. Crypto developers are extra high value targets because they likely access their wallet from the same machine they develop on. So I've taken a very hard stance on this, even feel we should do even better by disallowing updates for new releases (I realize adequate security here is not practical/feasible for most). To make matters worse: devs often install npm packages with sudo (and I have a canned response for sudo related issues telling them that they must now format their drives to fix it, and even that might not be enough as their bios and other embedded firmware could also be compromised). Meanwhile, yarn, a popular npm alternative, will NOT respect a package author's wishes to lock transitive dependencies. It's maddening (don't use yarn until/unless they fix this). The only time a dependency shouldn't be pinned is if you are also the author of that dependency. Anyway, people would say I'm fun at parties, but they stopped inviting me long ago.

jaquers 5y ago
How did you accomplish this practically? using '=' in package.json + shrinkwrap or vendoring in dependencies?
- davidmurdoch 5y ago
  We bundle most, and ship npm-shrinkwrap.json for these ones we don't. The shrinkwrap is pruned (via a a custom script, not npm's prune command, because of reasons ) at publish time, as npm installs all dev dependencies if they exist in the shrinkwrap file (despite being marked as "dev": true in said shrinkwrap).
datalopers 5y ago
Do you regret using node/npm on such a high-risk installation?
- davidmurdoch 5y ago
  When it was first built (6ish years ago? before I took over) there weren't any other practical options if you wanted an Ethereum simulator in JavaScript, which is a very common use case of ours. Though now that wasm is popular and there are Rust implementations of Ethereum, I still think being able step into and debug from within a JavaScript application is valuable. I think the ecosystem has just drank too much semver Kool-Aid. So no, I don't regret it, and I don't think the problem is unique to npm, I think the problem is exacerbated by its popularity and addiction to loose semver (the default when installing a package by name).
  - KyeRussell 5y ago
    Amen on semver. If you start pulling at that thread, the npm people start realising what a big mess they’ve gotten themselves in.
  - alisonkisk 5y ago
nhoughto 5y ago
I didn't know that about yarn transitives, do you have a link explaining more about the problem? I assumed a yarn.lock would.. lock, interesting to know more.
- leipert 5y ago
  I assume the author is talking about the following problem: I build library X and ship it with a lockfile to lock all transitive dependencies. User installs library X, even though I ship a lock file, it isn’t respected. (I assume this makes Resolution way to complex) If you ship a library where everything is locked, one should consider vendoring their dependencies. I think this is what npm does itself. (Or used to do). Or consider reducing your dependencies.
  - eyelidlessness 5y ago
    > I assume this makes Resolution way to complex I can’t speak to Yarn’s motivation, but I suspect for most Yarn users it’s not dissimilar to my own: pinning exact versions of transitive dependencies can be a big source of “audit fatigue” with NPM. It’s fine that authors wish to limit the surface area of version ranges they support, but for downstream users this can often mean being essentially held hostage with potential exposure to vulnerabilities and no reasonable alternative. Yarn should, like NPM, default to honoring all pinned dependencies. Both now have facilities to override those defaults (resolutions/overrides respectively), which is important. But there should also be a more gentle middle ground of “yes I want to assume this library meant to use semver, I understand I’m slightly relaxing the contract in the lockfile” in the form of a CLI flag… rather than manually writing potentially hundreds of lines of JSON. > Or consider reducing your dependencies. I mean, that sounds nice in theory. But the whole number of dependencies thing is a systemic design problem more than anything. No one is installing 1000s of dependencies. They’re installing ~5-10 and getting 1000s.
    - davidmurdoch 5y ago
      > Yarn should, like NPM, default to honoring all pinned dependencies. Both now have facilities to override those defaults (resolutions/overrides respectively), which is important. But there should also be a more gentle middle ground of “yes I want to assume this library meant to use semver, I understand I’m slightly relaxing the contract in the lockfile” in the form of a CLI flag… rather than manually writing potentially hundreds of lines of JSON. Yes, this! I want this, and for all nose package managers to share a lock file format.
lstamour 5y ago
My preferred fix for Yarn dependency tracking is to use zero installs[1] as there is no command to run and dependencies can be exactly what ships in the repo and nothing more (with the right flags). If accepting PRs from third-parties, the check-cache flag can run in CI to validate checksums from untrusted contributors — plus, you know, reading dependency source code when you have the time or reason to do so would be great within a PR review also. I wish more tools were able to concisely show you the differences between dependencies, but… sometimes dependencies have binaries and at that point you might need to fork or clone and build your own version of a dependency. I’d suggest only using dependencies when you can read and understand the code, but there are always limits. I can’t think of the last time I thought of a glibc dependency except when using Alpine or compiling something for Cgo. But it’s still something to keep in mind: that sometimes your project will be simpler and easier with fewer, smaller dependencies where you can read the code in full. 1: https://yarnpkg.com/features/zero-installs#does-it-have-security-implications https://yarnpkg.com/features/zero-installs#does-it-have-secu...
- davidmurdoch 5y ago
  Great tips for application developers. Though this still exposes you to potential transitive dependency supply-chain attacks on first-time installs, though it's likely very low risk at this point, especially if you're careful as you've described. For library authors that want to take reasonable precautions yarn makes it very hard. Though some suggest leaving transitive dependency updates up to the consumer makes yarn more secure, as they can update to the fixed packages sooner.. but I don't believe this benefit actually manifests often. I guess the moral of the story is that semver is really great on paper, but people suck at it, sometimes on purpose.
  - lstamour 5y ago
    In the extreme case, though, a library author can run their own build toolchain and create an executable npm module with zero declared dependencies, then version that as many times as they desire as long as their test cases guarantee that the new build still passes and dependencies have been thoroughly checked. By publishing the source code to git, those who desire to use a specific version could alternately embed the git URL in their dependencies instead of the pre-built npm package. Yarn Pack command can be used to run a build command[1] and Yarn Publish[2] in turn calls Yarn Pack, if I understand this correctly. 1: https://yarnpkg.com/advanced/lifecycle-scripts https://yarnpkg.com/advanced/lifecycle-scripts 2: https://yarnpkg.com/cli/npm/publish/#gatsby-focus-wrapper https://yarnpkg.com/cli/npm/publish/#gatsby-focus-wrapper That said… I agree that further work might be necessary here. I remember the first time I built an npm package and was confused when my lockfile in the package was completely ignored for dependency resolution. I don’t think this problem has gotten much attention, but the solution above might substantially solve the problem, if your dependencies are small enough to embed easily. It’s not ideal though, and I’d welcome other perspectives.
Gigachad 5y ago
Wouldn't it be more practical to do development inside a container or VM? Locking deps won't be enough if you haven't also audited them.
- fragmede 5y ago
  Sure, but the valuable crypto credentials need to get used somewhere and the paranoid (justifiably-so) will note that VM escapes vulnerabilities, while not common, do exist, and aren't just hypothetical. The current best practice, as I see it, is is a disposable chromebook, backed by a cold wallet which holds the keys to the multi-million dollar hoard.
glenngillen 5y ago
I appreciate your thoughtfulness, and someone who thinks deeply about things with a touch of self-deprecation is the exact kind of conversation I enjoy at parties ;)
- mmmpop 5y ago
alisonkisk 5y ago
bushbaba 5y ago
It's maddening the crypto world relies on Javascript so heavily. As a language it would never be my first, second, or even third choice for financial services.
mhio 5y ago
> Meanwhile, yarn, a popular npm alternative, will NOT respect a package author's wishes to lock transitive dependencies Do you mean locking transitive dependencies across an upgrade of a direct dependencies? How does npm do that?

jaquers 5y ago

How did you accomplish this practically? using '=' in package.json + shrinkwrap or vendoring in dependencies?

davidmurdoch 5y ago
We bundle most, and ship npm-shrinkwrap.json for these ones we don't. The shrinkwrap is pruned (via a a custom script, not npm's prune command, because of reasons ) at publish time, as npm installs all dev dependencies if they exist in the shrinkwrap file (despite being marked as "dev": true in said shrinkwrap).

datalopers 5y ago

Do you regret using node/npm on such a high-risk installation?

davidmurdoch 5y ago
When it was first built (6ish years ago? before I took over) there weren't any other practical options if you wanted an Ethereum simulator in JavaScript, which is a very common use case of ours. Though now that wasm is popular and there are Rust implementations of Ethereum, I still think being able step into and debug from within a JavaScript application is valuable. I think the ecosystem has just drank too much semver Kool-Aid. So no, I don't regret it, and I don't think the problem is unique to npm, I think the problem is exacerbated by its popularity and addiction to loose semver (the default when installing a package by name).
- KyeRussell 5y ago
  Amen on semver. If you start pulling at that thread, the npm people start realising what a big mess they’ve gotten themselves in.
- alisonkisk 5y ago

nhoughto 5y ago

I didn't know that about yarn transitives, do you have a link explaining more about the problem? I assumed a yarn.lock would.. lock, interesting to know more.

leipert 5y ago
I assume the author is talking about the following problem: I build library X and ship it with a lockfile to lock all transitive dependencies. User installs library X, even though I ship a lock file, it isn’t respected. (I assume this makes Resolution way to complex) If you ship a library where everything is locked, one should consider vendoring their dependencies. I think this is what npm does itself. (Or used to do). Or consider reducing your dependencies.
- eyelidlessness 5y ago
  > I assume this makes Resolution way to complex I can’t speak to Yarn’s motivation, but I suspect for most Yarn users it’s not dissimilar to my own: pinning exact versions of transitive dependencies can be a big source of “audit fatigue” with NPM. It’s fine that authors wish to limit the surface area of version ranges they support, but for downstream users this can often mean being essentially held hostage with potential exposure to vulnerabilities and no reasonable alternative. Yarn should, like NPM, default to honoring all pinned dependencies. Both now have facilities to override those defaults (resolutions/overrides respectively), which is important. But there should also be a more gentle middle ground of “yes I want to assume this library meant to use semver, I understand I’m slightly relaxing the contract in the lockfile” in the form of a CLI flag… rather than manually writing potentially hundreds of lines of JSON. > Or consider reducing your dependencies. I mean, that sounds nice in theory. But the whole number of dependencies thing is a systemic design problem more than anything. No one is installing 1000s of dependencies. They’re installing ~5-10 and getting 1000s.
  - davidmurdoch 5y ago
    > Yarn should, like NPM, default to honoring all pinned dependencies. Both now have facilities to override those defaults (resolutions/overrides respectively), which is important. But there should also be a more gentle middle ground of “yes I want to assume this library meant to use semver, I understand I’m slightly relaxing the contract in the lockfile” in the form of a CLI flag… rather than manually writing potentially hundreds of lines of JSON. Yes, this! I want this, and for all nose package managers to share a lock file format.

lstamour 5y ago

My preferred fix for Yarn dependency tracking is to use zero installs[1] as there is no command to run and dependencies can be exactly what ships in the repo and nothing more (with the right flags). If accepting PRs from third-parties, the check-cache flag can run in CI to validate checksums from untrusted contributors — plus, you know, reading dependency source code when you have the time or reason to do so would be great within a PR review also. I wish more tools were able to concisely show you the differences between dependencies, but… sometimes dependencies have binaries and at that point you might need to fork or clone and build your own version of a dependency. I’d suggest only using dependencies when you can read and understand the code, but there are always limits. I can’t think of the last time I thought of a glibc dependency except when using Alpine or compiling something for Cgo. But it’s still something to keep in mind: that sometimes your project will be simpler and easier with fewer, smaller dependencies where you can read the code in full. 1: https://yarnpkg.com/features/zero-installs#does-it-have-security-implications https://yarnpkg.com/features/zero-installs#does-it-have-secu...

davidmurdoch 5y ago
Great tips for application developers. Though this still exposes you to potential transitive dependency supply-chain attacks on first-time installs, though it's likely very low risk at this point, especially if you're careful as you've described. For library authors that want to take reasonable precautions yarn makes it very hard. Though some suggest leaving transitive dependency updates up to the consumer makes yarn more secure, as they can update to the fixed packages sooner.. but I don't believe this benefit actually manifests often. I guess the moral of the story is that semver is really great on paper, but people suck at it, sometimes on purpose.
- lstamour 5y ago
  In the extreme case, though, a library author can run their own build toolchain and create an executable npm module with zero declared dependencies, then version that as many times as they desire as long as their test cases guarantee that the new build still passes and dependencies have been thoroughly checked. By publishing the source code to git, those who desire to use a specific version could alternately embed the git URL in their dependencies instead of the pre-built npm package. Yarn Pack command can be used to run a build command[1] and Yarn Publish[2] in turn calls Yarn Pack, if I understand this correctly. 1: https://yarnpkg.com/advanced/lifecycle-scripts https://yarnpkg.com/advanced/lifecycle-scripts 2: https://yarnpkg.com/cli/npm/publish/#gatsby-focus-wrapper https://yarnpkg.com/cli/npm/publish/#gatsby-focus-wrapper That said… I agree that further work might be necessary here. I remember the first time I built an npm package and was confused when my lockfile in the package was completely ignored for dependency resolution. I don’t think this problem has gotten much attention, but the solution above might substantially solve the problem, if your dependencies are small enough to embed easily. It’s not ideal though, and I’d welcome other perspectives.

Gigachad 5y ago

Wouldn't it be more practical to do development inside a container or VM? Locking deps won't be enough if you haven't also audited them.

fragmede 5y ago
Sure, but the valuable crypto credentials need to get used somewhere and the paranoid (justifiably-so) will note that VM escapes vulnerabilities, while not common, do exist, and aren't just hypothetical. The current best practice, as I see it, is is a disposable chromebook, backed by a cold wallet which holds the keys to the multi-million dollar hoard.

glenngillen 5y ago

I appreciate your thoughtfulness, and someone who thinks deeply about things with a touch of self-deprecation is the exact kind of conversation I enjoy at parties ;)

mmmpop 5y ago

alisonkisk 5y ago

bushbaba 5y ago

It's maddening the crypto world relies on Javascript so heavily. As a language it would never be my first, second, or even third choice for financial services.

mhio 5y ago

> Meanwhile, yarn, a popular npm alternative, will NOT respect a package author's wishes to lock transitive dependencies Do you mean locking transitive dependencies across an upgrade of a direct dependencies? How does npm do that?