40 ms·

Node.js adds support for direct registry-less HTTPS imports

129 points by inglor 5y ago

inglor 5y ago
This is off by default due to security concerns but can be turned on with a flag. It's mainly in the community feedback phase so if there is anything you want to tell Node (be it "I want this!" or "This is a bad idea" - please do)
- Andrex 5y ago
  My layman's take is that it seems very similar to Log4j's vulnerability, but I don't really know for sure.
  - oauea 5y ago
    Please explain why you think that, because it seems completely unrelated and off-topic.
    - moritonal 5y ago
      Bit harsh? Log4j was caused by a URL leading to bad code being imported. This let's you import code via a URL, seems quite similar?
      - sodality2 5y ago
        Log4j was caused when arbitrary log strings were interpreted as code imports. In this case the flag has to be manually set and the code manually added as a dependency.
      - anamexis 5y ago
        The Log4j vulnerability allowed unsafe user input to trigger loading of arbitrary code. Basically every dependency manager out there will allow you to load "bad code" from a URL.
        tobyhinloopen 5y ago
        Log4j is no dependency manager
        anamexis 5y ago
        No, it's not, but this new Node.js feature is for loading dependencies
- mitchellst 5y ago
  Longtime node.js developer... I don't think we need this, and I don't think it's a good idea to add. The feature is _strange._ Which we know because it's behind a flag, i.e., it will have low utilization. It defies conventions—because the convention is to use a registry for packages. And it presents significant security concerns. (Which I don't really have to explain, as they're already cited as the reason to put it behind a flag.) Use of this feature seems like something you might see once in a career. (Indeed, I've seen a similar thing in another language employed only once... AND THERE WERE OTHER WAYS for that developer to have designed that system that would not have been so vulnerable.) When you see it, do you recognize what it is, understand and evaluate the risks it poses? Do you validate that the library sitting on the other side hasn't been taken over by a malicious actor? Do you validate that the library on the other side has up-to-date dependencies? If so, on what cadence do you do this, realistically? All the security tooling progress we've made (npm audit, etc.) seems shot in the foot by this. And I get that these concerns aren't dispositive on their own. I just don't see the argument on the other side that clearly. Is there some class of application that people really want to write in node that can't be written for lacking this? Do we really think this is a pattern that would be employed in well-made, secure codebases? I don't see the argument. Like I said, I've seen a feature like this used in another language. But I haven't seen it needed for any real application. I love me some good node ugly hacks. Still, the language features should be nudging us to write good software.
  - encryptluks2 5y ago
    Central authority != Good software. You are essentially advocating to provide a single point of failure that can and will be abused by nation-state hackers who just have to find one popular package and compromise the developer.
    - mitchellst 5y ago
      The argument is less about central authority than standard process. You don't have to use the official NPM registry. The point is that having a pipeline for modules with established conventions and automated means of audits and visibility on the dependency is a net good. Yes, there is such a thing as a supply chain attack. But even when that happens, the community puts out the call to patch immediately and points a finger at the offending package. Your HTTPS import from a random URL could fail silently. While NPM is a big target for supply chain attack, they know they are, so there are a lot of eyes on it. Not so with your HTTPS import.
- pimterry 5y ago
  In terms of feedback, there's an issue thread here covering some points already: https://github.com/nodejs/node/issues/41920 https://github.com/nodejs/node/issues/41920
- sdflhasjd 5y ago
  I recall when I was a completely PHP n00b (babbys first programming language), I might encounter an issue and search for the error on expertsexchange. Some of the answers would be really complicated, but there'd be that one answer: Just edit your php.ini and change `dangerous_option_you_will_die=true` - wow, such an easy fix. I'd know better later, but I'd still find horrendous options like these enabled in real software I would come accross in my professional career.
- forty 5y ago
  I have been doing nodejs backend developpement for the past 10 years. I'm not sure why we would need that honestly. Looks like it could create some mess if packages on npm all start importing files from random servers. Suddenly your CI relies on many servers to be up. And I don't know what's the problem with declaring things in package.json.
- danShumway 5y ago
  All of the things that make this enticing as a feature are large security risks, and the less risky version that people are pitching me doesn't have any of the enticing features. What people are pitching me on is the idea that these wouldn't be dynamic imports. They're saying they wouldn't be fetched during runtime, they'd be fetched before. They're saying they would be pinned. Well, you're describing a package system like npm. And for those purposes, as a way of defining static packages that get installed before runtime and can't be dynamically swapped out by the application -- npm is better than the system being proposed. The only way this would actually be interesting is if they were actually dynamic imports, but having them be dynamic imports would be very insecure. And for declarative imports that can be statically analyzed, this is inferior syntax to what we already have and encourages worse developer habits. Some problems: 1. Updating dependencies is harder, because imports across multiple files have to be synced. If a junior developer imports `https://lodash.com/v4 https://lodash.com/v4` in two files and later someone changes that to be `http://lodash.com/v5 http://lodash.com/v5` in one file, now I have dependency mismatches. The existing system doesn't have this problem because my dependencies are in one place. 2. Finding out which dependencies I'm using is harder. Maybe there's another tool that lists out all of the imports, that would be handy. But also, maybe the dependencies are all in one file with the requested version numbers right next to them. That seems a lot more convenient. 3. This forces me to actually run the Node program to install the dependencies, and it forces me to actually import them in a file before they'll be fetched. This is cumbersome, it would be good to be able to list out dependencies before they ever get imported, say for boilerplate imports that are used in lots of projects. 4. People have pointed out the lack of pinning. All of this is inferior to having all of the imports in one file that you can specify, maintain, and update without running the program or using an analyzer. I don't understand what the benefit is or why I should care about an import system that is just as inflexible as the current npm/Yarn package manager setup, but that also has what seems to be worse syntax and makes it harder for me to know what packages are being imported. I wish I understood more what problem this proposed syntax was actually trying to solve. I've read through the entire thread and I still don't understand why people want this. People in the Github thread are reassuring readers that this could work exactly the same way as node_modules. That's great, but if that's the case, what is the problem with node_modules that is prompting us to build a second system that is less organized and less explicit about what gets imported? We can already import URLs with both npm and Yarn. That's not a new feature, the only new feature is that now the URLs are spread across the entire codebase.
oefrha 5y ago
In my limited experience with Deno, it seems direct URL import just leads to a crappy, nonstandard, hard-to-upgrade version of a package.json in the form of, say, a deps.ts, once you get past a few dependencies. (And it’s very easy to get past a few dependencies in Deno since std modules need to be imported as URLs like everything else.)
- spiderice 5y ago
  That is my experience as well. I was really excited about this feature in Deno when I first heard about it. Then I started to get in to Deno, and it just seemed like it made everything harder. I thought it was going to be something like "here is a better way to do dependency management". But in practice it felt like "we got rid of dependency management. Figure it out on your own".
  - conaclos 5y ago
    I feel the same and import-maps do not solve the issue since they are not transitively loaded. In other word, if one of your dependency uses an import-map, then you have to copy-paste it in your own project to get its import-mapping.
progx 5y ago
So Deno is obsolete?
- mccorrinall 5y ago
  deno scripts need explicit permissions for using the file system, network etc that’s a pro in my book.
- ravenstine 5y ago
  No... If anything, this shows how ahead Deno is and how features like this in Node.js are too little too late. Deno's standard library is just better than Node's, IMO, and it's going to be hard for Node to play catchup, if that's even practical. In summary, it doesn't contain the cruft of the pre-ES2015 era that is still present in Node, even though Node has tried working around it. The global objects in Deno better replicate the browser, so it takes a lot less effort to do essentially the same things you would in a browser but in a DOMless environment. Deno is slightly more performant in Node in some areas, although this is usually negligible, and I don't have the JSPerf test on hand. I don't think Node is bad, by any means. If people are concerned about security... then don't use so many dependencies, or consider just not updating them until they've been demonstrated not to contain exploits; you don't actually need much code to essentially serve webpages. But I think Deno, over time, will at least match Node in popularity, maybe even more, because it has "modern" approaches right out of the box and doesn't need to move as slowly as Node in order to maintain a legacy. Although Deno misses some things here and there, I've seen features get added very quickly over the last couple years; at this point, it would take Node much longer to do the same things for fear of breaking something and because it's simply much older code. Then again, maybe I'll be wrong. Perhaps Node gets a second "big bang" and makes Deno obsolete. And I wouldn't care, either. All I want is a JavaScript language runtime with fewer concepts to learn that can also move quickly when something needs to be improved. If that ends up being Node, I'd be just as glad.
- qbasic_forever 5y ago
  Definitely not, deno does typescript out of the box without pulling in a bunch of npm dependencies and tools. It also is taking more of a go or python/batteries included view of building out a proper standard library vs. relying on the community to provide it. And of course deno has a much stricter execution sandbox model where you have to opt in to explicit permissions like file system access. IMHO we'll probably see node get more and more deno-like over time.
royjacobs 5y ago
Won't this make leftpad-type scenarios even _more_ likely? What is the benefit?
- zja 5y ago
  If node shares import syntax with browsers, it’s easier to write code that runs on both platforms. As far as leftpad, this isn’t that much different from using a package manager. You either cache your dependencies, or rely on a third party (npm repo/cdn).
- onion2k 5y ago
  When leftpad was in NPM there was an expectation that it wouldn't disappear suddenly. Package registries are thought to be a constant, reliable source of packages (even when they not quite that, as leftpad demonstrated). There would be no such expectation for a module that you include directly from an https URL. The main benefit I can see is that you can include code from your own "registry" very easily in unmanaged scripts. You can already do that because you can add a dependency to package.json using an https link already, but with this new feature you can include those dependencies in node scripts you aren't managing with a package file. You can do that. I'm not suggesting you should. I won't.
vorpalhex 5y ago
This is nice for many use cases, especially where running your own npm mirror is a pain. Hopefully you are controlling the http source though!
- qbasic_forever 5y ago
  Nice too in that you might not even need to touch or use npm. Imagine a node app without an enormous juggernaut of a node_modules folder!
999900000999 5y ago
I have mixed feelings here. On one hand do what you want, on the other hand you can already add https imports via package.json. Why in God's name would you not have a package.json. I think this would be awesome for hacking out prototypes, but if I saw this in production code I'd ether work to get it fixed or find a new job. Being reckless is fine in your spare time. It's not ok at work. My fear is this is going to push junior devs into sloppy practices.
- sneak 5y ago
  I have the distinct impression that whether you put the imports in package.json or in the source file itself, ~100% of the industry is engaging in sloppy practices now. I am terrified when trying to write secure es6. AIUI, when you add a module in go, the specific and exact version of the module is cryptographically locked in your project metadata. npm/yarn store the hashes in the lockfile but the package.json still uses those (usually) fuzzy matchers, and I'm not sure whether "npm i"/"yarn install" uses the lockfile exclusively by default. "npm help install" suggests that it uses package.json (which does not cryptographically secure the deps so that they match what you originally added) by default. (Additionally, tons of people install and deploy docker images by tag name alone, which is not cryptographically secure - different environment, same problem.) My mixed feelings about it are that we are slowly blurring the line between local and remote software, and pretty soon it will be nearly impossible to use most modern software in an offline environment (which is a hard requirement for certain security circumstances). Basically every single js developer in the world has granted Microsoft (owners of GitHub and NPM domains) the ability to remotely execute code on their machine at will. It's a lot of tooling and infrastructure to update and change in the event that the threat landscape changes significantly (e.g. war, national security mandates, etc).
  - rectang 5y ago
    Manually auditing an update of potentially hundreds of packages when package-lock.json gets regenerated is so challenging that many projects won't do it — and for those projects, the cryptographic hashes offer limited value. To make industry-wide improvements, we need to share resources for auditing of packages. There are many such endeavors underway at various stages of maturity. But this feature doesn't move that ball forwards. From the comments: https://github.com/nodejs/node/pull/36328#issuecomment-736034792 https://github.com/nodejs/node/pull/36328#issuecomment-73603... > It is insecure because there is no signature of the original file. Anybody that can take control of a domain name can take control of a server. Downloaded arbitrary code can't be added to a pool of pre-validated packages, except insofar as you control the server and only make pre-validated packages available on it exclusively to your own services.
neals 5y ago
Does anybody have an exmple of this and how / why I could use this?
- fredley 5y ago
  You can use it to shoot yourself in the foot.
- qbasic_forever 5y ago
  Presumably it's just like how deno does dependencies: https://deno.land/manual/examples/manage_dependencies https://deno.land/manual/examples/manage_dependencies As for why, it's simpler than adding a package.json and meticulously curating it. If you only have a few small dependencies this might make sense. You can also pull in stuff that isn't hosted on npm or properly packaged--for example just pull down a js file from someone's github repo. If you start getting a lot of dependencies then it might turn into a new nightmare to maintain. It's an experiment--people don't really know if it's good or bad until using it a bit.
  - deleted 5y ago
    [deleted]
mgkimsal 5y ago
Am I wrong in thinking this is similar to the below code? <?php include("https://otherdomain.com/somephpcode.txt https://otherdomain.com/somephpcode.txt"); ?> PHP got ridiculed for years for having this sort of stuff available (even if off by default eventually).
- deleted 5y ago
  [deleted]
- markstos 5y ago
  Sounds like it. These days, HTML is trusting JavaScript that's loaded from all over the place already. A "secure" database hosting service loads assets from 20 different domains on their login page. Loading code from remote domains can be abused and mis-used in any language. In this case, I think it's considered a "feature" because code was already being loaded remotely with NPM, who in turn got the code from the original author. Now you can load the code directly from the author. So instead of trusting both the author and NPM, you only need to trust the author. This could be considered a reduction in your attack surface area, and it's definitely helping with the NPM registry being a single-point-of-failure in the Node.js ecosystem.
  - dangerface 5y ago
    Npm won't allow you to require remote code from a user touched variable like the below. myRemoteLib = require(userGeneratedVariable)
    - eyelidlessness 5y ago
      I don’t know where you got that idea. NPM doesn’t have any say in CJS (or ESM) module resolution other than that they conform to relevant aspects of it. Node and other implementers of those module systems absolutely do allow dynamic require/import.
- nexuist 5y ago
  PHP got ridiculed for years for doing all the things all the other popular frameworks are coming around to support now. Server side rendering is particularly hilarious because it's like: oh, you're compiling .jsx files to .html now? Cool....can't imagine anyone ever did that before...
mrozbarry 5y ago
My impression is this is really just to have some sort of comparable feature to deno. You may think this somehow is more of a vulnerability, but the reality is the npm registry is already fairly insecure, we've seen npm packages get hijacked regularly. This also adds less dependence on npm, and maybe with all the packages that get hijacked there, this could be a reasonable move.
- junon 5y ago
  I'd really argue "regularly" is a bit much. It happens rarely and moving to URIs is only shifting the problem. I'd also argue the claim npm is insecure. How would you improve its security over what's there now? Lastly, there isn't really a dependence on npm at all. You were always able to 1) put URIs into package.json, and 2) make your own package manager. Node does not require npm to work. It just looks in node_modules by default, which npm uses to manage packages for you. They are completely decoupled. URI imports, on the other hand, have a lot of security problems and surface area and will probably face a load of criticism going forward.
  - dmitryminkovsky 5y ago
    > How would you improve its security over what's there now? Automated builds on NPM servers based on code in public repos, like Docker has (had?). The least they could do is require packages to be signed.
    - junon 5y ago
      How exactly does that help?
      - potatoz2 5y ago
        Controlled builds based on public repos prevents a malicious person (original author or not) from invisibly pushing a packages that doesn’t correspond to public sources. Signing packages prevents account takeovers from publishing bad packages. The remaining security problem is the author themselves coupled with too few eyes on public repos.
user-the-name 5y ago
So they are adding a hidden, targetable supply chain attack vector on purpose?
FlorianRappl 5y ago
What about TypeScript here? I think it does not yet understand such imports. However, especially for types (i.e., evaluation at compile-time) imports from a remote source would be good. Instead of keeping up-to-date via npm I could just write `import type { FooType } from 'https://remote-types-registry.com/foopackage@2 https://remote-types-registry.com/foopackage@2'` and wouldn't have to worry about typing updates. (surely this scenario is only valid when types are not shipped with the dependency)
- edgyquant 5y ago
  This is basically how Deno does things already
notpachet 5y ago
The people in the Github thread advocating for this feature seem to be pushing really hard on the point that this isn't any more insecure than NPM already is. But that's myopic. That's like saying that the lock is broken on the second floor window of your house, so you may as well unlock all the other windows. If a junior developer working for me proposed something like this I would pull them away from the computer and have a long coffee chat with them about basic architecture principles. Yes, we know there are problems with NPM. But why broaden the possible attack surface area by introducing another kind-of-sort-of-equivalent means of remote module loading? Now you're just going to have to spend time hardening two systems instead of one. At the end of the day this just feels like a misguided attempt to try and recapture some sense of dynamism from Deno. But if I wanted dynamism, I'd use Deno, with all of its pubescent awkwardness. I'm using Node.js stable for a reason. I spend a lot of cycles trying to defend Node from its many (often reasonable) detractors, and things like this make me feel like I'm wasting my breath...
- encryptluks2 5y ago
  - junon 5y ago
    Lul what? First of all, this tone is completely against the HN guidelines, so take your flag. Second of all, centralization is not often a part of a security model. Usually in such cases where availability is a concern you have a mirror set up, which is ridiculously easy to do with CouchDB and completely nullifies your arrogant point. Get off your high horse.
    - encryptluks2 5y ago
  - notpachet 5y ago
    My comment wasn't really about the merits of central authorities vs multiple authorities (although I would fall on the side of preferring a single package repository than having to worry about many of them). It was about having two equivalent mechanisms for solving the remote-code-loading problem, regardless of where you're actually loading it from.
bricss 5y ago
More attack vectors for attack ↯ vectors ⩘⩗ sake ‽ Idea for the new toolset start-up: - A cyberware to validate source code and its modules against https imports
BeefWellington 5y ago
My main problem with this is that they punted the integrity check work down the line. If this had been implemented as something akin to the integrity HTML attribute, e.g.: express = require( 'https://my_local_repo.tld/express/current/4.17.1', 'sha384-oqVuAfXRKap7fdgcCY5uykM6+R9GqQ8K/uxy9rx7HNQlGYl1kPzQho1wx4JwY8wC' ) It would be vastly superior. I'd go so far as to require it without an additional --unsafe-import type flag. However, I think they did not do this because the import of dependencies then comes into question. IMO this feature should not have been added until developers could guarantee the expected integrity hash matches. We've seen time and again what will happen now is hundreds of blog posts talking about this new feature, none of which will mention the integrity hashing functionality (because it's not there). When it does eventually get added, those posts will be the most popular anyone can find, and thus new users will end up using those eventually-wrong posts and implementing things in a less-secure way. Huge disservice to the developer community.
- sneak 5y ago
  It's rubygems all over again. I think only Nix and Go are doing dependencies right (cryptographically verified) these days. One upside is that it's https-only, which does include authentication, provided that you trust the holder of the certificate for my_local_repo.tld (in your example) to serve you the right files.
  - francislavoie 5y ago
    Don't forget about PHP. Composer is one of the best package managers around.
- jdcaron 5y ago
  Exactly, this kind if import without a hash validation is a big no for security reasons (unless you 100% trust your import source). This feature exists on the browser side with the script element: https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity https://developer.mozilla.org/en-US/docs/Web/Security/Subres...
  - 5y ago
jppope 5y ago
probably just use deno?
- x-shadowban 5y ago
  Seems like a lot of new node features are in response to competition with deno's offerings. Neat to watch.
mstade 5y ago
Direct link to the rendered docs for this feature: https://github.com/bmeck/node/blob/ceadb473e6d3a22f38b737108fb5ac943bf8be09/doc/api/esm.md#https-and-http-imports https://github.com/bmeck/node/blob/ceadb473e6d3a22f38b737108... It's unfortunate I think that the docs don't mention integrity checks or caching, but those things are mentioned in the PR thread. Integrity via policies which seems to be a new feature as well[1], perhaps designed to essentially obsolete things like package-lock.json et al? Caching is punted till later, presumably to not hold up progress on this particular feature. Seems to me these are actually steps in the right direction, paving some of the cow paths already created by npm and yarn. I for one would love to see package-lock.json etc make way for policies. Security is a concern for sure if there's no integrity checking on the packages loaded via https, but the same holds true for packages regardless of delivery method so I don't see how this is much worse security wise than just declaring dependencies in package.json. Policies does seem to be an effort to solve the integrity issue regardless of how the package is installed, which seems like a good idea to me. If the caching and integrity issues are resolved well then this would essentially mean package managers become optional, no? Seems pretty good to me! [1]: https://nodejs.org/dist/latest/docs/api/policy.html https://nodejs.org/dist/latest/docs/api/policy.html
jchw 5y ago
A lot of the complaints here claim that this is bad because it’s insecure. However, allow me to posit a question: what if I made a function that just fetches the script and evals it? Node.JS already involves running unsandboxed, practically untrusted arbitrary code, that nobody audits. I’d have more sympathy if not for the fact that you can pretty much emulate this with the permission level that Node.JS code runs today. Also, no… this is not like PHP. PHP include got lambasted because it compounded other PHP security issues. If you use ESM imports, the obvious way, there’s never going to be an accident where user input can control the URL, because the grammar makes it impossible. You can use the dynamic import statement, but at that point you’re off into the abyss.
- user-the-name 5y ago
  There is value in making insecure things hard and awkward to do, rather than providing them as first-class constructs.
  - jchw 5y ago
    While this is true, if this is going to be the future, it may as well be provided as a feature with the proper precautions. While this doesn’t fully sandbox code, it does at least seem to provide some decent baseline security features. If I needed to import code from a web address for some reason, this is certainly the way to go, and whether ideal or not, sometimes that is a need that crops up. That said, proper sandboxing and/or at least optional integrity checking would certainly be nice.
CGamesPlay 5y ago
Evaluating these things at runtime is absurd. It was absurd when deno wanted to do it and it’s absurd now. People keep equating the threat model of npm and url imports, but with npm the untrusted code gets downloaded at build time, so tools like snyk can scan it and flag problems. Is snyk supposed to intercept the live download and patch files now? Are the policy integrity checks they mention in the PR allow or block lists? For example, if my malicious trojan module does a dynamic import, can it bypass a saved integrity check by adding a cache bust url parameter? Honestly, I don’t think the idea of URL imports is inherently bad. Go manages with them pretty reasonably. However, go only processes them at build time. My docker image from node might run today and break tomorrow because it’s allowed to change the remote code it runs day to day.
- wperron 5y ago
  If you actually read the (admittedly long) thread, it's pretty clear that those imports are not resolved at runtime, and Deno has never evaluated imports at runtime either. Dependencies get evaluated and downloaded prior to startup, very similar to Go in fact.
  - CGamesPlay 5y ago
    I don't see that listed anywhere in the thread, so I'll ask here instead: Does prior to startup mean there's a separate build step, or are you saying it happens during the loading of the script. I'm pretty confident that the latter is what deno was doing, although I'm happy to be proven wrong. Are dynamic imports simply disallowed? Since it is impossible to download in advance a script to which you don't know the URL without running the program, they'd have to be, in order for your assertion to be true.
    - wperron 5y ago
      The whole conversation is a bit hard to follow, there was an issue initially, then this PR and then another Discussion that's linked somewhere in the PR. I'll let you go read those, I don't recall exactly what decisions were taken and what's the roadmap for this to become stable. Dynamic imports are... wild, to say the least. Unfortunately they will always be inherently harder to secure, and will most likely always be treated differently than static import statements. At least that was the consensus in Deno.
numlock86 5y ago
At this point they just try to make the ecosystem even worse and open as many attack vectors as possible, aren't they? Has it become some "funny meme" in the maintainer circle or something? Why there is no SRI for example?
dangerface 5y ago
Loading modules over https because why? are dependancy attacks currently too difficult? Thanks I hate it.
mrweasel 5y ago
Won't this effectively prevent an application to start if it cannot reach the internet? I haven't done much development with Node, but I've seen similar patterns with Java applications that attempt to pull in an XSD at start up, only to fail to boot completely, because the application server doesn't have direct internet access. It's a bit of a weird feature, which should only be used for a very limited number of project, in a niche which I'm not completely able to identify.
tehbeard 5y ago
This is, IMO, the weakest part of deno being implemented. With npm I can easily get metrics for how stale dependencies are or quickly check for security issues. The closest I found for deno was maybe trex? And that felt incredibly anemic.
danShumway 5y ago
Kind of an interesting read. First thoughts, I generally think this is bad practice. I mean, for one thing this guarantees you need an Internet connection to run your Node script, which seems pretty problematic. I'm also reading some really questionable security justifications from people on the issue tracker about why dynamic imports are actually fine and just as secure as importing packages from NPM. There are security improvements that could be made to NPM, particularly around package search, and NPM has suffered from people replacing existing package versions (although I think they're clamping down on this more now). But there is a really fundamental difference between taking that risk once during installation and taking that risk multiple times at runtime. Additionally, in regards to some of the comments saying that installing code directly from the source ("express.com" vs "npm.com") ignores the fact that this is already kind of possible with npm dependencies and promoting that method wouldn't have any of the other problems. ---- HOWEVER: You read further and there are actually some really interesting security risk mitigations being proposed that don't just boil down to saying "all imports are dangerous, why not do them here too." "Scopes" (which I was not aware was a concept that Node had any support for) are the idea that packages in a scope or a domain can't access any dependencies outside of them. Additionally, people are talking about mirroring Deno's system where these get cached locally and not re-fetched later, which is kind of similar to npm's system and reduces some risk. So you can load a module from HTTPS and in theory it only gets fetched once and it can't access the file system, cool. Additionally, `node_modules` dependencies can't use them, which is cool, means that it's less likely a random dependency starts doing this behind your back. ---- HOWEVER: Prototype pollution is still a thing, passing objects around across closures is still a thing, and as far as I know this is a big part of why it's difficult to sandbox Javascript modules. Scopes are really interesting, but I can't find a lot of information about them online, and their name conflicts with npm scopes, which are a different thing. And I can't find any info suggesting they've solved the bigger problems of sandboxing modules. And there's still the elephant in the room, which is that all of those security measures are stuff that could be done in a more general way that applies to normal modules? I'm not sure why I would only want scopes for HTTPS imports, assuming they even work securely for HTTPS imports in the first place, which I wish I could find more information about. It's cool that a module gets fetched once at runtime and not later, but that still means I need an Internet connection the first time the Node script runs. And it seems like "tell me what dependencies you need and I'll fetch them" is a good candidate for an install step, which Node has. I'm not totally against the idea, I can in theory think of situations where you might want to install something at runtime without knowing ahead of time what it is. But that's also really dangerous to do, and doesn't really fit with the caching model that is being proposed. ---- So I still feel pretty doubtful that this is a good feature to add, and it seems really dangerous at first glance, but maybe there are other use-cases that I don't understand or maybe there's something to scopes that I don't get. I will say that if I actually trusted the security of scopes as a way of sandboxing individual modules, then at that point I feel a lot better about dangerous imports in general. But I can't find any info on scopes, and as far as I know, systems like Deno gave up on this because it was really hard, and I also vaguely feel like effective module-level sandboxing in Node would be much bigger news than any import feature. But :shrug:, I could be missing something. This is based on only a quick read through the entire thread and a tiny bit of extra searching/reading the commit docs.
eyelidlessness 5y ago
A couple observations of interest to me: 1. This follows closely behind the recent addition of fetch support. I expected this would leverage that, but it does not. So I assume this was developed in greater isolation than I would have hoped. I don’t have time to look deeper now, but given overlapping responsibilities I would think fetch would be a better fit than the older http(s) APIs. 2. Again following close behind landing fetch, Node is gaining more and more feature overlap with Deno. It’s not clear yet, to me anyway, whether that’s a direct motivation… or a somewhat convenient coincidence of similar goals. If it is a direct motivation, I feel like API compatibility, while valuable, may not be the most valuable priority. At least speaking for myself, my primary attraction to Deno is its fast and build-/hassle-free TypeScript support. Everything else is icing on the cake.

129 points by inglor 5y ago

This is off by default due to security concerns but can be turned on with a flag. It's mainly in the community feedback phase so if there is anything you want to tell Node (be it "I want this!" or "This is a bad idea" - please do)

Andrex 5y ago
My layman's take is that it seems very similar to Log4j's vulnerability, but I don't really know for sure.
- oauea 5y ago
  Please explain why you think that, because it seems completely unrelated and off-topic.
  - moritonal 5y ago
    Bit harsh? Log4j was caused by a URL leading to bad code being imported. This let's you import code via a URL, seems quite similar?
    - sodality2 5y ago
      Log4j was caused when arbitrary log strings were interpreted as code imports. In this case the flag has to be manually set and the code manually added as a dependency.
    - anamexis 5y ago
      The Log4j vulnerability allowed unsafe user input to trigger loading of arbitrary code. Basically every dependency manager out there will allow you to load "bad code" from a URL.
      - tobyhinloopen 5y ago
        Log4j is no dependency manager
        anamexis 5y ago
        No, it's not, but this new Node.js feature is for loading dependencies
mitchellst 5y ago
Longtime node.js developer... I don't think we need this, and I don't think it's a good idea to add. The feature is _strange._ Which we know because it's behind a flag, i.e., it will have low utilization. It defies conventions—because the convention is to use a registry for packages. And it presents significant security concerns. (Which I don't really have to explain, as they're already cited as the reason to put it behind a flag.) Use of this feature seems like something you might see once in a career. (Indeed, I've seen a similar thing in another language employed only once... AND THERE WERE OTHER WAYS for that developer to have designed that system that would not have been so vulnerable.) When you see it, do you recognize what it is, understand and evaluate the risks it poses? Do you validate that the library sitting on the other side hasn't been taken over by a malicious actor? Do you validate that the library on the other side has up-to-date dependencies? If so, on what cadence do you do this, realistically? All the security tooling progress we've made (npm audit, etc.) seems shot in the foot by this. And I get that these concerns aren't dispositive on their own. I just don't see the argument on the other side that clearly. Is there some class of application that people really want to write in node that can't be written for lacking this? Do we really think this is a pattern that would be employed in well-made, secure codebases? I don't see the argument. Like I said, I've seen a feature like this used in another language. But I haven't seen it needed for any real application. I love me some good node ugly hacks. Still, the language features should be nudging us to write good software.
- encryptluks2 5y ago
  Central authority != Good software. You are essentially advocating to provide a single point of failure that can and will be abused by nation-state hackers who just have to find one popular package and compromise the developer.
  - mitchellst 5y ago
    The argument is less about central authority than standard process. You don't have to use the official NPM registry. The point is that having a pipeline for modules with established conventions and automated means of audits and visibility on the dependency is a net good. Yes, there is such a thing as a supply chain attack. But even when that happens, the community puts out the call to patch immediately and points a finger at the offending package. Your HTTPS import from a random URL could fail silently. While NPM is a big target for supply chain attack, they know they are, so there are a lot of eyes on it. Not so with your HTTPS import.
pimterry 5y ago
In terms of feedback, there's an issue thread here covering some points already: https://github.com/nodejs/node/issues/41920 https://github.com/nodejs/node/issues/41920
sdflhasjd 5y ago
I recall when I was a completely PHP n00b (babbys first programming language), I might encounter an issue and search for the error on expertsexchange. Some of the answers would be really complicated, but there'd be that one answer: Just edit your php.ini and change `dangerous_option_you_will_die=true` - wow, such an easy fix. I'd know better later, but I'd still find horrendous options like these enabled in real software I would come accross in my professional career.
forty 5y ago
I have been doing nodejs backend developpement for the past 10 years. I'm not sure why we would need that honestly. Looks like it could create some mess if packages on npm all start importing files from random servers. Suddenly your CI relies on many servers to be up. And I don't know what's the problem with declaring things in package.json.
danShumway 5y ago
All of the things that make this enticing as a feature are large security risks, and the less risky version that people are pitching me doesn't have any of the enticing features. What people are pitching me on is the idea that these wouldn't be dynamic imports. They're saying they wouldn't be fetched during runtime, they'd be fetched before. They're saying they would be pinned. Well, you're describing a package system like npm. And for those purposes, as a way of defining static packages that get installed before runtime and can't be dynamically swapped out by the application -- npm is better than the system being proposed. The only way this would actually be interesting is if they were actually dynamic imports, but having them be dynamic imports would be very insecure. And for declarative imports that can be statically analyzed, this is inferior syntax to what we already have and encourages worse developer habits. Some problems: 1. Updating dependencies is harder, because imports across multiple files have to be synced. If a junior developer imports `https://lodash.com/v4 https://lodash.com/v4` in two files and later someone changes that to be `http://lodash.com/v5 http://lodash.com/v5` in one file, now I have dependency mismatches. The existing system doesn't have this problem because my dependencies are in one place. 2. Finding out which dependencies I'm using is harder. Maybe there's another tool that lists out all of the imports, that would be handy. But also, maybe the dependencies are all in one file with the requested version numbers right next to them. That seems a lot more convenient. 3. This forces me to actually run the Node program to install the dependencies, and it forces me to actually import them in a file before they'll be fetched. This is cumbersome, it would be good to be able to list out dependencies before they ever get imported, say for boilerplate imports that are used in lots of projects. 4. People have pointed out the lack of pinning. All of this is inferior to having all of the imports in one file that you can specify, maintain, and update without running the program or using an analyzer. I don't understand what the benefit is or why I should care about an import system that is just as inflexible as the current npm/Yarn package manager setup, but that also has what seems to be worse syntax and makes it harder for me to know what packages are being imported. I wish I understood more what problem this proposed syntax was actually trying to solve. I've read through the entire thread and I still don't understand why people want this. People in the Github thread are reassuring readers that this could work exactly the same way as node_modules. That's great, but if that's the case, what is the problem with node_modules that is prompting us to build a second system that is less organized and less explicit about what gets imported? We can already import URLs with both npm and Yarn. That's not a new feature, the only new feature is that now the URLs are spread across the entire codebase.

oefrha 5y ago

In my limited experience with Deno, it seems direct URL import just leads to a crappy, nonstandard, hard-to-upgrade version of a package.json in the form of, say, a deps.ts, once you get past a few dependencies. (And it’s very easy to get past a few dependencies in Deno since std modules need to be imported as URLs like everything else.)

spiderice 5y ago
That is my experience as well. I was really excited about this feature in Deno when I first heard about it. Then I started to get in to Deno, and it just seemed like it made everything harder. I thought it was going to be something like "here is a better way to do dependency management". But in practice it felt like "we got rid of dependency management. Figure it out on your own".
- conaclos 5y ago
  I feel the same and import-maps do not solve the issue since they are not transitively loaded. In other word, if one of your dependency uses an import-map, then you have to copy-paste it in your own project to get its import-mapping.

progx 5y ago

So Deno is obsolete?

mccorrinall 5y ago
deno scripts need explicit permissions for using the file system, network etc that’s a pro in my book.
ravenstine 5y ago
No... If anything, this shows how ahead Deno is and how features like this in Node.js are too little too late. Deno's standard library is just better than Node's, IMO, and it's going to be hard for Node to play catchup, if that's even practical. In summary, it doesn't contain the cruft of the pre-ES2015 era that is still present in Node, even though Node has tried working around it. The global objects in Deno better replicate the browser, so it takes a lot less effort to do essentially the same things you would in a browser but in a DOMless environment. Deno is slightly more performant in Node in some areas, although this is usually negligible, and I don't have the JSPerf test on hand. I don't think Node is bad, by any means. If people are concerned about security... then don't use so many dependencies, or consider just not updating them until they've been demonstrated not to contain exploits; you don't actually need much code to essentially serve webpages. But I think Deno, over time, will at least match Node in popularity, maybe even more, because it has "modern" approaches right out of the box and doesn't need to move as slowly as Node in order to maintain a legacy. Although Deno misses some things here and there, I've seen features get added very quickly over the last couple years; at this point, it would take Node much longer to do the same things for fear of breaking something and because it's simply much older code. Then again, maybe I'll be wrong. Perhaps Node gets a second "big bang" and makes Deno obsolete. And I wouldn't care, either. All I want is a JavaScript language runtime with fewer concepts to learn that can also move quickly when something needs to be improved. If that ends up being Node, I'd be just as glad.
qbasic_forever 5y ago
Definitely not, deno does typescript out of the box without pulling in a bunch of npm dependencies and tools. It also is taking more of a go or python/batteries included view of building out a proper standard library vs. relying on the community to provide it. And of course deno has a much stricter execution sandbox model where you have to opt in to explicit permissions like file system access. IMHO we'll probably see node get more and more deno-like over time.

royjacobs 5y ago

Won't this make leftpad-type scenarios even _more_ likely? What is the benefit?

zja 5y ago
If node shares import syntax with browsers, it’s easier to write code that runs on both platforms. As far as leftpad, this isn’t that much different from using a package manager. You either cache your dependencies, or rely on a third party (npm repo/cdn).
onion2k 5y ago
When leftpad was in NPM there was an expectation that it wouldn't disappear suddenly. Package registries are thought to be a constant, reliable source of packages (even when they not quite that, as leftpad demonstrated). There would be no such expectation for a module that you include directly from an https URL. The main benefit I can see is that you can include code from your own "registry" very easily in unmanaged scripts. You can already do that because you can add a dependency to package.json using an https link already, but with this new feature you can include those dependencies in node scripts you aren't managing with a package file. You can do that. I'm not suggesting you should. I won't.

vorpalhex 5y ago

This is nice for many use cases, especially where running your own npm mirror is a pain. Hopefully you are controlling the http source though!

qbasic_forever 5y ago
Nice too in that you might not even need to touch or use npm. Imagine a node app without an enormous juggernaut of a node_modules folder!

999900000999 5y ago

I have mixed feelings here. On one hand do what you want, on the other hand you can already add https imports via package.json. Why in God's name would you not have a package.json. I think this would be awesome for hacking out prototypes, but if I saw this in production code I'd ether work to get it fixed or find a new job. Being reckless is fine in your spare time. It's not ok at work. My fear is this is going to push junior devs into sloppy practices.

sneak 5y ago
I have the distinct impression that whether you put the imports in package.json or in the source file itself, ~100% of the industry is engaging in sloppy practices now. I am terrified when trying to write secure es6. AIUI, when you add a module in go, the specific and exact version of the module is cryptographically locked in your project metadata. npm/yarn store the hashes in the lockfile but the package.json still uses those (usually) fuzzy matchers, and I'm not sure whether "npm i"/"yarn install" uses the lockfile exclusively by default. "npm help install" suggests that it uses package.json (which does not cryptographically secure the deps so that they match what you originally added) by default. (Additionally, tons of people install and deploy docker images by tag name alone, which is not cryptographically secure - different environment, same problem.) My mixed feelings about it are that we are slowly blurring the line between local and remote software, and pretty soon it will be nearly impossible to use most modern software in an offline environment (which is a hard requirement for certain security circumstances). Basically every single js developer in the world has granted Microsoft (owners of GitHub and NPM domains) the ability to remotely execute code on their machine at will. It's a lot of tooling and infrastructure to update and change in the event that the threat landscape changes significantly (e.g. war, national security mandates, etc).
- rectang 5y ago
  Manually auditing an update of potentially hundreds of packages when package-lock.json gets regenerated is so challenging that many projects won't do it — and for those projects, the cryptographic hashes offer limited value. To make industry-wide improvements, we need to share resources for auditing of packages. There are many such endeavors underway at various stages of maturity. But this feature doesn't move that ball forwards. From the comments: https://github.com/nodejs/node/pull/36328#issuecomment-736034792 https://github.com/nodejs/node/pull/36328#issuecomment-73603... > It is insecure because there is no signature of the original file. Anybody that can take control of a domain name can take control of a server. Downloaded arbitrary code can't be added to a pool of pre-validated packages, except insofar as you control the server and only make pre-validated packages available on it exclusively to your own services.

neals 5y ago

Does anybody have an exmple of this and how / why I could use this?

fredley 5y ago
You can use it to shoot yourself in the foot.
qbasic_forever 5y ago
Presumably it's just like how deno does dependencies: https://deno.land/manual/examples/manage_dependencies https://deno.land/manual/examples/manage_dependencies As for why, it's simpler than adding a package.json and meticulously curating it. If you only have a few small dependencies this might make sense. You can also pull in stuff that isn't hosted on npm or properly packaged--for example just pull down a js file from someone's github repo. If you start getting a lot of dependencies then it might turn into a new nightmare to maintain. It's an experiment--people don't really know if it's good or bad until using it a bit.
- deleted 5y ago
  [deleted]

mgkimsal 5y ago

Am I wrong in thinking this is similar to the below code? <?php include("https://otherdomain.com/somephpcode.txt https://otherdomain.com/somephpcode.txt"); ?> PHP got ridiculed for years for having this sort of stuff available (even if off by default eventually).

deleted 5y ago
[deleted]
markstos 5y ago
Sounds like it. These days, HTML is trusting JavaScript that's loaded from all over the place already. A "secure" database hosting service loads assets from 20 different domains on their login page. Loading code from remote domains can be abused and mis-used in any language. In this case, I think it's considered a "feature" because code was already being loaded remotely with NPM, who in turn got the code from the original author. Now you can load the code directly from the author. So instead of trusting both the author and NPM, you only need to trust the author. This could be considered a reduction in your attack surface area, and it's definitely helping with the NPM registry being a single-point-of-failure in the Node.js ecosystem.
- dangerface 5y ago
  Npm won't allow you to require remote code from a user touched variable like the below. myRemoteLib = require(userGeneratedVariable)
  - eyelidlessness 5y ago
    I don’t know where you got that idea. NPM doesn’t have any say in CJS (or ESM) module resolution other than that they conform to relevant aspects of it. Node and other implementers of those module systems absolutely do allow dynamic require/import.
nexuist 5y ago
PHP got ridiculed for years for doing all the things all the other popular frameworks are coming around to support now. Server side rendering is particularly hilarious because it's like: oh, you're compiling .jsx files to .html now? Cool....can't imagine anyone ever did that before...

mrozbarry 5y ago

My impression is this is really just to have some sort of comparable feature to deno. You may think this somehow is more of a vulnerability, but the reality is the npm registry is already fairly insecure, we've seen npm packages get hijacked regularly. This also adds less dependence on npm, and maybe with all the packages that get hijacked there, this could be a reasonable move.

junon 5y ago
I'd really argue "regularly" is a bit much. It happens rarely and moving to URIs is only shifting the problem. I'd also argue the claim npm is insecure. How would you improve its security over what's there now? Lastly, there isn't really a dependence on npm at all. You were always able to 1) put URIs into package.json, and 2) make your own package manager. Node does not require npm to work. It just looks in node_modules by default, which npm uses to manage packages for you. They are completely decoupled. URI imports, on the other hand, have a lot of security problems and surface area and will probably face a load of criticism going forward.
- dmitryminkovsky 5y ago
  > How would you improve its security over what's there now? Automated builds on NPM servers based on code in public repos, like Docker has (had?). The least they could do is require packages to be signed.
  - junon 5y ago
    How exactly does that help?
    - potatoz2 5y ago
      Controlled builds based on public repos prevents a malicious person (original author or not) from invisibly pushing a packages that doesn’t correspond to public sources. Signing packages prevents account takeovers from publishing bad packages. The remaining security problem is the author themselves coupled with too few eyes on public repos.

user-the-name 5y ago

So they are adding a hidden, targetable supply chain attack vector on purpose?

FlorianRappl 5y ago

What about TypeScript here? I think it does not yet understand such imports. However, especially for types (i.e., evaluation at compile-time) imports from a remote source would be good. Instead of keeping up-to-date via npm I could just write `import type { FooType } from 'https://remote-types-registry.com/foopackage@2 https://remote-types-registry.com/foopackage@2'` and wouldn't have to worry about typing updates. (surely this scenario is only valid when types are not shipped with the dependency)

edgyquant 5y ago
This is basically how Deno does things already

notpachet 5y ago

The people in the Github thread advocating for this feature seem to be pushing really hard on the point that this isn't any more insecure than NPM already is. But that's myopic. That's like saying that the lock is broken on the second floor window of your house, so you may as well unlock all the other windows. If a junior developer working for me proposed something like this I would pull them away from the computer and have a long coffee chat with them about basic architecture principles. Yes, we know there are problems with NPM. But why broaden the possible attack surface area by introducing another kind-of-sort-of-equivalent means of remote module loading? Now you're just going to have to spend time hardening two systems instead of one. At the end of the day this just feels like a misguided attempt to try and recapture some sense of dynamism from Deno. But if I wanted dynamism, I'd use Deno, with all of its pubescent awkwardness. I'm using Node.js stable for a reason. I spend a lot of cycles trying to defend Node from its many (often reasonable) detractors, and things like this make me feel like I'm wasting my breath...

encryptluks2 5y ago
- junon 5y ago
  Lul what? First of all, this tone is completely against the HN guidelines, so take your flag. Second of all, centralization is not often a part of a security model. Usually in such cases where availability is a concern you have a mirror set up, which is ridiculously easy to do with CouchDB and completely nullifies your arrogant point. Get off your high horse.
  - encryptluks2 5y ago
- notpachet 5y ago
  My comment wasn't really about the merits of central authorities vs multiple authorities (although I would fall on the side of preferring a single package repository than having to worry about many of them). It was about having two equivalent mechanisms for solving the remote-code-loading problem, regardless of where you're actually loading it from.

bricss 5y ago

More attack vectors for attack ↯ vectors ⩘⩗ sake ‽ Idea for the new toolset start-up: - A cyberware to validate source code and its modules against https imports

BeefWellington 5y ago

My main problem with this is that they punted the integrity check work down the line. If this had been implemented as something akin to the integrity HTML attribute, e.g.: express = require( 'https://my_local_repo.tld/express/current/4.17.1', 'sha384-oqVuAfXRKap7fdgcCY5uykM6+R9GqQ8K/uxy9rx7HNQlGYl1kPzQho1wx4JwY8wC' ) It would be vastly superior. I'd go so far as to require it without an additional --unsafe-import type flag. However, I think they did not do this because the import of dependencies then comes into question. IMO this feature should not have been added until developers could guarantee the expected integrity hash matches. We've seen time and again what will happen now is hundreds of blog posts talking about this new feature, none of which will mention the integrity hashing functionality (because it's not there). When it does eventually get added, those posts will be the most popular anyone can find, and thus new users will end up using those eventually-wrong posts and implementing things in a less-secure way. Huge disservice to the developer community.

sneak 5y ago
It's rubygems all over again. I think only Nix and Go are doing dependencies right (cryptographically verified) these days. One upside is that it's https-only, which does include authentication, provided that you trust the holder of the certificate for my_local_repo.tld (in your example) to serve you the right files.
- francislavoie 5y ago
  Don't forget about PHP. Composer is one of the best package managers around.
jdcaron 5y ago
Exactly, this kind if import without a hash validation is a big no for security reasons (unless you 100% trust your import source). This feature exists on the browser side with the script element: https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity https://developer.mozilla.org/en-US/docs/Web/Security/Subres...
- 5y ago

jppope 5y ago

probably just use deno?

x-shadowban 5y ago
Seems like a lot of new node features are in response to competition with deno's offerings. Neat to watch.

mstade 5y ago

Direct link to the rendered docs for this feature: https://github.com/bmeck/node/blob/ceadb473e6d3a22f38b737108fb5ac943bf8be09/doc/api/esm.md#https-and-http-imports https://github.com/bmeck/node/blob/ceadb473e6d3a22f38b737108... It's unfortunate I think that the docs don't mention integrity checks or caching, but those things are mentioned in the PR thread. Integrity via policies which seems to be a new feature as well[1], perhaps designed to essentially obsolete things like package-lock.json et al? Caching is punted till later, presumably to not hold up progress on this particular feature. Seems to me these are actually steps in the right direction, paving some of the cow paths already created by npm and yarn. I for one would love to see package-lock.json etc make way for policies. Security is a concern for sure if there's no integrity checking on the packages loaded via https, but the same holds true for packages regardless of delivery method so I don't see how this is much worse security wise than just declaring dependencies in package.json. Policies does seem to be an effort to solve the integrity issue regardless of how the package is installed, which seems like a good idea to me. If the caching and integrity issues are resolved well then this would essentially mean package managers become optional, no? Seems pretty good to me! [1]: https://nodejs.org/dist/latest/docs/api/policy.html https://nodejs.org/dist/latest/docs/api/policy.html

jchw 5y ago

A lot of the complaints here claim that this is bad because it’s insecure. However, allow me to posit a question: what if I made a function that just fetches the script and evals it? Node.JS already involves running unsandboxed, practically untrusted arbitrary code, that nobody audits. I’d have more sympathy if not for the fact that you can pretty much emulate this with the permission level that Node.JS code runs today. Also, no… this is not like PHP. PHP include got lambasted because it compounded other PHP security issues. If you use ESM imports, the obvious way, there’s never going to be an accident where user input can control the URL, because the grammar makes it impossible. You can use the dynamic import statement, but at that point you’re off into the abyss.

user-the-name 5y ago
There is value in making insecure things hard and awkward to do, rather than providing them as first-class constructs.
- jchw 5y ago
  While this is true, if this is going to be the future, it may as well be provided as a feature with the proper precautions. While this doesn’t fully sandbox code, it does at least seem to provide some decent baseline security features. If I needed to import code from a web address for some reason, this is certainly the way to go, and whether ideal or not, sometimes that is a need that crops up. That said, proper sandboxing and/or at least optional integrity checking would certainly be nice.

CGamesPlay 5y ago

Evaluating these things at runtime is absurd. It was absurd when deno wanted to do it and it’s absurd now. People keep equating the threat model of npm and url imports, but with npm the untrusted code gets downloaded at build time, so tools like snyk can scan it and flag problems. Is snyk supposed to intercept the live download and patch files now? Are the policy integrity checks they mention in the PR allow or block lists? For example, if my malicious trojan module does a dynamic import, can it bypass a saved integrity check by adding a cache bust url parameter? Honestly, I don’t think the idea of URL imports is inherently bad. Go manages with them pretty reasonably. However, go only processes them at build time. My docker image from node might run today and break tomorrow because it’s allowed to change the remote code it runs day to day.

wperron 5y ago
If you actually read the (admittedly long) thread, it's pretty clear that those imports are not resolved at runtime, and Deno has never evaluated imports at runtime either. Dependencies get evaluated and downloaded prior to startup, very similar to Go in fact.
- CGamesPlay 5y ago
  I don't see that listed anywhere in the thread, so I'll ask here instead: Does prior to startup mean there's a separate build step, or are you saying it happens during the loading of the script. I'm pretty confident that the latter is what deno was doing, although I'm happy to be proven wrong. Are dynamic imports simply disallowed? Since it is impossible to download in advance a script to which you don't know the URL without running the program, they'd have to be, in order for your assertion to be true.
  - wperron 5y ago
    The whole conversation is a bit hard to follow, there was an issue initially, then this PR and then another Discussion that's linked somewhere in the PR. I'll let you go read those, I don't recall exactly what decisions were taken and what's the roadmap for this to become stable. Dynamic imports are... wild, to say the least. Unfortunately they will always be inherently harder to secure, and will most likely always be treated differently than static import statements. At least that was the consensus in Deno.

numlock86 5y ago

At this point they just try to make the ecosystem even worse and open as many attack vectors as possible, aren't they? Has it become some "funny meme" in the maintainer circle or something? Why there is no SRI for example?

dangerface 5y ago

Loading modules over https because why? are dependancy attacks currently too difficult? Thanks I hate it.

mrweasel 5y ago

Won't this effectively prevent an application to start if it cannot reach the internet? I haven't done much development with Node, but I've seen similar patterns with Java applications that attempt to pull in an XSD at start up, only to fail to boot completely, because the application server doesn't have direct internet access. It's a bit of a weird feature, which should only be used for a very limited number of project, in a niche which I'm not completely able to identify.

tehbeard 5y ago

This is, IMO, the weakest part of deno being implemented. With npm I can easily get metrics for how stale dependencies are or quickly check for security issues. The closest I found for deno was maybe trex? And that felt incredibly anemic.

danShumway 5y ago

Kind of an interesting read. First thoughts, I generally think this is bad practice. I mean, for one thing this guarantees you need an Internet connection to run your Node script, which seems pretty problematic. I'm also reading some really questionable security justifications from people on the issue tracker about why dynamic imports are actually fine and just as secure as importing packages from NPM. There are security improvements that could be made to NPM, particularly around package search, and NPM has suffered from people replacing existing package versions (although I think they're clamping down on this more now). But there is a really fundamental difference between taking that risk once during installation and taking that risk multiple times at runtime. Additionally, in regards to some of the comments saying that installing code directly from the source ("express.com" vs "npm.com") ignores the fact that this is already kind of possible with npm dependencies and promoting that method wouldn't have any of the other problems. ---- HOWEVER: You read further and there are actually some really interesting security risk mitigations being proposed that don't just boil down to saying "all imports are dangerous, why not do them here too." "Scopes" (which I was not aware was a concept that Node had any support for) are the idea that packages in a scope or a domain can't access any dependencies outside of them. Additionally, people are talking about mirroring Deno's system where these get cached locally and not re-fetched later, which is kind of similar to npm's system and reduces some risk. So you can load a module from HTTPS and in theory it only gets fetched once and it can't access the file system, cool. Additionally, `node_modules` dependencies can't use them, which is cool, means that it's less likely a random dependency starts doing this behind your back. ---- HOWEVER: Prototype pollution is still a thing, passing objects around across closures is still a thing, and as far as I know this is a big part of why it's difficult to sandbox Javascript modules. Scopes are really interesting, but I can't find a lot of information about them online, and their name conflicts with npm scopes, which are a different thing. And I can't find any info suggesting they've solved the bigger problems of sandboxing modules. And there's still the elephant in the room, which is that all of those security measures are stuff that could be done in a more general way that applies to normal modules? I'm not sure why I would only want scopes for HTTPS imports, assuming they even work securely for HTTPS imports in the first place, which I wish I could find more information about. It's cool that a module gets fetched once at runtime and not later, but that still means I need an Internet connection the first time the Node script runs. And it seems like "tell me what dependencies you need and I'll fetch them" is a good candidate for an install step, which Node has. I'm not totally against the idea, I can in theory think of situations where you might want to install something at runtime without knowing ahead of time what it is. But that's also really dangerous to do, and doesn't really fit with the caching model that is being proposed. ---- So I still feel pretty doubtful that this is a good feature to add, and it seems really dangerous at first glance, but maybe there are other use-cases that I don't understand or maybe there's something to scopes that I don't get. I will say that if I actually trusted the security of scopes as a way of sandboxing individual modules, then at that point I feel a lot better about dangerous imports in general. But I can't find any info on scopes, and as far as I know, systems like Deno gave up on this because it was really hard, and I also vaguely feel like effective module-level sandboxing in Node would be much bigger news than any import feature. But :shrug:, I could be missing something. This is based on only a quick read through the entire thread and a tiny bit of extra searching/reading the commit docs.

eyelidlessness 5y ago

A couple observations of interest to me: 1. This follows closely behind the recent addition of fetch support. I expected this would leverage that, but it does not. So I assume this was developed in greater isolation than I would have hoped. I don’t have time to look deeper now, but given overlapping responsibilities I would think fetch would be a better fit than the older http(s) APIs. 2. Again following close behind landing fetch, Node is gaining more and more feature overlap with Deno. It’s not clear yet, to me anyway, whether that’s a direct motivation… or a somewhat convenient coincidence of similar goals. If it is a direct motivation, I feel like API compatibility, while valuable, may not be the most valuable priority. At least speaking for myself, my primary attraction to Deno is its fast and build-/hassle-free TypeScript support. Everything else is icing on the cake.