5 ms·

Show HN: Auto install npm dependencies as you code

60 points by siddharthkp 10y ago

KirinDave 10y ago
Can you turn off the uninstall feature? I envision, "Nooooo, I was just refactoring! The network here is so slow noooooo please!" Maven knows I've made the mistake of triggering a big download on a bad network.
- siddharthkp 10y ago
  Sure! Opened up an issue https://github.com/siddharthkp/auto-install/issues/1 https://github.com/siddharthkp/auto-install/issues/1
Rauchg 10y ago
Very cool! The Webpack equivalent: https://github.com/ericclemmons/npm-install-webpack-plugin https://github.com/ericclemmons/npm-install-webpack-plugin
- siddharthkp 10y ago
  Sweet!
randallsquared 10y ago
Given that in npm v3 the order of installation determines where things are installed, this seems like it might cause some interesting bugs that never happen on the developer's machine. :)
- Etzos 10y ago
  I'm not sure how this could happen as one shouldn't be depending on the directory structure in the node_modules/ folder anyway. Do you have an example of where this might cause issues?
  - randallsquared 10y ago
    Well, one shouldn't... :) I don't actually know that there's any problem that `rm node_modules; npm install` won't solve, since that would make things the same as a fresh install. All of the scenarios I could come up with were about long-running installs, rather than new deploys, so more likely to happen on dev than prod. ETA: actually wrong, see below.
    - Etzos 10y ago
      I can't think of any scenario when one would be relying on a specific structure being in the node_modules/ directory. And if there is such a need, then it's likely the code is doing something very strange or very special in which case this kind of package would not even be considered during creation (and I still think wouldn't cause problems). Even in long-running installs I still don't see a potential issue here. Do you have a small specific example you can think of?
      - randallsquared 10y ago
        Package A depends on lodash: ^4.6.1 Package B depends on lodash: ^4.14.2 ... you know what? I just installed semver to check my beliefs, and it turns out I'm just spreading FUD. Caret doesn't work the way I thought it did. Please disregard.
- deleted 10y ago
  [deleted]
- siddharthkp 10y ago
  Wow, I'll have to look that up. Thanks!
akamel 10y ago
So I can make a malicious module called expres and another one called expresss and screw with ppls machines?
- lima 10y ago
  This. Installing modules from npm is dangerous enough. Nice for education or playing around, unsuitable for a serious developers' workstation. Related: http://incolumitas.com/2016/06/08/typosquatting-package-managers/ http://incolumitas.com/2016/06/08/typosquatting-package-mana...
- micaksica 10y ago
  This. I love that the Node community enjoys innovating for convenience, but ideas like this one are less than half-baked from a security perspective. Just make a few typo'd popular packages, and use npm install scripts [1] and you have a very easy remote code execution vector on developer workstations. The bigger problem I see is that npm is a circus. No package signing and a ridiculous debate on why that's been going on for a year and a half [2]. Credentials leaks of popular modules. [3] When everything is a module and everyone is supposed to include modules vs. writing their own very simple functionality for things even like isArray polyfills [4] (24MM downloads a month!), you end up with the same attack surface that gives WordPress such a shitty reputation for security. It's not usually core, it's all the plugins by authors of unknown provenance and skill. WordPress gets pwned because there are a lot of plugins hastily written by new developers and used without audit by mom-and-pop web app shops and/or those that trust the code because they aren't capable of auditing it meaningfully. When you use an npm dependency, you are taking on all of their dependencies. You are trusting they don't leak creds, that npm has not been compromised, and that the chain underneath has been audited for malicious behavior. In reality this is impractical: go npm install express and see just how deep the dependency chain goes. Things like Snyk are required to just understand what might be vulnerable. [1] https://docs.npmjs.com/misc/scripts https://docs.npmjs.com/misc/scripts [2] https://github.com/node-forward/discussions/issues/29 https://github.com/node-forward/discussions/issues/29 [3] https://github.com/ChALkeR/notes/blob/master/Do-not-underestimate-credentials-leaks.md https://github.com/ChALkeR/notes/blob/master/Do-not-underest... [4] https://www.npmjs.com/package/isarray https://www.npmjs.com/package/isarray EDIT: Ironically, this module itself is vulnerable to code injection.
joshstrange 10y ago
I am very interested in this for projects that I'm just starting and don't expect to run on anything but my machine. It's an annoying break of flow to be writing code and say "Crap, I need request or lodash" and have to stop, npm install --save, require, then go back to what I'm doing. Yes I still have to require it in but for side projects/one-offs I find this pretty cool.
- throwanem 10y ago
  This might just be a question of workflow optimization. Have you considered waiting to do the installs until you've paused in writing code anyway, or using your editor's shell command execution capability (if any) to fire off these installs and let them run in background while you continue to work?
- siddharthkp 10y ago
  You feel me, brother! That's exactly what happened when I decided to create this :)
concise_unicorn 10y ago
Regular expressions are a very naive way of detecting calls to 'require'. For correctness you're better off recursively walking the AST. I've successfully used Detective in a couple of my personal projects to find all require statements. Relevant issue on Detective: https://github.com/substack/node-detective/issues/8 https://github.com/substack/node-detective/issues/8
- _RPM 10y ago
  The AST? What AST is that? You don't get access to an AST.
  - throwanem 10y ago
    The AST you get from any of a wide variety of parsers, of which esprima may still be only the most popular of many.
  - Etzos 10y ago
    It has to open and read a .js file already, it can certainly turn that into the representative AST for said file and then use the data from that. It will be slower, but it will also be more accurate and less likely to turn up false positives or miss things.
    - _RPM 10y ago
      It has to parse a javascript file, which isn't trivial. The reason they use regular expression is because implementing a javascript parser isn't an easy problem to solve fast, even though the grammar is available.
      - elmigranto 10y ago
        You downloading megabytes over network and performing disk IO with it. What's the problem with 100 milliseconds of file parsing?
      - Etzos 10y ago
nathancahill 10y ago
Just when you thought dependency hell couldn't get any worse, it becomes automatic.
tomc1985 10y ago
Dependency installation should be a bit frictional...
- siddharthkp 10y ago
  That's interesting, tell me more?
  - pc86 10y ago
    A typo should not start installing random software on my machine.
  - dack 10y ago
    I could be wrong, but I think he might mean that people should take care not to add too many dependencies to their project - if it's too easy, then it might result in unnecessary dependencies and brittle code. However, I'm not sure I agree with the statement - you could use this tool and still have the discipline not to pull in random packages.
    - dozzie 10y ago
      The idea is that people in general go down the currently-easier way, which is "add the dependency", leading to microdependencies and left-pad idiocy in npm case. If there is a friction, the balance is changed a little against pulling dependencies, at least those most trivial.
      - tomc1985 10y ago
        This!
  - dividuum 10y ago
    In any application with a lifecycle longer than your average "TodoMVC example" dependencies become a liability: You have to make or be sure than they are going to be around and still compatible with your current and future code. If you blindly add dependencies this problem gets more complicated.
qwertyuiop924 10y ago
I mean, wow. I've seen half-baked ideas before, but this takes the cake. Typos now equal remote code execution, and it's even easier to enter dependancy hell, because deps are now implicit.
- ed 10y ago
  The whole node ecosystem is a mess. Shrinkwraping is a pain, easily bottoms out due to node's small default stack limits, and often the only solution is to `rm -r node_modules` and rebuild the dependency graph from scratch.
- binarymax 10y ago
  Typos with npm have always allowed RCE. You can have typos while doing $npm install ....
  - cortesoft 10y ago
    Right, but when you are running npm install you know you are installing and so you know you should be careful, and only hit return if you are sure it is right. With this, you have to be careful the WHOLE time you are typing. You normally don't worry about hitting return in your editor causing RCE.
    - siddharthkp 10y ago
      Good point, any ideas on handling this better? Opened an issue here: https://github.com/siddharthkp/auto-install/issues/2 https://github.com/siddharthkp/auto-install/issues/2
      - cortesoft 10y ago
        I don't really know if there is a way to fix this issue... unless you had some sort of whitelist of acceptable packages, and a prompt if you try to install a non-whitelisted package. One other possibility is to have a delay, so that it waits an amount of time before installing, to give a chance to catch and fix the typo.
VOYD 10y ago
no.
VOYD 10y ago
no.
SrslyJosh 10y ago
Belongs in the Acme:: namespace. Oh wait, npm doesn't even have namespaces... Remind me again why everyone is using this shitheap?
joelg 10y ago
Lots of people addressing security and remote code execution by typo. Yes. You're right. If it scares you, don't use it. It's always possible to run malicious code by typo, and this is only a little different from installing dependencies from the terminal. Even when you do spell a package's name correctly, you still don't know for sure what you're installing. The guy just made a cool thing - it seems a little out-of-scope to freak out over security when npm was never really there.
- siddharthkp 10y ago
  Thanks for the support :) The concerns are fair though, just added a --secure flag which will install popular modules only (>10k downloads last month)
siddharthkp 10y ago
Thanks for pointing out the obvious risks due to typos. Added a --secure flag which will install popular modules only (>10k downloads last month) `auto-install --secure`
- mxstbr 10y ago
  That's one fast response time! Awesome!
  - siddharthkp 10y ago
    ;)

60 points by siddharthkp 10y ago

Can you turn off the uninstall feature? I envision, "Nooooo, I was just refactoring! The network here is so slow noooooo please!" Maven knows I've made the mistake of triggering a big download on a bad network.

siddharthkp 10y ago
Sure! Opened up an issue https://github.com/siddharthkp/auto-install/issues/1 https://github.com/siddharthkp/auto-install/issues/1

Rauchg 10y ago

Very cool! The Webpack equivalent: https://github.com/ericclemmons/npm-install-webpack-plugin https://github.com/ericclemmons/npm-install-webpack-plugin

siddharthkp 10y ago
Sweet!

randallsquared 10y ago

Given that in npm v3 the order of installation determines where things are installed, this seems like it might cause some interesting bugs that never happen on the developer's machine. :)

Etzos 10y ago
I'm not sure how this could happen as one shouldn't be depending on the directory structure in the node_modules/ folder anyway. Do you have an example of where this might cause issues?
- randallsquared 10y ago
  Well, one shouldn't... :) I don't actually know that there's any problem that `rm node_modules; npm install` won't solve, since that would make things the same as a fresh install. All of the scenarios I could come up with were about long-running installs, rather than new deploys, so more likely to happen on dev than prod. ETA: actually wrong, see below.
  - Etzos 10y ago
    I can't think of any scenario when one would be relying on a specific structure being in the node_modules/ directory. And if there is such a need, then it's likely the code is doing something very strange or very special in which case this kind of package would not even be considered during creation (and I still think wouldn't cause problems). Even in long-running installs I still don't see a potential issue here. Do you have a small specific example you can think of?
    - randallsquared 10y ago
      Package A depends on lodash: ^4.6.1 Package B depends on lodash: ^4.14.2 ... you know what? I just installed semver to check my beliefs, and it turns out I'm just spreading FUD. Caret doesn't work the way I thought it did. Please disregard.
deleted 10y ago
[deleted]
siddharthkp 10y ago
Wow, I'll have to look that up. Thanks!

akamel 10y ago

So I can make a malicious module called expres and another one called expresss and screw with ppls machines?

lima 10y ago
This. Installing modules from npm is dangerous enough. Nice for education or playing around, unsuitable for a serious developers' workstation. Related: http://incolumitas.com/2016/06/08/typosquatting-package-managers/ http://incolumitas.com/2016/06/08/typosquatting-package-mana...
micaksica 10y ago
This. I love that the Node community enjoys innovating for convenience, but ideas like this one are less than half-baked from a security perspective. Just make a few typo'd popular packages, and use npm install scripts [1] and you have a very easy remote code execution vector on developer workstations. The bigger problem I see is that npm is a circus. No package signing and a ridiculous debate on why that's been going on for a year and a half [2]. Credentials leaks of popular modules. [3] When everything is a module and everyone is supposed to include modules vs. writing their own very simple functionality for things even like isArray polyfills [4] (24MM downloads a month!), you end up with the same attack surface that gives WordPress such a shitty reputation for security. It's not usually core, it's all the plugins by authors of unknown provenance and skill. WordPress gets pwned because there are a lot of plugins hastily written by new developers and used without audit by mom-and-pop web app shops and/or those that trust the code because they aren't capable of auditing it meaningfully. When you use an npm dependency, you are taking on all of their dependencies. You are trusting they don't leak creds, that npm has not been compromised, and that the chain underneath has been audited for malicious behavior. In reality this is impractical: go npm install express and see just how deep the dependency chain goes. Things like Snyk are required to just understand what might be vulnerable. [1] https://docs.npmjs.com/misc/scripts https://docs.npmjs.com/misc/scripts [2] https://github.com/node-forward/discussions/issues/29 https://github.com/node-forward/discussions/issues/29 [3] https://github.com/ChALkeR/notes/blob/master/Do-not-underestimate-credentials-leaks.md https://github.com/ChALkeR/notes/blob/master/Do-not-underest... [4] https://www.npmjs.com/package/isarray https://www.npmjs.com/package/isarray EDIT: Ironically, this module itself is vulnerable to code injection.

joshstrange 10y ago

I am very interested in this for projects that I'm just starting and don't expect to run on anything but my machine. It's an annoying break of flow to be writing code and say "Crap, I need request or lodash" and have to stop, npm install --save, require, then go back to what I'm doing. Yes I still have to require it in but for side projects/one-offs I find this pretty cool.

throwanem 10y ago
This might just be a question of workflow optimization. Have you considered waiting to do the installs until you've paused in writing code anyway, or using your editor's shell command execution capability (if any) to fire off these installs and let them run in background while you continue to work?
siddharthkp 10y ago
You feel me, brother! That's exactly what happened when I decided to create this :)

concise_unicorn 10y ago

Regular expressions are a very naive way of detecting calls to 'require'. For correctness you're better off recursively walking the AST. I've successfully used Detective in a couple of my personal projects to find all require statements. Relevant issue on Detective: https://github.com/substack/node-detective/issues/8 https://github.com/substack/node-detective/issues/8

_RPM 10y ago
The AST? What AST is that? You don't get access to an AST.
- throwanem 10y ago
  The AST you get from any of a wide variety of parsers, of which esprima may still be only the most popular of many.
- Etzos 10y ago
  It has to open and read a .js file already, it can certainly turn that into the representative AST for said file and then use the data from that. It will be slower, but it will also be more accurate and less likely to turn up false positives or miss things.
  - _RPM 10y ago
    It has to parse a javascript file, which isn't trivial. The reason they use regular expression is because implementing a javascript parser isn't an easy problem to solve fast, even though the grammar is available.
    - elmigranto 10y ago
      You downloading megabytes over network and performing disk IO with it. What's the problem with 100 milliseconds of file parsing?
    - Etzos 10y ago

nathancahill 10y ago

Just when you thought dependency hell couldn't get any worse, it becomes automatic.

tomc1985 10y ago

Dependency installation should be a bit frictional...

siddharthkp 10y ago
That's interesting, tell me more?
- pc86 10y ago
  A typo should not start installing random software on my machine.
- dack 10y ago
  I could be wrong, but I think he might mean that people should take care not to add too many dependencies to their project - if it's too easy, then it might result in unnecessary dependencies and brittle code. However, I'm not sure I agree with the statement - you could use this tool and still have the discipline not to pull in random packages.
  - dozzie 10y ago
    The idea is that people in general go down the currently-easier way, which is "add the dependency", leading to microdependencies and left-pad idiocy in npm case. If there is a friction, the balance is changed a little against pulling dependencies, at least those most trivial.
    - tomc1985 10y ago
      This!
- dividuum 10y ago
  In any application with a lifecycle longer than your average "TodoMVC example" dependencies become a liability: You have to make or be sure than they are going to be around and still compatible with your current and future code. If you blindly add dependencies this problem gets more complicated.

qwertyuiop924 10y ago

I mean, wow. I've seen half-baked ideas before, but this takes the cake. Typos now equal remote code execution, and it's even easier to enter dependancy hell, because deps are now implicit.

ed 10y ago
The whole node ecosystem is a mess. Shrinkwraping is a pain, easily bottoms out due to node's small default stack limits, and often the only solution is to `rm -r node_modules` and rebuild the dependency graph from scratch.
binarymax 10y ago
Typos with npm have always allowed RCE. You can have typos while doing $npm install ....
- cortesoft 10y ago
  Right, but when you are running npm install you know you are installing and so you know you should be careful, and only hit return if you are sure it is right. With this, you have to be careful the WHOLE time you are typing. You normally don't worry about hitting return in your editor causing RCE.
  - siddharthkp 10y ago
    Good point, any ideas on handling this better? Opened an issue here: https://github.com/siddharthkp/auto-install/issues/2 https://github.com/siddharthkp/auto-install/issues/2
    - cortesoft 10y ago
      I don't really know if there is a way to fix this issue... unless you had some sort of whitelist of acceptable packages, and a prompt if you try to install a non-whitelisted package. One other possibility is to have a delay, so that it waits an amount of time before installing, to give a chance to catch and fix the typo.

VOYD 10y ago

no.

VOYD 10y ago

no.

SrslyJosh 10y ago

Belongs in the Acme:: namespace. Oh wait, npm doesn't even have namespaces... Remind me again why everyone is using this shitheap?

joelg 10y ago

Lots of people addressing security and remote code execution by typo. Yes. You're right. If it scares you, don't use it. It's always possible to run malicious code by typo, and this is only a little different from installing dependencies from the terminal. Even when you do spell a package's name correctly, you still don't know for sure what you're installing. The guy just made a cool thing - it seems a little out-of-scope to freak out over security when npm was never really there.

siddharthkp 10y ago
Thanks for the support :) The concerns are fair though, just added a --secure flag which will install popular modules only (>10k downloads last month)

siddharthkp 10y ago

Thanks for pointing out the obvious risks due to typos. Added a --secure flag which will install popular modules only (>10k downloads last month) `auto-install --secure`

mxstbr 10y ago
That's one fast response time! Awesome!
- siddharthkp 10y ago
  ;)