16 ms·

Caveat: I'm a node outsider, only forced to interact with it But there are a shocking number of install instructions that offer $(npm i -g) and if one is using

Caveat: I'm a node outsider, only forced to interact with it But there are a shocking number of install instructions that offer $(npm i -g) and if one is using Homebrew or nvm or a similar "user writable" node distribution, it won't prompt for sudo password and will cheerfully mangle the "origin" node_modules So, it's the same story as with python: yes, but only if the user is disciplined Now ruby drives me fucking bananas because it doesn't seem to have either concept: virtualenvs nor ./ruby_modules

andrewmcdonough 2y ago
Ruby has a number of solutions for this - rvm (the oldest, but less popular these days), rbenv (probably the most popular), chruby/gem_home (lightweight) or asdf (my personal choice as I can use the same tool for lots of languages). All of those tools install to locations that shouldn't need root.
- mdaniel 2y ago
  Yes, I am aware of all of those, although I couldn't offhand tell anyone the difference in tradeoffs between them. But I consider having to install a fresh copy of the whole distribution a grave antipattern. I'm aware that nvm and pyenv default to it and I don't like that I did notice how Homebrew sets env GEM_HOME=<Cellar>/libexec GEM_PATH=<Cellar>/libexec (e.g. <https://github.com/Homebrew/homebrew-core/blob/9f056db169d5f7e7cca264f406a370b3fb967ee4/Formula/f/fastlane.rb#L35-L36 https://github.com/Homebrew/homebrew-core/blob/9f056db169d5f...>) but, similar to my node experience, since I am a ruby outsider I don't totally grok what isolation that provides
MrJohz 2y ago
It's worth noting that Node allows two packages to have the same dependency at different versions, which means that `npm i -g` is typically a lot safer than a global `pip install`, because each package will essentially create its own dependency tree, isolated from other packages. In practice, NPM has a deduplication process that makes this more complicated, and so you can run into issues (although I believe other package managers can handle this better), but I rarely run into issues with this. That said, I agree that `npm i -g` is a poor system package manager, and you should typically be using Homebrew or whatever package manager makes the most sense on your system. That said, `npx` is a good alternative if you just want to run a command quickly to try it out or something like that.
- zahlman 2y ago
  >It's worth noting that Node allows two packages to have the same dependency at different versions Yes. It does this because JavaScript enables it - the default import syntax uses a file path. Python's default import syntax uses symbolic names. That allows you to do fun things like split a package across the filesystem, import from a zip file, and write custom importers, but doesn't offer a clean way to specify the version you want to import. So Pip doesn't try to install multiple versions either (which saves it the hassle of trying to namespace them). You could set up a system to make it work, but it'd be difficult and incredibly ugly. Some other language ecosystems don't have this problem because the import is resolved at compile time instead.
  - MrJohz 2y ago
    This is incorrect on several points. Firstly, Node's `require` syntax predates the modern JS `import` syntax. It is related to some attempts at the time to create tools that could act like a module system for the browser (in particular RequireJS), but it is distinct in that a lot of the rules about how modules would be resolved were specifically designed to make sense with NodeJS and the `node_modules` system. Secondly, although path imports are used for local imports, Node's `require` and `import` syntax both use symbolic names to refer to third-party packages (as well as built-in packages). If you have a module called `lodash`, you would write something like `import "lodash"`, and Node will resolve the name "lodash" to the correct location. This behaviour is in principle the same as Python's — the only change is the resolution logic, which is Node-specific, and not set by the Javascript language at all. The part of NodeJS that _does_ enable this behaviour is the scoped module installation. Conceptually, NPM install modules in this structure: * index.js (your code goes here) * node_modules (top level third-party modules go here) * react * index.js (react source code) * node_modules (react's dependencies go here) * lodash/index.js * vuejs * index.js (vuejs source code) * node_modules (vue's dependencies go here) * lodash/index.js * lodash * index.js Importantly, you can see that each dependency gets their own set of dependencies. When `node_modules/react/index.js` imports lodash, that will resolve to the copy of lodash installed within the react folder in the dependency tree. That way, React, VueJS, and the top-level package can all have their own, different versions of lodash. Of course in practice, you usually don't want lots of versions of the same library floating around, so NPM also has a deduping system that attempts to combine compatible modules (e.g. if the version ranges for React, Vue, and Lodash overlap, a version will be chosen that works for all modules). In addition, there are various ways of removing excess copies of modules — by default, NPM flattens the tree in a way that allows multiple modules to "see" the same imported module, and tools like PNPM use symlinks to remove duplicated modules. And you mention custom importers in Python, but I believe Yarn uses them already in NodeJS to do the fun things you talk about like importing from zip files that are vendored in a repository. All of the above would be possible in Python as well, without changing the syntax or the semantics of the `import` statement at all. However, it would probably break a lot of the ecosystem assumptions about where modules live and how they're packaged. And it's not necessarily required to fix the core Python packaging issues, so I don't think people in Python-land are investing much time in exploring this issue. Basically, no, there is no fundamental reason why Python and Node have to have fundamentally different import systems. The two languages process imports in a very similar way, with the main difference being whether packages are imported in a nested way (Node) or as a flattened directory (Python).
fny 2y ago
Because you don’t need virtualenvs or ruby_modules. You can have however many versions of the same gem installed it’s simply referenced by a gemfile, so for Ruby version X you are guaranteed one copy of gem version Y and no duplicates. This whole installing the same dependencies a million times across different projects in Python and Node land is completely insane to me. Ruby has had the only sane package manager for years. Cargo too, but only because they copied Ruby. Node has littered my computer with useless files. Python’s venv eat up a lot of space unnecessarily too.
- zahlman 2y ago
  In principle, venvs could hard-link the files from a common source, as long as the filesystem supports that. I'm planning to experiment with this for Paper. It's also possible to use .pth files (https://docs.python.org/3/library/site.html https://docs.python.org/3/library/site.html) to add additional folders to the current environment at startup. (I've heard some whispers that this causes a performance hit, but I haven't noticed. Python module imports are cached anyway.) Symlinks should work, too. (But I'm pretty sure Windows shortcuts would not. No idea about junctions.)
  - mdaniel 2y ago
    I wish the ecosystem made heavier use of .zip packages, which would gravely help with the logistics of your hardlink plan, in addition to slimming down 300MB worth of source code. The bad news is that (AIUI) the code needs to be prepared for use from a package and thus my retroactively just zipping them up will break things at runtime Take for example: $ du -hs $HOMEBREW_PREFIX/Cellar/ansible/11.1.0_1/libexec/lib/python3.13/site-packages/* | gsort --human 103M /usr/local/Cellar/ansible/11.1.0_1/libexec/lib/python3.13/site-packages/botocore 226M /usr/local/Cellar/ansible/11.1.0_1/libexec/lib/python3.13/site-packages/ansible_collections
    - zahlman 2y ago
      Wheels are zip files with a different extension. And the Python runtime can import Python code from them - that's crucial to the Pip bootstrapping process, in fact. But packages that stay zipped aren't a thing any more - because it's the installer that gets to choose how to install the package, and Pip just doesn't do that. Pip unpacks the wheel mainly because many packages are written with the assumption that the code will be unpacked. For example, you can't `open` a relative path to a data file that you include with your code, if the code remains in a zip file. Back in the day when eggs were still a common distribution format, it used to be common to set metadata to say that it's safe to leave the package zipped up. But it turned out to be rather difficult to actually be sure that was true. The wheel is also allowed to contain a folder with files that are supposed to go into other specified directories (specific ones described by the `sysconfig` standard library https://docs.python.org/3/library/sysconfig.html https://docs.python.org/3/library/sysconfig.html ), and Pip may also read some "entry point" metadata and create executable wrappers, and leave additional metadata to say things like "this package was installed with Pip". The full installation process for wheels as designed is described at https://packaging.python.org/en/latest/specifications/binary-distribution-format/ https://packaging.python.org/en/latest/specifications/binary... . (Especially see the #is-it-possible-to-import-python-code-directly-from-a-wheel-file section at the end.) The unzipped package doesn't just take up extra space due to unzipping, but also because of the .pyc cache files. A recent Pip wheel is 1.8 MiB packed and 15.2 MiB unpacked on my system; that's about 5.8 apparent MiB .py, 6.5 MiB .pyc, 2.1 MiB from wasted space in 4KiB disk blocks, and [s].8 MiB from a couple hundred folders.[/s] Sorry, most of that last bit is actually stub executables that Pip uses on Windows to make its wrappers "real" executables, because Windows treats .exe files specially.
8n4vidtmkvmk 2y ago
FWIW, you can usually just drop the `-g` and it'll install into `node_modules/.bin` instead, so it stays local to your project. You can run it straight out of there (by typing the path) or do `npm run <pkg>` which I think temporarily modifies $PATH to make it work.
- nicoburns 2y ago
  The `npx` command (which comes bundled with any nodejs install) is the way to do that these days.
  - 8n4vidtmkvmk 2y ago
    `npx` doesn't update package.json/package.lock though, right? So you might get a different version of the package once awhile. If it's an executable you depend on for your project, it makes sense to version it IMO.
    - nicoburns 2y ago
      You can do a (local) install using `npm install` and then execute the binary using `npx`. npx will also try to fetch the binary over the network if you don't have it installed, which is questionable behaviour in my opinion, but you can just cancel this if it starts doing it.
regularfry 2y ago
The mainline ruby doesn't but tools to support virtualenvs are around. They're pretty trivial to write: https://github.com/regularfry/gemsh/blob/master/bin/gemsh https://github.com/regularfry/gemsh/blob/master/bin/gemsh As long as you're in the ruby-install/chruby ecosystem and managed to avoid the RVM mess then the tooling is so simple that it doesn't really get any attention. I've worked exclusively with virtualenvs in ruby for years.

andrewmcdonough 2y ago

Ruby has a number of solutions for this - rvm (the oldest, but less popular these days), rbenv (probably the most popular), chruby/gem_home (lightweight) or asdf (my personal choice as I can use the same tool for lots of languages). All of those tools install to locations that shouldn't need root.

mdaniel 2y ago
Yes, I am aware of all of those, although I couldn't offhand tell anyone the difference in tradeoffs between them. But I consider having to install a fresh copy of the whole distribution a grave antipattern. I'm aware that nvm and pyenv default to it and I don't like that I did notice how Homebrew sets env GEM_HOME=<Cellar>/libexec GEM_PATH=<Cellar>/libexec (e.g. <https://github.com/Homebrew/homebrew-core/blob/9f056db169d5f7e7cca264f406a370b3fb967ee4/Formula/f/fastlane.rb#L35-L36 https://github.com/Homebrew/homebrew-core/blob/9f056db169d5f...>) but, similar to my node experience, since I am a ruby outsider I don't totally grok what isolation that provides

MrJohz 2y ago

It's worth noting that Node allows two packages to have the same dependency at different versions, which means that `npm i -g` is typically a lot safer than a global `pip install`, because each package will essentially create its own dependency tree, isolated from other packages. In practice, NPM has a deduplication process that makes this more complicated, and so you can run into issues (although I believe other package managers can handle this better), but I rarely run into issues with this. That said, I agree that `npm i -g` is a poor system package manager, and you should typically be using Homebrew or whatever package manager makes the most sense on your system. That said, `npx` is a good alternative if you just want to run a command quickly to try it out or something like that.

zahlman 2y ago
>It's worth noting that Node allows two packages to have the same dependency at different versions Yes. It does this because JavaScript enables it - the default import syntax uses a file path. Python's default import syntax uses symbolic names. That allows you to do fun things like split a package across the filesystem, import from a zip file, and write custom importers, but doesn't offer a clean way to specify the version you want to import. So Pip doesn't try to install multiple versions either (which saves it the hassle of trying to namespace them). You could set up a system to make it work, but it'd be difficult and incredibly ugly. Some other language ecosystems don't have this problem because the import is resolved at compile time instead.
- MrJohz 2y ago
  This is incorrect on several points. Firstly, Node's `require` syntax predates the modern JS `import` syntax. It is related to some attempts at the time to create tools that could act like a module system for the browser (in particular RequireJS), but it is distinct in that a lot of the rules about how modules would be resolved were specifically designed to make sense with NodeJS and the `node_modules` system. Secondly, although path imports are used for local imports, Node's `require` and `import` syntax both use symbolic names to refer to third-party packages (as well as built-in packages). If you have a module called `lodash`, you would write something like `import "lodash"`, and Node will resolve the name "lodash" to the correct location. This behaviour is in principle the same as Python's — the only change is the resolution logic, which is Node-specific, and not set by the Javascript language at all. The part of NodeJS that _does_ enable this behaviour is the scoped module installation. Conceptually, NPM install modules in this structure: * index.js (your code goes here) * node_modules (top level third-party modules go here) * react * index.js (react source code) * node_modules (react's dependencies go here) * lodash/index.js * vuejs * index.js (vuejs source code) * node_modules (vue's dependencies go here) * lodash/index.js * lodash * index.js Importantly, you can see that each dependency gets their own set of dependencies. When `node_modules/react/index.js` imports lodash, that will resolve to the copy of lodash installed within the react folder in the dependency tree. That way, React, VueJS, and the top-level package can all have their own, different versions of lodash. Of course in practice, you usually don't want lots of versions of the same library floating around, so NPM also has a deduping system that attempts to combine compatible modules (e.g. if the version ranges for React, Vue, and Lodash overlap, a version will be chosen that works for all modules). In addition, there are various ways of removing excess copies of modules — by default, NPM flattens the tree in a way that allows multiple modules to "see" the same imported module, and tools like PNPM use symlinks to remove duplicated modules. And you mention custom importers in Python, but I believe Yarn uses them already in NodeJS to do the fun things you talk about like importing from zip files that are vendored in a repository. All of the above would be possible in Python as well, without changing the syntax or the semantics of the `import` statement at all. However, it would probably break a lot of the ecosystem assumptions about where modules live and how they're packaged. And it's not necessarily required to fix the core Python packaging issues, so I don't think people in Python-land are investing much time in exploring this issue. Basically, no, there is no fundamental reason why Python and Node have to have fundamentally different import systems. The two languages process imports in a very similar way, with the main difference being whether packages are imported in a nested way (Node) or as a flattened directory (Python).

fny 2y ago

Because you don’t need virtualenvs or ruby_modules. You can have however many versions of the same gem installed it’s simply referenced by a gemfile, so for Ruby version X you are guaranteed one copy of gem version Y and no duplicates. This whole installing the same dependencies a million times across different projects in Python and Node land is completely insane to me. Ruby has had the only sane package manager for years. Cargo too, but only because they copied Ruby. Node has littered my computer with useless files. Python’s venv eat up a lot of space unnecessarily too.

zahlman 2y ago
In principle, venvs could hard-link the files from a common source, as long as the filesystem supports that. I'm planning to experiment with this for Paper. It's also possible to use .pth files (https://docs.python.org/3/library/site.html https://docs.python.org/3/library/site.html) to add additional folders to the current environment at startup. (I've heard some whispers that this causes a performance hit, but I haven't noticed. Python module imports are cached anyway.) Symlinks should work, too. (But I'm pretty sure Windows shortcuts would not. No idea about junctions.)
- mdaniel 2y ago
  I wish the ecosystem made heavier use of .zip packages, which would gravely help with the logistics of your hardlink plan, in addition to slimming down 300MB worth of source code. The bad news is that (AIUI) the code needs to be prepared for use from a package and thus my retroactively just zipping them up will break things at runtime Take for example: $ du -hs $HOMEBREW_PREFIX/Cellar/ansible/11.1.0_1/libexec/lib/python3.13/site-packages/* | gsort --human 103M /usr/local/Cellar/ansible/11.1.0_1/libexec/lib/python3.13/site-packages/botocore 226M /usr/local/Cellar/ansible/11.1.0_1/libexec/lib/python3.13/site-packages/ansible_collections
  - zahlman 2y ago
    Wheels are zip files with a different extension. And the Python runtime can import Python code from them - that's crucial to the Pip bootstrapping process, in fact. But packages that stay zipped aren't a thing any more - because it's the installer that gets to choose how to install the package, and Pip just doesn't do that. Pip unpacks the wheel mainly because many packages are written with the assumption that the code will be unpacked. For example, you can't `open` a relative path to a data file that you include with your code, if the code remains in a zip file. Back in the day when eggs were still a common distribution format, it used to be common to set metadata to say that it's safe to leave the package zipped up. But it turned out to be rather difficult to actually be sure that was true. The wheel is also allowed to contain a folder with files that are supposed to go into other specified directories (specific ones described by the `sysconfig` standard library https://docs.python.org/3/library/sysconfig.html https://docs.python.org/3/library/sysconfig.html ), and Pip may also read some "entry point" metadata and create executable wrappers, and leave additional metadata to say things like "this package was installed with Pip". The full installation process for wheels as designed is described at https://packaging.python.org/en/latest/specifications/binary-distribution-format/ https://packaging.python.org/en/latest/specifications/binary... . (Especially see the #is-it-possible-to-import-python-code-directly-from-a-wheel-file section at the end.) The unzipped package doesn't just take up extra space due to unzipping, but also because of the .pyc cache files. A recent Pip wheel is 1.8 MiB packed and 15.2 MiB unpacked on my system; that's about 5.8 apparent MiB .py, 6.5 MiB .pyc, 2.1 MiB from wasted space in 4KiB disk blocks, and [s].8 MiB from a couple hundred folders.[/s] Sorry, most of that last bit is actually stub executables that Pip uses on Windows to make its wrappers "real" executables, because Windows treats .exe files specially.

8n4vidtmkvmk 2y ago

FWIW, you can usually just drop the `-g` and it'll install into `node_modules/.bin` instead, so it stays local to your project. You can run it straight out of there (by typing the path) or do `npm run <pkg>` which I think temporarily modifies $PATH to make it work.

nicoburns 2y ago
The `npx` command (which comes bundled with any nodejs install) is the way to do that these days.
- 8n4vidtmkvmk 2y ago
  `npx` doesn't update package.json/package.lock though, right? So you might get a different version of the package once awhile. If it's an executable you depend on for your project, it makes sense to version it IMO.
  - nicoburns 2y ago
    You can do a (local) install using `npm install` and then execute the binary using `npx`. npx will also try to fetch the binary over the network if you don't have it installed, which is questionable behaviour in my opinion, but you can just cancel this if it starts doing it.

regularfry 2y ago

The mainline ruby doesn't but tools to support virtualenvs are around. They're pretty trivial to write: https://github.com/regularfry/gemsh/blob/master/bin/gemsh https://github.com/regularfry/gemsh/blob/master/bin/gemsh As long as you're in the ruby-install/chruby ecosystem and managed to avoid the RVM mess then the tooling is so simple that it doesn't really get any attention. I've worked exclusively with virtualenvs in ruby for years.