5 ms·

I had a situation recently where I saved a bunch of time on resolution during install of Python packages by breaking it up into smaller groups of packages to in

by memco 4y ago

I had a situation recently where I saved a bunch of time on resolution during install of Python packages by breaking it up into smaller groups of packages to install in separate steps. We have some packages in a private devpi server and some dependencies that can come from any pypi mirror. By forcing the private packages to be in their own install step it drastically cut down the time to resolve and start downloading packages. Does anyone know of comparisons of resolution metrics? I spent several hours of my week waiting on Python packages installation and would love to improve the situation. On a related note: our NPM install times are even worse: any tips to help there would be welcome.

shp0ngle 4y ago
I like pnpm, which is like npm, but does caching by default, and does some other optimizations. (p is for parallel, I think) The negative: sometimes it plays bad with some packages. Sometimes you need the (stupidly named) —shamefully-hoist, sometimes even that doesn’t help. pnpm by default does NOT have the flat structure in node_modules but puts into separate sub-folders; sometimes it plays bad with some dependencies that expect the flat structure (usually webpack-related cludges) But if you are starting fresh, or can spend some time on debugging, pnpm is amazing.
- 8n4vidtmkvmk 4y ago
  pnpm is plug n play... modules i think.
  - shp0ngle 4y ago
    Oh. Well that makes a bit less sense than “parallel”, honestly. Haha
edpenz 4y ago
pip-compile [0] is the best tool I've found for streamlining dependency resolution, but it may not provide any benefit for one-shot installs. [0] https://github.com/jazzband/pip-tools https://github.com/jazzband/pip-tools
- BiteCode_dev 4y ago
  For those who are wondering what is the benefit over pip freeze: pip freeze dumps you entire venv, while pip compile gets project dependancies, and infer the requirements file from that.
  - n8henrie 4y ago
    I'm not sure I get it -- if you had your core dependencies in setup.py / setup.cfg / pyproject.toml (for development) and then pip freeze (for creating a highly reproduced venv for applications), would that be similar? Why would one want to infer the requirements file? Would that be like `pip list --not-required --format-freeze`?

by memco 4y ago

shp0ngle 4y ago

I like pnpm, which is like npm, but does caching by default, and does some other optimizations. (p is for parallel, I think) The negative: sometimes it plays bad with some packages. Sometimes you need the (stupidly named) —shamefully-hoist, sometimes even that doesn’t help. pnpm by default does NOT have the flat structure in node_modules but puts into separate sub-folders; sometimes it plays bad with some dependencies that expect the flat structure (usually webpack-related cludges) But if you are starting fresh, or can spend some time on debugging, pnpm is amazing.

8n4vidtmkvmk 4y ago
pnpm is plug n play... modules i think.
- shp0ngle 4y ago
  Oh. Well that makes a bit less sense than “parallel”, honestly. Haha

edpenz 4y ago

pip-compile [0] is the best tool I've found for streamlining dependency resolution, but it may not provide any benefit for one-shot installs. [0] https://github.com/jazzband/pip-tools https://github.com/jazzband/pip-tools

BiteCode_dev 4y ago
For those who are wondering what is the benefit over pip freeze: pip freeze dumps you entire venv, while pip compile gets project dependancies, and infer the requirements file from that.
- n8henrie 4y ago
  I'm not sure I get it -- if you had your core dependencies in setup.py / setup.cfg / pyproject.toml (for development) and then pip freeze (for creating a highly reproduced venv for applications), would that be similar? Why would one want to infer the requirements file? Would that be like `pip list --not-required --format-freeze`?