7 ms·
Generate pip requirements.txt file based on imports of any project
- notnmeyer 2y agolooks great. similarly this is why i love ‘go mod tidy’. just use the thing and let the package manager sort it out after the fact.
- FlipFloopDev 2y agoSurprised I've never seen this, despite it existing for at least 9 years??
- mbank 2y agoI was looking for this and thought I was doing something wrong not finding anything... Great job! I do think though a "clean" development mode not needing this would be to work with a virgin virtual environment starting a new project and running pip freeze on that env.
- robblbobbl 2y agoNicely done
- hasante 2y agoIt dosent work 100% of the time & can get you gotchas that you need to fix - From my experience with it.
- Terretta 2y agoPlease consider consolidating python dependency management instead of fragmenting it: https://docs.astral.sh/uv/pip/compile/ https://docs.astral.sh/uv/pip/compile/ In other words, bring the thinking here. Whether it's new thinking, or decade old thinking, it's well time to converge. We've had decades of trying it like Heinz catsup varieties. Worth trying more wood behind fewer arrows.
- Spivak 2y agoYou're just picking a winner like every other Python dependency project. If it's not in a PEP do whatever the hell you want. Good ideas will get turned into future PEPs for tooling to standardize on. uv itself has two separate locking formats.
- halfcat 2y ago> If it's not in a PEP Yes exactly https://xkcd.com/927/ https://xkcd.com/927/
- imjonse 2y agoOne can start non-distruptively with uv by only using it as a pipx replacement at first ( uv tool). Nice and fast.
- Der_Einzige 2y agoPlease consider consolidating python dependency management instead of fragmenting it: https://github.com/mamba-org/mamba https://github.com/mamba-org/mamba In other words, bring the thinking here. Whether it's new thinking, or decade old thinking, it's well time to converge. We've had decades of trying it like Heinz catsup varieties. Worth trying more wood behind fewer arrows.
- kstrauser 2y agoWhile I don't know which (if any) Python package manager will "win", I'm confident it won't be Conda or similar. I hope and pray not, anyway.
- matrss 2y agoYes, the entire workflow of conda/mamba is a liability. Having environments detached from directories encourages bad practices (reusing environments for unrelated things, installing everything into the same environment and getting very broken dependency resolutions, that stuff) and it doesn't even have a lock file format, so environments can't be saved short of dumping its entire directory. Iff it must be the conda ecosystem, pixi (https://pixi.sh/latest/ https://pixi.sh/latest/) is a much better pick. But conda-style packages (or anything with install time dependency resolution really) also have the issue of being completely untestable. That makes them unsuitable at least for user-facing package management, if you care about your packages not randomly breaking without warning. I'd rather see every language converge on a single package manager that implements functional package management, i.e. guix or nix. One can dream...
- skeledrew 2y agoI've been using a conda+poetry flow for years and it's worked very well. Particularly because envs aren't tied to projects. I tried pixi for a few days in a project months ago and it was just breakage and untenable limitations all around. I prefer the freedom of using per project envs when I want, and shared envs other times.
- jonathrg 2y agoHow is uv, another flavor-of-the-month Python packaging tool, not contributing to fragmentation?
- danglymarsh 2y ago[flagged]
- alephxyz 2y agoSource? Never heard of it until today
- colechristensen 2y agoI looked into uv a while back and it indeed was considerably faster than pip, but the folks who created it seem like they're trying a business around python tooling and are in the early stages. There is a risk of a rug pull of some kind down the road when they decide they want to make money and that to me was too much of a risk at present. https://astral.sh/about https://astral.sh/about
- danglymarsh 2y ago[flagged]
- alephxyz 2y agoI'm not gonna dig up someone's history to figure out what you mean. If you're going to make a throwaway account to insult someone at least be explicit about your accusations.
- the_mitsuhiko 2y agouv unlike all other options has the best shot at becoming the one.
- kstrauser 2y ago
- whalesalad 2y agoThis is not fragmenting it. The requirements.txt file has been a steady and well used API for well over a decade. This tool just helps you produce one for a project that is missing one.
- ris 2y agoHow about you stop trying to create package managers for every single language/ecosystem in existence and instead converge on trying to solve the whole problem once and for all with Nix.
- verandaguy 2y agoAsking honestly since I've seen Nix and NixOS show up on the front page here a bunch over the years, but never used it: my impression of it is that it fills the same kind of niche as the Zig or Nim languages: conceptually pretty cool, but not widely adopted outside of a dedicated core user group. Is this really the case for Nix, or is it actually widely adopted, and this adoption is underreported? If it's not actually widely adopted, what do you think are the biggest obstacles in Nix's way?
- jghn 2y ago> Please consider consolidating python dependency management instead of fragmenting it You realize that this project predates uv by roughly a decade, right?
- Terretta 2y ago[flagged]
- chaps 2y agoCan you expand on what you mean, exactly?
- throwaway314155 2y agoThey thought they had a better advertising chance for `uv` than they actually did. As far as I can tell, `uv` is a replacement for `pip`. But the project linked by OP doesn't actually replace `pip`, but rather a small subset of functionality in `pip` - `pip freeze`. Unless `uv` has some sort of import scanning functionality, the suggestion to use `uv` instead doesn't really make any sense.
- Terretta 2y agoOn the contrary, read this about `uv` by the author of `rye`: https://lucumr.pocoo.org/2024/8/21/harvest-season/ https://lucumr.pocoo.org/2024/8/21/harvest-season/ Domination is a goal because it means that most investment will go into one stack. I can only re-iterate my wish and desire that Rye (and with it a lot of other tools in the space) should cease to exist once the dominating tool has been established. For me uv is poised to be that tool. It's not quite there today yet for all cases, but it will be in no time, and now is the moment to step up as a community and start to start to rally around it. Yes, I linked to an obscure feature uv supports. It's already a rather lot more.
- throwaway314155 2y ago
- tmslnz 2y agoHow is it different to pip-chill[1]? [1]: https://github.com/rbanffy/pip-chill https://github.com/rbanffy/pip-chill
- surye 2y agoI believe pip-chill still operates on packaged installed into the environment. This project seems to derive from the code itself, even if no packages are installed in the current environment.
- halfcat 2y agoI don’t know if this is how pipreqs works, but I’d be concerned about a typo in an import that inadvertently installs a library. I’ve found pip-tools [1] to be a nice middle ground between `pip freeze` and something heavier like poetry. It’s especially helpful in showing you where a library came from, when a library installs other libraries it depends upon. One could still have a typo in their requirements.in file with pip-tools, but changes there are much less frequent than imports in any Python file in your proejcf which could be a daily occurrence in some codebases. [1] https://github.com/jazzband/pip-tools https://github.com/jazzband/pip-tools
- mooreds 2y agoFrom the GH repo description: > Looking for maintainers to move this project forward. So not sure how maintained it is.
- v3ss0n 2y agoPlease consider using PDM instead. https://pdm-project.org/en/latest/ https://pdm-project.org/en/latest/
- joeyagreco 2y agocould you explain why?
- v3ss0n 2y agoIn short , it works like NPM. - reproducible python package dependency - reproducible python version - package locking - PEP compliant package management configuration via project.toml - automatic venv management - project configuration and management including build scripts in one place All you need is - pdm init # starts preparing env - pdm add package # install package locally, project wide - pdm install # reproduce
- wtallis 2y agoThis looks like it was copied from a readme file without regard for the context of this discussion thread. Instead of merely advertising, you should have posted something that actually refers to the capabilities and purpose of pipreqs and explained how PDM handles that better or obviates the need for a tool to do what pipreqs does.
- v3ss0n 2y agoI already stated the obvious differences. requirements files cannot capture a lot of metadata that is required to reproduce python setup , PDM deals a lot of important stuff in: Dependency Management in easy to edit pyproject.toml file , properly implementing the standard PEP specs 518 and 612 . Plus it automatically creates .venv with all the packages required when `pdm install` . Also it installs and manage the required Python verion in .venv. It also locks the proper package depedencies in pdm.lock file with proper hashes which pipreq or others can't . With pyproject.toml being the main config file you can also define build script , run scripts like you can do with `npm run dev` , using `pdm run dev` for example. Also you can define all the development tools and their config along with it. We used PDM , so it streamlines package and depedency, project management and venv config all in one pyproject.toml - and you can generate requirements.txt and pdm.lock which in turn builds your venv , can easily share with team (just one pyproject.toml file to share) .
- 999900000999 2y agoCan you have it work with pipfiles too ?
- remram 2y agoI had forgotten about pipfiles. Like their name implies (/s) they are not for pip but pipenv, a separate tool. https://github.com/pypa/pipfile https://github.com/pypa/pipfile "This format is still under active design and development", last commit "2 years ago". I think this is dead.
- 999900000999 2y agohttps://github.com/pypa/pipenv https://github.com/pypa/pipenv Pipenv was last updated 10 hours ago. Looks like it's still an active project to me.
- kstrauser 2y agoThat's the repo for the Pipfiles themselves. The (only?) thing using them is pipenv, which lives in a separate repo.
- remram 2y agoPipfiles is what we are talking about.
- kstrauser 2y agoTrue, but Pipfiles are basically whatever pipenv wants them to be. I'm not aware of anything but pipenv that uses them, although I'm sure there are more than zero such things. My point here being that I wouldn't take a lack of commits to the Pipfiles repo to mean that they're not still under development. More likely it's just being developed in the pipenv repo. But in any case, I'd vastly prefer the standard pyproject.toml over some file specific to mostly one tool.
- 2y ago
- sean_pedersen 2y agoI thought import names and PyPI names are not always equal, thus this can not work reliably, right?
- jonathrg 2y agoIt uses this file which maps import names to package names, but it's only 1152 packages and I'm not sure how it was generated. https://github.com/bndr/pipreqs/blob/master/pipreqs/mapping https://github.com/bndr/pipreqs/blob/master/pipreqs/mapping
- quietbritishjim 2y agoIn fairness that only seems to list those packages where the import name doesn't match the package name (e.g., it doesn't include numpy) so its overall coverage is a lot larger than that.
- deleted 2y ago[deleted]
- qwerty1793 2y agoEven worse, there are some different PyPI packages with the same import name. For example, `import snappy` is probably referring to the compression library https://pypi.org/project/python-snappy/ https://pypi.org/project/python-snappy/ but it could be this maths package https://pypi.org/project/snappy/ https://pypi.org/project/snappy/
- danielktdoranie 2y ago...it would also be nice if Debian (Ubunutu and other derivitives) stop shitting on pip in some ridiculous paranoid attempt to stop breaking apt packages (that don't break apt packages anyway) or because of "muh security".
- L3viathan 2y agoHuh? Messing with the system Python install absolutely can break your system. Just use venvs, or install separate Python versions for development e.g. via pyenv. You likely don't want to use the relatively outdated Python version shipped with Debian anyways.
- idoubtit 2y agoI used this last year, with a relative success. I was asked to fix a Python code written by an intern that was no longer there. The code used external libraries, but did not declare any. I had only a zip of the source code, without any library. pipreqs was able to identify the 22 required libraries. Unfortunately, there was a runtime crash because a library was at the wrong version. So I had to let a real Python developer handle the next steps. BTW, this tool is not a dependency manager. Many sibling comments seem to misunderstand the case and promote unrelated tools.
- remram 2y agoCan I have it find the requirements as of a specific date? If I find a script/notebook/project with no requirements.txt, I usually know when it was created. Being able to get the versions that were selected back then on the author's machine would be great for reproducibility.
- 0cf8612b2e1e 2y agoI have never realized until this moment that I want such a tool. Ideally, you would never find yourself in this position, but alas.
- matrss 2y agoFor the future, please pick a package manager that can give you a lock file alongside your code, so that you have a definitive record of the dependencies. Even if you have all versions as of the time of the last modification to the code, you don't know if the dependency resolution happened at that point in time, or if the environment was set up years prior and never updated. Nevertheless, this is what you are looking for: https://pypi.org/project/pypi-timemachine/ https://pypi.org/project/pypi-timemachine/
- remram 2y agoWhen I said "the author" I was not trying to mean myself. Agreed that in a perfect world I would never need this, but...
- d0mine 2y agoSee whether “exclude-newer” uv option might help if you’ve inherited a script without a dependencies lock file https://docs.astral.sh/uv/guides/scripts/#improving-reproducibility https://docs.astral.sh/uv/guides/scripts/#improving-reproduc...
- oprypin 2y agouv pip compile --exclude-newer=2019-12-02 https://docs.astral.sh/uv/pip/compile/ https://docs.astral.sh/uv/pip/compile/ • https://docs.astral.sh/uv/reference/cli/#uv-pip-compile https://docs.astral.sh/uv/reference/cli/#uv-pip-compile
- TZubiri 2y agolet's not fuck with supply chain vulnerabilities
- jonathrg 2y agoFeel free to elaborate.
- TZubiri 2y ago''' in python are there any libraries where the pip install command is different than import?" ChatGPT: Yes, there are some Python libraries where the `pip install` command differs from the module name you use for `import`. Below are a few examples of such cases: 1. *`pip install opencv-python`* - Import as: `import cv2` 2. *`pip install beautifulsoup4`* - Import as: `from bs4 import BeautifulSoup` 3. *`pip install PyMySQL`* - Import as: `import pymysql` 4. *`pip install python-dateutil`* - Import as: `import dateutil` 5. *`pip install python-dotenv`* - Import as: `import dotenv` 6. *`pip install google-auth`* - Import as: `import google.auth` 7. *`pip install Pillow`* - Import as: `from PIL import Image` ''' It seems that the project hardcodes the import to pypi names though. So it has a that sweetspot of being more impractical while still being insecure. https://github.com/bndr/pipreqs/blob/master/pipreqs/mapping https://github.com/bndr/pipreqs/blob/master/pipreqs/mapping If you install this in a company that takes security seriously, you should get warned or fired pretty much. It's one thing to import code from a random developer on the internet to do your job, we get a pass on that if it makes us more productive, but to import code that helps us with importing code? Red flag.
- skeledrew 2y agoAnything that takes away from users fully migrating to pyproject really needs to die.
- mikepurvis 2y agoBut pyproject itself isn't even taking a stance on the underlying question of dependency management, which can still be flit, poetry, uv, pipx, pipenv, and who knows what else. (I'm a poetry user myself, but I can see the writing on the wall that poetry is probably not going to be the winner of who ultimately ends up in the PEP.)
- skeledrew 2y agoThere isn't a stance on the build backend itself, beyond supporting the basics[0]. The spec itself however exists as the accepted successor to requirements.txt, and I think is actually implemented by all but the last tool in your list. [0] https://packaging.python.org/en/latest/specifications/pyproject-toml/ https://packaging.python.org/en/latest/specifications/pyproj...
- joeyagreco 2y agoand why is that?
- skeledrew 2y agoBecause it's wasted resources for anything that isn't in line with said spec[0]. There should ideally be only one explicit way[1] to do a thing. [0] https://peps.python.org/pep-0621/ https://peps.python.org/pep-0621/ [1] https://peps.python.org/pep-0020/ https://peps.python.org/pep-0020/
- hiccuphippo 2y agoThis project looks like a first step to migrate something to pyproject.
- skeledrew 2y ago
- bndr 2y agoOh wow, my package on the front page again. Glad that it's still being used. This was written 10 years ago when I was struggling with pulling and installing projects that didn't have any requirements.txt. It was frustrating and time-consuming to get everything up and running, so I decided to fix it, apparently many other developers had the same issue. [Update]: Though I do think the package is already at a level where it does one thing and it does it good. I'm still looking for maintainers to improve it and move it forward.
- deleted 2y ago[deleted]
- joshdavham 2y ago> This was written 10 years ago when I was struggling with pulling and installing projects that didn't have any requirements.txt And 10 years later this is still a common problem!
- v3ss0n 2y agoNot if you use proper python env and package management tools like pdm or poetry.
- spullara 2y agoWrap your python program in an LLM that just keeps installing things when it fails until it works :)
- andrewmcwatters 2y agoI don't spend a lot of time in Python, but my current understanding having read Python documentation and seeing some projects online is that you use pip and requirements.txt with --require-hashes, venv, and pyproject.toml to use a standard dependency management toolchain. Is that correct?
- thebigspacefuck 2y agoWould this be useful for identifying packages in a requirements.txt that aren’t used? Or is there another tool for that?
- manojlds 2y agomv and diff?
- timetraveller26 2y agogod, please no
- jcarrano 2y agoWhen I first started with Python, long ago, I looked into these kind of solutions, which didn't work so well, and wondered why the concept was not better developed. Later, with experience, I realized it is not am great idea, and more hassle than the benefits it brings. I don't think it is good idea to merrily write 10s of import statements and end up with loads of dependencies.
- xuraiis 2y agoAwesome project! Didn’t realise this existed when I took a stab at implementing a similar thing a few months back: https://github.com/nmichlo/pydependence https://github.com/nmichlo/pydependence using graph traversal of imports as well as naive support for lazy imports to generate optional extras as well as modify the pyproject.toml as part of pre-commit hooks. Edit: the main goal was to generate optional extras for different entry points in a large codebase, find missing or extra imports, resolve dependencies across multiple repos, and see which files reference the mapped packages. Ideally if you have many internal repos which are not published and you cannot correctly use dependency resolution, then you can generate requirements before you pass them onto something like UV.