3 ms·

Looking for a suggestion from the audience here. I lean on Makefile for automation wherever possible, even if it is just a centralized trigger command to run a

Looking for a suggestion from the audience here. I lean on Makefile for automation wherever possible, even if it is just a centralized trigger command to run a packaged script (eg scripts/dothething.py). One of my sticking points is how to ensure the Python virtual environment is installed and up to date before triggering any scripts. That is, I clone a new project, cd into it, and want to run some package command inside the expected environment (eg `poetry run scripts/dothething.py`). This will fail if the environment has not previously installed all of the defined dependencies. My annoyance is that requiring `poetry install` to run prior to every command is quite slow, and rarely has anything to do. Does anyone have a suggestion as to how to ensure the right thing is always done without the slow Poetry resolution? My dream is that in all of my projects you could, say `make test` and Python would optionally create the environment, install the dependencies, and run pytest in the fresh case. The typical development case would confirm that the environment exists and just run pytest.

323 4y ago
You can check that a venv exists by looking for .venv/pyvenv.cfg. Is that not enough? Unless you also need to check that all packages are installed and up to date.
- fbdab103 4y ago
  1) Confirming that the environment is in sync with my definition file (pyproject.toml or lockfile) is my true objective. If I suddenly add libraryFoo to pyproject.toml, but I forget to run `poetry install`, I want my `make test` command to note the discrepancy and correct the situation. 2) I think that has some brittleness of detecting where the environment lives (as per the author's post). For my use case, I would be fine assuming the defaults, but it is something else to go wrong Edit: Thinking out loud, I guess the quickest solution is to checksum the pyproject.toml file and store that somewhere. If current hash matches previous, do not run `poetry install`. While that would get me there, I am hoping for a first-party, standard solution.
  - 323 4y ago
    > I guess the quickest solution is to checksum the pyproject.toml This is the kind of stuff bazel/ninja do. https://bazel.build https://bazel.build https://ninja-build.org https://ninja-build.org
- jaccarmac 4y ago
  Thanks for the heads-up about that file; I use a similar development setup and have been slacking on wiring Make rules together because I thought I would have to touch my own files. It would be nice if the files Python installed had better timestamps, though. I'm the only one using Make on the team and install some tooling of my own, so have to chain upgrade Pip -> install deps -> install tools in the correct order. I'm not sure how to automate that order without running all three steps every time.
plonk 4y ago
As long as a venv exists and is activated, your Makefile can simply “pip install -r ./requirements.txt” and pip should make sure that all constraints in the requirements are met. Make sure that the package management tools are up-to-date first with “python -m pip install—upgrade pip setuptools wheel”. Pip is very fast now and shouldn’t take more than a second if all dependencies are already installed. The only part that’s too slow is installing a package from source.

323 4y ago

You can check that a venv exists by looking for .venv/pyvenv.cfg. Is that not enough? Unless you also need to check that all packages are installed and up to date.

fbdab103 4y ago
1) Confirming that the environment is in sync with my definition file (pyproject.toml or lockfile) is my true objective. If I suddenly add libraryFoo to pyproject.toml, but I forget to run `poetry install`, I want my `make test` command to note the discrepancy and correct the situation. 2) I think that has some brittleness of detecting where the environment lives (as per the author's post). For my use case, I would be fine assuming the defaults, but it is something else to go wrong Edit: Thinking out loud, I guess the quickest solution is to checksum the pyproject.toml file and store that somewhere. If current hash matches previous, do not run `poetry install`. While that would get me there, I am hoping for a first-party, standard solution.
- 323 4y ago
  > I guess the quickest solution is to checksum the pyproject.toml This is the kind of stuff bazel/ninja do. https://bazel.build https://bazel.build https://ninja-build.org https://ninja-build.org
jaccarmac 4y ago
Thanks for the heads-up about that file; I use a similar development setup and have been slacking on wiring Make rules together because I thought I would have to touch my own files. It would be nice if the files Python installed had better timestamps, though. I'm the only one using Make on the team and install some tooling of my own, so have to chain upgrade Pip -> install deps -> install tools in the correct order. I'm not sure how to automate that order without running all three steps every time.

plonk 4y ago

As long as a venv exists and is activated, your Makefile can simply “pip install -r ./requirements.txt” and pip should make sure that all constraints in the requirements are met. Make sure that the package management tools are up-to-date first with “python -m pip install—upgrade pip setuptools wheel”. Pip is very fast now and shouldn’t take more than a second if all dependencies are already installed. The only part that’s too slow is installing a package from source.