5 ms·

I have a single, simple script (not a package!) that has dependencies. Actually, I have a few of these, just sitting in /usr/local/bin so I can execute them whe

I have a single, simple script (not a package!) that has dependencies. Actually, I have a few of these, just sitting in /usr/local/bin so I can execute them whenever. How should I be managing environments for these scripts? Do I install dependencies in shared system python? Should I create a shared venv? Where should I store it? Any tools out there that make this decision for you and manage it? Just the fact that homebrew occasionally updates my installed Pythons and breaks everything makes me reluctant to use Python as a scripting language at all. It's pretty reliable when I have a folder dedicated to a project (poetry is good for this) but I'm pretty fed up with how brittle it all is.

DangitBobby 4y ago
In an amazing coincidence, I just found another front-page post exploring this same topic! For Python, it seems to suggest using nix-shell. 1. https://dbohdan.com/scripts-with-dependencies https://dbohdan.com/scripts-with-dependencies
- DangitBobby 4y ago
  My comment with an example of a single file script which can specify its own dependencies: https://news.ycombinator.com/item?id=34393630 https://news.ycombinator.com/item?id=34393630
wpietri 4y ago
I don't understand how there still isn't a good answer for this. It seems like such an obvious need, and for years I've heard go being promoted for solving this problem. I get why Python didn't start out with an answer for this, but in an era of 20 TB hard drives, I'm more than willing to burn space for ease of install.
fleekonpoint 4y ago
I use a separate directory and venv for each script. To execute the script, I use a shell script to call the venv's python interpreter. This is also how I use python scripts with cron/systemd. #!/bin/bash # myscript.sh venv/bin/python3 myscript.py You could also skip the shell script and use aliases in your .bashrc.
- DangitBobby 4y ago
  I do something kind of like this, but all of my scripts break when the underlying env is suddenly broken when e.g. brew updates python without asking me and breaks all existing environments. I'm sure I could come up with solutions that are very robust for my particular machine, but I would like something that allows me to share it as a gist and a teammate could just as easily use it, or I can use it myself on another machine without hassle. In other words, a solution contained within the script iteself and maybe one or two binaries outside that make it possible.
  - fleekonpoint 4y ago
    I see, it might be heavy handed but running them inside Docker containers might provide you with the isolation you're looking for. You could also build and share these images with your teammates. I've actually started using a lot of different CLI tools with Docker, especially when the tool isn't available for my OS.
saila 4y ago
If you don't mind adding a pyproject.toml, you could use pipx[1] to install these scripts. The directory structure would look like this: <name>/ pyproject.toml <name>.py The pyproject.toml would look like this (using poetry, but you could use a different tool for this): [tool.poetry] # NOTE: Set <name> to your project name name = "<name>" description = "My Script" version = "0.0.0" authors = ["Me <me@example.com>"] [tool.poetry.dependencies] python = "^3.11" requests = "^2.28.2" [tool.poetry.scripts] # NOTE: Change <name> to match your package name in [tool.poetry] <name> = "<name>:main" [build-system] requires = ["poetry-core"] build-backend = "poetry.core.masonry.api" Then you'd run `pipx install -e .` and the the executable script will be installed in ~/.local/bin. [1] https://pypa.github.io/pipx/ https://pypa.github.io/pipx/
- DangitBobby 4y ago
  Thank you for trying to provide a workable solution, it's really not bad, but it has some downsides for me. pipx itself is installed inside a python environment, so when brew breaks my pythons, it breaks pipx as well. Anytime brew breaks my pythons, I would need to do the install step again for every script (or write a tool myself which does it). Not a total deal breaker, but not really much better than my current situation which pretty much just assumes any version of `requests` or `fire` is acceptable. Because python itself is constantly being updated to break the base python environment on my machine, a workable solution would need to include the fact that the base python environment might need to have things installed.
  - noisy_boy 4y ago
    One option could be to have Python installed in a separate location specifically for this purpose and to NOT include it in PATH. Then it is "out-of-sight" of brew and such packages and sort. You can even make the entire location read-only once you are done with installation of Python + pipx etc.
    - DangitBobby 4y ago
      That would definitely work.
mixmastamyk 4y ago
Put them in ~/.local/bin, and that in your path. Then pip install reqs with —user. If you want this automated use pipx, but it is overkill and loses simplicity. The reason folks give dire warnings against simplicity is that sysad skills have plummeted in recent years. It’s easy to fix the rare conflict by putting the troublemaker in a venv. Outside of a large work project I never need to.
AndyKluger 4y ago
If you don't want to rely on the system or brew-installed Pythons for all your scripts, you might want to manage multiple Python installations with asdf-vm. --- Disclaimer before I continue with my own suggestions: I have made small code contributions to pip and pip-tools, and maintain a Zsh frontend to pip-tools+venv, called zpy. --- Suggestions: - each folder of code maps to its own default venv, and may map to more venvs for different Python runtimes - each folder of code has a requirements.in file with top-level dependencies, and a requirements.txt as a lock file - you can either add a shebang line for your script which explicitly invokes the venv's Python and link that file into ~/.local/bin/, or instead create an external launcher script in ~/.local/bin/. --- Here's an example of how that might be done using zpy functions: $ mkdir simple-scripts $ cd simple-scripts $ envin # or in subcommand form: zpy envin ==> creating -> ~/.local/share/venvs/280…/venv :: ~/Code/simple-scripts $ pipacs httpx ==> appending -> requirements.in :: ~/Code/simple-scripts httpx ==> compiling requirements.in -> requirements.txt :: ~/Code/simple-scripts anyio==3.6.2 # via httpcore certifi==2022.12.7 # via httpcore, httpx h11==0.14.0 # via httpcore httpcore==0.16.3 # via httpx httpx==0.23.3 # via -r requirements.in idna==3.4 # via anyio, rfc3986 rfc3986==1.5.0 # via httpx sniffio==1.3.0 # via anyio, httpcore, httpx ==> syncing requirements.txt -> env :: ~/Code/simple-scripts $ print -rl -- 'from httpx import get' 'print(get("https://ifconfig.co/json").json())' >do_it.py $ vpyshebang do_it.py # or: zpy vpyshebang do_it.py $ ln -s $PWD/do_it.py ~/.local/bin/do_it If a Python runtime update breaks your venvs, you can probably fix things up using zpy's pipup function.

DangitBobby 4y ago

In an amazing coincidence, I just found another front-page post exploring this same topic! For Python, it seems to suggest using nix-shell. 1. https://dbohdan.com/scripts-with-dependencies https://dbohdan.com/scripts-with-dependencies

DangitBobby 4y ago
My comment with an example of a single file script which can specify its own dependencies: https://news.ycombinator.com/item?id=34393630 https://news.ycombinator.com/item?id=34393630

wpietri 4y ago

I don't understand how there still isn't a good answer for this. It seems like such an obvious need, and for years I've heard go being promoted for solving this problem. I get why Python didn't start out with an answer for this, but in an era of 20 TB hard drives, I'm more than willing to burn space for ease of install.

fleekonpoint 4y ago

I use a separate directory and venv for each script. To execute the script, I use a shell script to call the venv's python interpreter. This is also how I use python scripts with cron/systemd. #!/bin/bash # myscript.sh venv/bin/python3 myscript.py You could also skip the shell script and use aliases in your .bashrc.

DangitBobby 4y ago
I do something kind of like this, but all of my scripts break when the underlying env is suddenly broken when e.g. brew updates python without asking me and breaks all existing environments. I'm sure I could come up with solutions that are very robust for my particular machine, but I would like something that allows me to share it as a gist and a teammate could just as easily use it, or I can use it myself on another machine without hassle. In other words, a solution contained within the script iteself and maybe one or two binaries outside that make it possible.
- fleekonpoint 4y ago
  I see, it might be heavy handed but running them inside Docker containers might provide you with the isolation you're looking for. You could also build and share these images with your teammates. I've actually started using a lot of different CLI tools with Docker, especially when the tool isn't available for my OS.

saila 4y ago

If you don't mind adding a pyproject.toml, you could use pipx[1] to install these scripts. The directory structure would look like this: <name>/ pyproject.toml <name>.py The pyproject.toml would look like this (using poetry, but you could use a different tool for this): [tool.poetry] # NOTE: Set <name> to your project name name = "<name>" description = "My Script" version = "0.0.0" authors = ["Me <me@example.com>"] [tool.poetry.dependencies] python = "^3.11" requests = "^2.28.2" [tool.poetry.scripts] # NOTE: Change <name> to match your package name in [tool.poetry] <name> = "<name>:main" [build-system] requires = ["poetry-core"] build-backend = "poetry.core.masonry.api" Then you'd run `pipx install -e .` and the the executable script will be installed in ~/.local/bin. [1] https://pypa.github.io/pipx/ https://pypa.github.io/pipx/

DangitBobby 4y ago
Thank you for trying to provide a workable solution, it's really not bad, but it has some downsides for me. pipx itself is installed inside a python environment, so when brew breaks my pythons, it breaks pipx as well. Anytime brew breaks my pythons, I would need to do the install step again for every script (or write a tool myself which does it). Not a total deal breaker, but not really much better than my current situation which pretty much just assumes any version of `requests` or `fire` is acceptable. Because python itself is constantly being updated to break the base python environment on my machine, a workable solution would need to include the fact that the base python environment might need to have things installed.
- noisy_boy 4y ago
  One option could be to have Python installed in a separate location specifically for this purpose and to NOT include it in PATH. Then it is "out-of-sight" of brew and such packages and sort. You can even make the entire location read-only once you are done with installation of Python + pipx etc.
  - DangitBobby 4y ago
    That would definitely work.

mixmastamyk 4y ago

Put them in ~/.local/bin, and that in your path. Then pip install reqs with —user. If you want this automated use pipx, but it is overkill and loses simplicity. The reason folks give dire warnings against simplicity is that sysad skills have plummeted in recent years. It’s easy to fix the rare conflict by putting the troublemaker in a venv. Outside of a large work project I never need to.

AndyKluger 4y ago

If you don't want to rely on the system or brew-installed Pythons for all your scripts, you might want to manage multiple Python installations with asdf-vm. --- Disclaimer before I continue with my own suggestions: I have made small code contributions to pip and pip-tools, and maintain a Zsh frontend to pip-tools+venv, called zpy. --- Suggestions: - each folder of code maps to its own default venv, and may map to more venvs for different Python runtimes - each folder of code has a requirements.in file with top-level dependencies, and a requirements.txt as a lock file - you can either add a shebang line for your script which explicitly invokes the venv's Python and link that file into ~/.local/bin/, or instead create an external launcher script in ~/.local/bin/. --- Here's an example of how that might be done using zpy functions: $ mkdir simple-scripts $ cd simple-scripts $ envin # or in subcommand form: zpy envin ==> creating -> ~/.local/share/venvs/280…/venv :: ~/Code/simple-scripts $ pipacs httpx ==> appending -> requirements.in :: ~/Code/simple-scripts httpx ==> compiling requirements.in -> requirements.txt :: ~/Code/simple-scripts anyio==3.6.2 # via httpcore certifi==2022.12.7 # via httpcore, httpx h11==0.14.0 # via httpcore httpcore==0.16.3 # via httpx httpx==0.23.3 # via -r requirements.in idna==3.4 # via anyio, rfc3986 rfc3986==1.5.0 # via httpx sniffio==1.3.0 # via anyio, httpcore, httpx ==> syncing requirements.txt -> env :: ~/Code/simple-scripts $ print -rl -- 'from httpx import get' 'print(get("https://ifconfig.co/json").json())' >do_it.py $ vpyshebang do_it.py # or: zpy vpyshebang do_it.py $ ln -s $PWD/do_it.py ~/.local/bin/do_it If a Python runtime update breaks your venvs, you can probably fix things up using zpy's pipup function.