9 ms·
Automate your Python project with Makefile (2021)
- bragr 4y agoI would not recommend make for python projects as make does not play nicely with virtualenvs and the environment variables that power them. You can make it work, but you'll be fighting it constantly.
- hprotagonist 4y ago“never activate venvs in a headless context” is sage advice and not just for make.
- kelseyfrog 4y agoAre you specifically talking about virtualenv or the concept of venvs in general? I've used make with conda, for example, for the last five years with no env issues at all, so some clarity or specific problem cases would help.
- kortex 4y agoWhat exactly are you doing that hits friction? You shouldn't be using "make" to "activate" your env - that should occur "outside" your makefile. It plays totally nice if you aren't trying to use make to set environmental state.
- cpburns2009 4y agoAn easy way to use virtual environments with Make is to create a wrapper script to activate the virtual environment and run a command in its context: # venv.sh venv="$1" shift . "$venv/bin/activate" "$@" Then in the Makefile I do: VENV := venv.sh ./venv install-reqs: ${VENV} pip -r requirements.txt
- bragr 4y agoThat works but the whole point of the article was to avoid creating extra scripts, and now you have the problem that you need to document the extra required environment setup before running make or you get opaque errors.
- rcarmo 4y agoHow so? I have a Makefile in all of my Python projects. Never had issues.
- bragr 4y agoPersonally it's the interactions between the life cycle management of the virtual environments and bulletproofing for use by all team members. For any non-trivial Python project, I end up with at least 2 virtual environments, one "normal" and one for installing unit tests/linting/other static checkers. If you're environments are already pull this project and goes to `make lint`, is the lint virtual environment extant, does it have all the required dependencies, and are those steps idempotent so as not to trigger a setup on every run. And then add on edge cases like, what if someone already had a virtual env active when you went to do the setup, what edge conditions does that introduce? Do you add something to your logic to detect extant virtual environments and deactivate them before trying other actions. All of these problems are solvable of course, but at what level of effort and for what benefit? And you pay all these costs every time you want to change something, especially if you've got to explain all the hacks to someone new, and you always run the risk that it works for you but not in a clean environment.
- fnordlord 4y agoI don't do much python anymore but recently, I decided to try moving from Makefile files to simple command-line python scripts for this type of thing. The python standard library has pretty much everything you need to replace `make` so there's no need for virtual environments and... 1. They're way more readable and intuitive 2. Cross-platform across Linux, Windows, MacOS, even if you have to branch off to platform-specific commands inside the script 3. I'd much rather maintain commands using the python standard library over complex command-line things with `sed`, `awk`, etc
- hackandtrip 4y agoAnother possibility is using the `scripts` in the `pyproject.toml`, as described here: https://python-poetry.org/docs/pyproject/#scripts https://python-poetry.org/docs/pyproject/#scripts
- hprotagonist 4y agothat’s generally not correct. entry points are how your package, once installed, is called from the command line; i wouldn’t wire up a docker image build step or twine publish that way.
- zer0w1re 4y agoThose two examples are the exact use-case that defining scripts in pyproject.toml are meant for. Users of my installed package would never need to run `twine publish` or build the project's docker image. That's only really needed by developers who would be working from the full project source including pyproject.toml.
- hprotagonist 4y agoentry points are installed to the interpreter site when you `pip install`: everyone gets them including your users.
- leblancfg 4y agoThat's specifically for the Poetry package manager, not Python in general.
- nomel 4y agoNeither is make. You’re using an external tool either way. The one purpose built for the task, and doesn’t have a bunch of archaic footguns [1], will probably give a better experience. 1. https://stackoverflow.com/questions/17965806/how-do-i-handle-files-with-spaces-in-my-makefile#:~:text=The%20very%20short%2C%20but%20IMO%20ultimately%20correct%2C%20answer,make%2C%20your%20%22anonymous%20developers%22%20simply%20cannot%20use%20them https://stackoverflow.com/questions/17965806/how-do-i-handle....
- arbol 4y agoMakefile quickly gets complicated when you have to pass in flags (--an-example-flag). e.g. dev: ./scripts/run-dev.sh $(if $(filter build,$(MAKECMDGOALS)),--build,) $(if $(filter restart,$(MAKECMDGOALS)),--restart,) ...etc Since the project is in python it would be better to write a python script and start the tasks as subprocesses. This means you can use the python argparser you're already familiar with. https://docs.python.org/3/library/subprocess.html https://docs.python.org/3/library/subprocess.html
- belthesar 4y agoI've been seeing a growing use of `just` for cases like this. It works similar in function to make, but it's a great deal cleaner and easier to use.
- eirki 4y agoUrl: https://github.com/casey/just https://github.com/casey/just
- pletnes 4y agoIt also backtracks up the directory tree so you don’t have to cd to run «just test».
- kapilvt 4y agoas someone who maintains projects both using just and make, I have to say just is awesome, and allows things like writing a given task directly in a scripting language. sadly its not universally available across operating systems and linux distros (ubuntu/debian was the last significant dev ecosystem where it wasn't native, re apt install). so for projects with wider contributor goals, I still default to make, despite its warts (which are many imo), cause it tends to work ootb.
- raffraffraff 4y agoI'm considering using it because the justfile can be anywhere in the directory tree above the point where you're running it. (This happens to be useful in certain situations like terraform env/region/stack trees. "Just" is also a command runner. While (as another person has pointed out) "make" is everywhere, "just" is easy to install.
- nathell 4y agoMake wasn’t designed to be a task runner. It can be used as one, but it doesn’t make (pun not intended) a very good one, and it doesn’t have the best syntax, either. Make is an artifact updater. Although it’s not its main focus, Peter Miller’s classical paper ‘Recursive Make Considered Harmful’ does a good job of explaining what it is. One great benefit of make is that it’s present everywhere, so there’s no additional hassle of installing an extra tool. Depending on the project, this hassle-freeness may or may not outweigh make’s relative incomfort as a task runner.
- taeric 4y agoWhat things were designed for is often a side show. We want to think that intent of creation matters, for some reason, when reality is full of that never really being the case. So, on the merits, what makes it a bad task runner between outputs? I agree it is somewhat obtuse, but I'm still mostly crying from trying to get nox and a few other things working. Githubs workflow syntax is as painful, all told. (Though, it has the very real constraint that it is "per repository" and you need something above it.)
- eska 4y agoWell first of all if you’re not using any dependency tracking logic anyway it provides 0 advantages over some scripts like make.sh test.sh push.sh lint.sh (where test.sh can simply call make.sh if you want).
- FreakLegion 4y ago> ...there is no more important proposition for every sort of history than that which we arrive at only with great effort but which we really should reach, -- namely that the origin of the emergence of a thing and its ultimate usefulness, its practical application and incorporation into a system of ends, are toto coelo separate; that anything in existence, having somehow come about, is continually interpreted anew, requisitioned anew, transformed and redirected to a new purpose...
- delaaxe 4y agoInvoke is like a Makefile but in Python: https://www.pyinvoke.org/index.html https://www.pyinvoke.org/index.html
- choeger 4y agoIf your make targets don't have dependencies you are probably using the wrong tool for the job.
- loloquwowndueo 4y agoThese “run something” targets should be marked as .PHONY so make realizes they will not produce a file. Otherwise a file named like the target will confuse make and turn your day into a sad one.
- pydry 4y agoI ran into this back when I used make as a command runner. After a short detour via just Ive been using a shell script with a big case statement since.
- kevincox 4y agoYup. I've found that in most cases a directory of script files ends up better. Or even better a directory of python files since shell kinda sucks.
- schemescape 4y agoUnrelated to Python, but is anyone aware of a tool like Make that can handle file path wildcards in prerequisites? For example, let's say I have a rule that consumes *.json to produce some target. I want it to rebuild that target if any *.json files are added or removed (or modified). As far as I'm aware, Make relies on file system timestamps, so even if it supported wildcards (I'm not sure if it does or not), it wouldn't be able to notice that foo.json has been deleted and a rebuild of the target is needed. I thought I'd ask here before I go build such a tool (to support incremental rebuilds of a static site). Edit to add that my particular use case has a non-prescriptive set of directories, nested a few levels deep. I'm actually realizing that that is probably a bigger hurdle for using something like Make (e.g. transform all *.md files, no matter how deep; but copy everything else, like images, verbatim; oh, and also then aggregate all *.md files into an Atom feed). Yes, I know this is asking a lot of something like Make!
- dwheeler 4y agoGNU make at least can do that.
- schemescape 4y agoInteresting! I had originally tried to do this with GNU Make (with the most naive approach you can imagine) and it wasn't able to notice that a prereq that matched the wildcard had been deleted (i.e. it thought the target was up to date, when I wanted it to rebuild). I'll read up on the wildcard function and see if that is what I was looking for: https://www.gnu.org/software/make/manual/html_node/Wildcard-Function.html https://www.gnu.org/software/make/manual/html_node/Wildcard-... Edit: A sibling comment also pointed out putting the wildcard'd files into a directory to help Make notice deletions. I'll give that a try first.
- bigmanwalter 4y agoDid you try searching stack overflow? https://stackoverflow.com/a/43402649 https://stackoverflow.com/a/43402649 If you dedicate a specific directory for all your JSON, then you can depend on that directory and it will do what you're asking for.
- Messier43 4y agoPipenv has script hooks as well. https://pipenv.pypa.io/en/latest/scripts/ https://pipenv.pypa.io/en/latest/scripts/ It also includes many other package.json features.
- BiteCode_dev 4y agoDon't. Use pydoit.org.
- digdugdirk 4y agoAs someone not up to date on the Python world, how does your suggestion compare with something like Poetry?
- BiteCode_dev 4y agopoetry is specialized in managing venv and packages, with the options to run some commands. doit is a task runner: you declare a name that groups several commands, and when you call the task with the name, it runs all the commands in order. It can accept bash commands, python functions, declare dependencies on other commands or on files, and cascade the tasks to follow those dependencies automatically. It's make, but in Python: nice syntax, works on windows/mac/linux, easy things are easy, hard things are possible.
- thedumbname 4y agoUse Makefile if you want to make your cross-platform build environment Linux only. You will have all kind of issues on Windows, Mac, *BSD, you name it.
- quesera 4y agoYou can resolve the differences between BSD and GNU make. Not sure about Windows. I'd expect that WSL bundles GNU make, no?
- jjgreen 4y agoMake is in POSIX, so stick to POSIX features and your Makefile is portable across all reasonable systems. For Windows, I guess you use a .bat file.
- raffraffraff 4y agoIs that more of an issue if you shell-out a lot in your Makefile? (I don't use mac or win)
- skratlo 4y agoIt's not true that Python doesn't have anything akin to package.json scripts. Bonus points: all is executed within project's virtualenv. https://python-poetry.org/docs/cli/#run https://python-poetry.org/docs/cli/#run
- MuffinFlavored 4y agois this separate to pip's "requirements.txt" usually?
- senand 4y agoYes, it's assuming poetry is used, whereby normally the packages needed by the project are defined in pyproject.toml
- kortex 4y agoVery different. Requirements.txt only handles dependency packages, and only handles on kind of dependency (no dev/release distinction). pyproject.toml does dev/release deps, it has package metadata, it can configure all your lint/test tooling, and it is extensible with plugins.
- MuffinFlavored 4y agoand they can't be combined, right? like package.json for example
- qprofyeh 4y agoHow does this compare to taskipy?
- paiute 4y agoWhile poetry is the way to go today, I've got my eyes on pdm. https://github.com/pdm-project/pdm https://github.com/pdm-project/pdm Pyproject.toml is a pep standard so it will be easy to move around tools.
- 4y ago
- jedberg 4y agoMakefiles are great for local task automation. In college I had a friend who started every homework, even the social science and English homework, with a makefile to compile his Latex files.
- devrc 4y agoI have been using Makefile for over 10 years in all of my projects, and here are some features I've always found lacking in Makefile: 1. There is no way to display documentation for commands and accepted parameters. Yes, you can write a special task that will display comments, but you have to copy it from project to project. 2. The need to pass named parameters when calling tasks. I want to write `make serve localhost 3000` instead of `make serve bind=localhost port=3000` 3. I've always had the need in different projects to use the same commands, so I had to copy tasks from project to project. I need a central place with commands that I can apply to any project. 4. The ability to write tasks in different languages. In some cases, it's easier to write in Python or TypeScript/Deno. 5. And most importantly, it is difficult to write commands in Makefile that can be used in different environments. For example, I need to run commands on different groups of servers: production and staging. This could look like: `make production TASK1 TASK2` or `make stage TASK1 TASK2`. In other words, the production/stage task sets up the parameters for executing tasks in a specific environment. It might be possible to call commands in this way with Makefile, but it seems too complicated. As a result, I decided to write my own utility for task automation: https://github.com/devrc-hub/devrc https://github.com/devrc-hub/devrc It solves all of the above problems and has other interesting features and also written in Rust .
- aldanor 4y agoI think Just (as in Justfile) also solves most of your points, is reasonably widely used, also written in Rust and has integration plugins for it in most editors.
- kayson 4y agoThe main reason I still use Make over other alternatives is that everyone already has it. No need to install something else to bootstrap a project. I put in a `make initialize` that takes care of everything: setting up the virtualenv, using our internal pip mirror, downloading dependencies, etc. It's just such low friction! Side note: Make is also really popular among self-hosters for ansible infrastructure setup.
- rcarmo 4y agoI’ve been doing this for decades now. Anything I do has “make deps”, “make serve”, etc. I don’t care that Make “wasn’t made for this”, it is ubiquitous and reliable at it.
- LeBit 4y agoI am surprised nobody is mentioning go-task https://github.com/go-task/task https://github.com/go-task/task. It is a great project. I love the management of dotfiles, the inclusion of other Taskfiles, the namespacing, the fact it is an easily downloadable binary that runs the same on Linux, macOS AND Windows. It is the superior option.
- aitchnyu 4y agoFor my side projects I use a bash script in the style of https://github.com/adriancooney/Taskfile https://github.com/adriancooney/Taskfile