7 ms·

Show HN: Jeeves – A Pythonic Alternative to GNU Make

55 points by yeti-sh 3y ago

Write a Python file named `jeeves.py` in your project directory with contents: import sh def lint(): """Lint your Python project.""" sh.mypy() This, together with pip install jeeves-shell[all] makes it possible to do j lint …which will run mypy for you, and, via the omnipotent `j` command, open ways for • Automation of routine tasks, • Standardization of your projects, • Implementation of best practices, • And more :) Github: https://github.com/jeeves-sh/jeeves-shell/ https://github.com/jeeves-sh/jeeves-shell/

LeonB 3y ago
It’s an omission and serious bug for this thread to not already contain the following comments: “Why would I consider this, given that make is perfect” And “Doesn’t eMacs org mode already do this?” — Seriously, looks nice. I’m a big fan of Jeeves and Wooster novels and I am a little concerned that Reginald Jeeves would not like to be portrayed as a kind of murderous robot. He’s far too civil to execute people (though he did use a kosh to knock out a police officer on at least one occasion).
- yeti-sh 3y ago
  Thanks! Perhaps this particular robot is running a much more civilized version of Skynet/Cyberdyne firmware. And is sure to be speaking impeccable English.
- deleted 3y ago
  [deleted]
- haolez 3y ago
  I wouldn't say make is perfect, but it's different from this kind of tool. Make is more geared towards file creation that depends on complex pipelines. It's not a task runner. In fact, you have to force it to be a task runner with .PHONY if you really need it to.
  - yeti-sh 3y ago
    With interpretable languages, this feature of Make is IMHO rarely called for. Task running is; Makefile still is a standard way to do project maintenance, both locally and CI. jeeves is an attempt to extract that feature of Make, and provide it in a more accessible way. By making .PHONY and .ONESHELL labels history, as a particular example.
- kelnos 3y ago
  > “Why would I consider this, given that make is perfect” Not even that; this doesn't seem to implement any of make's core features, so it's not actually a replacement for make. It's a replacement for shell scripts. (And no, I don't buy that the comparison is apt because some people use make as a command runner. Sure, they do, sometimes, but that doesn't mean jeeves is a make replacement. It's not.)
Guthur 3y ago
Considering how well python has standardised and unified it's build tool chain this seems like a great idea....oh wait. /sarcasm I would prefer something far more formal than Python.
arcanemachiner 3y ago
Reminds me of `just`. I love `just`. (Although I've never used make...) https://github.com/casey/just https://github.com/casey/just
- yeti-sh 3y ago
  Thanks for the hint, I didn't know about this project; will check it out.
- ospider 3y ago
  The problem of `just` is that it does not consider `mtime`, which makes it almost useless in my case. I do hate the quirky syntax of `make`, however, it does a great job of not repeating itself if files have not been changed.
  - ttymck 3y ago
    Well I don't think "just" is a build tool. It doesn't consider mtime because it has no reason to.
- benatkin 3y ago
  I looked through just again and it seems like a nice design, a little complex but powerful :) I might start using it. Edit: played with just and got confused about how it works. Not going to use it just yet.
  - arcanemachiner 3y ago
    I use it as a little dictionary of commonly used commands in my git repos. No more remembering commands to build a release or deploy a docker image. Now it's 'just release' and 'just deploy'.
duped 3y ago
How does it compare to waf (1)? (1) https://waf.io/ https://waf.io/
- yeti-sh 3y ago
  I've never used waf, but from its documentation it would seem that waf is a build system, implementing a concept of a DAG governing build of a project. That implies that waf is rather complicated. jeeves, on the other hand, does no such thing. It is a command runner: you write a function → it is converted into a shell command. I believe jeeves is easier to get started with, partially due to its simplicity and partially because it follows Python standards. For instance, waf uses a peculiar syntax to define command parameters: waf employs a peculiar method to configure command options: ctx.add_option('--foo', action='store', default=False, help='Silly test') while jeeves relies upon function arguments and type hints (as it is based upon Typer): def do(foo: Annotated[Option(help='Silly test')]): … While jeeves is easy to start with it also promises ability to scale: with modular packages for commands, subcommands, and installable/shareable plugins.
  - llimllib 3y ago
    Make is a build system, not a command runner, so you should probably change your copy; Jeeves doesn’t do what make does
    - yeti-sh 3y ago
      I wouldn't necessarily agree. I've seen may instances of Make particularly being used to run tedious commands — linting, testing, deployment, et cetera; for these things, jeeves can be a perfect replacement. A and B are usually called alternatives if they have an intersection in their feature sets; there are no perfect alternatives.
      - wavemode 3y ago
        That make can be used to run commands that don't build anything is just an artifact of how targets work. If something is marketed as a make alternative, most people would expect it to, at the very least, possess make's core feature, which is skipping targets that have already been built and whose inputs haven't changed. That's what makes something a build system and not just a command runner.
Arch-TK 3y ago
Looks like this has nothing to do with make. Where are the depedency graphs? Where are the suffix and pattern rules?
- yeti-sh 3y ago
  Indeed, they are out of scope. But I still would argue this does have something in common with Make — just as Make, jeeves can be used as a command runner, in which role Make is oftentimes used as well. Instead of `make` as entry point for all project-specific commands (like `make lint`, `make build`, `make deploy`), users can rely upon `j` — both locally and CI. Regarding your particular points: * dependency graphs aren't presently at scope, I haven't yet come up with the method of expressing them in Python which would have entirely satisfied me; * I am not so sure I would want to implement pattern rules due to complexity they bring about. Maybe just writing explicit Python code would be enough for cases where they're used. No strong opinion though.
  - culebron21 3y ago
    Having worked with Makefile and pushed it to limits, I'd say there are some deficiencies in the system you may try to tackle: 1. Databases. We ended up calling `psql -f some_script.sql && touch $my_task_name` and tracking changes with touch files. (Putting this into database on views or materialized views proved to be unsustainable.) 2. Datasets. If you just open a sqlite file, it's changed, and GNU Make thinks you must rebuild everything downstream. Datasets are mostly treated as row-order-independent, so hashing them as is does not always work. 3. Very expensive tasks that shouldn't be called always. Like my makefile had a script that parsed a million web pages, going around captchas via Tor, and touching the upstream files was to be avoided -- or if it happened, I had to manually touch the target, to avoid re-running that part. 4. Some targets can be updated and the result will always be new -- e.g. run a query to a live database, or news website. Some may produce the same. Would appreciate if any system has such a distinction. 5. Surprisingly, lots of alternative build systems don't do partial update. They only update every item in the deps graph. If you manage to get any of these right, you'd be praised.
- deleted 3y ago
  [deleted]
mhh__ 3y ago
I'm not sure if I really want python here. Make is ugly, but it's reasonably clean "algebraically". I find Python very difficult to reason about, and ugly — it's a language that doesn't seem to have been well designed in any grand sense.
- dmos62 3y ago
  I share your sentiments about Python. This is useful in a Python project though.
ketanmaheshwari 3y ago
Similar work: https://snakemake.github.io/ https://snakemake.github.io/
- yeti-sh 3y ago
  Thanks, will check this out!
dikei 3y ago
Remind me of * Fabric: https://www.fabfile.org/ https://www.fabfile.org/ * Invoke: https://www.pyinvoke.org/faq.html https://www.pyinvoke.org/faq.html
- geenat 3y ago
  RIP fabric. Makes me sad how fabric v2 jumped the complexity shark. v1 was so much more simple and elegant.
  - dikei 3y ago
    Apparently, Fabric v1 is replaced by Invoke now.
- yeti-sh 3y ago
  I looked at pyinvoke before I started jeeves. Roughly, as much as I can recall, there were a few reservations: - no type hints for docs & validation, I wanted them - Makefile in its basic form is very concise and doesn't require a @task decorator, I didn't want it either I didn't need dependency graph much.
  - mypalmike 3y ago
    The whole purpose of make is the dependency graph.
  - creshal 3y ago
    > I didn't need dependency graph much. So you made a Make alternative that isn't a Make alternative because you never needed Make in the first place?
    - yeti-sh 3y ago
      There are a lot of projects, both commercial and OSS, which have a Makefile for linting, version releases, etc, — and never use its dependency graph feature; or, their usage is so superficial that implementing it, say, - via direct function calls in Python code, - or using Typer callback functions …is easy. There are potential approaches worth exploring (annotations, decorators, …). I might come up with a list of options with syntax examples and ask for community's thoughts about it, — but at this point jeeves, as an MVP, is already useful for me and a few people I work with, and that motivated me to share it with the community.
theamk 3y ago
Note the examples use "sh" python library, which by default (1) captures stdout and stderr of all processes and (2) create tty for processs stdout. Those are really bad defaults in general, which is why I recommend everyone to avoid this library. The tty on stdout means many programs run in "interactive" rather then "batch" mode: programs which use pager get output truncated, auto-colors may get enabled and emit ESC controls into output streams (or not, depending on user's distro... fun!) But it is _especially_ bad for general tools runner. Because of forced capture, you are not going to get any output or errors until process completes, and maybe not even then. Also interactive prompts won't work, errors would be reordered compared to regular outputs and so on... I can't say anything about jeeves, but at least avoid "sh".
- yeti-sh 3y ago
  I like sh for brevity of its API. I often have to use `_tty_out=False`, but this is easy to fix once and for all commands in a script: my_sh = sh.bake(_tty_out=False) my_sh.do_whatever() The way how sh captures output can apparently be altered, say, by providing a callable to _out argument.
  - blacklight 3y ago
    You could have probably created your own little wrapper on top of subprocess.Popen and dispatch stdout and stderr around. No need for an external library with bad tty/piping defaults just because it has a nice API (which needs to be tweaked with _tty_out=False anyway if i want to pipe the output to another command. Btw does rich scrape the special formatting characters if piped?
    - yeti-sh 3y ago
      An example from the top of my head: compose = sh.docker.compose.bake('-f', 'deploy/dev.yml') … compose.down() … compose.up('--force-recreate', '-d') I feel this is a major improvement on top of Makefiles + shell commands in them. Nice API _matters_; it is ergonomics and therefore productivity. You can specify `_out=rich.print` and it will work but AFAIK it won't scrape ASCII terminal formatting characters. I had to fix these characters in my `sh` based scripts but do not consider that as a big deal.
a-dub 3y ago
i don't think you can call this an alternative to make, it doesn't seem to do any dependency processing (phony or file based) at all!
geenat 3y ago
How's the SSH interoperability? Transparent SSH interop is why I stick to Bash rather than moving to a Python solution.
- yeti-sh 3y ago
  Please elaborate. Do you mean submitting commands via ssh to remote servers from the script? The way I'd recommend to run shell commands from jeeves is `sh` library. It has features for ssh support: https://sh.readthedocs.io/en/latest/sections/contrib.html#ssh https://sh.readthedocs.io/en/latest/sections/contrib.html#ss...
benatkin 3y ago
I wouldn't want a stray python file, whether I was using in it a non-python project or a python project, to trip up editors. I think Earthly has the right idea with an Earthfile. https://docs.earthly.dev/docs/earthfile https://docs.earthly.dev/docs/earthfile It's a different tool for running commands, though.
- yeti-sh 3y ago
  You can also install a jeeves plugin with pip. Say, pip install jeeves-yeti-pyproject will provide you with jeeves, a bunch of commands, and a pack of dev dependencies which I personally happen to like and to use in my projects. I don't believe Make has plugins.
  - benatkin 3y ago
    That's some more indirection, besides putting a disconnected python file in the project directory. I'm not sure whether it's suggested to install it globally, as a development dependency with poetry/similar, or with pipx https://pypa.github.io/pipx/ https://pypa.github.io/pipx/ The "import sh" thing could have some users installing this package https://pypi.org/project/sh/ https://pypi.org/project/sh/ This in addition to it being known as "j". It has at least 3 names, jeeves, j, and sh.
    - yeti-sh 3y ago
      For a Python project, I'd recommend doing this as a dev dependency. * jeeves is the name of the project which I'm advertising, which converts a Python file to a set of commands; * `j` is the name of executable which aformentioned project exposes; * and `sh` is the library (the correct link to which you have provided) which is an optional dependency of `jeeves` and provides a more convenient interface to calling processes and executing commands from Python than `subprocess.run()`.
- shepherdjerred 3y ago
  Earthly is fantastic and I would recommend it, but they really need to improve sharing cache.
travellingprog 3y ago
Another alternative that I have liked for replacing Make as a task runner is Task: https://taskfile.dev/ https://taskfile.dev/
phil-martin 3y ago
The most comprehensive make alternative in python I've seen is Scons (https://scons.org/ https://scons.org/) It would be worth to see how they tackles some of the challenges you're looking into. Blurb from the website: SCons is an Open Source software construction tool. Think of SCons as an improved, cross-platform substitute for the classic Make utility with integrated functionality similar to autoconf/automake and compiler caches such as ccache. In short, SCons is an easier, more reliable and faster way to build software.
- pasc1878 3y ago
  Also see waf https://waf.io/ https://waf.io/ which is a similar build system as scons to replace make including dependencies
- chasinglogic 3y ago
  While I can't contest that SCons is comprehensive, I would never recommend it as a source of learning "what to do". SCons is not idiomatic Python and it abuses things like `eval` which gives it terrible performance. Source: I used to work for MongoDB and my full time job was to make SCons faster, which I eventually did by making it a Ninja generator (which has now been upstreamed). But the code is still pretty bad. Using SCons however is much nicer than using make / autoconf IMO, especially now that you can farm the builds out to Ninja.
  - phil-martin 3y ago
    Oh definitely agree on the non-idiomatic python. I was thinking less of "how to correctly implement and write the python code" and more of the "learn the history and decisions on why it might do things, and the user experience of defining a build in python". I really liked its consistency of repeatable build steps, the way it handled it's dependencies, and accepted the tradeoff it presented for slowness. But its been perhaps 10 years since I looked it, likely longer, and rose coloured glasses are applicable. And nice work making SCons faster!! not at easy thing to do at all.
- fero14041 3y ago
  An alternative to Scons could be Doit (<https://pydoit.org/ https://pydoit.org/>), which if I remember correctly was built as a faster alternative to Scons. See also reasons of some users to prefer the later to other mentioned here: <https://pydoit.org/stories.html https://pydoit.org/stories.html>.
linsomniac 3y ago
This looks really interesting and clever, I'm going to have to try using it for some of my python utilities. I really like how it handles arguments using click-like semantics. In some ways this is similar to a project I've been working on for the last month that is kind of a pythonic reimagining of Ansible and Cookiecutter. It's still a WIP, but I'm starting to successfully convert my YAML-style declarative snippet/cookiecutters into Python, and I'm working on improving the documentation now. Where Jeeves seems to excel at collections of small "scriptlets", uPlaybook is targeting declarative automation of larger tasks, including templating of configuration files, triggering restarts on changes, etc... https://github.com/linsomniac/uplaybook https://github.com/linsomniac/uplaybook
- pasc1878 3y ago
  Try the existing invoke which is a python task runner https://www.pyinvoke.org/ https://www.pyinvoke.org/
  - linsomniac 3y ago
    invoke is closer to Jeeves in that you define your tasks in a Python file. uPlaybook is an experiment investigating: What if Ansible had Python syntax rather than YAML? So it focuses on templating config files and declaring system state.
Animats 3y ago
They're thinking macro, not dependency graph. The point of "make" was supposed to be that it defined a dependency graph, then executed only the commands required to achieve the goals. Somewhere that got lost.
kelnos 3y ago
I don't really get the comparison to makefiles. The idea of make is that you express rules that lay out dependencies between files and other rules, and make (when given a target to build) figures out which rules do and don't need to be run based on what you want to build and what is and isn't up to date. Jeeves seems to be a thing where you define python functions and then Jeeves makes it so you can run them as 'subcommands' of the 'j' command line program, and to take arguments, etc. Seems like it'd be up to you to write your own code to handle figuring out if each function/subcommand has dependencies, and if those dependencies need to be run or built. Neat project, but has nothing to do with make. Sure, some people may use make as a command runner, but that doesn't mean this is a make replacement. At least not for the vast majority of things that people use make for.
Brian_K_White 3y ago
This is not an alternative to make* any more than some random half-baked build.sh is. * gnu or otherwise, and why specify that in the first place anyway? If it did actually do a suitable job of replacing gnu make, would it not also suitably replace bsd or any other make? Is a person who says something like that a good source of ideas so deep and quality that they probably do improve upon those who created unix? make may not be the final build system for the rest of eternity, but all this is, is "I don't really understand the full job make does, so here is something which only does a simpler job which I do understand, and I like python."
- yeti-sh 3y ago
  Saying 'GNU Make' particularly was an automatism. When I use Make I do use GNU Make specifically. I do not have experience with BSD Make, or other Make dialects. > did actually do a suitable job of replacing gnu make The title does not say _a replacement_, it says _an alternative_. I never intended to replace GNU Make; if I said that anywhere — that was a mistake on my part. > do improve upon those who created unix I did not intend to improve upon UNIX ideas or philosophy. The improvements I am aiming for are: - developer experience, - conciseness and maintainability of the code. …and these are being addressed for a narrow use case. For that use case, in my experience, Make is very often used. I argue that within the bounds of this use case, this is an alternative which can be of use to improve productivity and everyday experience. > I don't really understand the full job make does, so here is something which only does a simpler job which I do understand, and I like python I classify these assumptions about my understanding or misunderstanding as ad hominem and unprofessional.
  - zvr 3y ago
    The issue is that make(1) has lots of functionality... none of which is present here. In other words, make(1) does a lot of things, one of which is to run shell commands. If this the only thing you care about and your project offers, it would be better expressed as "Pythonic way to run commands" or something, with no mention of make.
    - 3y ago
culebron21 3y ago
If this is a translation of functions into CLI commands, there are enough of such crates. Old package `argh` and now `yaargh`[1] also translate functions into CLI commands in a very neat way, with just a decorator, and they don't need an external executable in the system to run, just `python your_file.py`. Makefile keeps dependency graph. I had a 100-entry 300-line Makefile, with graphviz drawing charts of it, and kept a huge, year-long project on my own, organized and running all partial updates smoothly. [1] https://github.com/ekimekim/yaargh https://github.com/ekimekim/yaargh
- yeti-sh 3y ago
  That's right; Google also published a similar package named `fire`, I used it a lot. > just `python your_file.py` That's actually one of the points. j lint is shorter than python scripts.py lint and the letter `j` is what the index finger of your right hand is pointing to on the keyboard. There is usually a tactile bump on that key. It is something you can type very quickly. You can also install shell completion to improve the experience further. Ergonomics matters.
  - culebron21 3y ago
    Well, you can do the same trick with argh & a symlink in /usr/bin. I'll agree this is some improvement in ergonomics if the executable supports tab completion. With makefile + argh I can do this: put the script as a dependency of data file, and rebuild it if code changes. P=python3 my_data_file.csv: my_script.py input_data.csv $P --flag1 --flag2 $^ $@ With your package, this becomes harder -- the file name and command name now are separated and must be checked for being in sync. So, the package adds some features, but does it at some cost of other ways of interaction.
    - yeti-sh 3y ago
      > argh & a symlink in /usr/bin. - Is putting a symlink to one of your project directories, probably residing in $HOME, to /usr/bin/ a good practice? - As an alternative one can put a symlink into ~/bin; but what if you have multiple different projects? - And they can have different jeeves commands and different plugins installed. For instance, `j lint` implementations in different projects are work on are completely different. Because these are different projects. Rather, I prefer to have jeeves automatically create the executable command in each virtual env, and its behavior in different virtual envs will be different.
marcosnils 3y ago
TIL about Jeeves, things like something that could play very nice with the Dagger python SDK (https://docs.dagger.io/sdk/python https://docs.dagger.io/sdk/python). Particularly since Dagger could help with bootstraping the python runtime and providing caching capabilities out of the box.
tpoacher 3y ago
Note that contrary to popular belief, there's nothing stopping one from declaring python as the main shell in a makefile.

55 points by yeti-sh 3y ago

LeonB 3y ago

It’s an omission and serious bug for this thread to not already contain the following comments: “Why would I consider this, given that make is perfect” And “Doesn’t eMacs org mode already do this?” — Seriously, looks nice. I’m a big fan of Jeeves and Wooster novels and I am a little concerned that Reginald Jeeves would not like to be portrayed as a kind of murderous robot. He’s far too civil to execute people (though he did use a kosh to knock out a police officer on at least one occasion).

yeti-sh 3y ago
Thanks! Perhaps this particular robot is running a much more civilized version of Skynet/Cyberdyne firmware. And is sure to be speaking impeccable English.
deleted 3y ago
[deleted]
haolez 3y ago
I wouldn't say make is perfect, but it's different from this kind of tool. Make is more geared towards file creation that depends on complex pipelines. It's not a task runner. In fact, you have to force it to be a task runner with .PHONY if you really need it to.
- yeti-sh 3y ago
  With interpretable languages, this feature of Make is IMHO rarely called for. Task running is; Makefile still is a standard way to do project maintenance, both locally and CI. jeeves is an attempt to extract that feature of Make, and provide it in a more accessible way. By making .PHONY and .ONESHELL labels history, as a particular example.
kelnos 3y ago
> “Why would I consider this, given that make is perfect” Not even that; this doesn't seem to implement any of make's core features, so it's not actually a replacement for make. It's a replacement for shell scripts. (And no, I don't buy that the comparison is apt because some people use make as a command runner. Sure, they do, sometimes, but that doesn't mean jeeves is a make replacement. It's not.)

Guthur 3y ago

Considering how well python has standardised and unified it's build tool chain this seems like a great idea....oh wait. /sarcasm I would prefer something far more formal than Python.

arcanemachiner 3y ago

Reminds me of `just`. I love `just`. (Although I've never used make...) https://github.com/casey/just https://github.com/casey/just

yeti-sh 3y ago
Thanks for the hint, I didn't know about this project; will check it out.
ospider 3y ago
The problem of `just` is that it does not consider `mtime`, which makes it almost useless in my case. I do hate the quirky syntax of `make`, however, it does a great job of not repeating itself if files have not been changed.
- ttymck 3y ago
  Well I don't think "just" is a build tool. It doesn't consider mtime because it has no reason to.
benatkin 3y ago
I looked through just again and it seems like a nice design, a little complex but powerful :) I might start using it. Edit: played with just and got confused about how it works. Not going to use it just yet.
- arcanemachiner 3y ago
  I use it as a little dictionary of commonly used commands in my git repos. No more remembering commands to build a release or deploy a docker image. Now it's 'just release' and 'just deploy'.

duped 3y ago

How does it compare to waf (1)? (1) https://waf.io/ https://waf.io/

yeti-sh 3y ago
I've never used waf, but from its documentation it would seem that waf is a build system, implementing a concept of a DAG governing build of a project. That implies that waf is rather complicated. jeeves, on the other hand, does no such thing. It is a command runner: you write a function → it is converted into a shell command. I believe jeeves is easier to get started with, partially due to its simplicity and partially because it follows Python standards. For instance, waf uses a peculiar syntax to define command parameters: waf employs a peculiar method to configure command options: ctx.add_option('--foo', action='store', default=False, help='Silly test') while jeeves relies upon function arguments and type hints (as it is based upon Typer): def do(foo: Annotated[Option(help='Silly test')]): … While jeeves is easy to start with it also promises ability to scale: with modular packages for commands, subcommands, and installable/shareable plugins.
- llimllib 3y ago
  Make is a build system, not a command runner, so you should probably change your copy; Jeeves doesn’t do what make does
  - yeti-sh 3y ago
    I wouldn't necessarily agree. I've seen may instances of Make particularly being used to run tedious commands — linting, testing, deployment, et cetera; for these things, jeeves can be a perfect replacement. A and B are usually called alternatives if they have an intersection in their feature sets; there are no perfect alternatives.
    - wavemode 3y ago
      That make can be used to run commands that don't build anything is just an artifact of how targets work. If something is marketed as a make alternative, most people would expect it to, at the very least, possess make's core feature, which is skipping targets that have already been built and whose inputs haven't changed. That's what makes something a build system and not just a command runner.

Arch-TK 3y ago

Looks like this has nothing to do with make. Where are the depedency graphs? Where are the suffix and pattern rules?

yeti-sh 3y ago
Indeed, they are out of scope. But I still would argue this does have something in common with Make — just as Make, jeeves can be used as a command runner, in which role Make is oftentimes used as well. Instead of `make` as entry point for all project-specific commands (like `make lint`, `make build`, `make deploy`), users can rely upon `j` — both locally and CI. Regarding your particular points: * dependency graphs aren't presently at scope, I haven't yet come up with the method of expressing them in Python which would have entirely satisfied me; * I am not so sure I would want to implement pattern rules due to complexity they bring about. Maybe just writing explicit Python code would be enough for cases where they're used. No strong opinion though.
- culebron21 3y ago
  Having worked with Makefile and pushed it to limits, I'd say there are some deficiencies in the system you may try to tackle: 1. Databases. We ended up calling `psql -f some_script.sql && touch $my_task_name` and tracking changes with touch files. (Putting this into database on views or materialized views proved to be unsustainable.) 2. Datasets. If you just open a sqlite file, it's changed, and GNU Make thinks you must rebuild everything downstream. Datasets are mostly treated as row-order-independent, so hashing them as is does not always work. 3. Very expensive tasks that shouldn't be called always. Like my makefile had a script that parsed a million web pages, going around captchas via Tor, and touching the upstream files was to be avoided -- or if it happened, I had to manually touch the target, to avoid re-running that part. 4. Some targets can be updated and the result will always be new -- e.g. run a query to a live database, or news website. Some may produce the same. Would appreciate if any system has such a distinction. 5. Surprisingly, lots of alternative build systems don't do partial update. They only update every item in the deps graph. If you manage to get any of these right, you'd be praised.
deleted 3y ago
[deleted]

mhh__ 3y ago

I'm not sure if I really want python here. Make is ugly, but it's reasonably clean "algebraically". I find Python very difficult to reason about, and ugly — it's a language that doesn't seem to have been well designed in any grand sense.

dmos62 3y ago
I share your sentiments about Python. This is useful in a Python project though.

ketanmaheshwari 3y ago

Similar work: https://snakemake.github.io/ https://snakemake.github.io/

yeti-sh 3y ago
Thanks, will check this out!

dikei 3y ago

Remind me of * Fabric: https://www.fabfile.org/ https://www.fabfile.org/ * Invoke: https://www.pyinvoke.org/faq.html https://www.pyinvoke.org/faq.html

geenat 3y ago
RIP fabric. Makes me sad how fabric v2 jumped the complexity shark. v1 was so much more simple and elegant.
- dikei 3y ago
  Apparently, Fabric v1 is replaced by Invoke now.
yeti-sh 3y ago
I looked at pyinvoke before I started jeeves. Roughly, as much as I can recall, there were a few reservations: - no type hints for docs & validation, I wanted them - Makefile in its basic form is very concise and doesn't require a @task decorator, I didn't want it either I didn't need dependency graph much.
- mypalmike 3y ago
  The whole purpose of make is the dependency graph.
- creshal 3y ago
  > I didn't need dependency graph much. So you made a Make alternative that isn't a Make alternative because you never needed Make in the first place?
  - yeti-sh 3y ago
    There are a lot of projects, both commercial and OSS, which have a Makefile for linting, version releases, etc, — and never use its dependency graph feature; or, their usage is so superficial that implementing it, say, - via direct function calls in Python code, - or using Typer callback functions …is easy. There are potential approaches worth exploring (annotations, decorators, …). I might come up with a list of options with syntax examples and ask for community's thoughts about it, — but at this point jeeves, as an MVP, is already useful for me and a few people I work with, and that motivated me to share it with the community.

theamk 3y ago

Note the examples use "sh" python library, which by default (1) captures stdout and stderr of all processes and (2) create tty for processs stdout. Those are really bad defaults in general, which is why I recommend everyone to avoid this library. The tty on stdout means many programs run in "interactive" rather then "batch" mode: programs which use pager get output truncated, auto-colors may get enabled and emit ESC controls into output streams (or not, depending on user's distro... fun!) But it is _especially_ bad for general tools runner. Because of forced capture, you are not going to get any output or errors until process completes, and maybe not even then. Also interactive prompts won't work, errors would be reordered compared to regular outputs and so on... I can't say anything about jeeves, but at least avoid "sh".

yeti-sh 3y ago
I like sh for brevity of its API. I often have to use `_tty_out=False`, but this is easy to fix once and for all commands in a script: my_sh = sh.bake(_tty_out=False) my_sh.do_whatever() The way how sh captures output can apparently be altered, say, by providing a callable to _out argument.
- blacklight 3y ago
  You could have probably created your own little wrapper on top of subprocess.Popen and dispatch stdout and stderr around. No need for an external library with bad tty/piping defaults just because it has a nice API (which needs to be tweaked with _tty_out=False anyway if i want to pipe the output to another command. Btw does rich scrape the special formatting characters if piped?
  - yeti-sh 3y ago
    An example from the top of my head: compose = sh.docker.compose.bake('-f', 'deploy/dev.yml') … compose.down() … compose.up('--force-recreate', '-d') I feel this is a major improvement on top of Makefiles + shell commands in them. Nice API _matters_; it is ergonomics and therefore productivity. You can specify `_out=rich.print` and it will work but AFAIK it won't scrape ASCII terminal formatting characters. I had to fix these characters in my `sh` based scripts but do not consider that as a big deal.

a-dub 3y ago

i don't think you can call this an alternative to make, it doesn't seem to do any dependency processing (phony or file based) at all!

geenat 3y ago

How's the SSH interoperability? Transparent SSH interop is why I stick to Bash rather than moving to a Python solution.

yeti-sh 3y ago
Please elaborate. Do you mean submitting commands via ssh to remote servers from the script? The way I'd recommend to run shell commands from jeeves is `sh` library. It has features for ssh support: https://sh.readthedocs.io/en/latest/sections/contrib.html#ssh https://sh.readthedocs.io/en/latest/sections/contrib.html#ss...

benatkin 3y ago

I wouldn't want a stray python file, whether I was using in it a non-python project or a python project, to trip up editors. I think Earthly has the right idea with an Earthfile. https://docs.earthly.dev/docs/earthfile https://docs.earthly.dev/docs/earthfile It's a different tool for running commands, though.

yeti-sh 3y ago
You can also install a jeeves plugin with pip. Say, pip install jeeves-yeti-pyproject will provide you with jeeves, a bunch of commands, and a pack of dev dependencies which I personally happen to like and to use in my projects. I don't believe Make has plugins.
- benatkin 3y ago
  That's some more indirection, besides putting a disconnected python file in the project directory. I'm not sure whether it's suggested to install it globally, as a development dependency with poetry/similar, or with pipx https://pypa.github.io/pipx/ https://pypa.github.io/pipx/ The "import sh" thing could have some users installing this package https://pypi.org/project/sh/ https://pypi.org/project/sh/ This in addition to it being known as "j". It has at least 3 names, jeeves, j, and sh.
  - yeti-sh 3y ago
    For a Python project, I'd recommend doing this as a dev dependency. * jeeves is the name of the project which I'm advertising, which converts a Python file to a set of commands; * `j` is the name of executable which aformentioned project exposes; * and `sh` is the library (the correct link to which you have provided) which is an optional dependency of `jeeves` and provides a more convenient interface to calling processes and executing commands from Python than `subprocess.run()`.
shepherdjerred 3y ago
Earthly is fantastic and I would recommend it, but they really need to improve sharing cache.

travellingprog 3y ago

Another alternative that I have liked for replacing Make as a task runner is Task: https://taskfile.dev/ https://taskfile.dev/

phil-martin 3y ago

The most comprehensive make alternative in python I've seen is Scons (https://scons.org/ https://scons.org/) It would be worth to see how they tackles some of the challenges you're looking into. Blurb from the website: SCons is an Open Source software construction tool. Think of SCons as an improved, cross-platform substitute for the classic Make utility with integrated functionality similar to autoconf/automake and compiler caches such as ccache. In short, SCons is an easier, more reliable and faster way to build software.

pasc1878 3y ago
Also see waf https://waf.io/ https://waf.io/ which is a similar build system as scons to replace make including dependencies
chasinglogic 3y ago
While I can't contest that SCons is comprehensive, I would never recommend it as a source of learning "what to do". SCons is not idiomatic Python and it abuses things like `eval` which gives it terrible performance. Source: I used to work for MongoDB and my full time job was to make SCons faster, which I eventually did by making it a Ninja generator (which has now been upstreamed). But the code is still pretty bad. Using SCons however is much nicer than using make / autoconf IMO, especially now that you can farm the builds out to Ninja.
- phil-martin 3y ago
  Oh definitely agree on the non-idiomatic python. I was thinking less of "how to correctly implement and write the python code" and more of the "learn the history and decisions on why it might do things, and the user experience of defining a build in python". I really liked its consistency of repeatable build steps, the way it handled it's dependencies, and accepted the tradeoff it presented for slowness. But its been perhaps 10 years since I looked it, likely longer, and rose coloured glasses are applicable. And nice work making SCons faster!! not at easy thing to do at all.
fero14041 3y ago
An alternative to Scons could be Doit (<https://pydoit.org/ https://pydoit.org/>), which if I remember correctly was built as a faster alternative to Scons. See also reasons of some users to prefer the later to other mentioned here: <https://pydoit.org/stories.html https://pydoit.org/stories.html>.

linsomniac 3y ago

This looks really interesting and clever, I'm going to have to try using it for some of my python utilities. I really like how it handles arguments using click-like semantics. In some ways this is similar to a project I've been working on for the last month that is kind of a pythonic reimagining of Ansible and Cookiecutter. It's still a WIP, but I'm starting to successfully convert my YAML-style declarative snippet/cookiecutters into Python, and I'm working on improving the documentation now. Where Jeeves seems to excel at collections of small "scriptlets", uPlaybook is targeting declarative automation of larger tasks, including templating of configuration files, triggering restarts on changes, etc... https://github.com/linsomniac/uplaybook https://github.com/linsomniac/uplaybook

pasc1878 3y ago
Try the existing invoke which is a python task runner https://www.pyinvoke.org/ https://www.pyinvoke.org/
- linsomniac 3y ago
  invoke is closer to Jeeves in that you define your tasks in a Python file. uPlaybook is an experiment investigating: What if Ansible had Python syntax rather than YAML? So it focuses on templating config files and declaring system state.

Animats 3y ago

They're thinking macro, not dependency graph. The point of "make" was supposed to be that it defined a dependency graph, then executed only the commands required to achieve the goals. Somewhere that got lost.

kelnos 3y ago

I don't really get the comparison to makefiles. The idea of make is that you express rules that lay out dependencies between files and other rules, and make (when given a target to build) figures out which rules do and don't need to be run based on what you want to build and what is and isn't up to date. Jeeves seems to be a thing where you define python functions and then Jeeves makes it so you can run them as 'subcommands' of the 'j' command line program, and to take arguments, etc. Seems like it'd be up to you to write your own code to handle figuring out if each function/subcommand has dependencies, and if those dependencies need to be run or built. Neat project, but has nothing to do with make. Sure, some people may use make as a command runner, but that doesn't mean this is a make replacement. At least not for the vast majority of things that people use make for.

Brian_K_White 3y ago

This is not an alternative to make* any more than some random half-baked build.sh is. * gnu or otherwise, and why specify that in the first place anyway? If it did actually do a suitable job of replacing gnu make, would it not also suitably replace bsd or any other make? Is a person who says something like that a good source of ideas so deep and quality that they probably do improve upon those who created unix? make may not be the final build system for the rest of eternity, but all this is, is "I don't really understand the full job make does, so here is something which only does a simpler job which I do understand, and I like python."

yeti-sh 3y ago
Saying 'GNU Make' particularly was an automatism. When I use Make I do use GNU Make specifically. I do not have experience with BSD Make, or other Make dialects. > did actually do a suitable job of replacing gnu make The title does not say _a replacement_, it says _an alternative_. I never intended to replace GNU Make; if I said that anywhere — that was a mistake on my part. > do improve upon those who created unix I did not intend to improve upon UNIX ideas or philosophy. The improvements I am aiming for are: - developer experience, - conciseness and maintainability of the code. …and these are being addressed for a narrow use case. For that use case, in my experience, Make is very often used. I argue that within the bounds of this use case, this is an alternative which can be of use to improve productivity and everyday experience. > I don't really understand the full job make does, so here is something which only does a simpler job which I do understand, and I like python I classify these assumptions about my understanding or misunderstanding as ad hominem and unprofessional.
- zvr 3y ago
  The issue is that make(1) has lots of functionality... none of which is present here. In other words, make(1) does a lot of things, one of which is to run shell commands. If this the only thing you care about and your project offers, it would be better expressed as "Pythonic way to run commands" or something, with no mention of make.
  - 3y ago

culebron21 3y ago

If this is a translation of functions into CLI commands, there are enough of such crates. Old package `argh` and now `yaargh`[1] also translate functions into CLI commands in a very neat way, with just a decorator, and they don't need an external executable in the system to run, just `python your_file.py`. Makefile keeps dependency graph. I had a 100-entry 300-line Makefile, with graphviz drawing charts of it, and kept a huge, year-long project on my own, organized and running all partial updates smoothly. [1] https://github.com/ekimekim/yaargh https://github.com/ekimekim/yaargh

yeti-sh 3y ago
That's right; Google also published a similar package named `fire`, I used it a lot. > just `python your_file.py` That's actually one of the points. j lint is shorter than python scripts.py lint and the letter `j` is what the index finger of your right hand is pointing to on the keyboard. There is usually a tactile bump on that key. It is something you can type very quickly. You can also install shell completion to improve the experience further. Ergonomics matters.
- culebron21 3y ago
  Well, you can do the same trick with argh & a symlink in /usr/bin. I'll agree this is some improvement in ergonomics if the executable supports tab completion. With makefile + argh I can do this: put the script as a dependency of data file, and rebuild it if code changes. P=python3 my_data_file.csv: my_script.py input_data.csv $P --flag1 --flag2 $^ $@ With your package, this becomes harder -- the file name and command name now are separated and must be checked for being in sync. So, the package adds some features, but does it at some cost of other ways of interaction.
  - yeti-sh 3y ago
    > argh & a symlink in /usr/bin. - Is putting a symlink to one of your project directories, probably residing in $HOME, to /usr/bin/ a good practice? - As an alternative one can put a symlink into ~/bin; but what if you have multiple different projects? - And they can have different jeeves commands and different plugins installed. For instance, `j lint` implementations in different projects are work on are completely different. Because these are different projects. Rather, I prefer to have jeeves automatically create the executable command in each virtual env, and its behavior in different virtual envs will be different.

marcosnils 3y ago

TIL about Jeeves, things like something that could play very nice with the Dagger python SDK (https://docs.dagger.io/sdk/python https://docs.dagger.io/sdk/python). Particularly since Dagger could help with bootstraping the python runtime and providing caching capabilities out of the box.

tpoacher 3y ago

Note that contrary to popular belief, there's nothing stopping one from declaring python as the main shell in a makefile.