3 ms·

Looks like this has nothing to do with make. Where are the depedency graphs? Where are the suffix and pattern rules?

yeti-sh 3y ago
Indeed, they are out of scope. But I still would argue this does have something in common with Make — just as Make, jeeves can be used as a command runner, in which role Make is oftentimes used as well. Instead of `make` as entry point for all project-specific commands (like `make lint`, `make build`, `make deploy`), users can rely upon `j` — both locally and CI. Regarding your particular points: * dependency graphs aren't presently at scope, I haven't yet come up with the method of expressing them in Python which would have entirely satisfied me; * I am not so sure I would want to implement pattern rules due to complexity they bring about. Maybe just writing explicit Python code would be enough for cases where they're used. No strong opinion though.
- culebron21 3y ago
  Having worked with Makefile and pushed it to limits, I'd say there are some deficiencies in the system you may try to tackle: 1. Databases. We ended up calling `psql -f some_script.sql && touch $my_task_name` and tracking changes with touch files. (Putting this into database on views or materialized views proved to be unsustainable.) 2. Datasets. If you just open a sqlite file, it's changed, and GNU Make thinks you must rebuild everything downstream. Datasets are mostly treated as row-order-independent, so hashing them as is does not always work. 3. Very expensive tasks that shouldn't be called always. Like my makefile had a script that parsed a million web pages, going around captchas via Tor, and touching the upstream files was to be avoided -- or if it happened, I had to manually touch the target, to avoid re-running that part. 4. Some targets can be updated and the result will always be new -- e.g. run a query to a live database, or news website. Some may produce the same. Would appreciate if any system has such a distinction. 5. Surprisingly, lots of alternative build systems don't do partial update. They only update every item in the deps graph. If you manage to get any of these right, you'd be praised.
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
shoo 3y ago
i agree. make is a tool for defining recipes to derive target files from dependency files. archetypal use case would be compiling and linking a bunch of C source and header files into an application binary. make isn't specialized to building C programs, make can also be used for other file-driven computation tasks such as defining data processing pipelines / graphs of processing that consume data files and produce data files. the rough essence of make is arguably: 1. there are targets, and recipes (written as a shell command) to derive the target 2. targets can express dependencies on other targets 3. targets are files in the filesystem. 4. add a bunch of language features on top (pattern rules, automatic variables, built-in functions, etc) to make it easier to write and maintain the recipes Another way of thinking about make is that it is a a way to define and evaluate gigantic pure-functional expressions, where each input and output file is a file, and where intermediate sub-expressions can be cached. for an alternative to make that supports 1+2+3+4, see e.g. https://gittup.org/tup/index.html https://gittup.org/tup/index.html for an alternative to make that supports 1+2+3 but explicitly does not aim to support 4, see e.g. https://ninja-build.org/manual.html https://ninja-build.org/manual.html another alternative to make that supports 1+2+3+4 but de-emphasizes 3, "targets are files", from the ruby community, is https://github.com/ruby/rake https://github.com/ruby/rake these days in projects it is quite common to see make used to do 1 but not 2, 3 nor 4. i.e. a makefile as a way to define a bunch of imperative shell recipes for named targets, where each named target has no relationship to a file, and dependencies between the targets aren't expressed or relevant. as a random example of this, here's the first google result searching for "makefile terraform": https://github.com/paulRbr/terraform-makefile/blob/master/Makefile https://github.com/paulRbr/terraform-makefile/blob/master/Ma... note all the targets are ".PHONY" targets, we're telling make to disable the default behaviour where every target corresponds to a file in the file system -- to turn off "3 targets are files in the filesystem". there's very sparing use of dependencies. so this is an example of using makes support for 1 defining recipes to produce targets and a little splash of 2 dependencies between targets, but not 3 nor 4. it looks like this "jeeves" project is aiming at this latter use case, not attempting to implement support for 2+3, the core of how make is typically used as a build tool
- yeti-sh 3y ago
  That's right. For projects which only use subset of Make features described by (1) jeeves can be considered an alternative for Make, and I would believe there are quite a few such projects.

Indeed, they are out of scope. But I still would argue this does have something in common with Make — just as Make, jeeves can be used as a command runner, in which role Make is oftentimes used as well. Instead of `make` as entry point for all project-specific commands (like `make lint`, `make build`, `make deploy`), users can rely upon `j` — both locally and CI. Regarding your particular points: * dependency graphs aren't presently at scope, I haven't yet come up with the method of expressing them in Python which would have entirely satisfied me; * I am not so sure I would want to implement pattern rules due to complexity they bring about. Maybe just writing explicit Python code would be enough for cases where they're used. No strong opinion though.

culebron21 3y ago
Having worked with Makefile and pushed it to limits, I'd say there are some deficiencies in the system you may try to tackle: 1. Databases. We ended up calling `psql -f some_script.sql && touch $my_task_name` and tracking changes with touch files. (Putting this into database on views or materialized views proved to be unsustainable.) 2. Datasets. If you just open a sqlite file, it's changed, and GNU Make thinks you must rebuild everything downstream. Datasets are mostly treated as row-order-independent, so hashing them as is does not always work. 3. Very expensive tasks that shouldn't be called always. Like my makefile had a script that parsed a million web pages, going around captchas via Tor, and touching the upstream files was to be avoided -- or if it happened, I had to manually touch the target, to avoid re-running that part. 4. Some targets can be updated and the result will always be new -- e.g. run a query to a live database, or news website. Some may produce the same. Would appreciate if any system has such a distinction. 5. Surprisingly, lots of alternative build systems don't do partial update. They only update every item in the deps graph. If you manage to get any of these right, you'd be praised.

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]

shoo 3y ago

i agree. make is a tool for defining recipes to derive target files from dependency files. archetypal use case would be compiling and linking a bunch of C source and header files into an application binary. make isn't specialized to building C programs, make can also be used for other file-driven computation tasks such as defining data processing pipelines / graphs of processing that consume data files and produce data files. the rough essence of make is arguably: 1. there are targets, and recipes (written as a shell command) to derive the target 2. targets can express dependencies on other targets 3. targets are files in the filesystem. 4. add a bunch of language features on top (pattern rules, automatic variables, built-in functions, etc) to make it easier to write and maintain the recipes Another way of thinking about make is that it is a a way to define and evaluate gigantic pure-functional expressions, where each input and output file is a file, and where intermediate sub-expressions can be cached. for an alternative to make that supports 1+2+3+4, see e.g. https://gittup.org/tup/index.html https://gittup.org/tup/index.html for an alternative to make that supports 1+2+3 but explicitly does not aim to support 4, see e.g. https://ninja-build.org/manual.html https://ninja-build.org/manual.html another alternative to make that supports 1+2+3+4 but de-emphasizes 3, "targets are files", from the ruby community, is https://github.com/ruby/rake https://github.com/ruby/rake these days in projects it is quite common to see make used to do 1 but not 2, 3 nor 4. i.e. a makefile as a way to define a bunch of imperative shell recipes for named targets, where each named target has no relationship to a file, and dependencies between the targets aren't expressed or relevant. as a random example of this, here's the first google result searching for "makefile terraform": https://github.com/paulRbr/terraform-makefile/blob/master/Makefile https://github.com/paulRbr/terraform-makefile/blob/master/Ma... note all the targets are ".PHONY" targets, we're telling make to disable the default behaviour where every target corresponds to a file in the file system -- to turn off "3 targets are files in the filesystem". there's very sparing use of dependencies. so this is an example of using makes support for 1 defining recipes to produce targets and a little splash of 2 dependencies between targets, but not 3 nor 4. it looks like this "jeeves" project is aiming at this latter use case, not attempting to implement support for 2+3, the core of how make is typically used as a build tool

yeti-sh 3y ago
That's right. For projects which only use subset of Make features described by (1) jeeves can be considered an alternative for Make, and I would believe there are quite a few such projects.