4 ms·

Are you familiar with Ninja? (which is absolutely excellent IMHO) http://martine.github.io/ninja/ http://martine.github.io/ninja/ > Ninja is a small build syst

Are you familiar with Ninja? (which is absolutely excellent IMHO) http://martine.github.io/ninja/ http://martine.github.io/ninja/ > Ninja is a small build system with a focus on speed. It differs from other build systems in two major respects: it is designed to have its input files generated by a higher-level build system, and it is designed to run builds as fast as possible. Obligatory: https://xkcd.com/927/ https://xkcd.com/927/

js2 13y ago
Concur, what haberman describes really sounds like ninja.
e12e 13y ago
I recently played a bit with ninja on a tiny c project -- as I was already using cmake, the transition was seamless, and even for such a very tiny project the speedup was tangible (but absolutely not relevant in any way, everything was building fast enough :). Also played a bit with tup, and it's quite nice too.
haberman 13y ago
TL;DR: I think Ninja is a step in the right direction, but I want to take it further. Yes, I came across Ninja when searching for prior art. While it's "philosophy" essays make it sound like it is exactly what I'm looking for, when I dug in deeper I found that it was still a ways away from what I had in mind. When you look at Ninja files, you find some of the things that I explicitly want to get rid of: variables and rules, evaluation and scoping. Their solution for C/C++ dependencies is to have special magic in the tool. Maybe the biggest difference between Ninja and what I am envisioning is that, although pared down and low-level, Ninja is still a command-line tool implemented as a language (.ninja files). What I have in mind is an embeddable library that deals only in flat, fully-expanded lists of tasks. At its core it is much more like an API than a tool (though wrapping it in a convenient command-line tool will of course be an important use case). Think about it this way. There are so many directions you might want to extend your build system: 1. you might want to run your jobs in a different way, for example by distributing them to a cluster of build workers. 2. you might want to run your builder as a daemon that watches the filesystem and receives notifications of changed files, instead of having to figure everything out from scratch every time. 3. You might have custom, per-language logic for calculating dynamic dependencies. 4. You might want to fingerprint your inputs and outputs and cache artifacts so that you can return cached results instead of doing redundant work (especially useful if you have a build cluster that many people share). 5. You might want to do strongly isolated and hermetic builds that run each command against a directory tree that only contains its supposed inputs. 6. you might want to closely integrate the build system with your IDE, so the IDE can show deep and up-to-date information about the progress of the build. 7. I think of Light Table-like environments as being the future of IDEs. Part of the philosophy of Light Table is that you can program your own custom editor interactions easily ("The future is specific": http://www.chris-granger.com/2012/05/21/the-future-is-specific/ http://www.chris-granger.com/2012/05/21/the-future-is-specif...). Programming a specific editing mode would often involve calculating artifacts derived from your source files to show in a different editor pane (for example, the assembly corresponding to your C, or a graphical representation of your source). To do this, you want something like "make", but programmable so you can generate and remove rules on the fly. With a Ninja-like approach, you would have to add these features to Ninja itself. But once you do this, Ninja is going to get even less small and simple. This is why I think the way forward is to have an API-based, embeddable builder engine. The core just knows how to take fully-expanded task specifications and evaluate the dependency graph to decide what jobs need to run. Then the logic for how to run jobs, record their results, possibly return cached results etc. can be easily mixed and matched. For example: 1. I would ship with a very simple command-line tool that acts mostly like "make" or "ninja": you run it, it looks at the filesystem to decide what to do, and then does it. When the build is finished the tool exits. 2. I would likely also ship with a daemon version of the same that watches the filesystem with inotify/dnotify and automatically keeps outputs up to date when inputs change. The daemon would likely support some kind of service API that you could use to programmatically query the list of tasks and jobs, and probably also an HTTP/HTML UI. 3. Anyone who wanted to integrate with a distributed builder service could embed the library but write completely custom code for actually executing the jobs. But here is the key thing: the Tasks that a project writes can stay the same no matter what fancy stuff gets built on top. Whether you are doing the dead-simplest local build from scratch or whether you have a long-lived daemon that distributes your build and heavily caches your artifacts, there is still a common data model for tasks and jobs, and still a common dependency-evaluating core that can be shared no matter where the library is embedded.

js2 13y ago

Concur, what haberman describes really sounds like ninja.

e12e 13y ago

I recently played a bit with ninja on a tiny c project -- as I was already using cmake, the transition was seamless, and even for such a very tiny project the speedup was tangible (but absolutely not relevant in any way, everything was building fast enough :). Also played a bit with tup, and it's quite nice too.

haberman 13y ago

TL;DR: I think Ninja is a step in the right direction, but I want to take it further. Yes, I came across Ninja when searching for prior art. While it's "philosophy" essays make it sound like it is exactly what I'm looking for, when I dug in deeper I found that it was still a ways away from what I had in mind. When you look at Ninja files, you find some of the things that I explicitly want to get rid of: variables and rules, evaluation and scoping. Their solution for C/C++ dependencies is to have special magic in the tool. Maybe the biggest difference between Ninja and what I am envisioning is that, although pared down and low-level, Ninja is still a command-line tool implemented as a language (.ninja files). What I have in mind is an embeddable library that deals only in flat, fully-expanded lists of tasks. At its core it is much more like an API than a tool (though wrapping it in a convenient command-line tool will of course be an important use case). Think about it this way. There are so many directions you might want to extend your build system: 1. you might want to run your jobs in a different way, for example by distributing them to a cluster of build workers. 2. you might want to run your builder as a daemon that watches the filesystem and receives notifications of changed files, instead of having to figure everything out from scratch every time. 3. You might have custom, per-language logic for calculating dynamic dependencies. 4. You might want to fingerprint your inputs and outputs and cache artifacts so that you can return cached results instead of doing redundant work (especially useful if you have a build cluster that many people share). 5. You might want to do strongly isolated and hermetic builds that run each command against a directory tree that only contains its supposed inputs. 6. you might want to closely integrate the build system with your IDE, so the IDE can show deep and up-to-date information about the progress of the build. 7. I think of Light Table-like environments as being the future of IDEs. Part of the philosophy of Light Table is that you can program your own custom editor interactions easily ("The future is specific": http://www.chris-granger.com/2012/05/21/the-future-is-specific/ http://www.chris-granger.com/2012/05/21/the-future-is-specif...). Programming a specific editing mode would often involve calculating artifacts derived from your source files to show in a different editor pane (for example, the assembly corresponding to your C, or a graphical representation of your source). To do this, you want something like "make", but programmable so you can generate and remove rules on the fly. With a Ninja-like approach, you would have to add these features to Ninja itself. But once you do this, Ninja is going to get even less small and simple. This is why I think the way forward is to have an API-based, embeddable builder engine. The core just knows how to take fully-expanded task specifications and evaluate the dependency graph to decide what jobs need to run. Then the logic for how to run jobs, record their results, possibly return cached results etc. can be easily mixed and matched. For example: 1. I would ship with a very simple command-line tool that acts mostly like "make" or "ninja": you run it, it looks at the filesystem to decide what to do, and then does it. When the build is finished the tool exits. 2. I would likely also ship with a daemon version of the same that watches the filesystem with inotify/dnotify and automatically keeps outputs up to date when inputs change. The daemon would likely support some kind of service API that you could use to programmatically query the list of tasks and jobs, and probably also an HTTP/HTML UI. 3. Anyone who wanted to integrate with a distributed builder service could embed the library but write completely custom code for actually executing the jobs. But here is the key thing: the Tasks that a project writes can stay the same no matter what fancy stuff gets built on top. Whether you are doing the dead-simplest local build from scratch or whether you have a long-lived daemon that distributes your build and heavily caches your artifacts, there is still a common data model for tasks and jobs, and still a common dependency-evaluating core that can be shared no matter where the library is embedded.