5 ms·

Piper: A proposal for a graphy pipe-based build system

70 points by mattsan 3y ago

comex 3y ago
> Incremental compilation is trivial due to each mapping having a single input and output, with the output file name being known before it is generated. If the build system only knows about a single input (the .c file), how do you deal with rebuilding when header files have changed?
- mattsan 3y ago
  Very good point! For my next amendment(s) to the idea, I will be attempting to think of a simple way to do M-N. It should naturally also cover 1-1, 1-N, and N-1.
epr 3y ago
>> Isn’t that just Make? > Yep! But what if we add mappings and collections? These are called pattern rules in make: https://www.gnu.org/software/make/manual/html_node/Pattern-Rules.html https://www.gnu.org/software/make/manual/html_node/Pattern-R... This is the best resource I've found on make to get a low friction build system up and running. Some parts are a bit dense if you're relatively new to make or used to someone else setting it up for you, but if you have the gnu make manual side-by-side for reference you should be good to go: https://make.mad-scientist.net/papers/advanced-auto-dependency-generation/ https://make.mad-scientist.net/papers/advanced-auto-dependen... If you've ever been frustrated with how time consuming it is to manually write make rules then the above method will probably be a night-and-day difference from what you're used to. I'd also strongly suggest either disabling builtin implicit rules with -r or reading up on all of them here: https://www.gnu.org/software/make/manual/html_node/Catalogue-of-Rules.html https://www.gnu.org/software/make/manual/html_node/Catalogue...
- mattsan 3y ago
  Great resources, thank you! I think I came across some of these ideas before and it might have been as simple as me resenting the syntax. I'll touch up on my `make` knowledge and try to make Piper more ergonomic!
  - epr 3y ago
    Please don't do that! Good reasons to borrow ideas from make would be familiarity for those coming from make, but definitely not because make is ergonomic. The entire reason you must put in tons of mental effort to grok this one true method of make usage I linked above is specifically because of how non-ergonomic make is out of the box. My comment was meant to be practical advice considering make's popularity. A while back I wanted to replace a legacy script that generated Makefiles for a project at work, and I started out by making something very similar to Piper in python so it was easy for coworkers, etc, but ended up finding the guide I mentioned in my above comment while doing gcc dependency generation. Since I was targeting make underneath anyways, and my needs were less complex than I thought, I ended up scrapping my redundant python build tool.
    - mattsan 3y ago
      Oops I worded it badly! I meant take some theory from Make and modify it to be more ergonomic before adding it into Piper. A la rethinking it all with new knowledge I gain.
      - mst 3y ago
        If you haven't already, I'd seriously suggest having a look at redo (the theory but also apenwarr's implementation at https://github.com/apenwarr/redo https://github.com/apenwarr/redo). It's not going in the same direction you are exactly but building a mental model of it in my head was salutary in terms of thinking through the purpose and implementation of build tools and hopefully it'll be similarly useful to you.
  - ilc 3y ago
    An idea: If piper is clearer for you, it can "compile" down into make, and then let gmake do the heavy lift, at least for some of the early prototyping.
    - r-w 3y ago
      Better to target Ninja, no? Since it’s basically designed for that.
- arthur2e5 3y ago
  Implicit rules and pattern rules have the little issue with header dependencies that I can never get over with. It always feels like I can only choose simplicity or accuracy in incremental builds. I suppose -MMD is the way for C, though the way it pulls in system header checks always feels a bit too far.
LanternLight83 3y ago
I think this is neat! Love the cute lil' characters, and I believe that simplified build systems are still fertile ground. Filtering and deriving from collections will be essential for larger projects. > $_IN: [...] For collections, this is a space separated list of filenames > [...] > I haven’t written any software for this Bit early for this sort of nit-picking, but that will need to be either a.) an escaped and quoted list or b.) an argv array that doesn't undergo word-splitting. Bonus consideration: What if the user's shell isn't sh? Perhaps it's out of scope (ie. this shouldn't support every use case of Make), but for what it's worth, they support all sorts of recipe interpreters.
- mattsan 3y ago
  Thank you! Re. a), b); I will probably add a "spread operator" syntax or something for collections. Maybe different symbols for the different types. Once I start prototyping I'll choose whatever is the least burden on the user (both in learning, comprehension, and in practice). Re. different shells, I have yet to experiment - simple as! When I write the code I'll spot the challenges. Obviously I'd like it to support everything, but I have no clue as to what the implications would be.
groby_b 3y ago
The hard part of build systems aren't map/reduce operations, but dependency discovery as you go - because suddenly, you're rebuilding your transform pipeline while you're executing it. That's where the fun dragons are. (I really think they're fun. But type 2 fun ;)
- convolvatron 3y ago
  amin vadhat wrote a little paper while in grad school about sniffing the workload while you were building it by hand and pooping out a makefile based on what you had done. seems like if you ran a makefile, recorded what the compiler looked at, you'd be in good shape wrt dependencies without having to parse the course or use other tools
  - emmelaich 3y ago
    There were similar tools to watch commercial installations and reproducing them outside of the proprietary tools. Also Red Hat labs had something which watched an installation including the users commands including sudo for later replay. If I find it's name I'll edit or reply to this.
  - pasc1878 3y ago
    That technology was available in the 1990s. Clearcase ran on a file system that recorded what files the build using clearmake looked at so your makefile just needed to know how to make a .o from a ,c file and all the dependencies were calculated. The build was on a server so you could get the results of other peoples (or automated) builds if they used the same dependencies. We stopped using that as the filesystem had to be on a server and so the builds were slower as involved network access and we did not have large repos where this might have been offset by a large number of build artifacts. (plus we did more on Windows)
    - groby_b 3y ago
      Plus let's not forget you needed roughly one fulltime Clearcase admin for every 10 engineers. It was a good idea with horrible execution :)
oDot 3y ago
I am working on a similar tool started from a filmmaking need to source-control large binary files. The current (semi-implemented) design is that each file or collection of files can be assigned a step in a pipeline and version controlled separately, and steps can be defined as dependencies of one another. This makes it possible to add "compute" steps that do not track any file but instead run a task, achieving a similar goal.
- mattsan 3y ago
  I think this will be the eventual goal with a simple syntax.
eka1 3y ago
There's this concept of "forward build system" that other people too have nibbled at. See: * http://neilmitchell.blogspot.com/2021/09/small-project-build-systems.html http://neilmitchell.blogspot.com/2021/09/small-project-build... * https://dl.acm.org/doi/abs/10.1145/3497775.3503687 https://dl.acm.org/doi/abs/10.1145/3497775.3503687 Primarily trying to reduce the high learning curve of complete build systems like bazel/please/buck. The author too seems to be heading in this direction
- eka1 3y ago
  Also https://taskfile.dev/ https://taskfile.dev/ seems to be in a similar vein. Simplifying the authoring of the build specification.
larsrc 3y ago
Being on the Bazel team, and having previously worked closely with the Piper team, this naming would be very confusing for me! That said: good to put your thoughts out there before going too deep into the implementation rabbit hole. Good luck with your idea. Build systems are fascinating.
- kohlerm 3y ago
  https://www.project-piper.io/ https://www.project-piper.io/ I was confused as well
- mxmlnkn 3y ago
  https://github.com/rhasspy/piper https://github.com/rhasspy/piper I was confused as well
- Am4TIfIsER0ppos 3y ago
  Also a mouse config tool: https://github.com/libratbag/piper https://github.com/libratbag/piper

70 points by mattsan 3y ago

> Incremental compilation is trivial due to each mapping having a single input and output, with the output file name being known before it is generated. If the build system only knows about a single input (the .c file), how do you deal with rebuilding when header files have changed?

mattsan 3y ago
Very good point! For my next amendment(s) to the idea, I will be attempting to think of a simple way to do M-N. It should naturally also cover 1-1, 1-N, and N-1.

epr 3y ago

>> Isn’t that just Make? > Yep! But what if we add mappings and collections? These are called pattern rules in make: https://www.gnu.org/software/make/manual/html_node/Pattern-Rules.html https://www.gnu.org/software/make/manual/html_node/Pattern-R... This is the best resource I've found on make to get a low friction build system up and running. Some parts are a bit dense if you're relatively new to make or used to someone else setting it up for you, but if you have the gnu make manual side-by-side for reference you should be good to go: https://make.mad-scientist.net/papers/advanced-auto-dependency-generation/ https://make.mad-scientist.net/papers/advanced-auto-dependen... If you've ever been frustrated with how time consuming it is to manually write make rules then the above method will probably be a night-and-day difference from what you're used to. I'd also strongly suggest either disabling builtin implicit rules with -r or reading up on all of them here: https://www.gnu.org/software/make/manual/html_node/Catalogue-of-Rules.html https://www.gnu.org/software/make/manual/html_node/Catalogue...

mattsan 3y ago
Great resources, thank you! I think I came across some of these ideas before and it might have been as simple as me resenting the syntax. I'll touch up on my `make` knowledge and try to make Piper more ergonomic!
- epr 3y ago
  Please don't do that! Good reasons to borrow ideas from make would be familiarity for those coming from make, but definitely not because make is ergonomic. The entire reason you must put in tons of mental effort to grok this one true method of make usage I linked above is specifically because of how non-ergonomic make is out of the box. My comment was meant to be practical advice considering make's popularity. A while back I wanted to replace a legacy script that generated Makefiles for a project at work, and I started out by making something very similar to Piper in python so it was easy for coworkers, etc, but ended up finding the guide I mentioned in my above comment while doing gcc dependency generation. Since I was targeting make underneath anyways, and my needs were less complex than I thought, I ended up scrapping my redundant python build tool.
  - mattsan 3y ago
    Oops I worded it badly! I meant take some theory from Make and modify it to be more ergonomic before adding it into Piper. A la rethinking it all with new knowledge I gain.
    - mst 3y ago
      If you haven't already, I'd seriously suggest having a look at redo (the theory but also apenwarr's implementation at https://github.com/apenwarr/redo https://github.com/apenwarr/redo). It's not going in the same direction you are exactly but building a mental model of it in my head was salutary in terms of thinking through the purpose and implementation of build tools and hopefully it'll be similarly useful to you.
- ilc 3y ago
  An idea: If piper is clearer for you, it can "compile" down into make, and then let gmake do the heavy lift, at least for some of the early prototyping.
  - r-w 3y ago
    Better to target Ninja, no? Since it’s basically designed for that.
arthur2e5 3y ago
Implicit rules and pattern rules have the little issue with header dependencies that I can never get over with. It always feels like I can only choose simplicity or accuracy in incremental builds. I suppose -MMD is the way for C, though the way it pulls in system header checks always feels a bit too far.

LanternLight83 3y ago

I think this is neat! Love the cute lil' characters, and I believe that simplified build systems are still fertile ground. Filtering and deriving from collections will be essential for larger projects. > $_IN: [...] For collections, this is a space separated list of filenames > [...] > I haven’t written any software for this Bit early for this sort of nit-picking, but that will need to be either a.) an escaped and quoted list or b.) an argv array that doesn't undergo word-splitting. Bonus consideration: What if the user's shell isn't sh? Perhaps it's out of scope (ie. this shouldn't support every use case of Make), but for what it's worth, they support all sorts of recipe interpreters.

mattsan 3y ago
Thank you! Re. a), b); I will probably add a "spread operator" syntax or something for collections. Maybe different symbols for the different types. Once I start prototyping I'll choose whatever is the least burden on the user (both in learning, comprehension, and in practice). Re. different shells, I have yet to experiment - simple as! When I write the code I'll spot the challenges. Obviously I'd like it to support everything, but I have no clue as to what the implications would be.

groby_b 3y ago

The hard part of build systems aren't map/reduce operations, but dependency discovery as you go - because suddenly, you're rebuilding your transform pipeline while you're executing it. That's where the fun dragons are. (I really think they're fun. But type 2 fun ;)

convolvatron 3y ago
amin vadhat wrote a little paper while in grad school about sniffing the workload while you were building it by hand and pooping out a makefile based on what you had done. seems like if you ran a makefile, recorded what the compiler looked at, you'd be in good shape wrt dependencies without having to parse the course or use other tools
- emmelaich 3y ago
  There were similar tools to watch commercial installations and reproducing them outside of the proprietary tools. Also Red Hat labs had something which watched an installation including the users commands including sudo for later replay. If I find it's name I'll edit or reply to this.
- pasc1878 3y ago
  That technology was available in the 1990s. Clearcase ran on a file system that recorded what files the build using clearmake looked at so your makefile just needed to know how to make a .o from a ,c file and all the dependencies were calculated. The build was on a server so you could get the results of other peoples (or automated) builds if they used the same dependencies. We stopped using that as the filesystem had to be on a server and so the builds were slower as involved network access and we did not have large repos where this might have been offset by a large number of build artifacts. (plus we did more on Windows)
  - groby_b 3y ago
    Plus let's not forget you needed roughly one fulltime Clearcase admin for every 10 engineers. It was a good idea with horrible execution :)

oDot 3y ago

I am working on a similar tool started from a filmmaking need to source-control large binary files. The current (semi-implemented) design is that each file or collection of files can be assigned a step in a pipeline and version controlled separately, and steps can be defined as dependencies of one another. This makes it possible to add "compute" steps that do not track any file but instead run a task, achieving a similar goal.

mattsan 3y ago
I think this will be the eventual goal with a simple syntax.

eka1 3y ago

There's this concept of "forward build system" that other people too have nibbled at. See: * http://neilmitchell.blogspot.com/2021/09/small-project-build-systems.html http://neilmitchell.blogspot.com/2021/09/small-project-build... * https://dl.acm.org/doi/abs/10.1145/3497775.3503687 https://dl.acm.org/doi/abs/10.1145/3497775.3503687 Primarily trying to reduce the high learning curve of complete build systems like bazel/please/buck. The author too seems to be heading in this direction

eka1 3y ago
Also https://taskfile.dev/ https://taskfile.dev/ seems to be in a similar vein. Simplifying the authoring of the build specification.

larsrc 3y ago

Being on the Bazel team, and having previously worked closely with the Piper team, this naming would be very confusing for me! That said: good to put your thoughts out there before going too deep into the implementation rabbit hole. Good luck with your idea. Build systems are fascinating.

kohlerm 3y ago
https://www.project-piper.io/ https://www.project-piper.io/ I was confused as well
mxmlnkn 3y ago
https://github.com/rhasspy/piper https://github.com/rhasspy/piper I was confused as well
Am4TIfIsER0ppos 3y ago
Also a mouse config tool: https://github.com/libratbag/piper https://github.com/libratbag/piper