5 ms·

Repobuild – Declarative-style build system, like Google Build

75 points by albertzeyer 13y ago

saljam 13y ago
Judging by his description of BUILD, it sounds like was a big influence on Go's build system. Well, no surprises there!
- tibbe 13y ago
  More likely Go's build system is influenced by Google's build system, which came first.
  - saljam 13y ago
    That's what I said. :-) BUILD had an influence on Go. Go is influenced by BUILD. One sentence is in the active voice, the other in the passive.
nercury 13y ago
This looks great! May be a reason for me to ditch CMake, but I kind of need Windows support :/. Is it much work to add? I wish someone made a similar way to share libraries like we can do in PHP (https://packagist.org https://packagist.org) or Ruby (http://bundler.io/ http://bundler.io/), but for C++.
ChuckMcM 13y ago
Interesting, if Chris can pull this off it will change the world in a huge way. Having used the build system inside of Google it was somewhat mesmerizing, you really could put together things pretty easily. I suspect there is something similar inside Amazon. But I completely agree that Github has done wonders for making many smaller projects accessible it has not significantly increased our ability to tie them together. That seems to be part of the motivation for the CCAN effort as well, although on a more language specific vector. But as a proof of concept the CPAN system (the one for perl) demonstrates just how powerful this function can be in enabling large scale rapid development. The flip side though, the monkey in the wrench as they say, is that the projects that participate have to conform to the rules of participation. In CPAN's case its a standardized dependency, build, and install models with required unit tests. So perhaps the first step here is to provide an incentive for clean build processes on a project.
- nercury 13y ago
  To expand on that, I was thinking about some standartization of sources (BUILD file in this case), standartize the way build tools could be plugged in, and also standartize the build outputs, so that they can be correctly included AND linked even without the sources. Then you can build a lot of things on top of that, not only source sharing and automatic dependency resolver, but also remote build servers, sharing already built libraries to save time, and easier integration of various editors, IDEs and other tools.
  - alook 13y ago
    This is really exciting - as a Java/Python guy who recently made forays into C++, I was disappointed at the state of dependency management and the lack of standardization across build tools. So many projects use submodules instead of cleanly declared dependencies. I hope that a project like this can motivate the C++ community to provide some more centralized support for creating shared libraries, managing third-party dependencies, and other areas where the language falls short today.
- bbq 13y ago
  Providing incentives seems key. I wonder why CPAN came to work so well. Possible incentives: - automated testing (TravisCI does this in isolation already) - automated distribution (build .deb, .msi, and .pkg files) - usage statistics? I'll have to think about this.
  - chris_va 13y ago
    If you come to any conclusions, let me know :).
    - bbq 13y ago
      Happily =)
- coolsunglasses 13y ago
  >Having used the build system inside of Google it was somewhat mesmerizing, you really could put together things pretty easily. I found using Leiningen for the first time to be a similar experience. I like it so much that I use it on non-Clojure projects too.
skybrian 13y ago
I'd be curious about how this compares to the Buck project, which is another Google-like build system: http://facebook.github.io/buck/ http://facebook.github.io/buck/
- chris_va 13y ago
  It's very similar. Here are a few other similar ones: https://github.com/chrisvana/repobuild/wiki/Similar-Build-Systems https://github.com/chrisvana/repobuild/wiki/Similar-Build-Sy... A few differences: * Buck requires installation (meaning clients need to have it installed), whereas Repobuild generates a Makefile. * Buck has a spiffy build daemon to make things faster. * Repobuild auto-install files from git. * Buck covers java better, but not other languages. * Buck went with a turning complete language, Repobuild intentionally went with only a config langauge (hotly debated religious difference)
fiatjaf 13y ago
There's a building collection of small pieces of Javascript code automatically fetchable from the browser in http://www.corslit.com/ http://www.corslit.com/. Does it sound like it? I think it should. Explanation here: http://williamcotton.com/another-way-to-publish-code http://williamcotton.com/another-way-to-publish-code
bagosm 13y ago
There is something I don't understand. You can add other project code to your own project, but how do you use that other code? In the .NET world where I live you can reference an Interface and let runtime fill that interface in (with a proper dependency injection method). For example you know you want an Interface of MailSender that has the method .SendEmail() and it gets filled with SMTPMailer which has SMTPMailer.SendEmail() Is this thing similar to that concept but in build-time? How do you know the interface/usage beforehand?
- mrich 13y ago
  Most open source libraries are written in C/C++. The interface is usually described in header files, you need them to compile your program. The implementation itself is contained in a library, which you link into your program. There is also the possibility to resolve the implementation at runtime via shared libraries.
- chris_va 13y ago
  "reference the Interface" requires that the interface files exist, and that they (plus the implementation) gets built into your project. That is what Repobuild covers. The actual mechanics of the .net runtime would stay the same.
srean 13y ago
You might find the blog post http://skalrant.blogspot.com/2013/05/build-systems.html http://skalrant.blogspot.com/2013/05/build-systems.html interesting. It argues that a declarative style is not suitable for many software development projects. Coming from the author who is also the author of an ML like language it makes for a good read. Declarative is surely elegant, but not always easy to work with under specific and not uncommon situations. Note, however, it touches on an orthogonal aspect of build systems. I think Repobuild's main thrust is in standardizing and assisting modularization whereas the post linked in this comment has a different focus, it argues for a non-declarative, fixed-point based style.
- thejash 13y ago
  Those are the same link. Can you please update one of them? I'd like to read both that you mentioned :)
stephen 13y ago
Well, 1) I wish him all the best and would love another good build tool alternative, but 2) I don't see his utopia of demonstrably making open-source building easier happening because the OS world is a bazaar, not a Google-esque cathedral. Google's build system is awesome because it can force everyone in the company to use it, going so far as to put every single project in one huge repository. And, then, crucially, the model assumes that any dev workstation can compile (or pull down via cache) every single upstream project. In the non-catherdal/bazaar world, take any non-trivial Java project, and all of it's upstream dependencies, and some are built with Ant, some with Maven, some with Buildr, some with Gradle, some require thrift installed, some require who knows what installed. And you can't just go back and add some meta-data of "oh sure, every Java project compiles src/main/java -> target"...their build files are going to have all sorts of crazy one-off steps (Maven plugins, whatever). There is no enforcement of "you must make your build use this BUILD file, and, btw, only program in Java, Python, or C." With all of Maven's warts, the jar-based repository with transitive dependencies is, IMO, it's major innovation. I just don't see a transitive upstream build-from-source model (like Google's) ever working in the bazaar that is Maven central. I don't at all mean to say Maven/Maven repository are perfect--for one, it's ridiculous how many broken dependencies are in Maven central. Naively, I had thought that was automatically enforced/not allowed, but, sigh, no, we all have to deal with, say, the ASM developers completely breaking their API from v3 and v4 and forever cursing the rest of us with jar hell.
- alook 13y ago
  Agreed. Transitively-resolved, platform-independent binaries are the big win coming from maven. Only once have I worked with a project where I required new system-level dependencies to compile & run a maven project; and I have never had a cross-platform issue when carrying projects across Mac & Linux. To be fair, the import-compiled-transitive-dependencies-as-binaries pattern that Maven likes to employ is probably less easily adapted to natively compiled languages like C/C++ than it is to managed runtime languages (Java/Erlang) or scripting languages (Ruby/Python). As much as it's a pain to work with "maven plugin spaghetti", it's at least deterministic and keeps things easy to figure out. There are those dependencies that you learn to remain wary of (asm/cglib/antlr/jline/etc), but at the end of the day I'm happy to work in an ecosystem where I don't have to compile every single line of third-party code myself.
stass 13y ago
I'd suggest to look at the freebsd ports build system. It is essentially declarative being built on top of Make language, and allows easy integration of various open source components and dependencies. If targeting multiple platforms is of an essence, NetBSD pkgsrc provides a great model also built on top of Make.
chris_va 13y ago
Author here (a bit late to the party). I'm happy to answer questions if anyone has any.
- nercury 13y ago
  How hard it would be to add Windows support? EDIT: Just a question, not a request.
  - chris_va 13y ago
    (I have not investigated it fully, so this may be inaccurate) Cmake: Repobuild basically generates a Makefile. You could change it to generate a CMakefile instead. That would not be a lot of work, probably a few (full) days to change and test it. Visual Studio: Generating a visual studio project file may be trickier. A lot of the work overlaps that of the Cmake stuff above (changing the output format). Realistically, a 2 week project (I would do xcode projects at the same time).
    - nercury 13y ago
      Ok, good info, I might try it. What I know for sure, CMake is not very adequate for separating work into different small, independant parts (somewhat "declarative" attept in CMake here: https://github.com/key-tools/key-machine-logs/blob/master/key-machine-logs/CMakeLists.txt https://github.com/key-tools/key-machine-logs/blob/master/ke...), and then including build targets from other CMake parts into one. Also, generating CMakeLists, then generating a project seems like an overkill :)

75 points by albertzeyer 13y ago

saljam 13y ago

Judging by his description of BUILD, it sounds like was a big influence on Go's build system. Well, no surprises there!

tibbe 13y ago
More likely Go's build system is influenced by Google's build system, which came first.
- saljam 13y ago
  That's what I said. :-) BUILD had an influence on Go. Go is influenced by BUILD. One sentence is in the active voice, the other in the passive.

nercury 13y ago

This looks great! May be a reason for me to ditch CMake, but I kind of need Windows support :/. Is it much work to add? I wish someone made a similar way to share libraries like we can do in PHP (https://packagist.org https://packagist.org) or Ruby (http://bundler.io/ http://bundler.io/), but for C++.

ChuckMcM 13y ago

Interesting, if Chris can pull this off it will change the world in a huge way. Having used the build system inside of Google it was somewhat mesmerizing, you really could put together things pretty easily. I suspect there is something similar inside Amazon. But I completely agree that Github has done wonders for making many smaller projects accessible it has not significantly increased our ability to tie them together. That seems to be part of the motivation for the CCAN effort as well, although on a more language specific vector. But as a proof of concept the CPAN system (the one for perl) demonstrates just how powerful this function can be in enabling large scale rapid development. The flip side though, the monkey in the wrench as they say, is that the projects that participate have to conform to the rules of participation. In CPAN's case its a standardized dependency, build, and install models with required unit tests. So perhaps the first step here is to provide an incentive for clean build processes on a project.

nercury 13y ago
To expand on that, I was thinking about some standartization of sources (BUILD file in this case), standartize the way build tools could be plugged in, and also standartize the build outputs, so that they can be correctly included AND linked even without the sources. Then you can build a lot of things on top of that, not only source sharing and automatic dependency resolver, but also remote build servers, sharing already built libraries to save time, and easier integration of various editors, IDEs and other tools.
- alook 13y ago
  This is really exciting - as a Java/Python guy who recently made forays into C++, I was disappointed at the state of dependency management and the lack of standardization across build tools. So many projects use submodules instead of cleanly declared dependencies. I hope that a project like this can motivate the C++ community to provide some more centralized support for creating shared libraries, managing third-party dependencies, and other areas where the language falls short today.
bbq 13y ago
Providing incentives seems key. I wonder why CPAN came to work so well. Possible incentives: - automated testing (TravisCI does this in isolation already) - automated distribution (build .deb, .msi, and .pkg files) - usage statistics? I'll have to think about this.
- chris_va 13y ago
  If you come to any conclusions, let me know :).
  - bbq 13y ago
    Happily =)
coolsunglasses 13y ago
>Having used the build system inside of Google it was somewhat mesmerizing, you really could put together things pretty easily. I found using Leiningen for the first time to be a similar experience. I like it so much that I use it on non-Clojure projects too.

skybrian 13y ago

I'd be curious about how this compares to the Buck project, which is another Google-like build system: http://facebook.github.io/buck/ http://facebook.github.io/buck/

chris_va 13y ago
It's very similar. Here are a few other similar ones: https://github.com/chrisvana/repobuild/wiki/Similar-Build-Systems https://github.com/chrisvana/repobuild/wiki/Similar-Build-Sy... A few differences: * Buck requires installation (meaning clients need to have it installed), whereas Repobuild generates a Makefile. * Buck has a spiffy build daemon to make things faster. * Repobuild auto-install files from git. * Buck covers java better, but not other languages. * Buck went with a turning complete language, Repobuild intentionally went with only a config langauge (hotly debated religious difference)

fiatjaf 13y ago

There's a building collection of small pieces of Javascript code automatically fetchable from the browser in http://www.corslit.com/ http://www.corslit.com/. Does it sound like it? I think it should. Explanation here: http://williamcotton.com/another-way-to-publish-code http://williamcotton.com/another-way-to-publish-code

bagosm 13y ago

There is something I don't understand. You can add other project code to your own project, but how do you use that other code? In the .NET world where I live you can reference an Interface and let runtime fill that interface in (with a proper dependency injection method). For example you know you want an Interface of MailSender that has the method .SendEmail() and it gets filled with SMTPMailer which has SMTPMailer.SendEmail() Is this thing similar to that concept but in build-time? How do you know the interface/usage beforehand?

mrich 13y ago
Most open source libraries are written in C/C++. The interface is usually described in header files, you need them to compile your program. The implementation itself is contained in a library, which you link into your program. There is also the possibility to resolve the implementation at runtime via shared libraries.
chris_va 13y ago
"reference the Interface" requires that the interface files exist, and that they (plus the implementation) gets built into your project. That is what Repobuild covers. The actual mechanics of the .net runtime would stay the same.

srean 13y ago

You might find the blog post http://skalrant.blogspot.com/2013/05/build-systems.html http://skalrant.blogspot.com/2013/05/build-systems.html interesting. It argues that a declarative style is not suitable for many software development projects. Coming from the author who is also the author of an ML like language it makes for a good read. Declarative is surely elegant, but not always easy to work with under specific and not uncommon situations. Note, however, it touches on an orthogonal aspect of build systems. I think Repobuild's main thrust is in standardizing and assisting modularization whereas the post linked in this comment has a different focus, it argues for a non-declarative, fixed-point based style.

thejash 13y ago
Those are the same link. Can you please update one of them? I'd like to read both that you mentioned :)

stephen 13y ago

Well, 1) I wish him all the best and would love another good build tool alternative, but 2) I don't see his utopia of demonstrably making open-source building easier happening because the OS world is a bazaar, not a Google-esque cathedral. Google's build system is awesome because it can force everyone in the company to use it, going so far as to put every single project in one huge repository. And, then, crucially, the model assumes that any dev workstation can compile (or pull down via cache) every single upstream project. In the non-catherdal/bazaar world, take any non-trivial Java project, and all of it's upstream dependencies, and some are built with Ant, some with Maven, some with Buildr, some with Gradle, some require thrift installed, some require who knows what installed. And you can't just go back and add some meta-data of "oh sure, every Java project compiles src/main/java -> target"...their build files are going to have all sorts of crazy one-off steps (Maven plugins, whatever). There is no enforcement of "you must make your build use this BUILD file, and, btw, only program in Java, Python, or C." With all of Maven's warts, the jar-based repository with transitive dependencies is, IMO, it's major innovation. I just don't see a transitive upstream build-from-source model (like Google's) ever working in the bazaar that is Maven central. I don't at all mean to say Maven/Maven repository are perfect--for one, it's ridiculous how many broken dependencies are in Maven central. Naively, I had thought that was automatically enforced/not allowed, but, sigh, no, we all have to deal with, say, the ASM developers completely breaking their API from v3 and v4 and forever cursing the rest of us with jar hell.

alook 13y ago
Agreed. Transitively-resolved, platform-independent binaries are the big win coming from maven. Only once have I worked with a project where I required new system-level dependencies to compile & run a maven project; and I have never had a cross-platform issue when carrying projects across Mac & Linux. To be fair, the import-compiled-transitive-dependencies-as-binaries pattern that Maven likes to employ is probably less easily adapted to natively compiled languages like C/C++ than it is to managed runtime languages (Java/Erlang) or scripting languages (Ruby/Python). As much as it's a pain to work with "maven plugin spaghetti", it's at least deterministic and keeps things easy to figure out. There are those dependencies that you learn to remain wary of (asm/cglib/antlr/jline/etc), but at the end of the day I'm happy to work in an ecosystem where I don't have to compile every single line of third-party code myself.

stass 13y ago

I'd suggest to look at the freebsd ports build system. It is essentially declarative being built on top of Make language, and allows easy integration of various open source components and dependencies. If targeting multiple platforms is of an essence, NetBSD pkgsrc provides a great model also built on top of Make.

chris_va 13y ago

Author here (a bit late to the party). I'm happy to answer questions if anyone has any.

nercury 13y ago
How hard it would be to add Windows support? EDIT: Just a question, not a request.
- chris_va 13y ago
  (I have not investigated it fully, so this may be inaccurate) Cmake: Repobuild basically generates a Makefile. You could change it to generate a CMakefile instead. That would not be a lot of work, probably a few (full) days to change and test it. Visual Studio: Generating a visual studio project file may be trickier. A lot of the work overlaps that of the Cmake stuff above (changing the output format). Realistically, a 2 week project (I would do xcode projects at the same time).
  - nercury 13y ago
    Ok, good info, I might try it. What I know for sure, CMake is not very adequate for separating work into different small, independant parts (somewhat "declarative" attept in CMake here: https://github.com/key-tools/key-machine-logs/blob/master/key-machine-logs/CMakeLists.txt https://github.com/key-tools/key-machine-logs/blob/master/ke...), and then including build targets from other CMake parts into one. Also, generating CMakeLists, then generating a project seems like an overkill :)