5 ms·

>make it easier to make it work on Windows from the start The problem with working on windows is that there's no shared standard (unless you count whatever irr

>make it easier to make it work on Windows from the start The problem with working on windows is that there's no shared standard (unless you count whatever irrelevant part of the POSIX spec that windows follows a shared standard). On windows there is no bash or posix shell, there is no gcc or a compiler which takes gcc-like options, there's no posix compliant or gnu compliant make. It's all just wrong. If you plan on making a build system, it has to take a lot of things and provide an abstraction layer over them to make them work for windows. I say forget it, it's a waste of effort, just make it work on linux and by extension (through WSL or mingw or whatever) you get windows. The alternative is worse as you have to add support for every special language to your build system. One of the best tests of a build system I've found has been to get a build system to build an executable and run it to produce an output which is then consumed as part of the build process. With pure make, this is purely awful and requires either generating a bunch of rules or lots of boilerplate, and generally it's easy to get wrong. With redo I got this working with relatively little work and got it to work with cross-builds. > cohesive whole build system is a much better platform for implementing hermetic builds In my adventures with redo, the biggest obstacle to hermetic builds that I've found aside from just the environment (env vars) has been compilers. Compilers suck at a: telling you what they're looking for and b: where they're looking for it. The best solution seems to be to strace gcc and look for calls to fstat or something like that to figure out which candidate header files it's looking for. And I hope you agree, that's an awful solution. Surely a better build system would need some form of overhaul to the interfaces provided by compilers? (Or, some re-implementation of some compiler features so you can spoon-feed a compiler the exact flags to ensure it behaves predictably even if the environment changes.) I guess unifying things could solve the environment problem, but I think I can solve it without unifying anything. >However, at least in the TFA redo TFA? >[1] Okay, so you mean dynamically generated .do files. I think in most cases you can avoid them, but I agree that having every depend-er explicitly depend on the .do file before depending on the file it's intended to redo is really error prone and annoying. There's at least one way to retrofit this into normal redo: by having some default.o.redo file or something and wrapping redo-ifchange and checking if any dependencies have a default*.redo file and depending on the corresponding .do file, although kludgy this at least avoids the possibility of errors if you add future depend-ers to the dependency. I agree redo is missing something here though. >Correct. And I should have mentioned that. But again, DJB did that in an indirect way, making the system "finicky" (in my opinion). Why finnicky? In the usual case you DON'T want anything other than just timestamps and hashes, it works for all cases, albeit slowly for some. If you NEED to customize it, you have to pull out the special tools but then you just wrap that in some little utility and mostly forget about it. > Well, I don't think going even more basic is good here. Hermetic builds are just too good to pass up, and that requires some complexity. What do YOU think are the obstacles to hermetic builds? I already know what I've found but I'm always looking for more input. I'm currently working with a few people to try to make redo work for strictly hermetic builds, or alternatively to design a better redo (or alternatively something else) which solves all the build problems I think exist. Unlike you, I don't really care if people like it or like using it, I've resigned myself to the fact that almost nobody cares about simple unix-esque software anymore and that if I want something like this, I'll have to design and write it myself.

ghoward 5y ago
> On windows there is no bash or posix shell Part of the language builtin to my build system provides ways of running commands without a shell. > there is no gcc or a compiler which takes gcc-like options, No, but there is MSVC. You just need to abstract the compiler and compiler flags. CMake did it, so why can't I? > there's no posix compliant or gnu compliant make. I don't need a make, and neither should redo. My build system will not be like CMake in that it will generate build files for something else; it will execute them itself. No make necessary. In fact, I could implement a make easily with my build system; just a parser would be necessary. > If you plan on making a build system, it has to take a lot of things and provide an abstraction layer over them to make them work for windows. I say forget it, it's a waste of effort, just make it work on linux and by extension (through WSL or mingw or whatever) you get windows. I don't think it's a waste of effort. People on Windows deserve a good build system too. In fact, not having a good one means that they are dependent on Microsoft. I'd like to help them reduce that dependence. > One of the best tests of a build system I've found has been to get a build system to build an executable and run it to produce an output which is then consumed as part of the build process. With pure make, this is purely awful and requires either generating a bunch of rules or lots of boilerplate, and generally it's easy to get wrong. With redo I got this working with relatively little work and got it to work with cross-builds. It's not hard to get make to do that. See [1] and [2]. > In my adventures with redo, the biggest obstacle to hermetic builds that I've found aside from just the environment (env vars) has been compilers. Yes, but however it's done, it's still easiest in a self-contained system, IMO. > Surely a better build system would need some form of overhaul to the interfaces provided by compilers? (Or, some re-implementation of some compiler features so you can spoon-feed a compiler the exact flags to ensure it behaves predictably even if the environment changes.) The build system should carefully control the environment. In fact, it should make sure the environment is empty. (There are details, but in general...) That's the biggest thing to make sure the compiler does not act weird. You should also do the build in a separate directory and copy needed files into the directory. Then check all commands and their arguments for possible files before running them. This is one reason why depending on an outside shell is not ideal. Hermetic builds won't ever be perfect, but you can make it 80% of the way with only a few things. > TFA? "The Famous Article". It refers to the link under discussion on HN. In this case, it refers to the apenwarr redo. > Okay, so you mean dynamically generated .do files. Not just that. Those were just an example of how dynamic dependencies are already awkward in redo. There are countless other uses for dynamic dependencies. To give you a taste, my build system will actually be able to build a distributed system, i.e., it will be able to be used for DevOps deployment, and it will be easy. Surely redo can as well, but as I've thought about how to do that, it seems awkward. Perhaps you can prove me wrong? > Why finnicky? In the usual case you DON'T want anything other than just timestamps and hashes, it works for all cases, albeit slowly for some. If you NEED to customize it, you have to pull out the special tools but then you just wrap that in some little utility and mostly forget about it. Finicky because as far as I can tell, the build database in redo is modified by multiple commands over a build. This can cause problems should a problem occur with the database mid-build. In a self-contained system, the database is in-memory during the entire build and only written out to disk once the build is done, which can then be used as a sort of "commit" of the build, marking it as finished. There are other reasons, but that's the first that comes to mind. > What do YOU think are the obstacles to hermetic builds? I already know what I've found but I'm always looking for more input. I think a good paper to read would be the Dolstra thesis on Nix. [3] If you consider Nix a build system (not just a package manager), it goes further than even Bazel. The thesis should be a good place to start. [1]: https://git.yzena.com/gavin/bc/src/branch/master/Makefile.in#L234-L259 https://git.yzena.com/gavin/bc/src/branch/master/Makefile.in... [2]: https://git.yzena.com/gavin/bc/src/branch/master/configure.sh#L1415 https://git.yzena.com/gavin/bc/src/branch/master/configure.s... [3]: https://edolstra.github.io/pubs/phd-thesis.pdf https://edolstra.github.io/pubs/phd-thesis.pdf

ghoward 5y ago

> On windows there is no bash or posix shell Part of the language builtin to my build system provides ways of running commands without a shell. > there is no gcc or a compiler which takes gcc-like options, No, but there is MSVC. You just need to abstract the compiler and compiler flags. CMake did it, so why can't I? > there's no posix compliant or gnu compliant make. I don't need a make, and neither should redo. My build system will not be like CMake in that it will generate build files for something else; it will execute them itself. No make necessary. In fact, I could implement a make easily with my build system; just a parser would be necessary. > If you plan on making a build system, it has to take a lot of things and provide an abstraction layer over them to make them work for windows. I say forget it, it's a waste of effort, just make it work on linux and by extension (through WSL or mingw or whatever) you get windows. I don't think it's a waste of effort. People on Windows deserve a good build system too. In fact, not having a good one means that they are dependent on Microsoft. I'd like to help them reduce that dependence. > One of the best tests of a build system I've found has been to get a build system to build an executable and run it to produce an output which is then consumed as part of the build process. With pure make, this is purely awful and requires either generating a bunch of rules or lots of boilerplate, and generally it's easy to get wrong. With redo I got this working with relatively little work and got it to work with cross-builds. It's not hard to get make to do that. See [1] and [2]. > In my adventures with redo, the biggest obstacle to hermetic builds that I've found aside from just the environment (env vars) has been compilers. Yes, but however it's done, it's still easiest in a self-contained system, IMO. > Surely a better build system would need some form of overhaul to the interfaces provided by compilers? (Or, some re-implementation of some compiler features so you can spoon-feed a compiler the exact flags to ensure it behaves predictably even if the environment changes.) The build system should carefully control the environment. In fact, it should make sure the environment is empty. (There are details, but in general...) That's the biggest thing to make sure the compiler does not act weird. You should also do the build in a separate directory and copy needed files into the directory. Then check all commands and their arguments for possible files before running them. This is one reason why depending on an outside shell is not ideal. Hermetic builds won't ever be perfect, but you can make it 80% of the way with only a few things. > TFA? "The Famous Article". It refers to the link under discussion on HN. In this case, it refers to the apenwarr redo. > Okay, so you mean dynamically generated .do files. Not just that. Those were just an example of how dynamic dependencies are already awkward in redo. There are countless other uses for dynamic dependencies. To give you a taste, my build system will actually be able to build a distributed system, i.e., it will be able to be used for DevOps deployment, and it will be easy. Surely redo can as well, but as I've thought about how to do that, it seems awkward. Perhaps you can prove me wrong? > Why finnicky? In the usual case you DON'T want anything other than just timestamps and hashes, it works for all cases, albeit slowly for some. If you NEED to customize it, you have to pull out the special tools but then you just wrap that in some little utility and mostly forget about it. Finicky because as far as I can tell, the build database in redo is modified by multiple commands over a build. This can cause problems should a problem occur with the database mid-build. In a self-contained system, the database is in-memory during the entire build and only written out to disk once the build is done, which can then be used as a sort of "commit" of the build, marking it as finished. There are other reasons, but that's the first that comes to mind. > What do YOU think are the obstacles to hermetic builds? I already know what I've found but I'm always looking for more input. I think a good paper to read would be the Dolstra thesis on Nix. [3] If you consider Nix a build system (not just a package manager), it goes further than even Bazel. The thesis should be a good place to start. [1]: https://git.yzena.com/gavin/bc/src/branch/master/Makefile.in#L234-L259 https://git.yzena.com/gavin/bc/src/branch/master/Makefile.in... [2]: https://git.yzena.com/gavin/bc/src/branch/master/configure.sh#L1415 https://git.yzena.com/gavin/bc/src/branch/master/configure.s... [3]: https://edolstra.github.io/pubs/phd-thesis.pdf https://edolstra.github.io/pubs/phd-thesis.pdf