3 ms·

> My understanding is that bazel is moving away from this, so that you can define toolchains by saying "here is a binary that serves the job of linking/compilin

> My understanding is that bazel is moving away from this, so that you can define toolchains by saying "here is a binary that serves the job of linking/compiling stuff". How do they ensure determinism in that case? Is it just an easy escape hatch so that new languages can be easily supported, with no actual guarantees of hermeticity? > The challenge with your idea is that you're basically saying "hey, we should sandbox and introspect <any number of fairly arbitrary and complex binaries> to intercept and modify their filesystem and network (at a minimum) accesses, across any number of versions and uses". I think my approach would certainly use a syscall whitelist. Any unsupported syscall would be a build failure, and presumably a bug report if it's a legitimate use. I suspect most build commands can get by with a pretty minimal set of syscalls (mainly basic filesystem access). At some point though, if you start supporting more and more syscalls, you start re-implementing VMs/containers, which sucks. This build system only stays simple if people don't try to do a bunch of wacky things with it :) Network accesses would probably get whitelisted by the user on a rule-by-rule basis for cases like "download these packages", with the outputs treated as always dirty. The tool would be responsible for running efficiently even if no work actually needs to be done. One weird/hard thing to support would be soft/hard linking. I'm not sure exactly what should be done there, but that might not be needed for early versions. > Even just handling conditionally rewriting file writes/reads based on guessing whether something is an input or re-used output isn't that easy in general. I'm not sure I understand this. One thing I should note is that in my scheme, you still have to specify output files for rules--you only get to skip specifying the inputs.

joshuamorton 7y ago
> Is it just an easy escape hatch so that new languages can be easily supported, with no actual guarantees of hermeticity? Yes, I mean in general some level of compiler hermeticity is assumed. You can verify it by checksumming everything (which bazel does), but you can easily destroy the performance/caching of bazel by modifying clang to have intentionally unpredictable results. > One weird/hard thing to support would be soft/hard linking. I'm not sure exactly what should be done there, but that might not be needed for early versions. I think bazels solution here is to just always make fat binaries. Or at least that's how it works if you're using blaze. > I'm not sure I understand this. One thing I should note is that in my scheme, you still have to specify output files for rules--you only get to skip specifying the inputs. Ah this helps somewhat, I think there are still potential ambiguities, one thing that bazel can do is statically determine all of the input sources without actually executing the build. You of course can't do this. This has a few nice properties: you get missing input errors before doing any actual building, you have a build graph you can statically analyze (deps queries are magic), and all the sandboxing can be done ahead of time (with symlinks to create a shadow filesystem) instead of ad-hoc, you can parallelize everything, meaning you can saturate all your machines the entire time. With strace, you have performance issues from the tracing itself, and you can't precompile all the dependencies beforehand you have to discover them as you go (this likely also hurts on memory since you'd need to keep all the partial compiler information in memory). I also find it easier to reason about stuff with explicit deps, but that's just me.
- zwegner 7y ago
  > I think bazels solution here is to just always make fat binaries. Or at least that's how it works if you're using blaze. Oh, I should've specified soft/hard filesystem links--they'd make handling a virtual filesystem rather complicated. Dynamic linking is another strange case, but it looks like dylib loading happens in userspace, through an open()/mmap() (at least in Linux), that would be caught through the normal filesystem hooks. > Ah this helps somewhat, I think there are still potential ambiguities, one thing that bazel can do is statically determine all of the input sources without actually executing the build. You of course can't do this. This has a few nice properties: you get missing input errors before doing any actual building, you have a build graph you can statically analyze (deps queries are magic), and all the sandboxing can be done ahead of time (with symlinks to create a shadow filesystem) instead of ad-hoc, you can parallelize everything, meaning you can saturate all your machines the entire time. OK, I started out thinking that there would be very little parallelism impact from my scheme, but I realize now the problem that dependencies for a build step are only discovered serially, as each one is built and the dependent process continues. I could imagine you could work around this with a speculation engine that uses the dependencies from the last run (possibly cancelling the speculated commands mid-flight if the dependent process doesn't end up opening their outputs, and if the speculated commands hasn't started writing files or anything). This would generally work fine, but would waste some work whenever you remove dependencies from a build step (e.g. delete an #include). But it does start to get messy! Another note: as far as not specifying inputs, I mean inputs beyond the command line (which obviously needs to be specified up front). So you would only need speculation like the above for things like auto-generated headers. The usual cases like "build this executable out of these objects which are each built from these source files" can be parallelized just fine with no speculation. Static analysis-type queries could be done after a build, but not in a clean tree. Not sure how much of a setback this would be--I'm not generally working in massive projects, and haven't yet felt the need for such queries. strace performance is a good point (I don't know what sort of overhead it has), and so is memory overhead. > I also find it easier to reason about stuff with explicit deps, but that's just me. I could go either way on this one--I generally prefer explicit to implicit, but I don't like repeating the explicit instructions twice (build rules and source code), with the failure mode being unreliable incremental builds. If I'm going to the trouble of explicitly listing dependencies, though, I'll probably go with the ridiculously simple make.py instead of messing with Bazel...

joshuamorton 7y ago

> Is it just an easy escape hatch so that new languages can be easily supported, with no actual guarantees of hermeticity? Yes, I mean in general some level of compiler hermeticity is assumed. You can verify it by checksumming everything (which bazel does), but you can easily destroy the performance/caching of bazel by modifying clang to have intentionally unpredictable results. > One weird/hard thing to support would be soft/hard linking. I'm not sure exactly what should be done there, but that might not be needed for early versions. I think bazels solution here is to just always make fat binaries. Or at least that's how it works if you're using blaze. > I'm not sure I understand this. One thing I should note is that in my scheme, you still have to specify output files for rules--you only get to skip specifying the inputs. Ah this helps somewhat, I think there are still potential ambiguities, one thing that bazel can do is statically determine all of the input sources without actually executing the build. You of course can't do this. This has a few nice properties: you get missing input errors before doing any actual building, you have a build graph you can statically analyze (deps queries are magic), and all the sandboxing can be done ahead of time (with symlinks to create a shadow filesystem) instead of ad-hoc, you can parallelize everything, meaning you can saturate all your machines the entire time. With strace, you have performance issues from the tracing itself, and you can't precompile all the dependencies beforehand you have to discover them as you go (this likely also hurts on memory since you'd need to keep all the partial compiler information in memory). I also find it easier to reason about stuff with explicit deps, but that's just me.

zwegner 7y ago
> I think bazels solution here is to just always make fat binaries. Or at least that's how it works if you're using blaze. Oh, I should've specified soft/hard filesystem links--they'd make handling a virtual filesystem rather complicated. Dynamic linking is another strange case, but it looks like dylib loading happens in userspace, through an open()/mmap() (at least in Linux), that would be caught through the normal filesystem hooks. > Ah this helps somewhat, I think there are still potential ambiguities, one thing that bazel can do is statically determine all of the input sources without actually executing the build. You of course can't do this. This has a few nice properties: you get missing input errors before doing any actual building, you have a build graph you can statically analyze (deps queries are magic), and all the sandboxing can be done ahead of time (with symlinks to create a shadow filesystem) instead of ad-hoc, you can parallelize everything, meaning you can saturate all your machines the entire time. OK, I started out thinking that there would be very little parallelism impact from my scheme, but I realize now the problem that dependencies for a build step are only discovered serially, as each one is built and the dependent process continues. I could imagine you could work around this with a speculation engine that uses the dependencies from the last run (possibly cancelling the speculated commands mid-flight if the dependent process doesn't end up opening their outputs, and if the speculated commands hasn't started writing files or anything). This would generally work fine, but would waste some work whenever you remove dependencies from a build step (e.g. delete an #include). But it does start to get messy! Another note: as far as not specifying inputs, I mean inputs beyond the command line (which obviously needs to be specified up front). So you would only need speculation like the above for things like auto-generated headers. The usual cases like "build this executable out of these objects which are each built from these source files" can be parallelized just fine with no speculation. Static analysis-type queries could be done after a build, but not in a clean tree. Not sure how much of a setback this would be--I'm not generally working in massive projects, and haven't yet felt the need for such queries. strace performance is a good point (I don't know what sort of overhead it has), and so is memory overhead. > I also find it easier to reason about stuff with explicit deps, but that's just me. I could go either way on this one--I generally prefer explicit to implicit, but I don't like repeating the explicit instructions twice (build rules and source code), with the failure mode being unreliable incremental builds. If I'm going to the trouble of explicitly listing dependencies, though, I'll probably go with the ridiculously simple make.py instead of messing with Bazel...