4 ms·

It's annoying, we all ran into it, but usually if you get weird results you just make clean and make again and move on. Deployments should always build from scr

It's annoying, we all ran into it, but usually if you get weird results you just make clean and make again and move on. Deployments should always build from scratch. All those fancy checks plus database could be added to make too if you really wanted. It still wouldn't support all the fancy features of redo, but improve your experience with all those existing projects based on make.

klodolph 6y ago
You don't have to build from scratch if you have high confidence that an incremental build is correct. You get this from modern build systems like Bazel.
- Joker_vD 6y ago
  Well, getting "weird results" is one particularly strong hint that the incremental build is not, in fact, correct. Weird linker errors, or better yet, weird resulting binaries. "I've just changed this routine and built the app, but when I run it the behaviour is the same?.. <attaches gdb> WTF, it has the old version of the routine! kill, q, make clean && make && gdb run, ah, now it works. Apparently, computers are haunted by the angry ghosts".
- iforgotpassword 6y ago
  I've done this for long enough to not have that confidence. It's not just whether the build system is correct, but also all the external things you do with it, random files laying around that aren't referenced by the build system but make it into the release bundle and mess up things at runtime, moon phase, whatnot. Even if Bazel is the ultimate, final, perfect build system, I don't trust everyone, including myself, to use it properly.
  - klodolph 6y ago
    I’ve used Bazel for long enough that I’ve regained that confidence. I don’t think it’s some “ultimate, final, perfect build system”, but it is a build system that doesn't have insidious failure modes when used improperly. Execution of build rules is sandboxed by default so you generally don’t run into the problem of implicit dependencies screwing up your builds—not that you can’t, but you generally don’t.
    - iforgotpassword 6y ago
      If I were running a one man show I might actually try, but debugging build problems when you're not the (only) author of the Makefile (equivalent) just isn't fun.
      - klodolph 6y ago
        I feel the opposite way. The larger the team is, the more likely I am to want to switch to Bazel (or something in its class).
    - jeffbee 6y ago
      You have to go far out of your way to make bazel not have implicit system deps, don't you? The default toolchain is to just call whatever cc is in the path, or whatever is in CC environment variable. Working on the toolchain rules / crosstool is very little practiced art and severely under-documented.
      - joshuamorton 6y ago
        Yes, see the "host mode".
      - klodolph 6y ago
        It is underdocumented, but you don’t have to go out of your way because you in most cases you can use prepackaged toolchains that other people have made. I have successfully written my own toolchain rules in Bazel. Yes, it is a pain. But it’s usually only necessary if you have some kind of unusual target—usually, someone else has written the toolchain rules for you. I've even witnessed a Bazel build system that made working DOS binaries. The entire set of rules was smaller than you might think.
- kabdib 6y ago
  It looked to me like putting toolchain and bespoke tool dependencies into Bazel rules was about as manual as any other build system (e.g., next to impossible in a cross-platform manner, and error-prone at best). The correct response for a compiler upgrade or a modification to a bespoke tool that generates a product is almost certainly a full rebuild unless you have very high confidence that you've identified ALL the dependencies. I believe this is not easily discoverable (e.g., you might have a dependency on "/usr/bin/cc", but not know enough about 'cc's internals to realize that you also have to have a dependency on "/usr/local/lib/cc1-arm-bfhsh-37.so" for some specific compiler option). I'd like to believe you, but I have scars from doing build systems and discovering all the wacky and wonderful things that tools do.
  - klodolph 6y ago
    Bazel sandboxes the build rules. If you need to discover dependencies, you can flip a switch and have all the missing dependencies be hard errors. Once you flip the switch and fix the errors, you leave it flipped on and you have confidence that you are not missing any dependencies. The entire C toolchain is normally an input to a Bazel build. You make a package (.tar.gz) containing the toolchain, put it somewhere accessible (http), and check a SHA-256 hash of the toolchain into your WORKSPACE file. When it changes, everything built with it is rebuilt. Getting complicated cross-platform toolchains working on Bazel is going to be hard, sure. But knowing whether you have hidden dependencies is not hard, because of the sandboxing.
- paulddraper 6y ago
  Bazel has plenty of its own correctness problems. Like https://github.com/bazelbuild/bazel/issues/3360 https://github.com/bazelbuild/bazel/issues/3360 . It used to be the case that file hashing wasn't coordinated well with a concurrently edited file, so changing source while building corrupts your cache.

klodolph 6y ago

You don't have to build from scratch if you have high confidence that an incremental build is correct. You get this from modern build systems like Bazel.

Joker_vD 6y ago
Well, getting "weird results" is one particularly strong hint that the incremental build is not, in fact, correct. Weird linker errors, or better yet, weird resulting binaries. "I've just changed this routine and built the app, but when I run it the behaviour is the same?.. <attaches gdb> WTF, it has the old version of the routine! kill, q, make clean && make && gdb run, ah, now it works. Apparently, computers are haunted by the angry ghosts".
iforgotpassword 6y ago
I've done this for long enough to not have that confidence. It's not just whether the build system is correct, but also all the external things you do with it, random files laying around that aren't referenced by the build system but make it into the release bundle and mess up things at runtime, moon phase, whatnot. Even if Bazel is the ultimate, final, perfect build system, I don't trust everyone, including myself, to use it properly.
- klodolph 6y ago
  I’ve used Bazel for long enough that I’ve regained that confidence. I don’t think it’s some “ultimate, final, perfect build system”, but it is a build system that doesn't have insidious failure modes when used improperly. Execution of build rules is sandboxed by default so you generally don’t run into the problem of implicit dependencies screwing up your builds—not that you can’t, but you generally don’t.
  - iforgotpassword 6y ago
    If I were running a one man show I might actually try, but debugging build problems when you're not the (only) author of the Makefile (equivalent) just isn't fun.
    - klodolph 6y ago
      I feel the opposite way. The larger the team is, the more likely I am to want to switch to Bazel (or something in its class).
  - jeffbee 6y ago
    You have to go far out of your way to make bazel not have implicit system deps, don't you? The default toolchain is to just call whatever cc is in the path, or whatever is in CC environment variable. Working on the toolchain rules / crosstool is very little practiced art and severely under-documented.
    - joshuamorton 6y ago
      Yes, see the "host mode".
    - klodolph 6y ago
      It is underdocumented, but you don’t have to go out of your way because you in most cases you can use prepackaged toolchains that other people have made. I have successfully written my own toolchain rules in Bazel. Yes, it is a pain. But it’s usually only necessary if you have some kind of unusual target—usually, someone else has written the toolchain rules for you. I've even witnessed a Bazel build system that made working DOS binaries. The entire set of rules was smaller than you might think.
kabdib 6y ago
It looked to me like putting toolchain and bespoke tool dependencies into Bazel rules was about as manual as any other build system (e.g., next to impossible in a cross-platform manner, and error-prone at best). The correct response for a compiler upgrade or a modification to a bespoke tool that generates a product is almost certainly a full rebuild unless you have very high confidence that you've identified ALL the dependencies. I believe this is not easily discoverable (e.g., you might have a dependency on "/usr/bin/cc", but not know enough about 'cc's internals to realize that you also have to have a dependency on "/usr/local/lib/cc1-arm-bfhsh-37.so" for some specific compiler option). I'd like to believe you, but I have scars from doing build systems and discovering all the wacky and wonderful things that tools do.
- klodolph 6y ago
  Bazel sandboxes the build rules. If you need to discover dependencies, you can flip a switch and have all the missing dependencies be hard errors. Once you flip the switch and fix the errors, you leave it flipped on and you have confidence that you are not missing any dependencies. The entire C toolchain is normally an input to a Bazel build. You make a package (.tar.gz) containing the toolchain, put it somewhere accessible (http), and check a SHA-256 hash of the toolchain into your WORKSPACE file. When it changes, everything built with it is rebuilt. Getting complicated cross-platform toolchains working on Bazel is going to be hard, sure. But knowing whether you have hidden dependencies is not hard, because of the sandboxing.
paulddraper 6y ago
Bazel has plenty of its own correctness problems. Like https://github.com/bazelbuild/bazel/issues/3360 https://github.com/bazelbuild/bazel/issues/3360 . It used to be the case that file hashing wasn't coordinated well with a concurrently edited file, so changing source while building corrupts your cache.