4 ms·

I agree with you on all grounds (I personally use make daily) however note that `make` checks dependencies via last updated time flag on files. I think it would

I agree with you on all grounds (I personally use make daily) however note that `make` checks dependencies via last updated time flag on files. I think it would have been a better default to cache based on hash of file. This way if document was modified, by hash wasn't changed, make shouldn't rebuild tasks that depend on this file.

enriquto 5y ago
But computing the hashes of all the files would take the same time as rebuilding everything, or maybe even slower. And you'd need to re-compute all these hashes even for a no-op. It's absurd. The nice thing about dates is that they can be checked instantaneously, and independently of file size. If a file has changed, its date has, so it's all the information you need.
- duped 5y ago
  Computing the SHA sum of a file is plenty fast on modern hardware, nowhere close to the same time as a rebuild.
  - slaymaker1907 5y ago
    Yep, I can take the sha256sum over my all my project's (vendored) dependencies in 10ms using the command: time find ./vendor -type f -print0 | sort -z | xargs -0 sha256sum | sort -k2 | sha256sum Note if using this code for practical purposes, you should be sure your collation is set appropriately. That's taken with 1280 files that are in total 19MB. For raw throughput, you can use BLAKE2 for even more speed https://www.blake2.net/ https://www.blake2.net/
    - enriquto 5y ago
      What % of that time is required for just computing the timestamps of these files? I bet it's a few orders of magnitude faster.
      - anamexis 5y ago
        Sure, but who cares?
    - Too 5y ago
      While 1280 files isn’t exactly small, it’s not really big either. Big as in where incremental builds really show their shine. This might be approaching niche but I’ve seen a few mega-projects with 100 times more than that (also mostly due to vendored deps). You really don’t want to read 10G from disk for every incremental build. The actual sha computation isn’t the bottleneck, disk is. Ninja on the other hand, using mtime, handles this without a sweat. The best solution would probably be to integrate the change detection with git, it is already content addressable and has lots of other smart things built in to make it fast.
- gnulinux 5y ago
  Rebuilding potentially has side-effects. You can have a build system that, if a file is changed, it makes HTTP calls, generate docker containers, store artifacts in S3 etc. Given this, computing the hash would be negligable amount of work. Besides make does iterate over all dependencies every time to check last modified date filesystem keeps. So, at that point one can compute hash.
  - IgorPartola 5y ago
    You shouldn’t have such a build system. Idempotence is a feature, not a bug.
    - afiori 5y ago
      idempotent does not mean side effects free; a compile-to-the-cloud system for distributed caching can be idempotent (ocaml is building support in this direction)
  - mmis1000 5y ago
    Side effects that target on non local dir seems shouldn't be included in the build step. It should be in a standalone script/ target/whatever. Or the build process will not be reproducible at all.
    - afiori 5y ago
      builds should not use non-local inputs, but they can have non-local outputs. the simplest case would be automatically publishing a package or image to a private registry
- anamexis 5y ago
  > If a file has changed, its date has, so it's all the information you need. Unless the date hasn't changed -- there is no hard guarantee that mtime will be accurate.
  - bityard 5y ago
    If the file's contents have changed but the timestamp didn't, then something is broken and _that_ is the thing to fix, not rewriting the fundamental mechanism underneath the oldest relevant build tool.
zvr 5y ago
There are always different preferences on deciding when something needs to be run again. File modification time is a reasonable default that serves many (most?) cases pretty well. Yes, file hashes might be another way. On the other hand, if I modify a comment, the file hash will change, but re-compilation is not necessary. In this case, you might need yet another way... You can always write your Makefiles to compute and depend on hashes or whatever else you might consider appropriate for your exact use case.

enriquto 5y ago

But computing the hashes of all the files would take the same time as rebuilding everything, or maybe even slower. And you'd need to re-compute all these hashes even for a no-op. It's absurd. The nice thing about dates is that they can be checked instantaneously, and independently of file size. If a file has changed, its date has, so it's all the information you need.

duped 5y ago
Computing the SHA sum of a file is plenty fast on modern hardware, nowhere close to the same time as a rebuild.
- slaymaker1907 5y ago
  Yep, I can take the sha256sum over my all my project's (vendored) dependencies in 10ms using the command: time find ./vendor -type f -print0 | sort -z | xargs -0 sha256sum | sort -k2 | sha256sum Note if using this code for practical purposes, you should be sure your collation is set appropriately. That's taken with 1280 files that are in total 19MB. For raw throughput, you can use BLAKE2 for even more speed https://www.blake2.net/ https://www.blake2.net/
  - enriquto 5y ago
    What % of that time is required for just computing the timestamps of these files? I bet it's a few orders of magnitude faster.
    - anamexis 5y ago
      Sure, but who cares?
  - Too 5y ago
    While 1280 files isn’t exactly small, it’s not really big either. Big as in where incremental builds really show their shine. This might be approaching niche but I’ve seen a few mega-projects with 100 times more than that (also mostly due to vendored deps). You really don’t want to read 10G from disk for every incremental build. The actual sha computation isn’t the bottleneck, disk is. Ninja on the other hand, using mtime, handles this without a sweat. The best solution would probably be to integrate the change detection with git, it is already content addressable and has lots of other smart things built in to make it fast.
gnulinux 5y ago
Rebuilding potentially has side-effects. You can have a build system that, if a file is changed, it makes HTTP calls, generate docker containers, store artifacts in S3 etc. Given this, computing the hash would be negligable amount of work. Besides make does iterate over all dependencies every time to check last modified date filesystem keeps. So, at that point one can compute hash.
- IgorPartola 5y ago
  You shouldn’t have such a build system. Idempotence is a feature, not a bug.
  - afiori 5y ago
    idempotent does not mean side effects free; a compile-to-the-cloud system for distributed caching can be idempotent (ocaml is building support in this direction)
- mmis1000 5y ago
  Side effects that target on non local dir seems shouldn't be included in the build step. It should be in a standalone script/ target/whatever. Or the build process will not be reproducible at all.
  - afiori 5y ago
    builds should not use non-local inputs, but they can have non-local outputs. the simplest case would be automatically publishing a package or image to a private registry
anamexis 5y ago
> If a file has changed, its date has, so it's all the information you need. Unless the date hasn't changed -- there is no hard guarantee that mtime will be accurate.
- bityard 5y ago
  If the file's contents have changed but the timestamp didn't, then something is broken and _that_ is the thing to fix, not rewriting the fundamental mechanism underneath the oldest relevant build tool.

zvr 5y ago

There are always different preferences on deciding when something needs to be run again. File modification time is a reasonable default that serves many (most?) cases pretty well. Yes, file hashes might be another way. On the other hand, if I modify a comment, the file hash will change, but re-compilation is not necessary. In this case, you might need yet another way... You can always write your Makefiles to compute and depend on hashes or whatever else you might consider appropriate for your exact use case.