4 ms·
Yep, I can take the sha256sum over my all my project's (vendored) dependencies in 10ms using the command: time find ./vendor -type f -print0 | sort -z | xargs
by slaymaker1907 5y ago
Yep, I can take the sha256sum over my all my project's (vendored) dependencies in 10ms using the command:
time find ./vendor -type f -print0 | sort -z | xargs -0 sha256sum | sort -k2
| sha256sum
Note if using this code for practical purposes, you should be sure your collation is set appropriately. That's taken with 1280 files that are in total 19MB.
For raw throughput, you can use BLAKE2 for even more speed https://www.blake2.net/ https://www.blake2.net/
- enriquto 5y agoWhat % of that time is required for just computing the timestamps of these files? I bet it's a few orders of magnitude faster.
- anamexis 5y agoSure, but who cares?
- Too 5y agoWhile 1280 files isn’t exactly small, it’s not really big either. Big as in where incremental builds really show their shine. This might be approaching niche but I’ve seen a few mega-projects with 100 times more than that (also mostly due to vendored deps). You really don’t want to read 10G from disk for every incremental build. The actual sha computation isn’t the bottleneck, disk is. Ninja on the other hand, using mtime, handles this without a sweat. The best solution would probably be to integrate the change detection with git, it is already content addressable and has lots of other smart things built in to make it fast.