3 ms·
It also contradicts itself. "Checksumming every output after building it is somewhat slow." "Mostly this is not too serious: the file is probably already in d
by tyingq 6y ago
It also contradicts itself.
"Checksumming every output after building it is somewhat slow."
"Mostly this is not too serious: the file is probably already in disk cache (since you just wrote it a moment ago!)"
- NateEag 6y agoYou quoted the wrong bit, if you're looking for his argument against hashing. The strong argument is in the section labeled: > Checksumming every input file before building is very slow
- tyingq 6y agoI'd be interested in comparing a fast CRC checksum versus their approach of stat() plus pulling a sequence number. I'm sure it's slower, but I'm curious how much slower for tens of thousands of files.
- jeffbee 6y agoFar and away the fastest CRC on my local box is Adler's checksum that uses the Intel hardware CRC32 instruction (introduced a long time ago, almost universally supported at this date). It only needs 40ms CPU time per GB, about 50x faster than md5sum. If you have a very new CPU you might have hardware SHA support.
- pmahoney 6y agoIt's not a contradiction to state something is not a problem most of the time. (Though the next point, calculating checksums prior to build, is much more significant.) Related example: on a cloud instance with SSD drive, I have scripts that generate a ~10GiB file, then immediately after (while it might still be in cache), calculating an md5sum still takes tens of seconds (maybe it wasn't in cache? I've not investigated deeply). It's just an example of a case that falls outside of that "Mostly" category.
- jeffbee 6y agoOn my local system md5sum takes 1.9 user CPU-seconds per GB. $ time dd if=/dev/zero bs=1048576 count=1024 | md5sum 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.104 s, 510 MB/s cd573cfaace07e7949bc0c46028904ff - real 0m2.108s user 0m1.891s sys 0m0.628s
- tyingq 6y agoCan you try cksum instead?
- jeffbee 6y ago3.3s here. Is it faster on your system?
- tyingq 6y agoNo, was hoping a non-cryptographic checksum would be faster. Perhaps cksum isn't the one. Edit: "sum -s" is faster for me, but the man page doesn't give much info on what the algorithm is.
- jeffbee 6y agoWell it returns 00000 for /dev/zero so it looks like it is a quite useless 16-bit check.
- JdeBP 6y agoIf you want to try measuring something more relevant, my redo implementation comes with a cubehash tool that uses the same hash as redo-ifchange et al. do. * http://jdebp.uk./Softwares/redo/ http://jdebp.uk./Softwares/redo/