I used this technique once to very useful effect in an automated dependency-checking tool. Essentially given an idempotent data-transformation process, you can

I used this technique once to very useful effect in an automated dependency-checking tool. Essentially given an idempotent data-transformation process, you can inject a dll into it and hook its filesystem calls, maintaining a list of all files it touches. Upon process termination, serialize out the filesystem state (size, timestamp, MD5) of each of those files, saving the data in a dependency file whose filename is the hash of the command line. When you run the same command line again, the dll can first look up the dependency file from the previous run, and if all files on the filesystem are in the same state as previously, you can short-circuit the process execution entirely. This is orders of magnitude faster in many cases for highly-parallel data build jobs where only some small percentage of the source data changes each time you run the build job, and has the advantage that you don't need manually maintain a list of dependencies for each process type (no new dependency can be added without changing one of the existing dependencies).