2 ms·

please provide evidence for this assertion.

by gjvc 2y ago

IshKebab 2y ago
Why are you so skeptical? Think about how it works and then you'll understand that cache invalidation bugs are completely inevitable. Hell, cache invalidation is notoriously difficult to get right even when you aren't building it on top of a complex tool that was never designed for aggressive caching. Just search the bugs for "hash": https://github.com/ccache/ccache/issues?q=is%3Aissue+hash+is%3Aclosed+label%3Abug https://github.com/ccache/ccache/issues?q=is%3Aissue+hash+is...
imtringued 2y ago
You can't cache based on the file contents alone. You will also need to cache based on all OS/compiler queries/variables/settings that the preprocessor depends on, since the header files might generate completely different content based on what ifdef gets triggered.
- mananaysiempre 2y ago
  And that’s not impossible, just tedious. One tricky (and often unimportant) part is negative dependencies—when the build depends on the fact that a header or library cannot be found in a particular directory on a search path (which happens all the time, if you think about it). As far as I know, no compilers will cooperate with you on this, so build systems that try to get this right have to trace the compiler’s system calls to be sure (Tup does something like this) or completely control and hash absolutely everything that the compiler could possibly see (Nix and IIUC Bazel).
  - zorgmonkey 2y ago
    In C++ the __has_include preprocessor expression has been standardized since C++17, I'm not certain if C has standardized it yet though.
    - mananaysiempre 2y ago
      It’s not about that, that’s not relevant to ccache at all. (And yes, C23 does have __has_include, though not a lot of compilers have C23 yet.) It’s about having potentially conflicting headers in the source file’s directory, in your -I directories, and in your /usr/include directories. Suppose a previous compile correctly resolved <libfoo.h> to /usr/include/libfoo.h, and that file remains unchanged, but since that time you’ve installed a private build of libfoo such that a new compile would instead resolve that to ~/.local/include/libfoo.h. What you want is to record not just that your compile opened /usr/include/libfoo.h (“positive dependencies” you get with -MD et al.), but that it tried $GITHOME/include/libfoo.h, ~/.local/include/libfoo.h, etc. before that and failed (“negative dependencies”), so that if any of those appear later you can force a recompile.
      - zorgmonkey 2y ago
        Oh yeah that can cause lots of weird problems. I've run into that sort of issue a lot when cross-compiling, cause often then you might have a system copy of a library and a different version for the target, that can be a real pain.
- amelius 2y ago
  Maybe run every build version in their own container?
- gjvc 2y ago
  please read the documentation before dispensing uninformed advice like this -- it works using the output of the preprocessor and optionally, file paths
cstrahan 2y ago
Here’s an example of a common Rust dependency which had a very subtle caching issue: https://github.com/dtolnay/thiserror/pull/279 https://github.com/dtolnay/thiserror/pull/279

please provide evidence for this assertion.

by gjvc 2y ago

please provide evidence for this assertion.

Why are you so skeptical? Think about how it works and then you'll understand that cache invalidation bugs are completely inevitable. Hell, cache invalidation is notoriously difficult to get right even when you aren't building it on top of a complex tool that was never designed for aggressive caching. Just search the bugs for "hash": https://github.com/ccache/ccache/issues?q=is%3Aissue+hash+is%3Aclosed+label%3Abug https://github.com/ccache/ccache/issues?q=is%3Aissue+hash+is...

imtringued 2y ago

You can't cache based on the file contents alone. You will also need to cache based on all OS/compiler queries/variables/settings that the preprocessor depends on, since the header files might generate completely different content based on what ifdef gets triggered.

mananaysiempre 2y ago
And that’s not impossible, just tedious. One tricky (and often unimportant) part is negative dependencies—when the build depends on the fact that a header or library cannot be found in a particular directory on a search path (which happens all the time, if you think about it). As far as I know, no compilers will cooperate with you on this, so build systems that try to get this right have to trace the compiler’s system calls to be sure (Tup does something like this) or completely control and hash absolutely everything that the compiler could possibly see (Nix and IIUC Bazel).
- zorgmonkey 2y ago
  In C++ the __has_include preprocessor expression has been standardized since C++17, I'm not certain if C has standardized it yet though.
  - mananaysiempre 2y ago
    It’s not about that, that’s not relevant to ccache at all. (And yes, C23 does have __has_include, though not a lot of compilers have C23 yet.) It’s about having potentially conflicting headers in the source file’s directory, in your -I directories, and in your /usr/include directories. Suppose a previous compile correctly resolved <libfoo.h> to /usr/include/libfoo.h, and that file remains unchanged, but since that time you’ve installed a private build of libfoo such that a new compile would instead resolve that to ~/.local/include/libfoo.h. What you want is to record not just that your compile opened /usr/include/libfoo.h (“positive dependencies” you get with -MD et al.), but that it tried $GITHOME/include/libfoo.h, ~/.local/include/libfoo.h, etc. before that and failed (“negative dependencies”), so that if any of those appear later you can force a recompile.
    - zorgmonkey 2y ago
      Oh yeah that can cause lots of weird problems. I've run into that sort of issue a lot when cross-compiling, cause often then you might have a system copy of a library and a different version for the target, that can be a real pain.
amelius 2y ago
Maybe run every build version in their own container?
gjvc 2y ago
please read the documentation before dispensing uninformed advice like this -- it works using the output of the preprocessor and optionally, file paths

cstrahan 2y ago

Here’s an example of a common Rust dependency which had a very subtle caching issue: https://github.com/dtolnay/thiserror/pull/279 https://github.com/dtolnay/thiserror/pull/279