I mean sure, you can work around any problem if you put enough effort into it. For this particular problem, in reality it introduces friction. e.g., one practic

I mean sure, you can work around any problem if you put enough effort into it. For this particular problem, in reality it introduces friction. e.g., one practical problem you face (if you've put sufficient work into your tooling to get to this point) is that your "dev" builds will now miss the cache far more often, because CI is running with different flags. Again, you can work around that... by (for example) spending more money and energy building with different flags. Nothing is impossible to do with enough investment, but these don't come free.