I mostly agree with you, but I think the strongest argument in favor of using them anyway is that you might add state/actions/third-party resources to your webs

I mostly agree with you, but I think the strongest argument in favor of using them anyway is that you might add state/actions/third-party resources to your website at some point, and adding those headers now is harmless and a way to make those changes safe.