So to protect against a single malicious server who might discover that we had previously loaded a cached resource, we shouldn't implement a cross-origin cache

So to protect against a single malicious server who might discover that we had previously loaded a cached resource, we shouldn't implement a cross-origin cache and have to make repeated requests, guaranteeing 3rd parties (the CDN) keep getting GET requests? You're just trading one problem (someone learning I previously requested a file) for another (leaking referrers to a CDN). Also, if you're loading "highly sensitive/top secret" data with a <link integrity="" href=""> or <script integrity="" src=""> tag, you have bigger problems.