What you're proposing doesn't sound that different from public key pinning (HPKP), where the web server tells the browser to distrust anyone any other certifica

What you're proposing doesn't sound that different from public key pinning (HPKP), where the web server tells the browser to distrust anyone any other certificates than the pinned one (or certificates from any other CAs). HPKP is deprecated now though.