It is common for an IdP to apply specific policies and controls dependent on which service provider the user is authenticating to. The IdP is specifically sayin

It is common for an IdP to apply specific policies and controls dependent on which service provider the user is authenticating to. The IdP is specifically saying "user X has access to application Y." That is why the Audience is a (required) component of a SAML Response.