Since the encryption is typically asymmetric, you do have per-SP encryption keys, since you encrypt using the key specified in the metadata provided by the SP.

Since the encryption is typically asymmetric, you do have per-SP encryption keys, since you encrypt using the key specified in the metadata provided by the SP. On the other hand, the SP verifies the signature using the public key specified in the metadata of the IdP. So, to have per-SP signing key, you need per-SP metadata, which is an additional complication.