Not only overdraw but also having to be careful with how you craft the scene. SM pretty much allows for arbitrary scenes. Also small nitpick: you don't filter

Not only overdraw but also having to be careful with how you craft the scene. SM pretty much allows for arbitrary scenes. Also small nitpick: you don't filter the shadow map, but the resulting shadow response.