They may have had access, but they didn't have the means to change your trusted devices or perform other sensitive operations. I'm often asked to confirm my App

They may have had access, but they didn't have the means to change your trusted devices or perform other sensitive operations. I'm often asked to confirm my Apple ID password when making changes. If I can just make up a new one, this basically means the PIN is all I need now.