Regardless of who is doing the job, you're still going to need some device on your person (or otherwise readily accessible) that can be used to confirm that you

Regardless of who is doing the job, you're still going to need some device on your person (or otherwise readily accessible) that can be used to confirm that you are actually you, and whatever that device is will become a single point of failure. The real issue is there needs to be some simple standard workflow for when the device (be it smartphone or otherwise) fails. And in this case the government pretty much already is providing that service, or at least the backbone for it. "You lost your phone? Well send us a picture of your driver's license or passport plus a selfie in [this] pose from a different trusted number and, after we try calling your old number just to make sure it's really lost, we'll use the new number." Financial institutions already have the infrastructure for photo-id confirmation for KYC regulations, and selfie verification is widely used for dating apps. Yeah if someone breaks your phone, steals your id, and can deepfake you then they can probably steal your identity, but someone in that position can probably already steal your identity.