Presumably you'll know if your phone has gone missing and can then deactivate it as an authentication token long before someone could combine it with guessing y

Presumably you'll know if your phone has gone missing and can then deactivate it as an authentication token long before someone could combine it with guessing your PIN. There has been a secure token business operating for years without a major problem due to token loss. Migration to using your phone instead of a separate token is a natural transition and doesn't change the fundamentals much.