Use ldap as your centralised auth, add a schema extension for ssh public keys, configure sshd/pam/nss to use ldap (and the keys contained therein) for users/aut

Use ldap as your centralised auth, add a schema extension for ssh public keys, configure sshd/pam/nss to use ldap (and the keys contained therein) for users/auth. Use groups or a host attribute in ldap to manage access to specific servers or groups of servers. Bonus: you can manage sudo access from ldap too.