It might be like shooting with cannon into sparrows, but FreeIPA can define policies, where you can say which service on which host or hostgroup is allowed to a

It might be like shooting with cannon into sparrows, but FreeIPA can define policies, where you can say which service on which host or hostgroup is allowed to authenticate which user (or group). So you can have an user, which is defacto "on the remote server", but the only thing he can do is to login into specified sftp service. For smaller scales, it is surely possible to do something similar, without FreeIPA, only with PAM.