We've got 9-10 production servers. My methodology is allow what you want and deny everything else. So with iptables specific ports are open for public access an

We've got 9-10 production servers. My methodology is allow what you want and deny everything else. So with iptables specific ports are open for public access and all else is dropped. For complete access to the servers we use openvpn. OpenVPN is setup on one server in the production environment that acts as a gateway to the rest of the servers. Only those with vpn keys can gain access. Each developer is given a specific IP and access is restricted through iptables based on what their requirements are. Knockd a port knocking daemon is used as a way to open a hole in the firewall for 60 seconds to make the connection to openvpn. I have a backdoor incase openvpn or knockd go down. The great thing with openvpn is later on you can disable keys if need be. User accounts are created across the servers for users however ssh keys are only added to the specific servers they may need access to. No passwords, people use insecure passwords, they can be cracked. SSH keys with 2048 bits work nicely.