Minimally, require a password. This password would be separate from your Windows/Linux/OSX account password. You could also use a non-standard port or configur

Minimally, require a password. This password would be separate from your Windows/Linux/OSX account password. You could also use a non-standard port or configure some sort of port knocking system, or just close the VNC port to the outside world and allow connections over VPN or SSH tunnels only.