One thing to note is that depending on how your network is structured and firewalled, if _one_ server has a graphical env of some sort (maybe a ML/image process

One thing to note is that depending on how your network is structured and firewalled, if _one_ server has a graphical env of some sort (maybe a ML/image processing server that has graphical environments installed)... root access could be nasty nonetheless. I never install GUIs on my servers, but when deploying a image processing server, while installing another package to debug CUDA/drivers, I've unintentionally added a full windowing system without realising it.