I still don’t know understand why astronomers had to leave a a specially built building. Can’t you reorganize and move away the two bullies and keep most people

I still don’t know understand why astronomers had to leave a a specially built building. Can’t you reorganize and move away the two bullies and keep most people where they are to minimize disruption?