All the names and places have been changed to protect the guilty: When I was a young data engineer in 2014 at a very large telco in Canada the real time resour

All the names and places have been changed to protect the guilty: When I was a young data engineer in 2014 at a very large telco in Canada the real time resource allocation of network bandwidth would work flawlessly from 6am to 11pm, but would have a random panic every day at 11.25 pm, where the cluster would enter a degraded mode until just after 6am the next day. This was very weird, but I trace the issue to a bunch of NFS drives which started being used after 11pm. We figured some job somewhere needed to run then and there was nothing to do about it but migrate the hardware. After a couple of months, a lot of planning and tens of millions we moved the data from those drives to some shiny new beefy servers and the problems went away. Years later I worked with someone else from that telco. Turns out he was in a team that did nothing but monitor the allocation of jobs on the compute cluster in real time and decide which machine each job ended up on. Their work hours were from 6am to 11pm.