Fun fact, on GCP Kubernetes you can have green lightbulbs on every single dashboard, and your entire site can be down anyway. Our CI/CD was leaking "review" de

Fun fact, on GCP Kubernetes you can have green lightbulbs on every single dashboard, and your entire site can be down anyway. Our CI/CD was leaking "review" deployments, I forgot about them until one day I upgraded a node and the entire site went down, even though everything was green. Turned out there is some sort of naximum amount of nginx entries in ingress and we were hitting it. That was some frantic debugging, solution was just to delete the spurious review deployments.