The issue was the Docker daemon was affected due to poor resource configuration. It should run at a higher priority and with guaranteed resources so it can't be

The issue was the Docker daemon was affected due to poor resource configuration. It should run at a higher priority and with guaranteed resources so it can't be impacted by user mode tasks. k8s has been slow to catch up in this area but finally has priorities and preemption. That said Docker normally doesn't run in a pod so it's also a matter of setting the node total allocatable CPU to something reasonable and ensuring all pods spawned by kubelet are nested under a namespace that has a lower priority than the system tasks (kubelet itself, Docker if you use it etc).