Fun day for people running bare metal GPU nodes, where teams have been training models for months, and now it must be abruptly aborted to apply security patches

Fun day for people running bare metal GPU nodes, where teams have been training models for months, and now it must be abruptly aborted to apply security patches... is that something that can be resumed, or do they have to restart from scratch?