4 ms·

Something struck with me here that I've been thinking about. OP says a human should never wait for a pod. Agreed, it is annoying and sometimes means waiting for

Something struck with me here that I've been thinking about. OP says a human should never wait for a pod. Agreed, it is annoying and sometimes means waiting for an EC2 and the pod. We have jobs that users initiate that use 80+GB of memory and a few dozen cores. We run only one pod per node because the next size up EC2 costs a fortune and performance tops out on our current size. These jobs are triggered via a button click that trigger a lambda that submits a job to the cluster. If it is a fresh node, user has to wait for the 1gb container to download from ECR. But it is the same container that the automated jobs that kick off every few minutes also uses, to rarely is there any waiting. But sometimes there is. Should we be running some sort of clustering job scheduler that gets the job request and distributes work amongst long running pods in the cluster? My fear is that we just creat another layer of complexity and still end up waiting for the EC2, waiting for the pod to download, waiting for the agent now running on this pod to join the work distribution cluster. However, we probably could be more proactive with this because we could spin up an extra pod+EC2 when the work cluster is 1:1 job:ec2. Thoughts? We're in the process of moving to Karpenter, so all this may be solved for us very soon with some clever configuration.

p_l 3y ago
If you don't want to change the setup too much, consider running your nodes off an AMI with pre-loaded image. Maybe also ensure how exactly the images are layered, so if necessary you can reduce amount of "first boot patch" download.
Too 3y ago
There is a difference between waiting and waiting. For an hourly batch job that already takes 10 minutes to run, the extra time for pod scheduling and container downloading is negligible anyway. What you shouldn’t do is put pod scheduling in places where thousands of users per minute expect sub-second latency. In your case, if the time for starting up the EC2 becomes a bigger factor than the job itself, you can add placeholder pods that just sleep, while requiring exactly that machine config but request 0 cpus, just to make sure it stays online.
hackermeows 3y ago
Have you looked into using AWS Batch for these jobs, it can handle this no problem

p_l 3y ago

If you don't want to change the setup too much, consider running your nodes off an AMI with pre-loaded image. Maybe also ensure how exactly the images are layered, so if necessary you can reduce amount of "first boot patch" download.

Too 3y ago

There is a difference between waiting and waiting. For an hourly batch job that already takes 10 minutes to run, the extra time for pod scheduling and container downloading is negligible anyway. What you shouldn’t do is put pod scheduling in places where thousands of users per minute expect sub-second latency. In your case, if the time for starting up the EC2 becomes a bigger factor than the job itself, you can add placeholder pods that just sleep, while requiring exactly that machine config but request 0 cpus, just to make sure it stays online.

hackermeows 3y ago

Have you looked into using AWS Batch for these jobs, it can handle this no problem