3 ms·

Happy to hear more about what we can do around jobs! Disclosure: I work at Google on Kubernetes.

bryanlarsen 10y ago
Not the OP, but I'll chime in with what we're missing when we took a look at it. AFAICT there is no way to tell k8s to retry a job X number of times; you have to futz around with deadlines.
- gtaylor 10y ago
  Job ordering/dependencies! Job1a must finish successfully before Job1b and Job1c. Alternatively, all Jobs matching this query must succeed before this job. Failures can be OK if a configurable threshold of success is met. Exceeding the thresh fails the dependent jobs before they get to run. You can do all of this with supervisor pods, but since you asked...
kozikow 10y ago
- When I used them last time, there was no python api. New https://github.com/kubernetes-incubator/client-python https://github.com/kubernetes-incubator/client-python is a big improvement. Http API required sidecar proxy container. - Retry control is course grained. IIUC the only option are "no retry at all" or "terminate if job have been active for X seconds". If my jobs tend to be long and failure happens early, second option can lead to hundreds of retries. Each job retry is expensive, as it creates a new pod. I remember ending up with 100s pods.
kozikow 10y ago
I just rewrote some part of my system to kubernetes job. Gist is here: https://gist.github.com/kozikow/86ee26998c5de1cf3780b6bc0cd35ec0 https://gist.github.com/kozikow/86ee26998c5de1cf3780b6bc0cd3... . - As mentioned previously, retry count limit to job/pod would be useful. - Minor nit: Deleting old job with the pod is cumbersome with current python API. Either delete_namespaced_job with orphan_dependencies: False, but I would need to check if job exists beforehand (deleting if job does not exist fails). On the other hand delete_collection_namespaced_job lets me delete job by label selector, but I also need to manually clean up the pod. - Another nit: When exporting stackdriver logs collected by GKE from a job to big query, each log ends up as a separate logsource.

Not the OP, but I'll chime in with what we're missing when we took a look at it. AFAICT there is no way to tell k8s to retry a job X number of times; you have to futz around with deadlines.

gtaylor 10y ago
Job ordering/dependencies! Job1a must finish successfully before Job1b and Job1c. Alternatively, all Jobs matching this query must succeed before this job. Failures can be OK if a configurable threshold of success is met. Exceeding the thresh fails the dependent jobs before they get to run. You can do all of this with supervisor pods, but since you asked...

kozikow 10y ago

- When I used them last time, there was no python api. New https://github.com/kubernetes-incubator/client-python https://github.com/kubernetes-incubator/client-python is a big improvement. Http API required sidecar proxy container. - Retry control is course grained. IIUC the only option are "no retry at all" or "terminate if job have been active for X seconds". If my jobs tend to be long and failure happens early, second option can lead to hundreds of retries. Each job retry is expensive, as it creates a new pod. I remember ending up with 100s pods.

kozikow 10y ago

I just rewrote some part of my system to kubernetes job. Gist is here: https://gist.github.com/kozikow/86ee26998c5de1cf3780b6bc0cd35ec0 https://gist.github.com/kozikow/86ee26998c5de1cf3780b6bc0cd3... . - As mentioned previously, retry count limit to job/pod would be useful. - Minor nit: Deleting old job with the pod is cumbersome with current python API. Either delete_namespaced_job with orphan_dependencies: False, but I would need to check if job exists beforehand (deleting if job does not exist fails). On the other hand delete_collection_namespaced_job lets me delete job by label selector, but I also need to manually clean up the pod. - Another nit: When exporting stackdriver logs collected by GKE from a job to big query, each log ends up as a separate logsource.