5 ms·

Heard from a Googler that the internal infrastructure (Borg) is simply not optimized for quick startup. Launching a new Borg job often takes multiple minutes be

by kccqzy 4y ago

Heard from a Googler that the internal infrastructure (Borg) is simply not optimized for quick startup. Launching a new Borg job often takes multiple minutes before the job runs. Not surprising at all.

epberry 4y ago
Echoing this. The SRE book is also highly revealing about how Google request prioritization works. https://sre.google/sre-book/load-balancing-datacenter/ https://sre.google/sre-book/load-balancing-datacenter/ My personal opinion is that Google's resources are more tightly optimized than AWS and they may try to find the 99% best allocation versus the 95% best allocation on AWS.. and this leads to more rejected requests. Open to being wrong on this.
valleyjo 4y ago
As another comment points out, GPU resources are less common so it takes longer to create, which makes sense. In general, start up times are pretty quick on GCP as other comments also confirm.
dekhn 4y ago
booting VMs != starting a borg job.
- kccqzy 4y ago
  The technology may be different but the culture carries over. People simply don't have the habit to optimize for startup time.
readams 4y ago
Borg is not used for gcp vms, though.
- dilyevsky 4y ago
  It is used but borg scheduler does not manage vm startups
dekhn 4y ago
A well-configured isolated borg cluster and well-configured job can be really fast. If there's no preemption (IE, no other job that is kicked off and gets some grace period), the packages are already cached locally, and there is no undue load on the scheduler, the resources are available, and it's a job with tasks, rather than multiple jobs, it will be close to instantaneous. I spend a significant fraction of my 11+ years there clicking Reload on my job's borg page. I was able to (re-)start ~100K jobs globally in about 15 minutes.
- fragmede 4y ago
  Psh someone's bragging about not being at batch priority.
  - dekhn 4y ago
    I ran at -1
jsolson 4y ago
This is mostly not true in cases where resources are actually available (and in GCE if they're not the API rejects the VM outright, in general). To the extent that it is true for Borg when the job schedules immediately, it's largely due to package (~container layers, ish) loading. This is less relevant today (because reasons), and also mostly doesn't apply to GCE as the relevant packages are almost universally proactively made available on relevant hosts. The origin for the info that jobs take "minutes" likely involves jobs that were pending available resources. This is a valid state in Borg, but GCE has additional admission control mechanisms aimed at avoiding extended residency in pending. As dekhn notes, there are many factors that contribute to VM startup time. GPUs are their own variety of special (and, yes, sometimes slow), with factors that mostly don't apply to more pedestrian VM shapes.

by kccqzy 4y ago

epberry 4y ago

Echoing this. The SRE book is also highly revealing about how Google request prioritization works. https://sre.google/sre-book/load-balancing-datacenter/ https://sre.google/sre-book/load-balancing-datacenter/ My personal opinion is that Google's resources are more tightly optimized than AWS and they may try to find the 99% best allocation versus the 95% best allocation on AWS.. and this leads to more rejected requests. Open to being wrong on this.

valleyjo 4y ago

As another comment points out, GPU resources are less common so it takes longer to create, which makes sense. In general, start up times are pretty quick on GCP as other comments also confirm.

dekhn 4y ago

booting VMs != starting a borg job.

kccqzy 4y ago
The technology may be different but the culture carries over. People simply don't have the habit to optimize for startup time.

readams 4y ago

Borg is not used for gcp vms, though.

dilyevsky 4y ago
It is used but borg scheduler does not manage vm startups

dekhn 4y ago

A well-configured isolated borg cluster and well-configured job can be really fast. If there's no preemption (IE, no other job that is kicked off and gets some grace period), the packages are already cached locally, and there is no undue load on the scheduler, the resources are available, and it's a job with tasks, rather than multiple jobs, it will be close to instantaneous. I spend a significant fraction of my 11+ years there clicking Reload on my job's borg page. I was able to (re-)start ~100K jobs globally in about 15 minutes.

fragmede 4y ago
Psh someone's bragging about not being at batch priority.
- dekhn 4y ago
  I ran at -1

jsolson 4y ago

This is mostly not true in cases where resources are actually available (and in GCE if they're not the API rejects the VM outright, in general). To the extent that it is true for Borg when the job schedules immediately, it's largely due to package (~container layers, ish) loading. This is less relevant today (because reasons), and also mostly doesn't apply to GCE as the relevant packages are almost universally proactively made available on relevant hosts. The origin for the info that jobs take "minutes" likely involves jobs that were pending available resources. This is a valid state in Borg, but GCE has additional admission control mechanisms aimed at avoiding extended residency in pending. As dekhn notes, there are many factors that contribute to VM startup time. GPUs are their own variety of special (and, yes, sometimes slow), with factors that mostly don't apply to more pedestrian VM shapes.