menage

14 ms·

91.
by menage 11y ago
The GFS cell used back in 2004 for staging Borg binaries to production was /gfs/seven/, for the same reason :-)
92.
by menage 11y ago
We did try to write a paper on Borg way back in ~2008 but it got bogged down by internal disagreements on the style and approach ...
93.
by menage 11y ago
One important point that the author seems to have misunderstood is that Borg was the predecessor to the other two systems, not the successor. Borg went into production (running a bunch of websearch dedicated clusters) in late 2004, long b
94.
by menage 11y ago
There are multiple definitions of the word 'container'. In the case of Borg, 'container' referred to the kernel resource isolation component, which is completely orthogonal to how the apps were packaged. Borg aggressivel
95.
by menage 11y ago
But you can make guarantees to the critical jobs (up to the total size of the machine) and then let batch jobs (with less time-sensitive requirements) run best-effort in the slack. In the event that there ends up being no best-effort resour
96.
by menage 11y ago
It won't - for any single given service, running on a dedicated system will probably be able to squeeze out a tiny bit more performance than running on a shared system with containers, for whatever fraction of the time that you're
97.
by menage 11y ago
The borgmon process would be a completely separate job on separate machines (generally with a lot fewer instances). The logsaver would also be a separate job, although typically running co-located 1:1 with instances of the actual service jo
98.
by menage 11y ago
Most Borg containers were in fact single processes (or more correctly, a shell wrapper around a single process)
99.
by menage 11y ago
Yes, I think it's safe to say that Google operates at scale. A lot of user-facing services at Google have to be over-provisioned in order to handle the cyclical usage patterns (the daily query peak is far higher than the average for mo
100.
by menage 11y ago
Primarily because dedicated servers are a lot less efficient. The more different things you can pack on a machine while still ensuring that the high-priority/low-latency jobs get prompt access to the resources that they've reserve
101.
by menage 12y ago
Being able to ensure that your important job gets a certain fraction of the CPU, memory or I/O resources regardless of what other jobs on the same machine are doing is very much an instance of 'resource isolation' - in terms
102.
by menage 12y ago
Google had been using primitive kernel containers (based on cpusets and fake NUMA emulation) since early 2007 - this was quite some time prior to getting cgroups into the mainline kernel.
103.
by menage 12y ago
If you want unrelated jobs to share library binaries, you need to stick to explicit release schedules for those libraries, which means more coordination between the various app teams and the teams that own the libraries. When the size of th
104.
by menage 12y ago
Yes, we contributed the core cgroups system, and a good chunk of the memory and I/O cgroup subsystems. Google didn't have much need for namespacing though - when you have complete control over all the code that's running, it&
105.
by menage 13y ago
[I used to be a GMail SRE] Correct, it's pretty much impossible for an outage to not be noticed and the GMail on-call being automatically paged. SREs at GMail are engineers, yes, but they're very much directly involved with fixing
106.
by menage 13y ago
Presumably the flywheel axis can be rotated internally.
107.
by menage 13y ago
I left the cluster management group over three years ago, so I've not had much chance to work on / think about containers since then. This code grew symbiotically with Google's kernel patches (big chunks of which were open-so
108.
by menage 13y ago
cgroups (and lmctfy) does support limiting memory usage on a per-application basis, as well as a bunch of other resources (ability to run on particular CPUs, access certain network ports, disk/IO, etc). You can also prevent application
109.
by menage 13y ago
Great to see some of this code finally making it out into the world, although it seems to have changed a bit over the last few years. (I still recognize a few low-level functions.) Who came up with the name?
110.
by menage 13y ago
We started building a large-scale datacenter automation system in 2003, and by late 2005 it was deployed on most machines but it became apparent that achieving the high density of job packing we wanted was going to be impossible by relying
111.
by menage 13y ago
LXC didn't really exist in a usable form when Google started work on kernel-based resource isolation.
112.
by menage 13y ago
(Original cgroups developer here, although I've since moved on from Google and don't have time to play an active role anymore.) It's true that cgroups are a complex system, but they were developed to solve a complex group of
113.
by menage 13y ago
Most apps running on Google servers are aware that they're running in a shared environment, so they don't need the overhead of virtualized network interfaces. So I doubt that there will be any specific support for network namespac
114.
by menage 14y ago
Fundamentally, the cgroups framework is just a way of creating some arbitrary kernel state and associating a set of processes with that state. For most cgroup subsystems, the kernel state is something to do with resource usage, but it can