3 ms·
Nice info. I've wondered for quite a while about the impact of the G level of compile parallelization :). A couple of followup questions, if I may: - How long
by exikyut 7y ago
Nice info. I've wondered for quite a while about the impact of the G level of compile parallelization :). A couple of followup questions, if I may:
- How long do incremental and non-incremental builds take with the build configuration you describe?
- How does core<->engineer allocation work? Is 500 cores an average number? High-priority work presumably commands more access, but I'm curious if allocation is generally team-specific or engineer-specific (eg, more responsibilities = more resources). (I guess the reason I ask this is that 500 cores sounds kind of impressive and I'm going "surely they'll run out of cores if they're handing them out like candy???"), but then thinking about it that's only like 11 computers...)
- I'm curious if you happen to know how Chromium gets built (IIUC Android builds it 2 (or is it 3?) times). It's tricky to distill (presumably/understandably deliberately so) how many cores the CI/trybot/etc infra is using. Overall my curiosity is what the core/time ratios/graphs are, ie how many cores translates to how short of a build.
- As an aside, I vaguely recall absorbing by osmosis several years ago that Chromium takes approximately 15 minutes to build on a Very Cool™ local workstation (HP Z series). Not sure how out of date this info is. Do engineers still do test builds on local hardware (and if yes, how long does this take? probably longer than 500 cores, heh); or is everything done in throwaway cloud instances nowadays?
Thanks in advance for whatever answers you're happy to provide :)
- jart 7y agoGoogle has unlimited resources. Cores and codebase size don't matter to them. Only concern is what's on the critical path, e.g. final link step. It's very rare for builds to not be incremental. AFAIK JavaScript is the only language that can't be compiled incrementally. Probably the only engineers who would need to run a build solely on local hardware are ones that help support open source tools.
- joshuamorton 7y agoJs can and does get compiled incrementally if people do it correctly. It's fully supported by blaze. It's just that a bunch of old codebases globbed everything, so it appeared no incremental.
- jart 7y agoIs that with the js_binary/library rules or something new? Last time I checked, all the work that goes into compiling the JS sources and minifying them into a single "binary" blob happens in the js_binary definition, and the js_library definitions traditionally served only to construct a total ordering of raw source source file dependencies.
- joshuamorton 7y agoTwo partial answers: there's a newer better thing thats now the default and splits the building, but even under old js_library, there were some gains from not globbing everything together...