6 ms·

Chromium continuous build infrastructure engineer here. As he says, this is slightly out of date: we forked WebKit to make Blink in 2013, and actually merged B

Chromium continuous build infrastructure engineer here. As he says, this is slightly out of date: we forked WebKit to make Blink in 2013, and actually merged Blink into Chromium proper just this week! It is no longer managed by gclient/DEPS, but instead developed alongside Chromium in the same repository. But many other things (pdfium, v8, webrtc, and tons of smaller libraries) are still pulled in via DEPS in exactly the way described. I'd just like to say this about our development, testing, and release process: it really really works. We keep moving fast, everyone knows what's going on throughout the codebase because everyone is working on master, and there are never any big ugly merge conflicts to resolve when new features land. There are, of course, problems with the system. We don't have enough resources to run every test for every commit, so they end up batched and we have to rely on humans and heuristics to determine which CL in the batch to revert. The set of configurations on the try-bots doesn't exactly match up with the set of configurations that are run against tip-of-tree. And flaky tests are, of course, a perennial problem for everyone. But we're working on making all of these better, and huge progress has been made in the last couple years. Ask me anything if you have any questions about our build and test infrastructure!

robotnoises 11y ago
Do you think a version of this system could work in a team that has mostly "b-players?" I just wonder how much of this stays afloat because the caliber of the engineers is higher than that of your typical company.
- phasmantistes 11y ago
  I don't think any of our decisions are driven by or dependent upon the caliber of our engineers. For example: a) No merges; and the only branches are release branches. This makes sense in almost any context. Whether your team is big or small, high-caliber or college sophomores, this works. The advantages are numerous. No one works on a feature branch in isolation, only to have tons of merge conflicts and surprises when they try to bring it back in. Bug fixes and features are never landed on a release branch and then not merged into master for the next release. Development doesn't slow to a crawl while you try to stabilize master for a release. b) Keeping dependencies near tip-of-tree. Chromium has two kinds of dependencies: third-party libraries (e.g. yasm, lighttpd) and things we develop on our own but in separate repositories (e.g. pdfium, v8, boringssl). The former tend to have stable releases produced by their own maintainers; the latter is being developed in parallel and you want to be as up-to-date as possible. Most projects are similar: you have third-party dependencies (e.g. python packages) and your own libraries (like a common set of api interfaces). You want to keep the ones you control as close to tip-of-tree as possible so that you can move fast and integrate your own changes quickly. The caveat to all of this "move quickly" stuff is "test Test TEST". If you are constantly rolling dependencies forward, stuff will break. Run all the tests before landing a dependency change. Revert dependency changes with impunity: since they are effectively merges of many commits made to the other repo, they are more likely to have breaking changes than any single commit to your main repo. So pretest, test, and revert. None of this requires amazing engineers. It just requires a dedication to quality and good testing hygiene.
  - neikos 11y ago
    I wonder though, there /have/ to be merge conflicts? I suppose you continually rebase on the HEAD of master to keep up to date? So instead of taking care of conflicts at the end of a 'dev cycle' you take care of it in the middle. Did I understand that correctly?
    - phasmantistes 11y ago
      The difference is that, in our flow, the merge conflicts are on the scale of a a single developer making a single self-contained commit. In a flow with feature branches, the merge conflicts are on a scale of the entire branch conflicting with any other changes made on master. But yes we also provide tools (https://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/depot_tools.html https://commondatastorage.googleapis.com/chrome-infra-docs/f...) to make things like rebasing (see git rebase-update) your local branches on top of master easier.
  - dankohn1 11y ago
    I'm sorry if this is obvious, but I would love to understand exactly what you mean by "the only branches are release branches". What I think you mean is that the only branches are release branches and the development branches on each dev's machine. At my startup, we use feature branches, and we expect devs to continually merge master into their branch while it's under development (ideally daily), and to fix any resulting conflicts or test failures. We have them push these branches to the repo (which triggers our continuous integration server), but they're treated as private works in progress. Then, when they finish and pass code review, we're confident that they will merge seamlessly, since they've been merging master all along. (We also use git merge --squash for the final merge to avoid seeing all the intermediate commits on the master commit log.) I'm trying to understand if your practices are meaningfully different. Do developers have private branches on their machines for weeks on end? Are they encouraged to regularly merge master into those private branches?
ttr314 11y ago
Being Google, what exactly keeps you from running every test for every commit? Maybe I am being a bit naive here, but as the owner of multiple data centers across the globe, shouldn't you have enough resources to do that, even for a project as big as Chromium?
- bengoodger 11y ago
  Many but not all things can be usefully parallelized. The build, and most unit tests and integration tests can, though we struggle with the latter where real hardware is required. Some performance tests are also run on real hardware. This doesn't scale as well.
- phasmantistes 11y ago
  Chromium is open source, and will run try jobs for anyone from anywhere around the world (we won't accept your commit without lots of checks, but we'll run your tests). That means that we don't get to rely on a lot of Google-internal technology, including our production data centers. Also, we are compiling and testing on Mac, Windows, Linux, Android, iOS, ChromeOS, and historical versions of all of the above. That's something that a mostly linux-server-based company doesn't do a lot of. So we maintain our own datacenters with our own hardware, and don't get to take advantage of all of Google's scale.
IanWhalen 11y ago
I'm curious whether you've got any interesting projects underway to work on the flaky test issue? I've been looking at FBs test lifecycle approach with some interest (http://goo.gl/QtUNxY http://goo.gl/QtUNxY), but would love to hear about any other approaches to solving the same problem. And FWIW on building MongoDB with Evergreen (https://evergreen.mongodb.com https://evergreen.mongodb.com) we've found that a stepback approach seems to give us the finest granularity with minimal cost in identifying the introduction of an error - we batch commits for execution, but when a specific test fails we start running just that test on each previous commit until we hit a passing execution. It obviously doesn't work perfectly in the face of flaky tests (see above question) but it seems to do pretty well.
- phasmantistes 11y ago
  We are working on a few things. The first is automatic retries of fast tests. If a test runs quickly and fails, it costs us little to try again just to make sure. Most of our unit tests are configured to run up to 3 times. Another thing is keeping track of a database of individual test case passes/failures across all time. This will let us automatically mark tests as flaky if they fail often, and ignore their results programatically rather than requiring a human to manually mark the test as ignorable. A third thing is, obviously, automatically filing bugs against the owners/authors of tests which have been marked flaky. This is controversial -- often a test is just fine until one of its underlying libraries has a race condition introduced, and the real person to fix it should be the author of that change, not the author of the test. But it is still a step in the right direction much of the time. Many people subscribe to the philosophy that "a flaky test is worse than no test", because you think it is giving you information when in fact it is giving you none. I subscribe to a slightly different philosophy: "A test with a known flaky rate is hugely valuable". If you know how often a test flakes (statistically), then you can measure variances from that rate to detect changes. Of course, a flaky test with an unknown rate of flaky is still useless. Hence the second initiative above: measuring the rate of flake of everything.
  - cpeterso 11y ago
    Mozilla has a big problem with flaky tests (aka "intermittents") too. The SpiderMonkey VM can run tests in a deterministic mode to make tests less flaky by making things like GC more predictable. Firefox has an experimental "chaos mode" that takes the opposite approach. It purposely randomizes behavior by adjusting thread priorities, changing hash table iteration order, and randomize timer durations. Unfortunately, many flaky tests fail in chaos mode, so it is not enabled by default. http://robert.ocallahan.org/2014/03/introducing-chaos-mode.html http://robert.ocallahan.org/2014/03/introducing-chaos-mode.h...
mwcampbell 11y ago
This is only tangentially related to the topics covered in the article and your comment, but I'm curious about the Chromium project's decision to mostly use static linking. I've always found it interesting that Chrome is shipped mostly as one big binary (or on Windows, two: chrome.dll and chrome_child.dll), whereas IE and Safari use several (one for the rendering engine, one for the JS engine, one or two for networking, etc.). My guess is that Chrome uses one big binary to minimize startup time, since the OS can just map that binary into memory and go. Am I correct about the rationale? Another way to ask the same thing is this: what would be the downsides of using a component build for releases and not just for development? If you used a component build for releases, you'd have greater parity between dev and production, and that's generally a good thing. I also notice that there are a few modules that get their own DLLs, at least on Windows. For example, in Chrome 45, we have chrome_elf.dll, libegl.dll, libexif.dll, and libglesv2.dll; I'd think these libraries could be statically linked just like the few hundred others. Are there good reasons for these exceptions? Or are these just little details that tend to get overlooked given the rapid pace of development?
- phasmantistes 11y ago
  Since all I've worked on for the last three years is the infrastructure, and not the binary that lands on your desktop/phone, I'm not really qualified to answer that. (Which is just a fancy way of saying I've got no earthly idea :)) But since I've seen @bengoodger in this thread, maybe he can answer your questions.
- aboodman 11y ago
  I can't speak to the current situation, but originally Chrome was one monolithic dll for exactly the reason you describe: we wanted renderer launches to be as fast as possible and having the code mostly already paged in helped a lot. Over time it became impossible to link and so we had to switch to having a separate renderer dll. I'm not sure why there are now a handful of separate dlls.
- justinschuh 11y ago
  Like Aaron mentioned, the renderer was originally split into a separate DLL because we hit the limits of Microsoft's linker at the time. Later, we ended up relying on that split for disabling win32k inside the sandbox (otherwise we would have had to deal with a bunch of broken USER and GDI initializations). As for the remaining DLLs, it mostly comes down to licensing concerns. We don't include proprietary code in our main binaries, and in order to simplify IP concerns it's often best to split such code into other DLLs. For example, you asked about libegl.dll and libglesv2.dll, which I believe include a licensed copy of the SwiftShader library. It's also worth pointing out that the performance impact of having a single, static binary in Windows is much less significant than it used to be. So, were we starting Chrome today I'm not so sure we'd make the same decision.
swah 11y ago
So, engineers work on the master branch, test locally and push to master? And the buildbot detects commits that break the build after the fact?
- phasmantistes 11y ago
  Engineers base their local development branches on top of origin/master. They can make as many local commits on that branch as they like. Then they upload the diff of that entire branch vs. its upstream (origin/master, unless they're doing chained changes) to the code review system. As it goes through code review, it is tested on the trybots. Once all the trybots pass and it has received approval from the reviewers, it is landed on master and fully tested again.
  - kelnos 11y ago
    Maybe I'm still not understanding fully, but that really sounds just like having branches, except they aren't treated by the system as such. There's still the possibility that while developer A's change is going through code review and testing, developer B's changes (which conflict with developer A's) make it through and get landed. I'd agree that having long-lived branches is often bad, but that's more a matter of discipline than anything else. I could certainly start working on a feature locally using Chromium's model, take my time, and get far enough behind the tip of the tree that by the time I want to submit for code review and testing, my stuff doesn't apply to the new tip. Branches vs. no branches doesn't seem to be a factor here.
    - phasmantistes 11y ago
      There are no branches in the golden-source-of-truth repository (except for release branches, but those are never merged back into master and no development is done directly on them). The only branches that exist are on the developer's local machine, and only exist for that developer's personal preference and bookkeeping. Yes, it is totally possible for two developers to be working in the same place and try to land their changes at the same time, resulting in a conflict. This happens in one of three ways: a) dev B's change fails tests, because the try bots which try to test dev B's change have already sync'd dev As change (this is the most common) b) dev B's change passes all the tests, but fails to commit because dev A's change landed seconds earlier. In this case the system sends dev B an email asking them to rebase their change and try again. Simple. c) dev B's change passes all the tests, then lands and causes tip-of-tree tests to fail because dev A's change landed seconds earlier and they disagree on the name of an API method or something. This happens rarely, and is easily resolved by reverting one of the two and asking the author to try again. Anyway, the point is that the dev branches are incredibly short lived: they exist only on the developer's machine and end up as a single, coherent, self-contained commit on master. If you get far enough behind that your stuff doesn't apply on master, it is your responsibility to clean it up, and no one else even knows that you had a conflict.
Ecco 11y ago
You say you have to rely on humans and heuristics to determine which CL to revert. Couldn't you automatically re-run just the failing tests on the batch? More importantly, you seem to be running all the tests, as often as you can. A commit changing a text label shouldn't require running all the tests, that just doesn't scale. Have you consider splitting Chromium into different projects, where commits for one project would result in the tests of that project, and only that project?
- phasmantistes 11y ago
  We actually are working on systems which re-run the failing tests on all CLs within a failing batch. Unfortunately, we haven't been able to do this previously due to resource constraints. In particular, if a batch had 10 commits in it, you can either run 10 additional compiles and tests (didn't have the resources), or you can binary-search/bisect to find the failure (takes too long). It is often faster to use a heuristic to say "Test_Foo failed, and Foo_Src was modified in change X, so revert change X". We don't like relying on heuristics, but until recently it was our only choice. We actually don't run all the tests all the time. We have a good dependency system (called Gyp/GN, similar to Bazel) which knows which tests rely on which source files. When the infrastructure compiles a new change, it only runs the tests which depend on source files touched by that change.
JoshTriplett 11y ago
Now that Chromium has moved over to Git, any plans or timeline to migrate Chromium over to Gerrit, rather than Reitveld?
- phasmantistes 11y ago
  Heh, that's a contentious topic. Personally, I would like to do so. I think I speak correctly when I say that most of the Infrastructure team would like to do so, to increase security/auditability and ease the maintenance burden on ourselves. However, large portions of the team are opposed to Gerrit because a) it doesn't integrate as well with our other systems (such as trybots and cq) and b) the UI has some issues. We are working with people to try to fix those, but there's no particular timeline for switching, or even a guarantee that we will.
  - JoshTriplett 11y ago
    > a) it doesn't integrate as well with our other systems (such as trybots and cq) and b) the UI has some issues. Is there a tracking bug for all the integration and UI issues? How can folks working on Chromium but without an @google.com help?
    - phasmantistes 11y ago
      I think this is the best tracking bug: https://code.google.com/p/chromium/issues/detail?id=465929 https://code.google.com/p/chromium/issues/detail?id=465929 There are three things going on: 1) Rietveld exists 2) A new Polymer-based UI for Rietveld exists 3) Gerrit exists The bug linked above is about evaluating the ability to make the Polymer-based UI (which is pretty and functional) work on top of Gerrit instead of on top of Rietveld. This is one of the possible paths forward for getting people to use Gerrit, and the one for which external contributors could be the most help. Other paths forward exist: making Rietveld and the polymer Rietveld UI so good that no one wants to switch to Gerrit anymore; working directly on the Gerrit UI; or convincing devs that the Gerrit UI is good enough. But I think the bug and description above is the path on which external contributors could do the most good.
oso2k 11y ago
Could I ask you a dumb question? What does CL stand for?
- sdhillon 11y ago
  changelist
  - oso2k 11y ago
    Thought so. I've seen CL used in odd ways on the v8-users list so I got confused. Thanks!
maximilianburke 11y ago
What are the policies behind the commit queues? Is it mandatory for a change to be queued?
- phasmantistes 11y ago
  The commit queue can only be bypassed by people who are 'committers' and therefore have direct push access to the repo. Becoming a committer requires landing multiple changes, demonstrating competence and desire to continue to contribute, and being nominated and seconded by your peers. This is true for all contributors, even Google employees. Even committers are heavily encouraged to use the commit queue, and everyone does. Something like 96% of commits land via the CQ. It's simply a bad idea not to. The main use-case for bypassing the queue is quickly reverting a change which somehow slipped through, and even those usually go through the queue anyway with a special flag that bypasses all of the tests.
dx211 11y ago
Do Chrome for PC and Chrome for iOS come from the same code base, and if so, how do you automate testing for closed systems like iOS? (I ask because from my totally anecdotal perspective, Chrome for PC is a rock-solid dream of an application, but Chrome on my iPad crashes if it sees an animated GIF from across the room.)
- darinf 11y ago
  It is unfortunate that Apple does not allow us to bring all of Chromium--our full rendering stack (Blink, etc.)--to iOS. If they did, we could build a much better browser for iOS. Instead, we are forced to use their provided WebView, and that means living with its limitations and bugs.
nickpsecurity 11y ago
Does Google currently use or plan to use static analysis tools that prove absence of key bug classes? https://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis#C.2C_C.2B.2B https://en.wikipedia.org/wiki/List_of_tools_for_static_code_... I know in safety-critical the Astree and SPARK tools have been particularly good at verifying code. Something like Polyspace might be more appropriate for Chrome, though.
- wolfgke 11y ago
  SPARK is based on Ada. Astree also seems to be limited only to C. Chrome, to the opposite, is written in millions of lines of C++.
  - nickpsecurity 11y ago
    The reason I mentioned SPARK and Astree is because they're excellent at what they do. People Googling them will understand how these approaches work to prove absence of bugs rather than test for their presence. Then, they might experiment with some of the others on the list. I mentioned PolySpace as an example of something more appropriate for Chrome because it supports C++ and is used in high integrity applications.
- justinschuh 11y ago
  We've invested quite a bit of energy in static analysis tools over the years, but so far the return hasn't been there. There are still ongoing efforts, and we'll see if those pan out, but up to this point we've just run into a very low hit rate and a false positive rate that drowns everything else out from a security perspective. Formal methods are a different question, but I can't imagine the size of the undertaking needed to convert even a subset of Chromium over to formal methods. Instead, we've focused pretty heavily on more conventional anti-exploit mitigations applied to existing C++ code. Our sandboxing is the most obvious one, but we have a variety of other protections on top of typical stuff like ASLR and NX (e.g. heap partitioning, a safe integer handling toolkit, etc). We're also leading efficient control-flow integrity work in clang, and busily working it up to a stable release. And then we have fuzzing as part of our continuous integration, running across thousands of cores of instrumented builds.
perlgeek 11y ago
> We don't have enough resources to run every test for every commit, so they end up batched and we have to rely on humans and heuristics to determine which CL in the batch to revert. You could automatically bisect with just the failing test to find which commit to revert. At least if it's not a flapping test.

robotnoises 11y ago

Do you think a version of this system could work in a team that has mostly "b-players?" I just wonder how much of this stays afloat because the caliber of the engineers is higher than that of your typical company.

phasmantistes 11y ago
I don't think any of our decisions are driven by or dependent upon the caliber of our engineers. For example: a) No merges; and the only branches are release branches. This makes sense in almost any context. Whether your team is big or small, high-caliber or college sophomores, this works. The advantages are numerous. No one works on a feature branch in isolation, only to have tons of merge conflicts and surprises when they try to bring it back in. Bug fixes and features are never landed on a release branch and then not merged into master for the next release. Development doesn't slow to a crawl while you try to stabilize master for a release. b) Keeping dependencies near tip-of-tree. Chromium has two kinds of dependencies: third-party libraries (e.g. yasm, lighttpd) and things we develop on our own but in separate repositories (e.g. pdfium, v8, boringssl). The former tend to have stable releases produced by their own maintainers; the latter is being developed in parallel and you want to be as up-to-date as possible. Most projects are similar: you have third-party dependencies (e.g. python packages) and your own libraries (like a common set of api interfaces). You want to keep the ones you control as close to tip-of-tree as possible so that you can move fast and integrate your own changes quickly. The caveat to all of this "move quickly" stuff is "test Test TEST". If you are constantly rolling dependencies forward, stuff will break. Run all the tests before landing a dependency change. Revert dependency changes with impunity: since they are effectively merges of many commits made to the other repo, they are more likely to have breaking changes than any single commit to your main repo. So pretest, test, and revert. None of this requires amazing engineers. It just requires a dedication to quality and good testing hygiene.
- neikos 11y ago
  I wonder though, there /have/ to be merge conflicts? I suppose you continually rebase on the HEAD of master to keep up to date? So instead of taking care of conflicts at the end of a 'dev cycle' you take care of it in the middle. Did I understand that correctly?
  - phasmantistes 11y ago
    The difference is that, in our flow, the merge conflicts are on the scale of a a single developer making a single self-contained commit. In a flow with feature branches, the merge conflicts are on a scale of the entire branch conflicting with any other changes made on master. But yes we also provide tools (https://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/depot_tools.html https://commondatastorage.googleapis.com/chrome-infra-docs/f...) to make things like rebasing (see git rebase-update) your local branches on top of master easier.
- dankohn1 11y ago
  I'm sorry if this is obvious, but I would love to understand exactly what you mean by "the only branches are release branches". What I think you mean is that the only branches are release branches and the development branches on each dev's machine. At my startup, we use feature branches, and we expect devs to continually merge master into their branch while it's under development (ideally daily), and to fix any resulting conflicts or test failures. We have them push these branches to the repo (which triggers our continuous integration server), but they're treated as private works in progress. Then, when they finish and pass code review, we're confident that they will merge seamlessly, since they've been merging master all along. (We also use git merge --squash for the final merge to avoid seeing all the intermediate commits on the master commit log.) I'm trying to understand if your practices are meaningfully different. Do developers have private branches on their machines for weeks on end? Are they encouraged to regularly merge master into those private branches?

ttr314 11y ago

Being Google, what exactly keeps you from running every test for every commit? Maybe I am being a bit naive here, but as the owner of multiple data centers across the globe, shouldn't you have enough resources to do that, even for a project as big as Chromium?

bengoodger 11y ago
Many but not all things can be usefully parallelized. The build, and most unit tests and integration tests can, though we struggle with the latter where real hardware is required. Some performance tests are also run on real hardware. This doesn't scale as well.
phasmantistes 11y ago
Chromium is open source, and will run try jobs for anyone from anywhere around the world (we won't accept your commit without lots of checks, but we'll run your tests). That means that we don't get to rely on a lot of Google-internal technology, including our production data centers. Also, we are compiling and testing on Mac, Windows, Linux, Android, iOS, ChromeOS, and historical versions of all of the above. That's something that a mostly linux-server-based company doesn't do a lot of. So we maintain our own datacenters with our own hardware, and don't get to take advantage of all of Google's scale.

IanWhalen 11y ago

I'm curious whether you've got any interesting projects underway to work on the flaky test issue? I've been looking at FBs test lifecycle approach with some interest (http://goo.gl/QtUNxY http://goo.gl/QtUNxY), but would love to hear about any other approaches to solving the same problem. And FWIW on building MongoDB with Evergreen (https://evergreen.mongodb.com https://evergreen.mongodb.com) we've found that a stepback approach seems to give us the finest granularity with minimal cost in identifying the introduction of an error - we batch commits for execution, but when a specific test fails we start running just that test on each previous commit until we hit a passing execution. It obviously doesn't work perfectly in the face of flaky tests (see above question) but it seems to do pretty well.

phasmantistes 11y ago
We are working on a few things. The first is automatic retries of fast tests. If a test runs quickly and fails, it costs us little to try again just to make sure. Most of our unit tests are configured to run up to 3 times. Another thing is keeping track of a database of individual test case passes/failures across all time. This will let us automatically mark tests as flaky if they fail often, and ignore their results programatically rather than requiring a human to manually mark the test as ignorable. A third thing is, obviously, automatically filing bugs against the owners/authors of tests which have been marked flaky. This is controversial -- often a test is just fine until one of its underlying libraries has a race condition introduced, and the real person to fix it should be the author of that change, not the author of the test. But it is still a step in the right direction much of the time. Many people subscribe to the philosophy that "a flaky test is worse than no test", because you think it is giving you information when in fact it is giving you none. I subscribe to a slightly different philosophy: "A test with a known flaky rate is hugely valuable". If you know how often a test flakes (statistically), then you can measure variances from that rate to detect changes. Of course, a flaky test with an unknown rate of flaky is still useless. Hence the second initiative above: measuring the rate of flake of everything.
- cpeterso 11y ago
  Mozilla has a big problem with flaky tests (aka "intermittents") too. The SpiderMonkey VM can run tests in a deterministic mode to make tests less flaky by making things like GC more predictable. Firefox has an experimental "chaos mode" that takes the opposite approach. It purposely randomizes behavior by adjusting thread priorities, changing hash table iteration order, and randomize timer durations. Unfortunately, many flaky tests fail in chaos mode, so it is not enabled by default. http://robert.ocallahan.org/2014/03/introducing-chaos-mode.html http://robert.ocallahan.org/2014/03/introducing-chaos-mode.h...

mwcampbell 11y ago

This is only tangentially related to the topics covered in the article and your comment, but I'm curious about the Chromium project's decision to mostly use static linking. I've always found it interesting that Chrome is shipped mostly as one big binary (or on Windows, two: chrome.dll and chrome_child.dll), whereas IE and Safari use several (one for the rendering engine, one for the JS engine, one or two for networking, etc.). My guess is that Chrome uses one big binary to minimize startup time, since the OS can just map that binary into memory and go. Am I correct about the rationale? Another way to ask the same thing is this: what would be the downsides of using a component build for releases and not just for development? If you used a component build for releases, you'd have greater parity between dev and production, and that's generally a good thing. I also notice that there are a few modules that get their own DLLs, at least on Windows. For example, in Chrome 45, we have chrome_elf.dll, libegl.dll, libexif.dll, and libglesv2.dll; I'd think these libraries could be statically linked just like the few hundred others. Are there good reasons for these exceptions? Or are these just little details that tend to get overlooked given the rapid pace of development?

phasmantistes 11y ago
Since all I've worked on for the last three years is the infrastructure, and not the binary that lands on your desktop/phone, I'm not really qualified to answer that. (Which is just a fancy way of saying I've got no earthly idea :)) But since I've seen @bengoodger in this thread, maybe he can answer your questions.
aboodman 11y ago
I can't speak to the current situation, but originally Chrome was one monolithic dll for exactly the reason you describe: we wanted renderer launches to be as fast as possible and having the code mostly already paged in helped a lot. Over time it became impossible to link and so we had to switch to having a separate renderer dll. I'm not sure why there are now a handful of separate dlls.
justinschuh 11y ago
Like Aaron mentioned, the renderer was originally split into a separate DLL because we hit the limits of Microsoft's linker at the time. Later, we ended up relying on that split for disabling win32k inside the sandbox (otherwise we would have had to deal with a bunch of broken USER and GDI initializations). As for the remaining DLLs, it mostly comes down to licensing concerns. We don't include proprietary code in our main binaries, and in order to simplify IP concerns it's often best to split such code into other DLLs. For example, you asked about libegl.dll and libglesv2.dll, which I believe include a licensed copy of the SwiftShader library. It's also worth pointing out that the performance impact of having a single, static binary in Windows is much less significant than it used to be. So, were we starting Chrome today I'm not so sure we'd make the same decision.

swah 11y ago

So, engineers work on the master branch, test locally and push to master? And the buildbot detects commits that break the build after the fact?

phasmantistes 11y ago
Engineers base their local development branches on top of origin/master. They can make as many local commits on that branch as they like. Then they upload the diff of that entire branch vs. its upstream (origin/master, unless they're doing chained changes) to the code review system. As it goes through code review, it is tested on the trybots. Once all the trybots pass and it has received approval from the reviewers, it is landed on master and fully tested again.
- kelnos 11y ago
  Maybe I'm still not understanding fully, but that really sounds just like having branches, except they aren't treated by the system as such. There's still the possibility that while developer A's change is going through code review and testing, developer B's changes (which conflict with developer A's) make it through and get landed. I'd agree that having long-lived branches is often bad, but that's more a matter of discipline than anything else. I could certainly start working on a feature locally using Chromium's model, take my time, and get far enough behind the tip of the tree that by the time I want to submit for code review and testing, my stuff doesn't apply to the new tip. Branches vs. no branches doesn't seem to be a factor here.
  - phasmantistes 11y ago
    There are no branches in the golden-source-of-truth repository (except for release branches, but those are never merged back into master and no development is done directly on them). The only branches that exist are on the developer's local machine, and only exist for that developer's personal preference and bookkeeping. Yes, it is totally possible for two developers to be working in the same place and try to land their changes at the same time, resulting in a conflict. This happens in one of three ways: a) dev B's change fails tests, because the try bots which try to test dev B's change have already sync'd dev As change (this is the most common) b) dev B's change passes all the tests, but fails to commit because dev A's change landed seconds earlier. In this case the system sends dev B an email asking them to rebase their change and try again. Simple. c) dev B's change passes all the tests, then lands and causes tip-of-tree tests to fail because dev A's change landed seconds earlier and they disagree on the name of an API method or something. This happens rarely, and is easily resolved by reverting one of the two and asking the author to try again. Anyway, the point is that the dev branches are incredibly short lived: they exist only on the developer's machine and end up as a single, coherent, self-contained commit on master. If you get far enough behind that your stuff doesn't apply on master, it is your responsibility to clean it up, and no one else even knows that you had a conflict.

Ecco 11y ago

You say you have to rely on humans and heuristics to determine which CL to revert. Couldn't you automatically re-run just the failing tests on the batch? More importantly, you seem to be running all the tests, as often as you can. A commit changing a text label shouldn't require running all the tests, that just doesn't scale. Have you consider splitting Chromium into different projects, where commits for one project would result in the tests of that project, and only that project?

phasmantistes 11y ago
We actually are working on systems which re-run the failing tests on all CLs within a failing batch. Unfortunately, we haven't been able to do this previously due to resource constraints. In particular, if a batch had 10 commits in it, you can either run 10 additional compiles and tests (didn't have the resources), or you can binary-search/bisect to find the failure (takes too long). It is often faster to use a heuristic to say "Test_Foo failed, and Foo_Src was modified in change X, so revert change X". We don't like relying on heuristics, but until recently it was our only choice. We actually don't run all the tests all the time. We have a good dependency system (called Gyp/GN, similar to Bazel) which knows which tests rely on which source files. When the infrastructure compiles a new change, it only runs the tests which depend on source files touched by that change.

JoshTriplett 11y ago

Now that Chromium has moved over to Git, any plans or timeline to migrate Chromium over to Gerrit, rather than Reitveld?

phasmantistes 11y ago
Heh, that's a contentious topic. Personally, I would like to do so. I think I speak correctly when I say that most of the Infrastructure team would like to do so, to increase security/auditability and ease the maintenance burden on ourselves. However, large portions of the team are opposed to Gerrit because a) it doesn't integrate as well with our other systems (such as trybots and cq) and b) the UI has some issues. We are working with people to try to fix those, but there's no particular timeline for switching, or even a guarantee that we will.
- JoshTriplett 11y ago
  > a) it doesn't integrate as well with our other systems (such as trybots and cq) and b) the UI has some issues. Is there a tracking bug for all the integration and UI issues? How can folks working on Chromium but without an @google.com help?
  - phasmantistes 11y ago
    I think this is the best tracking bug: https://code.google.com/p/chromium/issues/detail?id=465929 https://code.google.com/p/chromium/issues/detail?id=465929 There are three things going on: 1) Rietveld exists 2) A new Polymer-based UI for Rietveld exists 3) Gerrit exists The bug linked above is about evaluating the ability to make the Polymer-based UI (which is pretty and functional) work on top of Gerrit instead of on top of Rietveld. This is one of the possible paths forward for getting people to use Gerrit, and the one for which external contributors could be the most help. Other paths forward exist: making Rietveld and the polymer Rietveld UI so good that no one wants to switch to Gerrit anymore; working directly on the Gerrit UI; or convincing devs that the Gerrit UI is good enough. But I think the bug and description above is the path on which external contributors could do the most good.

oso2k 11y ago

Could I ask you a dumb question? What does CL stand for?

sdhillon 11y ago
changelist
- oso2k 11y ago
  Thought so. I've seen CL used in odd ways on the v8-users list so I got confused. Thanks!

maximilianburke 11y ago

What are the policies behind the commit queues? Is it mandatory for a change to be queued?

phasmantistes 11y ago
The commit queue can only be bypassed by people who are 'committers' and therefore have direct push access to the repo. Becoming a committer requires landing multiple changes, demonstrating competence and desire to continue to contribute, and being nominated and seconded by your peers. This is true for all contributors, even Google employees. Even committers are heavily encouraged to use the commit queue, and everyone does. Something like 96% of commits land via the CQ. It's simply a bad idea not to. The main use-case for bypassing the queue is quickly reverting a change which somehow slipped through, and even those usually go through the queue anyway with a special flag that bypasses all of the tests.

dx211 11y ago

Do Chrome for PC and Chrome for iOS come from the same code base, and if so, how do you automate testing for closed systems like iOS? (I ask because from my totally anecdotal perspective, Chrome for PC is a rock-solid dream of an application, but Chrome on my iPad crashes if it sees an animated GIF from across the room.)

darinf 11y ago
It is unfortunate that Apple does not allow us to bring all of Chromium--our full rendering stack (Blink, etc.)--to iOS. If they did, we could build a much better browser for iOS. Instead, we are forced to use their provided WebView, and that means living with its limitations and bugs.

nickpsecurity 11y ago

Does Google currently use or plan to use static analysis tools that prove absence of key bug classes? https://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis#C.2C_C.2B.2B https://en.wikipedia.org/wiki/List_of_tools_for_static_code_... I know in safety-critical the Astree and SPARK tools have been particularly good at verifying code. Something like Polyspace might be more appropriate for Chrome, though.

wolfgke 11y ago
SPARK is based on Ada. Astree also seems to be limited only to C. Chrome, to the opposite, is written in millions of lines of C++.
- nickpsecurity 11y ago
  The reason I mentioned SPARK and Astree is because they're excellent at what they do. People Googling them will understand how these approaches work to prove absence of bugs rather than test for their presence. Then, they might experiment with some of the others on the list. I mentioned PolySpace as an example of something more appropriate for Chrome because it supports C++ and is used in high integrity applications.
justinschuh 11y ago
We've invested quite a bit of energy in static analysis tools over the years, but so far the return hasn't been there. There are still ongoing efforts, and we'll see if those pan out, but up to this point we've just run into a very low hit rate and a false positive rate that drowns everything else out from a security perspective. Formal methods are a different question, but I can't imagine the size of the undertaking needed to convert even a subset of Chromium over to formal methods. Instead, we've focused pretty heavily on more conventional anti-exploit mitigations applied to existing C++ code. Our sandboxing is the most obvious one, but we have a variety of other protections on top of typical stuff like ASLR and NX (e.g. heap partitioning, a safe integer handling toolkit, etc). We're also leading efficient control-flow integrity work in clang, and busily working it up to a stable release. And then we have fuzzing as part of our continuous integration, running across thousands of cores of instrumented builds.

perlgeek 11y ago

> We don't have enough resources to run every test for every commit, so they end up batched and we have to rely on humans and heuristics to determine which CL in the batch to revert. You could automatically bisect with just the failing test to find which commit to revert. At least if it's not a flapping test.