3 ms·

My notes say it was 193k at the start. The final dashboard when we stopped said "216,626 / 216,890 = 99.8%; 264 to go". The other correction I would make is th

by rsc 2y ago

My notes say it was 193k at the start. The final dashboard when we stopped said "216,626 / 216,890 = 99.8%; 264 to go". The other correction I would make is that this post does not mention Nilton Volpato, who had written an earlier Buildifier and graciously accepted replacing his implementation with a new one and then taking over ownership for that new implementation as well. (Eventually ownership moved to Laurent's team.) It looks like it was just under 2,000 commits. We did pretty extensive testing, by having Blaze load a BUILD file and its transitive closure and then dump that parsed form back out to a binary format. Any automated commit had to preserve that parsed-and-dumped binary format bit for bit. The slowest part of the testing was waiting for Blaze to do all the loads. Every day I would prepare and test as many files as I could, break them into CLs (think PRs), mail Rob a shell script he could run to approve them all, and go to bed. Then I'd get up early in the morning (5am ET) to submit the changes, because there were various cached indexes that got updated when BUILD files got submitted, and it seemed better to send them when not many people would be working. That scheme worked until a system did fall over and someone got paged, and then after that I agreed to only submit the large changes during business hours. :-)

laurentlb 2y ago
Thanks for the precisions! I've added some updates at the bottom of the post.
haburka 2y ago
You didn’t have Rosie to automatically split up your changes and send them out yet?? That must have been rough. LSCs are way easier now
- ammar2 2y ago
  Yeah I'm surprised by that as well. As far as I remember, Rosie started out in 2010 and people were using in 2012. Maybe the clustering/splitting didn't support this use-case or it wasn't well-known enough?
  - singron 2y ago
    From my memory, extremely large but semantically simple changes still used global approvers since it wasn't considered worth the effort to get approval from hundreds or thousands of individual OWNERs using rosie. Also, with a change of this magnitude, especially one affecting BUILD files, it might not have been possible to create the mega-CL that rosie uses, but I don't remember specific limitations on that.
    - jrockway 2y ago
      That's exactly what I remember. I did a few LSCs; one I used Rosie just to see how it worked (I recall wanting to use it for all my CLs because it submits after someone clicks approve and the tests pass), but most of the time it was easier to find a global approver and submit the whole thing atomically.
      - dartos 2y ago
        I always appreciate these googler threads talking about all the Google internal tools. Reminds me of all the impenetrable jargon around me when I was new
- rsc 2y ago
  (Rosie is a code janitor program at Google that takes a change that affects files all across the tree and automatically cuts it into individual changes that can be mailed out to the individual teams as CLs/PRs for their approval.) Rosie existed but very much wanted to break up the CL into independent per-directory CLs, and since I was editing one file in every directory in the entire tree, that would have been 200,000 independent CLs. I broke the list up by top-level directory or sub-directory and hit 100+ directories at a time. Rosie also really wants to run each affected directory's tests, and I did not, because at scale flaky tests and such would be a significant source of false positives. The bit-for-bit check on the internal parsed representation of the meaning of the BUILD file proved that the changes were no-ops. That was better than any tests of the code in the directory. I was already automating everything else, including deciding which files to change, reverting edits in files that were concurrently modified (they got swept into the next attempt), and the testing. Running a shell command to actually make the CLs was not difficult. And it generating the approval script trivial too. Rosie is great but it wasn't the right tool for this job.

by rsc 2y ago

laurentlb 2y ago

Thanks for the precisions! I've added some updates at the bottom of the post.

haburka 2y ago

You didn’t have Rosie to automatically split up your changes and send them out yet?? That must have been rough. LSCs are way easier now

ammar2 2y ago
Yeah I'm surprised by that as well. As far as I remember, Rosie started out in 2010 and people were using in 2012. Maybe the clustering/splitting didn't support this use-case or it wasn't well-known enough?
- singron 2y ago
  From my memory, extremely large but semantically simple changes still used global approvers since it wasn't considered worth the effort to get approval from hundreds or thousands of individual OWNERs using rosie. Also, with a change of this magnitude, especially one affecting BUILD files, it might not have been possible to create the mega-CL that rosie uses, but I don't remember specific limitations on that.
  - jrockway 2y ago
    That's exactly what I remember. I did a few LSCs; one I used Rosie just to see how it worked (I recall wanting to use it for all my CLs because it submits after someone clicks approve and the tests pass), but most of the time it was easier to find a global approver and submit the whole thing atomically.
    - dartos 2y ago
      I always appreciate these googler threads talking about all the Google internal tools. Reminds me of all the impenetrable jargon around me when I was new
rsc 2y ago
(Rosie is a code janitor program at Google that takes a change that affects files all across the tree and automatically cuts it into individual changes that can be mailed out to the individual teams as CLs/PRs for their approval.) Rosie existed but very much wanted to break up the CL into independent per-directory CLs, and since I was editing one file in every directory in the entire tree, that would have been 200,000 independent CLs. I broke the list up by top-level directory or sub-directory and hit 100+ directories at a time. Rosie also really wants to run each affected directory's tests, and I did not, because at scale flaky tests and such would be a significant source of false positives. The bit-for-bit check on the internal parsed representation of the meaning of the BUILD file proved that the changes were no-ops. That was better than any tests of the code in the directory. I was already automating everything else, including deciding which files to change, reverting edits in files that were concurrently modified (they got swept into the next attempt), and the testing. Running a shell command to actually make the CLs was not difficult. And it generating the approval script trivial too. Rosie is great but it wasn't the right tool for this job.