13 ms·
What I don't understand is how they accomplish larger collaborative changes. The paper says: "Almost all development occurs at the 'head' of the repository, no
by weinzierl 8y ago
What I don't understand is how they accomplish larger collaborative changes. The paper says:
"Almost all development occurs at the 'head' of the repository, not on branches."
Googler Rachel Potvin made an even stronger statement in her presentation about "The Motivation for a Monolithic Codebase" [1]:
"Branching for development at Google is exceedingly rare [..]"
In the related ACM paper she published with Josh Levenberg there is the statement that:
"Development on branches is unusual and not well supported at Google, though branches are typically used for releases."
I my world when we have to make a bigger change we create a branch and only merge it into the trunk when it is good enough to be
integrated. The branch enables us to work on that change together.
I don't understand how they do this at google. As far as I understand in their model they either have to
- give up on collaboration and always have just a single developer work on a change.
- share code by other means.
- check in unfinished work to the trunk for collaboration and constantly break trunk.
[1] https://youtu.be/W71BTkUbdqE?t=904 https://youtu.be/W71BTkUbdqE?t=904
[2] https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext# https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...
- mateuszf 8y agoYou don't have to check in unfinished work, you can break your work in parts and make people work on independent parts, every one of which moves progress forward incrementally. Well, you could actually call that "unfinished" because in the beginning the code doesn't accomplish the task, but progressively it will become more useful.
- weinzierl 8y ago> you can break your work in parts You can when you can but you can't when you can't. I 100% agree with you that we should work this way whenever possible and we should work hard to keep our code in a state that lets us cleanly divide work. In my experience it is not always possible to split up work that way. Think of untangling dependencies of a larger part of the code as an example.
- ithkuil 8y agoSometimes you think you can't because you either haven't learned the right tricks, or because it takes more effort and thus you opt to fork off your work in a separate long standing branch in order to optimize your development velocity at the expense of possible surprising costs during merge (if other people also make your same choice). Other times it's genuinely necessary to make a long standing branch. In those cases, you just do it. Trunk based development should not be a dogma, just a different default choice.
- jsty 8y agoYou could also gate experimental code behind feature flags which aren't set in prod
- weinzierl 8y agoI agree that feature flags can be a solution sometimes. The presentation and the paper I linked to in my question discusses this, but they also mention large-scale refactorings and this is where I don't see how feature flags can help. For example: How do they untangle a wad of code that is large enough that it takes longer than a few days and more than a single developer to get the code back into a state that is acceptable for trunk? The changes required for this kind of refactorings can be all over the place, regardless of any organizational boundaries in your code. I can't see how changes of this nature can be put behind feature flags.
- richardwhiuk 8y agoYou don't do that - you lots of small incremental refactors. I think Google also have tools to make a large refactor into a small change. (i.e. you create the new API, migrate stuff to use it, deprecate the old one, migrate the rest, retire the old one).
- ThaJay 8y ago- build an interface to the code to be changed - make all code that uses said code use the interface instead - build the feature switch into the interface
- VikingCoder 8y agoTry like hell to use dependency injection in the first place.
- Cedricgc 8y agohttps://trunkbaseddevelopment.com/branch-by-abstraction/ https://trunkbaseddevelopment.com/branch-by-abstraction/
- sigsergv 8y agoI think they don't branch code because perforce branches are horrible and merging them back to HEAD is extremely painful process for monorep.
- raverbashing 8y agoSo, use a better tool?
- philosopher1234 8y agoOr, don’t branch. Is branching so essential?
- lordnacho 8y agoIsn't it essential for mental organisation? How do you think about what's different about a set of changes without some sort of DAG?
- hopler 8y agoMake one change at a time.
- yjftsjthsd-h 8y agoJust do one thing at a time? Today, I am working on X; my commits are for X, and details are in the commit message.
- dharmab 8y agoThat breaks as soon as you have to interrupt working on Nice To Have Feature X to working on Inportant Bugfix/CVE Y.
- addicted 8y agoHow often does that happen to an individual developer though? Once a month? In an averagely well run company even that may be towards the higher end. Should your entire development strategy be based on a once a month occurrence?
- _cs2017_ 8y agoCan you clarify why branching helps collaboration? In other words, why is it harder to commit to trunk when you have several developers working on a feature?
- weinzierl 8y agoBranching enables me to share unfinished work with my collaborators. Sometimes I don't want to commit to trunk yet but still share code and collaborate on a part of the code base.
- majikandy 8y agoMake the changes smaller and more valuable and do it on trunk.
- weinzierl 8y agoThis should always be Plan A but in my experience it is not always possible. Think of untangling dependencies between a large number of components as an example.
- fergus_google 8y agoUntangle one dependency at a time.
- hknd 8y agoSo you can prepare a change for trunk, and share the change with your colleagues to collaborate on. Don't see why branching should be needed for this.
- weinzierl 8y agoThat's what I meant in my question with "share code by other means". It works but in my opinion it is a large pain and I can't believe people at Google work by sending patches back and forth.
- wahnfrieden 8y agoThere’s a wealth of reading available to you if you look up “trunk-based development” as a keyword. Likewise with “continuous integration” (the actual practice, not the build tooling). Jez Humble for instance has written extensively on this.
- weinzierl 8y agoI know about this but haven't found an answer to my question. This was in part my motivation to post this question here.
- jorlow 8y agoYou develop features behind compile or runtime flags and keep it off until it's ready to ship. This is what chromium.org does so that might be a more accessible way to see it in practice.
- hinkley 8y agoYhere are a bunch of ways to land code that doesn’t get run yet, by adding new functions or default parameters and conditional logic.
- afpx 8y agoI found this for you: https://trunkbaseddevelopment.com/ https://trunkbaseddevelopment.com/ I only scanned through it, but it seems similar to the de facto way of doing things before distributed version control systems became popular (in the late 2000s?).
- weinzierl 8y agoI know about this and it doesn't answer my question. This was in part my motivation to post this question here.
- ownagefool 8y agoI think you mistunderstood that link is arguing for. It's basically github flow with tagging on what you release, only goes into a bit more details and discussed alternatives and suffers a "bit from too much information" https://guides.github.com/introduction/flow/ https://guides.github.com/introduction/flow/ Is a cleaner and more obvious guide. The idea is you have a constantly usable master, and your branches should be short lived so you don't hit a brick wall trying to get reviews and merge on your massive change sets. Ultimately it means you want to test and review your change before it goes into master as opposed to creating "production", "staging" and "develop" branches, which largely just kick the can down the road and is a different way to solve that "what's deployed where" issue.
- afpx 8y agoThanks - I should have read through it more.
- tieze 8y agoIt might be a practical thing. I've heard from a Googler (a couple of years back) that getting changes in can take ages, and by the time the change lands, there's a good chance that there are merge conflicts, and the cycle starts over. Branches would make this even more painful.
- hknd 8y agoDepends on the size of the change. Small changes are preferred in most trunk-based-dev companies. FYI: There are (multiple) tools in Facebook and Google which are an abstraction on top of their VCS. (e.g. which feels more like git, where you can work on a stream of changes which depend on each other without actually pushing anything to head)
- nradov 8y agoBranches actually make this much less painful. Just reverse merge from the trunk back to each branch on a frequent basis.
- hknd 8y agoYou'll have to remember that all of those big companies have their tools and processes customized for their scale. Example: Instead of branching you would just create a `changelist` (a commit, a set of changes to files) and work on that. You can show it to your colleagues. You can build and test it. You can send the id to anyone to have a look at it, or test it themselves. You can have multiple changelists depending on each other, without being commited. This might be an interesting read for you: https://paulhammant.com/2014/01/08/googles-vs-facebooks-trunk-based-development/ https://paulhammant.com/2014/01/08/googles-vs-facebooks-trun...
- hopler 8y agoYou can use git forks or whatever for development. This philosophy just says that you only push to production environment from one standard head trunk.
- mcguire 8y ago"Google use Perforce for their trunk (with additional tooling), and many (but not all) developers use Git on their local workstation to gain local-branching with an inhouse developed bridge for interop with Perforce. "Branches & Merge Pain "TL;DR: the same "They don’t have merge pain, because as a rule developers are not merging to/from branches. At least up to the central repo’s server they are not. On workstations, developers may be merging to/from local branches, and rebasing when the push something that’s “done” back to the central repo. "Release engineers might cherry-pick defect fixes from time to time, but regular developers are not merging (you should not count to-working-copy merges)"
- CamouflagedKiwi 8y agoUnfinished work is not typically checked into master (and it's certainly not regularly broken). What is more common is that very large changes are checked in as a series of individually compatible changes, and often broken up across the repository (there are of course tools to help with this). It's relatively rare for multiple developers to work on a single changelist; it's much more common to break the work into separate changelists. Haven't worked there for some years now so I'm a bit rusty on some of the detail.
- ravibhatt 8y agoIt is amazing to see how we declare something as too obvious or natural. Like trunk based development, so obvious that questioning it makes one a fool :-) Git and its model was the best thing few years back. Now since google is doing all its dev in the main trunk/master, it must be correct and more intelligent. Wouldn't it be a case that they went with what they had at a certain time and continue to use it as everyone is used to it and it still works? Not sure if google analysed if branching was bad and then chose trunk based development? I cannot understand how a company that has a well defined process doing branches, is doing it wrong? or how it is so not optimal etc. I guess it is a matter of processes and culture. None of the great companies are great because their source control strategy (or code) was excellent. We developers always over analyse everything and come up with excellent logic and some of us are gifted with words more than others.
- geoffhill 8y agoNone of the other replies try to explain specifics of how this works, so let me illustrate an example of two teams collaborating to add Feature X to the monorepo without branching: 1) Team A checks in their code to provide Feature X. Their code is not used anywhere in the codebase yet, however full unit test coverage exists for the public API; this is required for code review. 2) Team B checks in their code to turn on Feature X in their product, gated under a command-line flag which by default uses the old behavior. 3) Team B checks in an integration test that flips the flag and makes sure everything works as planned. 4) If Team B requires changes to Feature X to get expected behavior, they communicate those changes to Team A and someone from either team (using available human resources) makes the changes. 5) Team B checks in a small change to flip the flag by default. 6) Team B monitors their product. If things go awry, only the very latest change is reverted and repeat (4). 7) Once stability is achieved, Team B checks in a change to remove the flag.
- radicality 8y agoI've been at FB for a few years now (similar style) and this model of a monolithic repo, no branching, and simply submitting 'difs' (a list of changesets / patches) which get merged directly into master after the 'diff' is accepted and you land it seems much easier to me. Maybe it's just because I got used to it, but now whenever I have to touch it I find the branch-based development confusing.