15 ms·

How I learned to stop worrying and push to master

145 points by sparklingmango 5y ago

keithnz 5y ago
we use a cherry pick model all developers develop features, we then cherry pick what features we want in the next release, do a release branch, merge in features, test (and if necessary, fix) then merge into main.
deleted 5y ago
[deleted]
Thaxll 5y ago
There are people pushing to master without review? I mean it's a common thing?
- GuestHNUser 5y ago
  This new to me as well. I am very curious what situations people are in where they don't/can't wait for a PR into master. If you reading this do this, I'd be interested in hearing what the benefit of pushing to master is for you. Is it purely for velocity? And if so why is velocity that important?
  - GuestHNUser 5y ago
    Coming back to my comment having finished the article in full. I wonder how the author's code base will develop using a weekly reflection on the code committed instead of PRs. I'm not sure how well their process will handle someone's poor design decisions. My experience has been that poor decisions tend to stay in code once they are there, and especially as more things become dependent on those decisions. Not that reviews will catch everything, but that second set of eyes can go a long way.
- zebraflask 5y ago
  Yes, there are! Familiar with this approach, pretty common with greenfield development. After causing a few outages and issues that you have to hear about, one learns fairly quickly not to write code that needs endless PR reviews.
  - GuestHNUser 5y ago
    I'm curious to know how does your team avoid shooting itself in the foot with unreviewed code? As in, is there some process that keeps unmaintainable elements from sneaking in?
    - zebraflask 5y ago
      Trust people to know what they're doing? A lot of PR reviews tend to devolve into almost anything but solving the issue at hand.
      - GuestHNUser 5y ago
        Fair enough. My experience with reviews has been they are quick quality checks, and at least for teams I have been on, were beneficial. I can see how they could become more of a ritual process than a beneficial one though, especially after seeing some other comments in the thread.
      - tetraca 5y ago
        Surprising. We do a PR and review for nearly every block of work we commit. Most pass without comment. The ones that have comments usually have good suggestions, or actually catch an error.
secondcoming 5y ago
> The more time that passes from the point a branch is broken off from develop to the time it is merged back in, the more opportunity there is for other branches to diverge in drastic ways. When a release is made you should merge those changes into other branches too. Then merge-conflicts or behavioural changes are found, and fixed, earlier.
- notreallyserio 5y ago
  This can be tricky. I worked somewhere that did this and it ends up forcing some devs to deal with multiple conflicts a week while they're working on a branch, when they could instead keep an eye on what is going on in main and deal with the conflicts on their own schedule.
  - nicoburns 5y ago
    The problem here seems to be devs working on a branch for a whole week. If you break down tasks smaller than this and merge them separately then you end up with far fewer merge conflicts.
  - secondcoming 5y ago
    I'd put that under the 'Too Bad' category. If a branch is so far behind reality/master then any testing done on it is quite likely to be invalid, and it won't have any hotfixes that were deployed in the meantime. YMMV
gsk22 5y ago
Isn't there a middle ground here? If you have CI/CD, you can eliminate release branches by just making PRs directly into main. The idea of post-merge code review is horrifying to me. I guess this is the "move fast and break things" attitude in action. Deploy now, catch bugs...sometime, maybe (TM)!
- sdfaksdfjalj 5y ago
  This is the way it's done at many major tech companies. Each individual commit is reviewed then merged directly into main/master. I've never used them but the thought of feature branches seems absurd versus simply merging small changes. Very rarely can a "feature" not be broken down into small self-contained changes.
- Starlevel001 5y ago
  > Deploy now, catch bugs...sometime, maybe (TM)! Given the state of literally all software I use, this seems to be the default behaviour.
- Ozzie_osman 5y ago
  I've found that to be a great middle ground on most teams. If you want to stretch it a little more, you could selectively do post-merge reviews for things that might be low risk (ie a UI change that's behind a feature flag that only your team sees), and keep riskier changes (like a big refactor, a data migration, etc) on the pre-merge review flow.
  - deckiedan 5y ago
    We have pre merge review - but for trivial stuff, text / label changes, etc, put a "tiny" tag on the PR, and if urgent paste it to the slack channel asking for a glance and nod review... At least having another pair of eyes check text-only changes has caught so many typos, etc, and only takes a couple minutes.
- PontiacParade 5y ago
  I agree with the middle ground comment. That is how we tend to do things where I work. We have a modified GitFlow: main: is the source of truth and is what is in production. develop: is the constantly moving branch we make PRs against. You can commit directly here which is discouraged but it isn't a hard rule. ticket: is a branch for each JIRA ticket not each feature. release: We don't make these and just use tags on main. Each "release" is a merge from develop to main and that gets deployed. hot fix: These are made against main and merged back to develop when they are used. It is rare enough I have to look up our "official" procedure. With that we can easily use PRs, release code in small hidden chunks, do code reviews, etc. Seems like the big win they got was releasing small hidden chunks of a feature and deploying it to staging. They also gave up some nice things as well like code review before merging.
tmerr 5y ago
I absolutely believe in this after working both ways. With feature branches I would waste time rebasing, resolving merge conflicts, to then come up with the perfect pile of commit messages before sharing with the team. Committing to master forces early collaboration & tighter feedback loops. Edit: I don't know about post merge code reviews, that seems like a risky idea to me at scale
- montroser 5y ago
  Early collaboration and tighter feedback loops are key. But you can make that all happen as a matter of culture, without needing committing to master as a forcing function.
dan-robertson 5y ago
Maybe it’s fine to put code reviews later in the process but I think I’d be worried about the frequency of changes that break the build. Is there some way to avoid that? Debugging failures only to discover that the build was broken (in the sense of a test failure, but also could be a JavaScript syntax error I suppose) underneath you is surely bad for velocity. Certainly I think there are times when having multiple people pushing straight to a feature branch is good but I’d worry about a codebase where features touch so much of the same stuff that merges are so painful. But maybe OP’s environment is just alien to me and there are reasons for both of these things.
- njovin 5y ago
  If you're using something that supports it (like Github) a good middle-ground would be auto-merging PRs. Branch from main -> code -> PR w/ automerge -> move on to the next task. This would give you speedy merges to main branch with the protection of the per-PR build checks.
  - dan-robertson 5y ago
    Obviously I don’t know much about the normal git process (the one I use is a little unusual) but what is the alternative to auto-merging PRs? Do people need to manually hit merge once tests are automatically run and code reviewed or is there some smaller set of people with the ability to merge? Do people ever do code review after a merge to make sure it worked properly?
3np 5y ago
> This all came together for me when I was catching up on YouTube and stumbled across Dave Farley's video Continuous Integration vs Feature Branch Workflow I really wish he didn’t overload the term "Continuous Integration” to also mean "workflow without feature branches". It will surely cause a lot of confusion to those who aren’t fully down with the concepts already. I can already foresee a small startup where the CTO-by-confidence/coincidence and one of the "senior" devs are having extremely heated circular arguments about the pros and cons of CI, not even talking about the same thing. OPs "trunk-based development" seems like a more suitable term for what they’re describing.
- couchand 5y ago
  I'm not sure what distinction you're drawing. Continuous integration is entirely synonymous with trunk-based development. It's true that as an industry we've overloaded the term Continuous Integration to mean build servers running automated tests (I just did so in a comment in this thread!), but that's where the overload is.
  - deleted 5y ago
    [deleted]
  - 3np 5y ago
    The overload is the absence of feature-branches and PR/MRs with review. A git-flow-like model and CI are fully compatible. OPs model is explicitly incompatible.
hatchnyc 5y ago
I have been doing this for years. I have my team push changes directly to master (they have the option to use a feature branch and code review if they feel it is necessary of course). Once per release cycle, we have meet, I put the diff off all changes since the last release and we review every change going out as a group, we talk about what is being changed, why, and people can explain their changes. Sometimes something needs to be fixed, and we can fix it right there. - The quality of these reviews is better than any code review I’ve seen on feature branch reviews - We review the whole collection of changes going out, on rare occasions when two changes conflict, you catch these - Everyone keeps up to date with what is changing in the code base and why - If for whatever reason there’s an issue after the deployment, it’s easier to fix because everyone has fresh in their head what has changed in this release
- lgunsch 5y ago
  Does the team use a mono-repo? Or have multiple repos and track changes for that week to review? I'm very curious. Curious enough to even try it.
  - hatchnyc 5y ago
    Yes, actually there are two small additional repos, but we only very occasionally change and review. I think it would be a bit more difficult with many repos. Perhaps you could make it work, but it’s quite convenient to pull up everything in a single diff in an IDE. On the other hand, I usually treat a repository as a “deployable unit“ more or less, so I guess releases would be scoped to a repository as well. Also, I’ve done this with smaller teams, 5-6 people. If you had three or four times as many that might make for a long meeting.
- apineda 5y ago
  How often are your release cycles? Wouldn't these group reviews take a lot of time (half day)?
  - hatchnyc 5y ago
    Usually takes about two hours every two weeks, but I figure we’d have lost that time in offline code reviews anyway. I’ve had people point out that I have a small team. They have a point, I’m not sure how this would scale to a large team—but maybe if a team is too large to review their code together they could be split into smaller groups. Also, I think these help replace what might have been other meetings as well. Sometimes product people like to sit in and listen so they know exactly what's being released--I don't think they'd ever participate or get much value from a normal code review.
redis_mlc 5y ago
This is based on 20+ years of experience with committing to master as directly as possible. In the cvs/svn age, what was found at the SV companies I worked at was that the developers we had were not smart enough to work on per-developer branches and merge back correctly. After several outages and about a month of soul-searching at each place, we decided that committing to master and running a test suite caused less problems. Everybody was happy and productive. In the git age, I've noticed two patterns that are master-related: 1) For docs and devops, committing to master in github works fine since there's usually just one or two committers to each repo. It also lets you use the UI edit button for docs. 2) For application development, very short-lived branches for patches works fine. Note that in the current interview environment, interviewers will not accept the above explanations and will literally lose their minds and insult you - because IT is a fad-driven world. (They assume every repo commit is for a major application with dozens of developers.) Yet it worked fine for me for 20+ years, longer than most of their careers.
bcoughlan 5y ago
I honestly don't encounter non-trivial merge conflicts in practice on a team of 5 developers. Our repos are scoped roughly to be team-sized so the velocity is low enough to know what everyone is working on. I guess some of this advice applies better to repos where a large number of people are working on it. I couldn't imagine giving up the quality gate factor of PRs. Carving out the time to dissect changes catches so many bugs (although it can be received harshly sometimes compared to face to face). Also pushing to master vs. long lived feature braches is a false dichotomy. You can have small PRs on short-lived branches that may not be a complete feature but can be merged without making the main branch unreleasable. There is also the political factor to consider in companies where product and sales people control the selected work items. Once something is in a working state there is pressure to move on to the next thing. Fighting for quality before it is in a publishable state is a devs best defence against later rework. The CD community is overly obsessed with velocity. Of course removing obstacles can lead to a smoother faster workflow. Take it to the extreme and it becomes a dopamine hit activity, the goal is to merge changes fast and we become unable to take the time to think deeply and reflect since it is clear that we are valued by our rate of commits over smart decisions.
- xupybd 5y ago
  "Once something is in a working state there is pressure to move on to the next thing. Fighting for quality before it is in a publishable state is a devs best defence against later rework." This insight is true and depressing. It's not the best way to focus effort. You fight for quality on a new feature that may or may not get use. By the time you know how well received the feature is it's too late to allocate resources to improving the code quality. So if you don't get it right the first time, this code might cost you months of wasted time when trying to make changes to it in the future. It would be far better if time was allocated to going back and figuring out what parts of the code are causing problems and going back and fixing them. Spending time removing the features that don't get used. How do you get the political buy in to do this? I have no idea.
  - twobitshifter 5y ago
    On the flip side, if you’ve ever been on the manager side this can drive you crazy. Something passes all the tests and you get the arguments about why it suboptimal for future usage by a reviewer. Often something we don’t know we’ll ever do and complete bike shedding. I favor getting the code in and then refactoring later based on yngni. Can that come back to bite you? Absolutely, but not as many times as you’d have been warned, so the cost benefit works out.
quickthrower2 5y ago
My favourite workflow is feature branches for a about 1-2 days work. If the feature takes longer then split it up into multiple code/code review/merge to master iterations. Use a feature flag if required. This keeps merge conflicts low and keeps screwing up master to a minimum.
- shearnie 5y ago
  This is what we do also. Monster PR's get glancing reviews and are risky and diruptive to everyone else's feature branches when they merge latest. So the smaller the better. Quicker integrations. Less mess. Less risk.
- usrusr 5y ago
  That sounds nice, where I work there's a deeply ingrained problem of long lived feature branches. In that situation every little nudge towards earlier merges is valuable, so we call that merge target branch "develop", only to be less intimidating. Master formally exists, but it's just a dead bookmark pointing at the tag of the latest release, existing only to make develop appear more inviting.
plasma 5y ago
My workflow personally and in teams: 1) Consider main branch deployable any time, so don’t push changes that can’t be deployed 2) You can commit to main/master if it’s a reasonably small change not needing review 3) PRs uses for more complex changes, easier to review 4) Deploys off main branch 5) Tests run on all branches Works well 99% of the time, can fall down when a large chain is queued for deploy (just merged) and someone wants to push a minor change but now it’s got to be everything (unless you revert temporarily).
- seattle_spring 5y ago
  > 2) You can commit to main/master if it’s a reasonably small change not needing review I earnestly do not believe there exists any change small enough to not need a review.
  - couchand 5y ago
    CI should be sufficient review for most code changes. Manually review architecture and design, but automatically verify implementation.
    - GauntletWizard 5y ago
      The review should be perfunctory, but a second set of eyes is always needed
      - maximilianroos 5y ago
        That seems inherently contradictory? Particularly if you put some cost on a) not being able to merge and build on the feature in another branch, b) disturbing someone to do something perfunctory
        GauntletWizard 5y ago
        It is an inherent contraction. Having good testing and fast flow is a contradiction. The advantage of a second set of eyes is simply a sanity check... from a second pair of eyes. It's all a balancing act, but it does prevent a large set of failure classes that are basically "This dude went crazy"
excitednumber 5y ago
Doesn’t this article need a trigger warning
ncmncm 5y ago
This looks to me like you have just renamed master, and only the release manager or somebody gets to push to that one. Or, rather, cherry-pick to it.
forgingahead 5y ago
Personal preferences: 1. For solo devs/founders, push directly to master. Iterate quickly to serve customers, focus on growth and being "not-dead" by default.[0] 2. The moment you have a 2nd dev working (most likely because you have some sense of product-market fit and some revenue growth), then create feature PRs off of master. Review apps on each feature branch (Heroku supports this easily). 3. 3-5 devs: Have a "develop" branch and PRs go into "develop". "develop" deploys to a staging app, which is tested, and if all is well, "develop" can be merged into master which deploys to prod. 4. > 5 devs: Then you can use the full Gitflow model, with develop/releases/master splits in your branches. I find that the above works well to find the nice balance between productivity and risk-management. This also works nicely whether you are a consultant/services company working project by project with a client, or whether you're building a product startup. Doing the full Gitflow model as a solo dev is unproductive, and committing to master with a larger team is asking for disaster, especially if your app is critical to your customer business needs. [0]: http://www.paulgraham.com/aord.html http://www.paulgraham.com/aord.html
efficax 5y ago
Nuts! 600 engineers push to the repo i work on everyday. Good luck handling that with trunk based development
- msftie 5y ago
  Well now you have to tell us why and how 600 people are sharing a repo! That’s a lot. Granted at Microsoft thousands of people worked in the same codebase. But in my recent experience we’ve generally been working on a repo per project, which typically maps to a small team.
  - seattle_spring 5y ago
    > Well now you have to tell us why and how 600 people are sharing a repo Any answer you could possibly get from this question will eventually boil down to project repos vs monorepo. There are pros and cons to each, which are more or less meaningless depending on the amount of developers working in parallel.
    - msftie 5y ago
      I figured as much, and just wondered about the challenges of trunk based development with a large contributor pool. If so many teams are working in different subtrees of the monorepo, it seems like trunk based development could be just fine.
- charcircuit 5y ago
  What's the issue with it? I imagine the architecture of the code base matters significantly in how it can scale.
denvaar 5y ago
> With Trunk-Based Development (or Continuous Integration), developers are encouraged to push their code to the main branch frequently. Not just when a feature is finished, but every time there is new meaningful working code. I'm a big fan of checking in small amounts of meaningful code and utilizing feature flags when necessary. Like others have mentioned tight feedback loops are key. I have never felt like I needed to ditch branches though. It honestly sounds like a nightmare to commit directly to main in a team setting. On the other hand, maybe it would force developers to think twice about what they're changing.
timemachine 5y ago
I’ve been reading Dave Farley’s new book, “Modern Software Engineering” He has a few rants related to GitFlow vs Trunk development. Many of his points agree with OP in that merging is a big pain point in GitFlow. I’ve both used strategies on many different projects. Regardless of the development strategy I’ve seen nasty merge parties. The way to avoid those merges is to reduce your batch size and keep your un-integrated changes to a narrow scope. If you need to make changes out side of that scope. Stash your work; create a new branch; make the change; let your teammates know what you did; before going back to your other branch. You can then integrate that fix back to your local copy. But the important thing is that your team mates can also sync that one off change to their local copy too. The worse thing is when two developers find the same bug and fix it simultaneously in different commits. Trunk-based and GitFlow both have this problem. Stick to the scope of work that was coordinated in your standup meeting for the day and let your coworkers know if you need to go outside of that scope. Be conscientious. (Complete aside: Try to do trunk based development in a Perforce code base and you will learn a lot about reducing your the batch size of your commits and communicating the scope of code changes. Perforce requires you to be team oriented when developing)
kgeist 5y ago
We have 50 devs (including inexperienced juniors) and 1-3 releases per day from different teams, pushing untested changes to master would be problematic because one team (maybe with a less important feature) could break/delay everything for the others, and go figure who broke what (it used to happen with trunk-based development, hence the switch to Git flow). Sometimes there is a problem and we need to make a hotfix and redeploy, but how do you do it when 5 teams just pushed random untested crap to main? Our feature branches don't diverge much, each team/release has its ows staging environment (around 15 right now IIRC), and it's a rule to refresh them with new changes from master every day. Yes sometimes there's conflicts when two features are to be released on the same day (it wasn't caught during one of the "refreshings"), but it doesn't happen often because during PI planning we discuss possible interdependencies between teams/releases in advance to resolve problems long before merge, so those conflicts tend to be trivial. All features are required to be split into smaller subreleases which shouldn't take more than a week or two to make, so there isn't enough time for branch diverge anyway. And what happens when business requirements change and the feature is cancelled, do you unmerge all that? Sometimes priorities change and a very important client needs a feature to be released sooner, so we change the release plan accordingly, and how do you do it when everyone already pushed their untested, possibly broken stuff to master? It probably also depends on business needs, in our case we deal with statistics which drives our clients' decisions (who to fire or promote), lack of testing/review/unsupervised merges to master would be a disaster for our business.
GVRV 5y ago
I, for one, will never understand how Trunk Based Development (TBD) is considered "sane default" these days. The power of version control isn't just in a record of history, it's also in branching; and most often, I've noticed developers move to TBD because they don't understand the intricacies of their version control system and how to leverage it for a proper async parallel development workflow. You don't need to adopt GitFlow or another workflow verbatim, understand how you want to deliver software and work within the team so that you can adapt it to your requirements. The points made by the author are confusing to me. Quality Assurance was under-resourced. They had a huge job of checking and re-checking every feature to verify that there were no regressions. After merging a feature into develop, they had to check again to see if there were any new issues that were introduced by bad merges or conflicting feature requirements. If this was the case and they were fine with QA testing just the `master` branch after moving to TBD, maybe QA shouldn't have been testing their feature branches in the original workflow. Just use branches for proper code review and then QA only steps in after the branch is merged? The threshold of conflict was amplified by the time that passed between when a branch was cut from develop to the time when it was merged back. For bigger features, a branch's life could last one or even two weeks. The more time that passed, the greater divergence there would be from the other code. Feature branches should be short-lived, as atomic as possible. And if you're working on a big feature, you have to update your branch frequently with upstream changes. Merges of Doom only happen if you're not following version control best practices. This also requires a little bit of planning upfront (especially if you're working in parallel on a single feature), but forcing that thought is a good thing. It also seems like they attributed moving to Kanban as only being possible due to the move to TBD, but it's not like it's impossible with a proper branching workflow. So, the author made the switch to TBD and attributed it to increased velocity and better _overall morale_, but I think they're just enjoying the seemingly greener grass across the fence for a while.
- huetius 5y ago
  I agree with almost all of your post. I would only offer that you consider that the most important task of a developer in most organizations is to eliminate complexity beyond a bare minimum. Having a simple, safe, and predictable version control workflow is within that purview, even if it means most people do not use or remain ignorant of the full power of the tools at their disposal. All other points stand, and the simple workflow doesn’t have to be TBD.
deathanatos 5y ago
GitFlow, broken by design. 1. Do all your development and testing on one branch (develop). Vet its HEAD commit. 2. Once that's good to go, merge that into a different branch (master), and deploy a completely different commit to production! The vast majority of devs that I have spoken to believe that, after merging develop with master that develop == master, and have no controls in place to actually guarantee that. & in case you're thinking "I thought they were?" * merge to master (master) # what shipped |\ | * PR #184 (develop) # what was used by devs Different commits == potentially different trees. (And, in a large enough company with enough commits and time, "potentially" drifts towards certainty.) A good shop will deploy master somewhere sane like a QA env first… but still. It's brain-dead. Truck-based development is simpler, less often crashes the minds of devs who can't be arsed to learn git, and what you test/dev == what you deploy.
- zoomablemind 5y ago
  > ...Truck-based development is simpler... Trunk-based, that is. For a moment I confused the 'truck' proposition with the 'bus factor' reasoning [1], which of course, is from a different context. [1]: https://en.wikipedia.org/wiki/Bus_factor https://en.wikipedia.org/wiki/Bus_factor
- lilyball 5y ago
  As I understand it, in Git Flow, the result of the merge into main should be identical to the tree that was in develop, meaning there's no difference, it's just a matter of maintaining the appropriate history. Under normal conditions this should be trivially true. Develop forked off from main, no commits on main, therefore the merge back into main has no changes on the first-parent side and is equal to the second parent. The two ways to screw this up are: 1. You did a hotfix release and didn't merge the hotfix back into develop properly, or 2. You made changes on a release branch and forgot to merge that back into develop. In both cases, the fact that the results of merging develop into main produced a different tree than what was in develop (or rather, in the current release branch) should be a signal that you screwed up somewhere. Having said all that, I'm willing to bet that 98% of places that do Git Flow don't actually have any checks to ensure that the tree on main is identical to the tree on the release branch. And this is because the model is actually rather complex, most people don't understand Git properly, and the accessible documentation about Git Flow doesn't even bother to mention the possibility that the merge into main could produce a different tree.
johnhowardstein 5y ago
It can be fine, but the moment you need to roll back a change, you'll wish for feature branches. If you reach for cherry-pick, you've failed. Anyway, branches and merge-requests take almost zero extra time and effort above pushing to master, with the added benefit that you can test and deploy a branch from your CI/CD separately.
galaxyLogic 5y ago
I think there is something wrong with the basic idea of git and similar systems, which is that anybody can just change anything because they work in parallel meaning changes they make can conflict. Therefore we have conflict resolution, but shouldn't we try to make it less likely for conflicts to arise in the first place? Instead I think we need module-ownership and tools supporting that. At any given time every module should be assigned to an owner-programmer or small owner-team. Only they can change their modules. Others can request changes, or create their own copy of that module to modify, but not modify code owned by someone else willy-nilly. If programmers cannot modify modules owned by others there will be no merge-conflicts, right? I wonder why this kind of code-ownership approach isn't more widely practiced and why there doesn't seem to be much tool-support for it?
- albertopv 5y ago
  So, back to Visual SourceSafe?
  - galaxyLogic 5y ago
    My experience with VSS was pretty good but I'm thinking of something more radical. VSS etc. allow you to temporarily lock a file so no-one else can edit it while you have it locked out. I'm thinking that instead of temporary lock-outs we should have persistent module ownership. Only the owners can modify the code of their modules, and perhaps temporarily grant commit-rights for their module to others. Preferably the owner should have a deputy or two who would take over if the owner gets sick. Super-user can grant and take away ownership to any module. Super-user should have a deputy or two as well. Here's the metaphor: In New York City and all cities you have traffic lights and there are parking rules, and pedestrian crossings and bike-lanes and some streets are one-directional. What would happen if all the rules and traffic-lights were taken out? You could still get from A to B, but probably on average much slower because you would get stuck in traffic-jams much more often. Perhaps counter-intuitively creating rules which restrict how you can drive and park your car do not make you move slower, they make traffic more efficient. Traffic jam is like a merge-conflict. Two cars merging on to the same narrow street from opposite directions. One of them has to back out. And then so do all cars behind it. Not fun. Git etc. are a bit like city without traffic lights, one-directional streets and parking restrictions. Anybody can do whatever they want, branch and branch and merge and resolve conflicts. It gives you the impression of great flexibility and freedom, but so would a city without traffic rules. Yet all cities have realized they need to restrict what people do on their streets, to eliminate "traffic-merge-conflicts". Now naturally you can say that your project has rules in place as to who can modify what code and when (do you?). But I think there should be tool-support for that and persistent module-ownership instead of "module communism" where everybody owns everything. When everybody owns everything no-one is responsible. I'm not a big rules-guy but I think traffic lights do more good than bad.
albertopv 5y ago
In my experience you can't use feature flags for everything. Also, if you are having so much trouble merging because of lot of changes and commits, that's a smell of a bad design or a project too big that must be broken up.
sam_lowry_ 5y ago
Cancel him, he should push to main.
bacro 5y ago
I think GitFlow is helpful for mobile apps development. We cannot get into production every time we want, we are at mercy of app store review process. Also, if there is a bug in production it is very difficult to get a fix for all users once a version is deployed. I wonder if anyone uses successfully Trunk-based development for mobile apps development and if you could share your experience against GitFlow (pros and cons)
- agsnu 5y ago
  Generally the app store review process is much less of a problem than it used to be, with happy path being 24-48 hours rather than 1 week+. Of course everything is context dependent (team size, codebase size, testing maturity, frequency of release), but I guess we found GitFlow seemed to be better aligned with modelling the "versioned software" approach - i.e. with releases determined by features, everybody knows/cares about what is in version 2.1 etc, you might have multiple releases all on the go at the same time with development happening for 2.1.1 bug fixes, 2.2 minor features, and maybe master has moved on to stuff that will ship in 3.x etc. Definitely with less frequent releases (App Store Review taking 1+ weeks and being unpredictable probably had greater influence on release frequency) and a large team, we ended up with releases feeling quite painful - there was a lot of pressure to land features close to the deadline because the next release might not be for a while, meaning there was more churn on release branches as people tried to stabilise things that were borderline "ready". Git Flow's release branching model added overhead - people might forget to merge back to develop, deal with conflicts or semantic brokenness if you had different targeted fixes on the release branch to mainline development (e.g. let's just disable this functionality for now on release branch and push it to the next version, fix forward on develop, merge release branch back to develop -> now it's disabled there too, have to unwind). We switched many years ago (probably after reading https://barro.github.io/2016/02/a-succesful-git-branching-model-considered-harmful/ https://barro.github.io/2016/02/a-succesful-git-branching-mo... ) to a time-based continuous release model - periodically cut a release from master (there are lots of teams shipping mobile apps on a monthly, 2 weekly or even weekly cadence these days), with trunk-based development and cactus branching for releases. If we need to hotfix just need to go to the most recent release tag/branch and ship a fix based off that point, no merging back between branches. If you need to fix a bug in the release, it's on you to make sure an appropriate fix is applied in both places, through e.g. cherry-picking, no need to worry about merging anything back anywhere. Together with some other actions (cultural focus on reducing post-branch churn and investment in testing capability to gain confidence in release candidates faster, set in the context of a goal to increase release cadence; increased usage of feature flags; etc), we ended up significantly improving release frequency & predictability, which reduced the time for stakeholders to get changes shipped and visible to users from when they were "done" (especially valuable for small changes, of course). Nobody ever really understood the utility of the "production" branch from GitFlow. tl;dr GitFlow seemed to add overhead with little value; we switched to trunk based development and didn't look back.
ay 5y ago
We made something quite similar work rather well at $work. However, there is a somewhat subtle trick. The staging/prod candidate is built from master/main branch 21 days old, plus potentially some (few) cherrypicks. This allows to fearlessly commit to master, and then the “T-21” has 21 days of completely predictable future, that can be changed by doing cherrypicks. So we can run intense testing on “T-0” aka main, find any issues and add them to be cherry-picked into the T-21. The bonus is that when the fixes “arrive” to T-21, the cherry-picks become void and stop being applied. Thus, absent the bugs, the staging/prod code automatically converges to main/master over time. And yes, we do the reviews of the commits that go into master - but from the T-21 point of view they are 21 days in the future! So there is ample time for any reaction. Would folks be interested in a more detailed write up ?
- acjacobson 5y ago
  I would be interested in learning more - this sounds completely novel to any way I have worked before.
  - ay 5y ago
    Ok I will post a reply here + do “show hn” or some such…
- gulbrandr 5y ago
  > Would folks be interested in a more detailed write up ? Yes please.
  - ay 5y ago
    Ok I update the reply here when I have something to share !
claytonaalves 5y ago
Well here is the workflow we use here at our company: Branch feature from master/main. Keep feature branch in sync with master/main (merging from master/main everyday). This minimizes conflicts when we finally get to merge feature branch to master. Any refactor made in feature branch may be cherrypicked to master any time. This reduces differences between feature branch and master/main, resulting in less code to review.
alkonaut 5y ago
Allowing direct push to master is fine so long as you 1) have a small team so it's not very congested 2) have a short enough build/test cycle that you can enforce running tests locally. As soon as your test suite grows to the point that users aren't likely to run ALL tests before pushing new changes, you can't. Also, if you want to have code reviews at all, you want them pre-merge. If you have irreversible changes (For example: you have code that writes a serialized format and once you review the code that changes it, people have already used the code, even if only in testing/staging - but you now have data serialized in the bad wire format that you may need to be able to correct, and the correction code after review will need to be maintained forever unless you accept the loss of the data). I think: commit directly to master to scaffold and iterate quickly on a greenfield project with 1-3 devs. Then start doing feature branches and PR's to master.
- Gareth321 5y ago
  Exactly this. One particularly haughty CEO I worked for came to this exact same revelation. "But what about bugs?" "Just tell the developers not to add any bugs!" I'm not even kidding. That's what he said. Continuous release is fine for applications which aren't mission critical AND you have users who are accepting of an increase in bugs. Back in the real world, it's just not a good idea. We're all trying to calibrate the right balance between QA and output, and there are many ways to find the right balance. I remain unconvinced that "YOLO" is ever the right process.
  - deleted 5y ago
    [deleted]
  - teknopaul 5y ago
    Trunk dev and continuous releasing every push to trunk is not something I have heard suggested. If you want CD from some particular branch it would be silly to suggest the active dev branch. Trunk dev is about how and when new code is merged. It says nothing about when it is released. It's incorrect to think that any people doing trunk dev deploy from trunk willy nilly.
mentos 5y ago
What is wrapping all new features in a #ifdef NEW_FEATURE //code #endif called? My process over the last 7 years working in game dev has slowly evolved to this where I'll ask a contractor to implement a feature on the main branch but make sure that it is completely toggleable via a #define NEW_FEATURE I thought I was pretty clever until I read an article on HN a few years ago that this is exactly what Google does lol
- kkirsche 5y ago
  I think you are describing build-time feature flags or feature toggles. Similarly, this type of thing can be managed at runtime via integrations or implementing a database backed version yourself.
- erezsh 5y ago
  Yeah, and let me tell you, it's an absolute nightmare to get to work when you need two features that weren't written to compile at the same time. I'm not saying it's necessarily a bad idea. Just that it isn't a replacement for abstraction. If you have more than a couple of #ifdef blocks per feature, you might be setting yourself up for future pain.
- jan_g 5y ago
  This approach works surprisingly well for most features, but care should be taken to remove those 'ifs' once the feature is stable/complete. Or, when implemented cleanly, they could also be left in place and used as long lived feature toggles that can be driven by configs (or user settings, ...). The hairy part is when the new feature or change cuts across different parts of the code and then, given enough time, those 'ifs' pollute the codebase. Solution here is to be mindful of how the system grows, keeping things isolated and (perhaps counter-intuitive) favor duplication over shared logic/libraries. Regardless if it's about one big thing (i.e. monolith) or multiple small things (different projects even).
deleted 5y ago
[deleted]
dpark 5y ago
Ah, so what they really learned is that Gitflow is garbage. Yeah, long lived working branches are an enormous pain in the ass. I’m not convinced they are ever worth the hassle. Certainly for small teams they are not. I am highly doubtful of the “commit first, test and code review later” model, though. Maybe this works for a very small, tight team who are all very highly skilled and care a ton about engineering quality. But this model can fall apart quickly. Bad code blocks the whole team and everyone pays the cost. You eventually end up with someone babysitting the build and test process to keep it moving and then some bright mind asks why you don’t put this stuff before checkin.
deleted 5y ago
[deleted]
lolive 5y ago
Am i the only one who worked on a project with multiple feature branches in parallel by different subteams, with non trivial code merge conflicts (and data model conflicts) and a upper management that constantly reshuffle the delivery dates for these features. Plus maintenance branches to support previous versions with a hotfix branch (and also backporting some newer features to older version). That might be unusual, but even GitFlow in that case is very poor handling those kind of deliveries.
sirwhinesalot 5y ago
I really feel like most of the processes we have in software development are unnecessary if not detrimental. CI/CD is the best and most impactful thing in years and optimizing for it is the best you can do IMO. Anything that gets in the way of CI/CD you want to avoid. Anything that helps with CI/CD you want more of it. Using that as a guide post, trunk-based development is better than feature branches. Kanban is better than Scrum. Monoliths vs Microservices is less clear cut, depends on how costly the monolith is to build vs how annoying all the services are to deploy.

we use a cherry pick model all developers develop features, we then cherry pick what features we want in the next release, do a release branch, merge in features, test (and if necessary, fix) then merge into main.

deleted 5y ago

[deleted]

Thaxll 5y ago

There are people pushing to master without review? I mean it's a common thing?

GuestHNUser 5y ago
This new to me as well. I am very curious what situations people are in where they don't/can't wait for a PR into master. If you reading this do this, I'd be interested in hearing what the benefit of pushing to master is for you. Is it purely for velocity? And if so why is velocity that important?
- GuestHNUser 5y ago
  Coming back to my comment having finished the article in full. I wonder how the author's code base will develop using a weekly reflection on the code committed instead of PRs. I'm not sure how well their process will handle someone's poor design decisions. My experience has been that poor decisions tend to stay in code once they are there, and especially as more things become dependent on those decisions. Not that reviews will catch everything, but that second set of eyes can go a long way.
zebraflask 5y ago
Yes, there are! Familiar with this approach, pretty common with greenfield development. After causing a few outages and issues that you have to hear about, one learns fairly quickly not to write code that needs endless PR reviews.
- GuestHNUser 5y ago
  I'm curious to know how does your team avoid shooting itself in the foot with unreviewed code? As in, is there some process that keeps unmaintainable elements from sneaking in?
  - zebraflask 5y ago
    Trust people to know what they're doing? A lot of PR reviews tend to devolve into almost anything but solving the issue at hand.
    - GuestHNUser 5y ago
      Fair enough. My experience with reviews has been they are quick quality checks, and at least for teams I have been on, were beneficial. I can see how they could become more of a ritual process than a beneficial one though, especially after seeing some other comments in the thread.
    - tetraca 5y ago
      Surprising. We do a PR and review for nearly every block of work we commit. Most pass without comment. The ones that have comments usually have good suggestions, or actually catch an error.

secondcoming 5y ago

> The more time that passes from the point a branch is broken off from develop to the time it is merged back in, the more opportunity there is for other branches to diverge in drastic ways. When a release is made you should merge those changes into other branches too. Then merge-conflicts or behavioural changes are found, and fixed, earlier.

notreallyserio 5y ago
This can be tricky. I worked somewhere that did this and it ends up forcing some devs to deal with multiple conflicts a week while they're working on a branch, when they could instead keep an eye on what is going on in main and deal with the conflicts on their own schedule.
- nicoburns 5y ago
  The problem here seems to be devs working on a branch for a whole week. If you break down tasks smaller than this and merge them separately then you end up with far fewer merge conflicts.
- secondcoming 5y ago
  I'd put that under the 'Too Bad' category. If a branch is so far behind reality/master then any testing done on it is quite likely to be invalid, and it won't have any hotfixes that were deployed in the meantime. YMMV

gsk22 5y ago

Isn't there a middle ground here? If you have CI/CD, you can eliminate release branches by just making PRs directly into main. The idea of post-merge code review is horrifying to me. I guess this is the "move fast and break things" attitude in action. Deploy now, catch bugs...sometime, maybe (TM)!

sdfaksdfjalj 5y ago
This is the way it's done at many major tech companies. Each individual commit is reviewed then merged directly into main/master. I've never used them but the thought of feature branches seems absurd versus simply merging small changes. Very rarely can a "feature" not be broken down into small self-contained changes.
Starlevel001 5y ago
> Deploy now, catch bugs...sometime, maybe (TM)! Given the state of literally all software I use, this seems to be the default behaviour.
Ozzie_osman 5y ago
I've found that to be a great middle ground on most teams. If you want to stretch it a little more, you could selectively do post-merge reviews for things that might be low risk (ie a UI change that's behind a feature flag that only your team sees), and keep riskier changes (like a big refactor, a data migration, etc) on the pre-merge review flow.
- deckiedan 5y ago
  We have pre merge review - but for trivial stuff, text / label changes, etc, put a "tiny" tag on the PR, and if urgent paste it to the slack channel asking for a glance and nod review... At least having another pair of eyes check text-only changes has caught so many typos, etc, and only takes a couple minutes.
PontiacParade 5y ago
I agree with the middle ground comment. That is how we tend to do things where I work. We have a modified GitFlow: main: is the source of truth and is what is in production. develop: is the constantly moving branch we make PRs against. You can commit directly here which is discouraged but it isn't a hard rule. ticket: is a branch for each JIRA ticket not each feature. release: We don't make these and just use tags on main. Each "release" is a merge from develop to main and that gets deployed. hot fix: These are made against main and merged back to develop when they are used. It is rare enough I have to look up our "official" procedure. With that we can easily use PRs, release code in small hidden chunks, do code reviews, etc. Seems like the big win they got was releasing small hidden chunks of a feature and deploying it to staging. They also gave up some nice things as well like code review before merging.

tmerr 5y ago

I absolutely believe in this after working both ways. With feature branches I would waste time rebasing, resolving merge conflicts, to then come up with the perfect pile of commit messages before sharing with the team. Committing to master forces early collaboration & tighter feedback loops. Edit: I don't know about post merge code reviews, that seems like a risky idea to me at scale

montroser 5y ago
Early collaboration and tighter feedback loops are key. But you can make that all happen as a matter of culture, without needing committing to master as a forcing function.

dan-robertson 5y ago

Maybe it’s fine to put code reviews later in the process but I think I’d be worried about the frequency of changes that break the build. Is there some way to avoid that? Debugging failures only to discover that the build was broken (in the sense of a test failure, but also could be a JavaScript syntax error I suppose) underneath you is surely bad for velocity. Certainly I think there are times when having multiple people pushing straight to a feature branch is good but I’d worry about a codebase where features touch so much of the same stuff that merges are so painful. But maybe OP’s environment is just alien to me and there are reasons for both of these things.

njovin 5y ago
If you're using something that supports it (like Github) a good middle-ground would be auto-merging PRs. Branch from main -> code -> PR w/ automerge -> move on to the next task. This would give you speedy merges to main branch with the protection of the per-PR build checks.
- dan-robertson 5y ago
  Obviously I don’t know much about the normal git process (the one I use is a little unusual) but what is the alternative to auto-merging PRs? Do people need to manually hit merge once tests are automatically run and code reviewed or is there some smaller set of people with the ability to merge? Do people ever do code review after a merge to make sure it worked properly?

3np 5y ago

> This all came together for me when I was catching up on YouTube and stumbled across Dave Farley's video Continuous Integration vs Feature Branch Workflow I really wish he didn’t overload the term "Continuous Integration” to also mean "workflow without feature branches". It will surely cause a lot of confusion to those who aren’t fully down with the concepts already. I can already foresee a small startup where the CTO-by-confidence/coincidence and one of the "senior" devs are having extremely heated circular arguments about the pros and cons of CI, not even talking about the same thing. OPs "trunk-based development" seems like a more suitable term for what they’re describing.

couchand 5y ago
I'm not sure what distinction you're drawing. Continuous integration is entirely synonymous with trunk-based development. It's true that as an industry we've overloaded the term Continuous Integration to mean build servers running automated tests (I just did so in a comment in this thread!), but that's where the overload is.
- deleted 5y ago
  [deleted]
- 3np 5y ago
  The overload is the absence of feature-branches and PR/MRs with review. A git-flow-like model and CI are fully compatible. OPs model is explicitly incompatible.

hatchnyc 5y ago

I have been doing this for years. I have my team push changes directly to master (they have the option to use a feature branch and code review if they feel it is necessary of course). Once per release cycle, we have meet, I put the diff off all changes since the last release and we review every change going out as a group, we talk about what is being changed, why, and people can explain their changes. Sometimes something needs to be fixed, and we can fix it right there. - The quality of these reviews is better than any code review I’ve seen on feature branch reviews - We review the whole collection of changes going out, on rare occasions when two changes conflict, you catch these - Everyone keeps up to date with what is changing in the code base and why - If for whatever reason there’s an issue after the deployment, it’s easier to fix because everyone has fresh in their head what has changed in this release

lgunsch 5y ago
Does the team use a mono-repo? Or have multiple repos and track changes for that week to review? I'm very curious. Curious enough to even try it.
- hatchnyc 5y ago
  Yes, actually there are two small additional repos, but we only very occasionally change and review. I think it would be a bit more difficult with many repos. Perhaps you could make it work, but it’s quite convenient to pull up everything in a single diff in an IDE. On the other hand, I usually treat a repository as a “deployable unit“ more or less, so I guess releases would be scoped to a repository as well. Also, I’ve done this with smaller teams, 5-6 people. If you had three or four times as many that might make for a long meeting.
apineda 5y ago
How often are your release cycles? Wouldn't these group reviews take a lot of time (half day)?
- hatchnyc 5y ago
  Usually takes about two hours every two weeks, but I figure we’d have lost that time in offline code reviews anyway. I’ve had people point out that I have a small team. They have a point, I’m not sure how this would scale to a large team—but maybe if a team is too large to review their code together they could be split into smaller groups. Also, I think these help replace what might have been other meetings as well. Sometimes product people like to sit in and listen so they know exactly what's being released--I don't think they'd ever participate or get much value from a normal code review.

redis_mlc 5y ago

This is based on 20+ years of experience with committing to master as directly as possible. In the cvs/svn age, what was found at the SV companies I worked at was that the developers we had were not smart enough to work on per-developer branches and merge back correctly. After several outages and about a month of soul-searching at each place, we decided that committing to master and running a test suite caused less problems. Everybody was happy and productive. In the git age, I've noticed two patterns that are master-related: 1) For docs and devops, committing to master in github works fine since there's usually just one or two committers to each repo. It also lets you use the UI edit button for docs. 2) For application development, very short-lived branches for patches works fine. Note that in the current interview environment, interviewers will not accept the above explanations and will literally lose their minds and insult you - because IT is a fad-driven world. (They assume every repo commit is for a major application with dozens of developers.) Yet it worked fine for me for 20+ years, longer than most of their careers.

bcoughlan 5y ago

I honestly don't encounter non-trivial merge conflicts in practice on a team of 5 developers. Our repos are scoped roughly to be team-sized so the velocity is low enough to know what everyone is working on. I guess some of this advice applies better to repos where a large number of people are working on it. I couldn't imagine giving up the quality gate factor of PRs. Carving out the time to dissect changes catches so many bugs (although it can be received harshly sometimes compared to face to face). Also pushing to master vs. long lived feature braches is a false dichotomy. You can have small PRs on short-lived branches that may not be a complete feature but can be merged without making the main branch unreleasable. There is also the political factor to consider in companies where product and sales people control the selected work items. Once something is in a working state there is pressure to move on to the next thing. Fighting for quality before it is in a publishable state is a devs best defence against later rework. The CD community is overly obsessed with velocity. Of course removing obstacles can lead to a smoother faster workflow. Take it to the extreme and it becomes a dopamine hit activity, the goal is to merge changes fast and we become unable to take the time to think deeply and reflect since it is clear that we are valued by our rate of commits over smart decisions.

xupybd 5y ago
"Once something is in a working state there is pressure to move on to the next thing. Fighting for quality before it is in a publishable state is a devs best defence against later rework." This insight is true and depressing. It's not the best way to focus effort. You fight for quality on a new feature that may or may not get use. By the time you know how well received the feature is it's too late to allocate resources to improving the code quality. So if you don't get it right the first time, this code might cost you months of wasted time when trying to make changes to it in the future. It would be far better if time was allocated to going back and figuring out what parts of the code are causing problems and going back and fixing them. Spending time removing the features that don't get used. How do you get the political buy in to do this? I have no idea.
- twobitshifter 5y ago
  On the flip side, if you’ve ever been on the manager side this can drive you crazy. Something passes all the tests and you get the arguments about why it suboptimal for future usage by a reviewer. Often something we don’t know we’ll ever do and complete bike shedding. I favor getting the code in and then refactoring later based on yngni. Can that come back to bite you? Absolutely, but not as many times as you’d have been warned, so the cost benefit works out.

quickthrower2 5y ago

My favourite workflow is feature branches for a about 1-2 days work. If the feature takes longer then split it up into multiple code/code review/merge to master iterations. Use a feature flag if required. This keeps merge conflicts low and keeps screwing up master to a minimum.

shearnie 5y ago
This is what we do also. Monster PR's get glancing reviews and are risky and diruptive to everyone else's feature branches when they merge latest. So the smaller the better. Quicker integrations. Less mess. Less risk.
usrusr 5y ago
That sounds nice, where I work there's a deeply ingrained problem of long lived feature branches. In that situation every little nudge towards earlier merges is valuable, so we call that merge target branch "develop", only to be less intimidating. Master formally exists, but it's just a dead bookmark pointing at the tag of the latest release, existing only to make develop appear more inviting.

plasma 5y ago

My workflow personally and in teams: 1) Consider main branch deployable any time, so don’t push changes that can’t be deployed 2) You can commit to main/master if it’s a reasonably small change not needing review 3) PRs uses for more complex changes, easier to review 4) Deploys off main branch 5) Tests run on all branches Works well 99% of the time, can fall down when a large chain is queued for deploy (just merged) and someone wants to push a minor change but now it’s got to be everything (unless you revert temporarily).

seattle_spring 5y ago
> 2) You can commit to main/master if it’s a reasonably small change not needing review I earnestly do not believe there exists any change small enough to not need a review.
- couchand 5y ago
  CI should be sufficient review for most code changes. Manually review architecture and design, but automatically verify implementation.
  - GauntletWizard 5y ago
    The review should be perfunctory, but a second set of eyes is always needed
    - maximilianroos 5y ago
      That seems inherently contradictory? Particularly if you put some cost on a) not being able to merge and build on the feature in another branch, b) disturbing someone to do something perfunctory
      - GauntletWizard 5y ago
        It is an inherent contraction. Having good testing and fast flow is a contradiction. The advantage of a second set of eyes is simply a sanity check... from a second pair of eyes. It's all a balancing act, but it does prevent a large set of failure classes that are basically "This dude went crazy"

excitednumber 5y ago

Doesn’t this article need a trigger warning

ncmncm 5y ago

This looks to me like you have just renamed master, and only the release manager or somebody gets to push to that one. Or, rather, cherry-pick to it.

forgingahead 5y ago

Personal preferences: 1. For solo devs/founders, push directly to master. Iterate quickly to serve customers, focus on growth and being "not-dead" by default.[0] 2. The moment you have a 2nd dev working (most likely because you have some sense of product-market fit and some revenue growth), then create feature PRs off of master. Review apps on each feature branch (Heroku supports this easily). 3. 3-5 devs: Have a "develop" branch and PRs go into "develop". "develop" deploys to a staging app, which is tested, and if all is well, "develop" can be merged into master which deploys to prod. 4. > 5 devs: Then you can use the full Gitflow model, with develop/releases/master splits in your branches. I find that the above works well to find the nice balance between productivity and risk-management. This also works nicely whether you are a consultant/services company working project by project with a client, or whether you're building a product startup. Doing the full Gitflow model as a solo dev is unproductive, and committing to master with a larger team is asking for disaster, especially if your app is critical to your customer business needs. [0]: http://www.paulgraham.com/aord.html http://www.paulgraham.com/aord.html

efficax 5y ago

Nuts! 600 engineers push to the repo i work on everyday. Good luck handling that with trunk based development

msftie 5y ago
Well now you have to tell us why and how 600 people are sharing a repo! That’s a lot. Granted at Microsoft thousands of people worked in the same codebase. But in my recent experience we’ve generally been working on a repo per project, which typically maps to a small team.
- seattle_spring 5y ago
  > Well now you have to tell us why and how 600 people are sharing a repo Any answer you could possibly get from this question will eventually boil down to project repos vs monorepo. There are pros and cons to each, which are more or less meaningless depending on the amount of developers working in parallel.
  - msftie 5y ago
    I figured as much, and just wondered about the challenges of trunk based development with a large contributor pool. If so many teams are working in different subtrees of the monorepo, it seems like trunk based development could be just fine.
charcircuit 5y ago
What's the issue with it? I imagine the architecture of the code base matters significantly in how it can scale.

denvaar 5y ago

> With Trunk-Based Development (or Continuous Integration), developers are encouraged to push their code to the main branch frequently. Not just when a feature is finished, but every time there is new meaningful working code. I'm a big fan of checking in small amounts of meaningful code and utilizing feature flags when necessary. Like others have mentioned tight feedback loops are key. I have never felt like I needed to ditch branches though. It honestly sounds like a nightmare to commit directly to main in a team setting. On the other hand, maybe it would force developers to think twice about what they're changing.

timemachine 5y ago

I’ve been reading Dave Farley’s new book, “Modern Software Engineering” He has a few rants related to GitFlow vs Trunk development. Many of his points agree with OP in that merging is a big pain point in GitFlow. I’ve both used strategies on many different projects. Regardless of the development strategy I’ve seen nasty merge parties. The way to avoid those merges is to reduce your batch size and keep your un-integrated changes to a narrow scope. If you need to make changes out side of that scope. Stash your work; create a new branch; make the change; let your teammates know what you did; before going back to your other branch. You can then integrate that fix back to your local copy. But the important thing is that your team mates can also sync that one off change to their local copy too. The worse thing is when two developers find the same bug and fix it simultaneously in different commits. Trunk-based and GitFlow both have this problem. Stick to the scope of work that was coordinated in your standup meeting for the day and let your coworkers know if you need to go outside of that scope. Be conscientious. (Complete aside: Try to do trunk based development in a Perforce code base and you will learn a lot about reducing your the batch size of your commits and communicating the scope of code changes. Perforce requires you to be team oriented when developing)

kgeist 5y ago

We have 50 devs (including inexperienced juniors) and 1-3 releases per day from different teams, pushing untested changes to master would be problematic because one team (maybe with a less important feature) could break/delay everything for the others, and go figure who broke what (it used to happen with trunk-based development, hence the switch to Git flow). Sometimes there is a problem and we need to make a hotfix and redeploy, but how do you do it when 5 teams just pushed random untested crap to main? Our feature branches don't diverge much, each team/release has its ows staging environment (around 15 right now IIRC), and it's a rule to refresh them with new changes from master every day. Yes sometimes there's conflicts when two features are to be released on the same day (it wasn't caught during one of the "refreshings"), but it doesn't happen often because during PI planning we discuss possible interdependencies between teams/releases in advance to resolve problems long before merge, so those conflicts tend to be trivial. All features are required to be split into smaller subreleases which shouldn't take more than a week or two to make, so there isn't enough time for branch diverge anyway. And what happens when business requirements change and the feature is cancelled, do you unmerge all that? Sometimes priorities change and a very important client needs a feature to be released sooner, so we change the release plan accordingly, and how do you do it when everyone already pushed their untested, possibly broken stuff to master? It probably also depends on business needs, in our case we deal with statistics which drives our clients' decisions (who to fire or promote), lack of testing/review/unsupervised merges to master would be a disaster for our business.

GVRV 5y ago

I, for one, will never understand how Trunk Based Development (TBD) is considered "sane default" these days. The power of version control isn't just in a record of history, it's also in branching; and most often, I've noticed developers move to TBD because they don't understand the intricacies of their version control system and how to leverage it for a proper async parallel development workflow. You don't need to adopt GitFlow or another workflow verbatim, understand how you want to deliver software and work within the team so that you can adapt it to your requirements. The points made by the author are confusing to me. Quality Assurance was under-resourced. They had a huge job of checking and re-checking every feature to verify that there were no regressions. After merging a feature into develop, they had to check again to see if there were any new issues that were introduced by bad merges or conflicting feature requirements. If this was the case and they were fine with QA testing just the `master` branch after moving to TBD, maybe QA shouldn't have been testing their feature branches in the original workflow. Just use branches for proper code review and then QA only steps in after the branch is merged? The threshold of conflict was amplified by the time that passed between when a branch was cut from develop to the time when it was merged back. For bigger features, a branch's life could last one or even two weeks. The more time that passed, the greater divergence there would be from the other code. Feature branches should be short-lived, as atomic as possible. And if you're working on a big feature, you have to update your branch frequently with upstream changes. Merges of Doom only happen if you're not following version control best practices. This also requires a little bit of planning upfront (especially if you're working in parallel on a single feature), but forcing that thought is a good thing. It also seems like they attributed moving to Kanban as only being possible due to the move to TBD, but it's not like it's impossible with a proper branching workflow. So, the author made the switch to TBD and attributed it to increased velocity and better _overall morale_, but I think they're just enjoying the seemingly greener grass across the fence for a while.

huetius 5y ago
I agree with almost all of your post. I would only offer that you consider that the most important task of a developer in most organizations is to eliminate complexity beyond a bare minimum. Having a simple, safe, and predictable version control workflow is within that purview, even if it means most people do not use or remain ignorant of the full power of the tools at their disposal. All other points stand, and the simple workflow doesn’t have to be TBD.

deathanatos 5y ago

GitFlow, broken by design. 1. Do all your development and testing on one branch (develop). Vet its HEAD commit. 2. Once that's good to go, merge that into a different branch (master), and deploy a completely different commit to production! The vast majority of devs that I have spoken to believe that, after merging develop with master that develop == master, and have no controls in place to actually guarantee that. & in case you're thinking "I thought they were?" * merge to master (master) # what shipped |\ | * PR #184 (develop) # what was used by devs Different commits == potentially different trees. (And, in a large enough company with enough commits and time, "potentially" drifts towards certainty.) A good shop will deploy master somewhere sane like a QA env first… but still. It's brain-dead. Truck-based development is simpler, less often crashes the minds of devs who can't be arsed to learn git, and what you test/dev == what you deploy.

zoomablemind 5y ago
> ...Truck-based development is simpler... Trunk-based, that is. For a moment I confused the 'truck' proposition with the 'bus factor' reasoning [1], which of course, is from a different context. [1]: https://en.wikipedia.org/wiki/Bus_factor https://en.wikipedia.org/wiki/Bus_factor
lilyball 5y ago
As I understand it, in Git Flow, the result of the merge into main should be identical to the tree that was in develop, meaning there's no difference, it's just a matter of maintaining the appropriate history. Under normal conditions this should be trivially true. Develop forked off from main, no commits on main, therefore the merge back into main has no changes on the first-parent side and is equal to the second parent. The two ways to screw this up are: 1. You did a hotfix release and didn't merge the hotfix back into develop properly, or 2. You made changes on a release branch and forgot to merge that back into develop. In both cases, the fact that the results of merging develop into main produced a different tree than what was in develop (or rather, in the current release branch) should be a signal that you screwed up somewhere. Having said all that, I'm willing to bet that 98% of places that do Git Flow don't actually have any checks to ensure that the tree on main is identical to the tree on the release branch. And this is because the model is actually rather complex, most people don't understand Git properly, and the accessible documentation about Git Flow doesn't even bother to mention the possibility that the merge into main could produce a different tree.

johnhowardstein 5y ago

It can be fine, but the moment you need to roll back a change, you'll wish for feature branches. If you reach for cherry-pick, you've failed. Anyway, branches and merge-requests take almost zero extra time and effort above pushing to master, with the added benefit that you can test and deploy a branch from your CI/CD separately.

galaxyLogic 5y ago

I think there is something wrong with the basic idea of git and similar systems, which is that anybody can just change anything because they work in parallel meaning changes they make can conflict. Therefore we have conflict resolution, but shouldn't we try to make it less likely for conflicts to arise in the first place? Instead I think we need module-ownership and tools supporting that. At any given time every module should be assigned to an owner-programmer or small owner-team. Only they can change their modules. Others can request changes, or create their own copy of that module to modify, but not modify code owned by someone else willy-nilly. If programmers cannot modify modules owned by others there will be no merge-conflicts, right? I wonder why this kind of code-ownership approach isn't more widely practiced and why there doesn't seem to be much tool-support for it?

albertopv 5y ago
So, back to Visual SourceSafe?
- galaxyLogic 5y ago
  My experience with VSS was pretty good but I'm thinking of something more radical. VSS etc. allow you to temporarily lock a file so no-one else can edit it while you have it locked out. I'm thinking that instead of temporary lock-outs we should have persistent module ownership. Only the owners can modify the code of their modules, and perhaps temporarily grant commit-rights for their module to others. Preferably the owner should have a deputy or two who would take over if the owner gets sick. Super-user can grant and take away ownership to any module. Super-user should have a deputy or two as well. Here's the metaphor: In New York City and all cities you have traffic lights and there are parking rules, and pedestrian crossings and bike-lanes and some streets are one-directional. What would happen if all the rules and traffic-lights were taken out? You could still get from A to B, but probably on average much slower because you would get stuck in traffic-jams much more often. Perhaps counter-intuitively creating rules which restrict how you can drive and park your car do not make you move slower, they make traffic more efficient. Traffic jam is like a merge-conflict. Two cars merging on to the same narrow street from opposite directions. One of them has to back out. And then so do all cars behind it. Not fun. Git etc. are a bit like city without traffic lights, one-directional streets and parking restrictions. Anybody can do whatever they want, branch and branch and merge and resolve conflicts. It gives you the impression of great flexibility and freedom, but so would a city without traffic rules. Yet all cities have realized they need to restrict what people do on their streets, to eliminate "traffic-merge-conflicts". Now naturally you can say that your project has rules in place as to who can modify what code and when (do you?). But I think there should be tool-support for that and persistent module-ownership instead of "module communism" where everybody owns everything. When everybody owns everything no-one is responsible. I'm not a big rules-guy but I think traffic lights do more good than bad.

albertopv 5y ago

In my experience you can't use feature flags for everything. Also, if you are having so much trouble merging because of lot of changes and commits, that's a smell of a bad design or a project too big that must be broken up.

sam_lowry_ 5y ago

Cancel him, he should push to main.

bacro 5y ago

I think GitFlow is helpful for mobile apps development. We cannot get into production every time we want, we are at mercy of app store review process. Also, if there is a bug in production it is very difficult to get a fix for all users once a version is deployed. I wonder if anyone uses successfully Trunk-based development for mobile apps development and if you could share your experience against GitFlow (pros and cons)

agsnu 5y ago
Generally the app store review process is much less of a problem than it used to be, with happy path being 24-48 hours rather than 1 week+. Of course everything is context dependent (team size, codebase size, testing maturity, frequency of release), but I guess we found GitFlow seemed to be better aligned with modelling the "versioned software" approach - i.e. with releases determined by features, everybody knows/cares about what is in version 2.1 etc, you might have multiple releases all on the go at the same time with development happening for 2.1.1 bug fixes, 2.2 minor features, and maybe master has moved on to stuff that will ship in 3.x etc. Definitely with less frequent releases (App Store Review taking 1+ weeks and being unpredictable probably had greater influence on release frequency) and a large team, we ended up with releases feeling quite painful - there was a lot of pressure to land features close to the deadline because the next release might not be for a while, meaning there was more churn on release branches as people tried to stabilise things that were borderline "ready". Git Flow's release branching model added overhead - people might forget to merge back to develop, deal with conflicts or semantic brokenness if you had different targeted fixes on the release branch to mainline development (e.g. let's just disable this functionality for now on release branch and push it to the next version, fix forward on develop, merge release branch back to develop -> now it's disabled there too, have to unwind). We switched many years ago (probably after reading https://barro.github.io/2016/02/a-succesful-git-branching-model-considered-harmful/ https://barro.github.io/2016/02/a-succesful-git-branching-mo... ) to a time-based continuous release model - periodically cut a release from master (there are lots of teams shipping mobile apps on a monthly, 2 weekly or even weekly cadence these days), with trunk-based development and cactus branching for releases. If we need to hotfix just need to go to the most recent release tag/branch and ship a fix based off that point, no merging back between branches. If you need to fix a bug in the release, it's on you to make sure an appropriate fix is applied in both places, through e.g. cherry-picking, no need to worry about merging anything back anywhere. Together with some other actions (cultural focus on reducing post-branch churn and investment in testing capability to gain confidence in release candidates faster, set in the context of a goal to increase release cadence; increased usage of feature flags; etc), we ended up significantly improving release frequency & predictability, which reduced the time for stakeholders to get changes shipped and visible to users from when they were "done" (especially valuable for small changes, of course). Nobody ever really understood the utility of the "production" branch from GitFlow. tl;dr GitFlow seemed to add overhead with little value; we switched to trunk based development and didn't look back.

ay 5y ago

We made something quite similar work rather well at $work. However, there is a somewhat subtle trick. The staging/prod candidate is built from master/main branch 21 days old, plus potentially some (few) cherrypicks. This allows to fearlessly commit to master, and then the “T-21” has 21 days of completely predictable future, that can be changed by doing cherrypicks. So we can run intense testing on “T-0” aka main, find any issues and add them to be cherry-picked into the T-21. The bonus is that when the fixes “arrive” to T-21, the cherry-picks become void and stop being applied. Thus, absent the bugs, the staging/prod code automatically converges to main/master over time. And yes, we do the reviews of the commits that go into master - but from the T-21 point of view they are 21 days in the future! So there is ample time for any reaction. Would folks be interested in a more detailed write up ?

acjacobson 5y ago
I would be interested in learning more - this sounds completely novel to any way I have worked before.
- ay 5y ago
  Ok I will post a reply here + do “show hn” or some such…
gulbrandr 5y ago
> Would folks be interested in a more detailed write up ? Yes please.
- ay 5y ago
  Ok I update the reply here when I have something to share !

claytonaalves 5y ago

Well here is the workflow we use here at our company: Branch feature from master/main. Keep feature branch in sync with master/main (merging from master/main everyday). This minimizes conflicts when we finally get to merge feature branch to master. Any refactor made in feature branch may be cherrypicked to master any time. This reduces differences between feature branch and master/main, resulting in less code to review.

alkonaut 5y ago

Allowing direct push to master is fine so long as you 1) have a small team so it's not very congested 2) have a short enough build/test cycle that you can enforce running tests locally. As soon as your test suite grows to the point that users aren't likely to run ALL tests before pushing new changes, you can't. Also, if you want to have code reviews at all, you want them pre-merge. If you have irreversible changes (For example: you have code that writes a serialized format and once you review the code that changes it, people have already used the code, even if only in testing/staging - but you now have data serialized in the bad wire format that you may need to be able to correct, and the correction code after review will need to be maintained forever unless you accept the loss of the data). I think: commit directly to master to scaffold and iterate quickly on a greenfield project with 1-3 devs. Then start doing feature branches and PR's to master.

Gareth321 5y ago
Exactly this. One particularly haughty CEO I worked for came to this exact same revelation. "But what about bugs?" "Just tell the developers not to add any bugs!" I'm not even kidding. That's what he said. Continuous release is fine for applications which aren't mission critical AND you have users who are accepting of an increase in bugs. Back in the real world, it's just not a good idea. We're all trying to calibrate the right balance between QA and output, and there are many ways to find the right balance. I remain unconvinced that "YOLO" is ever the right process.
- deleted 5y ago
  [deleted]
- teknopaul 5y ago
  Trunk dev and continuous releasing every push to trunk is not something I have heard suggested. If you want CD from some particular branch it would be silly to suggest the active dev branch. Trunk dev is about how and when new code is merged. It says nothing about when it is released. It's incorrect to think that any people doing trunk dev deploy from trunk willy nilly.

mentos 5y ago

What is wrapping all new features in a #ifdef NEW_FEATURE //code #endif called? My process over the last 7 years working in game dev has slowly evolved to this where I'll ask a contractor to implement a feature on the main branch but make sure that it is completely toggleable via a #define NEW_FEATURE I thought I was pretty clever until I read an article on HN a few years ago that this is exactly what Google does lol

kkirsche 5y ago
I think you are describing build-time feature flags or feature toggles. Similarly, this type of thing can be managed at runtime via integrations or implementing a database backed version yourself.
erezsh 5y ago
Yeah, and let me tell you, it's an absolute nightmare to get to work when you need two features that weren't written to compile at the same time. I'm not saying it's necessarily a bad idea. Just that it isn't a replacement for abstraction. If you have more than a couple of #ifdef blocks per feature, you might be setting yourself up for future pain.
jan_g 5y ago
This approach works surprisingly well for most features, but care should be taken to remove those 'ifs' once the feature is stable/complete. Or, when implemented cleanly, they could also be left in place and used as long lived feature toggles that can be driven by configs (or user settings, ...). The hairy part is when the new feature or change cuts across different parts of the code and then, given enough time, those 'ifs' pollute the codebase. Solution here is to be mindful of how the system grows, keeping things isolated and (perhaps counter-intuitive) favor duplication over shared logic/libraries. Regardless if it's about one big thing (i.e. monolith) or multiple small things (different projects even).

deleted 5y ago

[deleted]

dpark 5y ago

Ah, so what they really learned is that Gitflow is garbage. Yeah, long lived working branches are an enormous pain in the ass. I’m not convinced they are ever worth the hassle. Certainly for small teams they are not. I am highly doubtful of the “commit first, test and code review later” model, though. Maybe this works for a very small, tight team who are all very highly skilled and care a ton about engineering quality. But this model can fall apart quickly. Bad code blocks the whole team and everyone pays the cost. You eventually end up with someone babysitting the build and test process to keep it moving and then some bright mind asks why you don’t put this stuff before checkin.

deleted 5y ago

[deleted]

lolive 5y ago

Am i the only one who worked on a project with multiple feature branches in parallel by different subteams, with non trivial code merge conflicts (and data model conflicts) and a upper management that constantly reshuffle the delivery dates for these features. Plus maintenance branches to support previous versions with a hotfix branch (and also backporting some newer features to older version). That might be unusual, but even GitFlow in that case is very poor handling those kind of deliveries.

sirwhinesalot 5y ago

I really feel like most of the processes we have in software development are unnecessary if not detrimental. CI/CD is the best and most impactful thing in years and optimizing for it is the best you can do IMO. Anything that gets in the way of CI/CD you want to avoid. Anything that helps with CI/CD you want more of it. Using that as a guide post, trunk-based development is better than feature branches. Kanban is better than Scrum. Monoliths vs Microservices is less clear cut, depends on how costly the monolith is to build vs how annoying all the services are to deploy.