14 ms·
A development process to ship features fast
- xs83 4y agoNot writing tests early on simply backloads the effort to the point of where deploying anything is a crap shoot. Most of the other things I can agree with but tests is literally 101 for my teams
- x3n0ph3n3 4y agoI couldn't disagree more with what they say about feature branches. > merging the feature branch with the main branch takes a lot of time Why would that be the case? `git merge` is not a slow and expensive process. Is the problem the per-branch code review process that teams often engage in? They didn't even discuss how code is reviewed (feature demos are not code reviews). > when the feature is done, merging branches back might lead to new bugs and instability Feature branches should regularly be pulling in changes from the main branch. This seems like it would be a problem whenever your local main gets behind the upstream main, either way. Lots of teams will squash changes before merging feature branches, and run the test suite on it before it's merged. > it is more difficult to do incremental refactoring Smaller PRs and feature branches. There have been a few posts lately espousing the removal of workflows that have evolved over time to solve real problems in development practices. To me, it seems like a lot of cowboy coders not wanting to be constrained by sustainable development methodologies.
- P5fRxh5kUvp2th 4y agoI've seen feature branches attacked a lot and the common refrain is always the same. They live for weeks and then are hard to merge back in. This is why you should be merging main back into your feature branch daily, so imo this is just a strawman. That's assuming it's not reasonable to have smaller features. Sometimes it's not. You should strive for feature branches that live for a day or two, but if you MUST have it longer, just merge back in.
- peter422 4y ago> tests are great in the long run, but can slow you down in the short run. This is just silly, especially given you mention freezing releases and having an entire QA team go over the product. It would be much faster to have no QA team, no freezing of releases and instead have a robust test infrastructure. It’s fine that this set of rules works for your company, you like them and are familiar with them, but that’s all this list is.
- threeseed 4y agoThe context is that of an early stage startup. At that point most products are basic enough for developers to test the product themselves. And if the codebase is undergoing lots of churn and heavy refactoring then tests unquestionably will slow you down. Maybe it is better to only begin automated tests when you reach product-market-fit.
- sbdncuvh 4y agoWhy is everyone in this thread forgetting what tests actually do? Tests force you to write better code. Code that is easier to change. Code that is more robust. You don't need 100% coverage or perfect tests to gain this benefit. A good test suite enables you to make large refractors safely and quickly. The most scary refractors I've ever had to do are ones that involve large changes to code that has no tests. Often those refractors that allow company level pivots require adding huge amounts of tests to a legacy code base before I can even get started. Adding tests to code that has none is the worst. The code is coupled all over the shop and requires 100 times more mocking and stubbing to get under tests when you add this stuff retroactively. Tests speed up development velocity and improve your codebase. Just don't get stupid about it and you won't negatively impact startup performance.
- osigurdson 4y agoI generally feel the highest value tests are at the lowest level (code with few dependencies) as well as end-to-end integration tests. It gets a little murkier in middle layers since alternate implementations of dependencies (aka mocks) need to be created, or one has to deal with complex setup of dependencies or in some cases re-testing the same logic that has already been tested at a lower level. I'm not suggesting middle layer testing is not valuable, shouldn't be done, or to never use mocks, etc., it is just much more difficult and more creativity is required to add value in this area. For example, in some situations middle layer testing requires the creation of various kinds of utilities and frameworks in order to find a reasonable benefit vs effort sweet spot. Dogmatic approaches (such as mocking all dependencies in all situations) feel like they are productive at first but can end up being costly as each mock is an alternate implementation of an interface and doesn't necessary capture the behaviour of the concrete implementation.
- haskellandchill 4y ago> Request at least one demo per week Sigh, such a misguided quota.
- dchuk 4y agoI’ve led a fairly large product org in recent years, and frequent demos of deployed software is the number one way to ensure you’re shipping software frequently and consistently
- haskellandchill 4y agoFrequent demos are great.
- deleted 4y ago[deleted]
- wnolens 4y agoOn my team it's used as a false deadline, causing us to ship low quality code without sufficient tests, ultimately trading long term delivery dates for short term ones.
- imaginarypedro 4y agoI tend to agree with you that prescriptive notions like this are a little misguided, but I like the spirit of what he's after here.
- alasdair_ 4y ago"no" is a reasonable response to such a request :)
- dr_dshiv 4y ago“ We created and use Growthflags to implement feature flags. ”
- ramesh31 4y ago> Avoid Feature Branches This is a big one. It seems the industry (at least to my awareness) has almost totally rejected Gitflow for trunk based development in the last few years. It’s a beautiful idea in theory, but a complete headache in reality. > Use feature flags Be careful here. Adding feaure flags means you now effectively have N^N builds (where N is the number of flags) in production, each with their own possible quirks and integration issues. Keep them to an absolute minimum, and remove them as soon as possible.
- milosmns 4y agoWait why N² and not 2.pow(n)?
- P5fRxh5kUvp2th 4y agogitflow is not feature flow, that seems to be a common misunderstanding.
- imaginarypedro 4y agoThere are a lot of good practices there and I commend the author for sharing. I disagree with code freezes + manual QA verification -- why just dip your toes into CICD? -- and the advice about writing tests. "We don't have time to test now" results in a vicious circle.
- tomasreimers 4y ago> Break features into small, incremental changes Stacking is a wonderful workflow to accomplish this: https://graphite.dev/blog/post/N8SOs49y4bYdV1Vjf5qE https://graphite.dev/blog/post/N8SOs49y4bYdV1Vjf5qE
- ec109685 4y agoIt's an interesting approach, but it seems like by the time PR 1 is reviewed, much of it could be out of date as you continuously refactor along the way. Also what happens if during the review of PR #1, the reviewer suggests something profound that would fundamentally change the direction of future commits?
- alasdair_ 4y agoFeature branches are fine. Just merge master into your branch on a daily basis and the concerns about merging go away.
- BarryMilo 4y agoAgreed. I'm not using feature branches because it's fun, I'm doing it because it reduces the time wasted on figuring out how git/our product works. Stay up to date and you won't have an issue.
- andrewstuart 4y agoStartup founders should be explicitly telling their developers to not write tests, not use CI/CD, not use feature branches. All that stuff can come after you have proven your software makes money / users want to use it. Just get the damn thing written in the quickest, most crappy way necessary for speed. The code doesn't need to be organised well. It doesn't need to be documented. The only thing that matters is to do your best to avoid security holes. No tests. Stop testing, apart from manual runthroughs that the CEO can do in between trying to raise funding. My eyes roll when I hear of startups creating "good processes" for development at any level, let alone CI/CD and tests.
- pier25 4y agoI totally agree. Stop thinking like a developer, think like a business owner. Build the MVP so that it's good enough to start solving whatever it is you're solving and get your first customers. If you can't get customers, what's the point of having a sophisticated codebase with tests, etc?
- presentation 4y agoNot even solving, just making it sellable.
- andrewstuart 4y ago>> If you can't get customers, what's the point of having a sophisticated codebase with tests, etc? And in fact every minute spent on writing and running tests (and there's always ALOT) and CI/CD is wasted, wastes money and time, leading to longer development time until the software can be put in front of users. Therefore CI/CD and tests and all their associated processes are a red flag and increases the likelihood a startup will fail to bring its software to market. Developers who have always worked in well organised environments have really deep rooted trouble getting their head around this way of thinking. If you say to an experienced developer "no tests! no CI/CD!", it's likely they will immediately decide you are an unprofessional hack.
- 4y ago
- thijser 4y agoThis sounds a lot like trunk based development: feature flags, incremental changes, no long-lived feature branches. I think better tooling around git will help a lot with speeding up development teams.
- mchusma 4y agoI think this depends on the stage, but I don't think for us deploying weekly instead of 4x per week (M-Th). Would help. We also found feature flags slowed us down a lot in the early stages. I feel these should only be done for some features (e.g. an explicit AB test, a large feature with a lot of technical risk). Feature flags create a ton of complexity, and are often not worth it.
- wpietri 4y agoAgreed on both counts. At my last startup, there was a release every few hours. We used feature flags for anything where the code's future was uncertain, especially experiments. But once the experiment was resolved, we mercilessly removed the flag, refactored the code to reflect what we'd learned, and added tests.
- anorsich 4y agoDeploying 4x is better, but not every team can do it without sacrificing quality. That essay was written based on 3 three pre Series A companies, which didn't have right processes in place and was struggling to ship fast enough. Weekly deploys is a first step. It's difficult to impossible to deploy more often if you don't have good tests coverage.
- wpietri 4y agoThere's some good advice here, but this is ridiculous: > But, on another side tests slows you down as you build features. When your team writes tests they need 2x more time to release any feature. Tests are an investment in fast future development. Last I did a startup, we did not write tests for anything throwaway or experimental. But once we decided some chunk of code was a keeper (e.g., experiment had a positive result), we wrote tests. With just a team of 6, we were releasing a few times a day, not just once a week. And we had no "QA freeze" nonsense, because our tests were solid enough that we didn't need manual QA to release. That said, I hired for people who were good at writing tests, and we did pair programming with frequent pair rotation, which gave us a big quality boost and helped keep our test suite lean and effective. I can believe that some startup teams have low enough levels of testing skill that "they need 2x more time to release any feature".
- Kamq 4y ago> Tests are an investment in fast future development. I mean, they are, but what's the roi on that investment? Not all investments are good. 90% of start-ups fail, the chunk of code that's a keeper still has a 90% chance of being thrown away.
- wpietri 4y agoSure, and that's why I explicitly said "experiment had a positive result". Before you decide a particular chunk of code is a keeper, you should have data that gives you confidence. Before you have signs of product-market fit, I think people should write throwaway code and then throw it away. Once you have something worth investing in, I think it's time to invest. Having high-quality code in areas where you're confident it's valuable enables rapid experimentation in the areas where you're not sure yet.
- rco8786 4y agoI’m having a hard time reconciling “use CI/CD” with “deploy weekly” and “freezing code before release” is certainly not in service of “developing weirdly fast”. Some reasonable advice in here otherwise.
- presentation 4y agoMy company actually does this - we just found that a) we wanted to be able to tell our clients about new features instead of them “stumbling on them,” which makes more sense when you’re dealing with low numbers of high value high touch clients; and b) we wanted to have greater confidence that our code was working properly in the absence of a thorough test suite. Feature flags could help with a) and a good test suite would help with b) though.
- rco8786 4y agoTo be clear I’m not saying it’s “bad”. Just that the article purports to be about how develop things fast. And purposefully slowing down your releases is not that
- Hnrobert42 4y agoWith feature toggles, you can deploy continuously, but release weekly.
- rco8786 4y agoRespectfully, that is worse than just deploying once a week (which is totally fine btw. Lots of reasons to do that) You’re still waiting to release features that are done and turning on a whole week’s worth of features at one time, but it’s even worse because you spent a bunch of extra time building it to have feature flags.
- q7xvh97o2pDhNrh 4y agoI think you're correct only in the scenario where the feature-flags are binary on/off switches. The modern "feature flag" is more of a knob that can be slowly turned up, along with ideally a way to send shadow traffic through the feature even before any % of real traffic is allowed through. In that proper implementation of feature-flags, the overall benefits (easy rollback, graduated rollout, selective rollout based on user characteristics, better ability to manage scale) outweigh the rather-trivial engineering cost of wrapping a few if-clauses around the feature.
- explaingarlic 4y agoMonorepos are unreasonably effective for a product that you can guarantee won't scale to silly proportions. They also provide you some great capabilities for introducing new team members to your solution, bringing aboard contractors if something breaks or you face some regulatory issues later on, etc.
- dqpb 4y ago> for a product that you can guarantee won't scale to silly proportions Both Google and Facebook use monorepos
- dinosaurdynasty 4y agoBoth companies have custom in-house proprietary software to support their monorepos
- askafriend 4y agoIf you're successful, you can just hire people to help build out this custom internal infrastructure. If you're not successful, then you won't have any scaling problems.
- explaingarlic 4y agoYou mean for their core solutions? Not everything at FAANG serves half of humanity :P
- deleted 4y ago[deleted]
- rehevkor5 4y agoThe stuff under "Freeze the code before release", such as a code freeze where devs can't make any changes, seems to be the opposite of the article's goal. Your release should be a specific commit ID, not a branch name, so it doesn't matter if people continue to develop and push changes. If you need to make changes specific to the release, use a release branch. No need to stop devs from moving on.
- danpalmer 4y ago> weirdly fast > deploy weekly These are at odds with each other. In my previous role at a startup we deployed on average twice per day per developer. The automation necessary to make this hands off and reliable wasn’t actually that much. It rarely caused issues.
- ChrisMarshallNY 4y agoThis appears to be for an online-based product. I write apps (which may include some fairly significant backend stuff). Some of these suggestions apply, but not all. I ship weirdly fast, anyway,
- rhacker 4y agoWe're doing all of these things and we're weirdly very slow. But his first paragraph mentioned motivation, which I think no one has: no pay increases, people quitting (and not just quietly), customers upset, no stock options and just all around no one likes anyone else at the company any more. I think the pandemic broke us.
- kareemsabri 4y agoI agree with most of these, and vehemently disagree with a couple. I think we move pretty damn fast, we average 5 commits per dev per day. I very much agree with break features into many small pull requests. We merge constantly even (especially) if the code path won't be hit yet. Just get it into production. Makes the code review faster, the complexity lower, and lets you focus on one thing at a time. I think no tests makes you slower. We paid a bit of cost to start writing tests, but now we can deploy 10 times a day because we know the mission critical stuff is working. It's more about testing the right stuff. Decide what can break, and leave it untested. What can't break should be covered by tests. We do stuff with money, all that is robustly tested. The UI isn't. Code freezing / weekly releasing is counter to CI/CD. Just make your deployment take less than 10 minutes, and ship code constantly.
- brightball 4y agoAgree with this strongly. Weekly releases reduce efficiency and lead to more complex production issues as you work to unpack which part of the batch of things you just deployed is causing a problem. Use feature flags to decouple Deployment from Release and you’re fine.
- Gigachad 4y agoFeature flags are good but I hate how they enable weird customer specific workflows. Imo the mindset should be that all feature flags will eventually be removed when testing is finished. But management sees the capability and says add a flag for customer Y that changes how this page works.
- brightball 4y agoAgreed. IMO the best approach is a flag that can let you enable for specific customers (alpha or internal sample customers), all beta customers and then all customers. And after it’s enabled for everybody, remove the flag.
- 4y ago
- bcrosby95 4y agoIf you're writing both a client and server, manually testing corner cases can be anywhere from difficult to impossible. Ideally the client won't let them occur. But you should still verify the server doesn't give up the goods. I guess you could "manually" test by using the command line? Sounds like a lot of work. In my experience, more work than automating it.
- dlevine 4y agoI have worked for a lot of startups, and the best ones have deployed as often as possible. One company (which grew massively and is now public) allowed developers to push code whenever they wanted. Once your code was merged into master and the required tests had passed, you could push to staging and then production whenever you wanted. However, you were responsible for making sure that your push didn't break anything. Developers took a lot of responsibility, and when something broke, the last guy who pushed usually fixed it really fast (and if not someone else did). Releases went out several times a day. At my current job (fast-growing mid-stage startup), any developer can push when they want, but it works a bit differently. Once the release is staged, everyone who has a commit on staging is responsible for verifying their feature. Then the developer goes through a "release checklist," which involves finding a buddy and jointly running through a list of manual tests to validate key features. After that, they can push. They probably push a few times a week. I feel like code freeze/manual QA/weekly push is something of an anti-pattern. Developers are disconnected from the process of releasing their code, and take less ownership. While in practice manual QA should lead to finding more bugs, in practice it can lead to more defects.
- awinter-py 4y agosimply skip app store review saves 48 hours on releases
- iopq 4y agoNot clicking anything that sounds like "Use this one weird trick to ..."
- gyulai 4y agoI wonder if this development process is also something that bankers don't want you to know about.
- robertwt7 4y agoFrom my own experience, weiting test is a huuge advantage for a small time trade off. I didn’t write much in the previous company and we regret it every single day. New dev won’t write any as well because there’re just no discipline. Also tbh feature branch is not really bad? How long does it take for one to create branch and open PR for reviews? It was a nightmare for everyone to merge in 1 branch then merge back to master when we want to lol. FF 100% helps, but we shouldn’t compromise tests and PRs I believe (it also helps when we need to rollback stuff obviously)
- SunghoYahng 4y agoCan anyone give me some other articles similar to this one?
- JimmieMcnulty 4y agoThis is written as if git doesn’t exist, and Agile were never invented. If branching/merging is giving you this much trouble, you’re doing something very, very wrong. Pause, actually learn git, then resume. Also, testing is so easy, if you’re doubling your dev time writing tests, that’s another thing you need to dedicate time to for learning, because it should add maybe 10% more if you’re competent.
- WHATDOESIT 4y agoHow does learning Git solve the problem that the other team members changed the code so much I have to rewrite half my work again? Happens especially in agile environments - waterfall is planned upfront so usually you don't change what you already did.
- Xylakant 4y agoHow does any tool help if your colleagues don’t talk to you? Or you don’t talk to your colleagues?
- WHATDOESIT 4y agoWell if I commit my changes ASAP to the main branch, there's much less/zero divergence at any given time. This is not about talking - I and the other teammates might be perfectly aware of it happening and still - we are agile so their work has to be done immediately, mine too, the problem with feature branches remains. I actually tried to simply have my team commit to the main branch, and it worked out quite well - we never had problems with code divergence. I guess it doesn't work so well for larger teams not working on MVPs, though (we were 5 developers).
- Xylakant 4y agoEvery time you have a local working copy, you essentially have a feature branch. Their local modifications and your local modifications can be incompatible - pushing to the same branch only moves the problem to “who pushed first”. At least with separate branches, you can observe their changes and pull them in at a suitable time.
- BerislavLopac 4y ago> Avoid Feature Branches The argument in favour of this advice only checks out when a feature branch is kept out of sync with the main branch. In my experience, the process that works the best is to rebase the feature branch on top of the main every time there is a divergence, resolving all the conflicts right then and there. > Use a monorepo Definitely - if you have a monolithic implementation that is deployed as a single unit. Otherwise, monorepos start getting in your way pretty soon. > Use feature flags There are pros and cons for this. In my experience, feature flags work best in the front-end, where you want to hide features from certain users at certain times. > Deploy weekly The deployment strategy depends primarily of the type of the product you're delivering. For a standard, vanilla SaaS, the continuous deployment is the most efficient.
- ricardobeat 4y agoI wish more people were comfortable with rebasing. This is my favorite workflow too, but I've never been in a team where we're able to do that as a default, unfortunately it requires an understanding of git that's deeper than the average dev deems worth.
- BerislavLopac 4y agoNot necessarily. A lot of developers despise the idea of using git GUIs, but e.g. GitKraken makes rebases - even with conflicts - a breeze.
- jaxn 4y agoPlus you get to say out loud “let’s get cracking’” every time you open it up. Just me?
- gjulianm 4y agoBut you don't really need rebasing in this case, right? Merging the main branch into the feature branch would work too, and it's not too difficult.
- 4y ago
- jongjong 4y agoI agree with all of them except monorepos. A monorepo might be nice in the beginning for the first 6 months but in the long run, it encourages (facilitates/enables) developers to build tightly coupled components so the code base ends up looking like spaghetti and none of the components end up being generic enough to be reusable and that's where the medium and long term costs start adding up. Business domain logic creeps into all the components. I've seen it happen multiple times at different companies. Monorepos don't make sense at face value if you want your code to be modular. How can bringing together multiple modules into a single monolith possibly help to improve modularity. Plato said it best: "You see, it’s not, I think, the function of heat to cool things down, but the opposite." "Nor of dryness to make things wet, but the opposite."
- ricardobeat 4y agoReusability is a mirage. If you have a monorepo, you don't need to care about reusability outside of that context. Unless you're creating components to be shared with the world, the tight coupling is a benefit, not a disadvantage. Not to say 'spaguetti' code is not an issue, but these are a little bit orthogonal to each other. It's very easy, for example, to grow complexity two/three-fold by trying to make a component more generic and reusable before the actual need arises. And when it does, the original interface you predicted is rarely a good fit.
- bckr 4y agoSoftware engineering is programming over time. At some point the business needs will change. At that point, someone will wish that the Apache server didn't talk directly to the database about last year's invoice minutiae.
- jongjong 4y agoThis is spot on. Also, it's one thing to claim that "code maintainability doesn't matter" or that "wasting time constantly refactoring code doesn't matter," it's another to claim "Monorepos are good for software maintainability and help developers make efficient use of their time." I actually do think that code maintainability probably doesn't matter much in the context of many big corporations since they often have market monopolies and can afford to hire huge numbers of developers; they can afford to waste a lot of developer time... That's fair enough, but I don't see why they should be lecturing the rest of us about what good software development practices are. That would be like a rich person lecturing a group of starving people about money not being important. This lesson is completely irrelevant to that audience. Anyone who listens to them will starve to death.
- ricardobeat 4y agoMost of the advice is on point, except for the first one: "Avoid feature branches" I think this is mistaking cause and effect. The problem is having long lived branches, not the fact you have a branch. IMO the best advice is to avoid long-running branches. This is easy to achieve by slicing features into smaller units, and using feature flags, both recommendations already featured in the list.
- gocartStatue 4y agoI have a friend that says "I'm not doing branches". He actually does branches, but they never live longer than say 2 hours. Maybe that's a confusing wording, but for people used to weeks or months-long feature branches, short-lived can still be unclear. The clearest helper "rule" for a team I came up with, was "at the end of the day, there is no branch in a repository except main". Takes time to get there, but simplifies a lot.
- lucideer 4y agoThese are largely subjective choices that have no positive nor negative bearing on how fast you ship. > Avoid Feature Branches > Break features into small, incremental changes This makes efficient & organised writing of code easier but makes product development harder or at least very artificially limited. If you want to ship fast this will help. If you want to ship quality & value, this will likely hamper you. This needs to be compromise - feature branches can be tricky to manage well, but it's better to learn how to than avoid it. It will allow you to manage complexity and meet your target audience's needs more effectively. > Use a monorepo Absolutely arbitrary and has zero bearing on speed of delivery. Monorepos vs monoliths vs multi-repo microarch is as efficient / in-efficient as your choice of app architecture, team structure, tooling, software ecosystem, etc. It depends on far too many variables to be a blanket positive. > Automate CI/CD pipeline > will make it easy to release changes multiple times a day > Deploy weekly There's a lot of contradictions here. Deploy/release much more often than weekly. Please. > Freeze the code before release [...] This is not to improve the speed OK. > Write tests (or not?) Make up your mind. > Request at least one demo per week This is generally reasonable advice, but the cadence seems too frequent. Demos are expensive w.r.t. developer's time & energy - they should be frequent but still leave some time for actual dev in between. > Use feature flags This is nice if it's simple (if you're following the "Break features into small, incremental changes" advice I guess you have no large/complex features). If you do have any large features, feature flags can sometimes come with side-effects, impacting performance / stability / security of your overall application. Follow this with care. > Define owners This sounds like good advice until you read the author's description: > avoid the shared responsibility trap, when no one is responsible This sounds like a bad manager or dysfunctional company - shared responsibility is a good thing actually. If your engineers aren't motivated to own code, that's a different problem. > The feature owner knows explicitly that no one else is going to complete their work. This sounds like a very dysfunctional team. If you have this problem, work on your culture, don't try and fix it with technical solutions. Overall, code ownership is good as long as it's shared. Bus factor is a thing. --- This all seems like the common mistake of mis-attributing success to a large but irrelevant change that coincides with many small beneficial incremental learnings. Imagined example of the above: say the author moved from a disorganised mess of many disparate repos to a fresh-start carefully-planned monorepo: things didn't improve because it's a monorepo, things improved because someone sat down and planned out the organisation of code to fit their needs (needs which probably evolved to that point of maturity over many small repo-structure changes before). Correct connclusion here isn't "monorepos are good", it's "sitting down and re-thinking/re-organising/planning code structure periodically is good".
- danbruc 4y agoIf you want to implement something quickly and even more so if you want to maintain that speed in the long run, then there is one thing that will turn all the things the article talks about into mere noise - design things properly. Especially data models. After that comes implementing things probably, i.e. no shortcuts and hacks. The things the article talks about are not bad advice but I do not think any of them are critical for being fast besides getting and incoperating feedback as early as possible. This is of course not limited to feedback on an implementation, you of course also want early feedback on your design before you even start implementing something.
- hirefullstackde 4y ago[dead]
- greghinch 4y ago1. I don't understand how you can merge/deploy quickly without good test coverage. Unless you don't mind breaking things in production. 2. Feature flagging is pretty key to make the described process possible, and about halfway down it turns out of course this post is really an ad for a feature flagging SaaS product. Nothing wrong with that necessarily, but worth knowing that the motive behind this is to sell you a product.