10 ms·

Balancing “If it ain’t broke, don’t fix it” vs. “Release early and often”

128 points by yarapavan 4y ago

hulitu 4y ago
> Customers often ask me, why pour time into updates when what I have runs just fine? The answer is that a bug in your old version might assert itself at the worst possible time and force you to forego sleep for the next five days to recover Why not do proper testing ? I know it's expensive. And when you have an OS as an init system is even difficult. It is sad the the UNIX philosophy is dying being replaced with the Windows philosophy.
- codegeek 4y ago
  You cannot test everything to perfection. Shit will break in production. Question is: how to minimize shit breaking in production and hopefully eliminating critical bugs that can cause revenue/data loss.
peteradio 4y ago
How do people disentangle "ship the incorrect thing and let customers let us know what they really want" vs "ship the correct thing but built incorrectly and let customers know why its wrong". I hate the second with a burning passion because it seems like its bound to give the impression that you really don't know what you are doing. I've been put into situations too many times where I just could not get the point across that "yes I understand that we may not be giving the customers what they want, but what you are telling me to build is clearly not correct" its maddening. Sorry if its off topic but it seems at least adjacent to the articles balance. This seems to happen with PMs who do not have a deep enough background in the technical detail and yet reject the feedback given because it will reflect that poor understanding. "Agile" environments seem to enable this kind of behavior. I'm beginning to think I need to pivot to PM in my career so I can avoid the madness as a developer.
- sumtechguy 4y ago
  That depends on your customer. Some are cool with 'something is better than nothing'. Some are not. Some also are very 'I put it in the contract just do it that way' others are 'I just want the right thing'. Some just do not know what they want at all and just want you to do it and put all the liability off onto you but do not really want to spell out what that means at all. Being a PM is that turned up to 11 but now you do not control the code at all.
  - peteradio 4y ago
    I guess what I'm talking about is this: 1) PO wants feature X for customer P. 2) Architect in meeting with PM and PO indicates that feature X is a simple addition to Y 3) Engineer is assigned X and says no this isn't a simple addition to Y because on deeper inspection it will fail for cases 1,2,3, we would need a different architecture. 4) PM says build it the naive way, let the customers find case 1,2,3 before we fix, we are an Agile team after all. 5) I quit.
    - antupis 4y ago
      I would do 4.5) escalate to PM boss everytime 1,2,3 happens.
    - helge9210 4y ago
      > PM says build it the naive way Why PM has an option to choose here? > will fail for cases 1,2,3 a) tip QA to test these cases b) add cases 1,2,3 to "Known issues" After all that is the reason they call it "beta": "Cause it beta then nothing"
    - skellera 4y ago
      Seems like you’re working with a PM who thinks agile is an excuse to release bad software. MVP should still solve a customer pain point. To play devils advocate though, maybe case 1,2,3 are low enough risk to release. Having metrics set to watch if these are actually big problems could be “good enough.”
    - 4y ago
henriquecm8 4y ago
Release early, don't fix it
- _8j50 4y ago
  This is the way.
- tomalaci 4y ago
  Creating bugs? No, I am creating learning opportunities for more junior developers.
  - devteambravo 4y ago
    Are you my manager?
- BurningFrog 4y ago
  "Move Fast and Break Things" at maximum speed!
jbverschoor 4y ago
Those are two different things.. Nothing to balance about. But put it in a graph or drawing, and it means it's true and supported by 'science'
silentsea90 4y ago
How about use Feature flags and ship daily? Decouple your release-schedule/deployments from your product iteration/launch timeline.
- allendoerfer 4y ago
  Never miss an opportunity to add complexity!
  - generalizations 4y ago
    Unofficial RedHat motto.
  - int0x2e 4y ago
    Feature flags or another way to release bleeding edge features to only some of your users can sometimes be the best way to iterate over new features quickly when your most user base is risk-averse
  - silentsea90 4y ago
    Centralized coordination complexity v/s decentralized FF rollout complexity. The bigger the team contributing to a deploy, the better the latter gets because engineers rolling out their FF have all the context they need.
- alldayeveryday 4y ago
  I think I know your meaning, though I would rephrase it as decoupling your deployments from your releases/launches.
  - silentsea90 4y ago
    Edited.
sylware 4y ago
Grotesque generalities. The devil hides in the details: planned obsolescence and redhad (IBM) with its glibc symbol versionning frenzy and not a clean ELF ABI set of libs (as an example). Yep. Open source is not from enough, lean and stable in time open source is required.
torstenvl 4y ago
Simple: Release Early. Release Often until it Ain't Broke. Don't change things that Ain't Broke. I don't think there's much consternation over actual bugfixes. There's consternation over constantly-shifting UIs and APIs. The only time there's any tension between those two things is when people conflate bugs with poor design. The messy but correct thing to do when you've correctly implemented a poor design is to maintain backward compatibility while also offering the option to use the newer, more correct behavior. See, e.g., strcat() → strncat() → strlcat()
- pabs3 4y ago
  Preferably also provide assistance to people using the old things, in the form of making the old API a wrapper for the new API where possible, adding warnings for the old API, adding migration tools etc.
- klodolph 4y ago
  (I think it would be strcat() → strlcat(), since strncat() is not really safer / newer.)
  - david2ndaccount 4y ago
    Just always use memcpy, you should always know the size of your buffers and if you don’t you will get a buffer overflow at some point.
    - klodolph 4y ago
      I've heard this advice before, and the problem with this advice is that people who just use memcpy in practice often write code with bugs in it. So the advice is bad advice. Good advice will result in code that has fewer bugs, or bugs with lower severity.
      - kevin_thibedeau 4y ago
        It can also lead to accidentally quadratic behavior which is why the modernized string ops that get so much hate these days rely on one iteration over the input.
jrochkind1 4y ago
I think libraries (used as dependencies by other software) and "top-level" software/applications (used by users directly, whether that user is a developer or not) -- have different pressures and concerns. Libraries should be much more conservative. I don't think "release early and often" was said about libraries. And yes, it's tricky that it's not always a clear line between these two categories. A unix command line utility is sort of top-level software, but also likely to be use in a scripted/automated fashion. So it's not always cut and dry, but the more you are aware of software depending on your software, the more careful you should be with releases.
- wongarsu 4y ago
  A great example of an end-user application that is like a library is Microsoft Excel. Sure, the UX can change, but the formula engine has to be extremely conservative, staying bug-compatible with basically every version, or all hell breaks lose.
- JohnFen 4y ago
  I agree. As a developer using libraries and component, I really hate "release early and often". As a user, I also hate "release early and often". In both roles, releasing early and often effectively means I'm always using beta software, with all the headaches using such software brings. However, as part of an internal development process -- that is, before customers see the release, "fail fast" is a totally legitimate and decent approach.
- wpietri 4y ago
  I think it depends a lot on where the library is in its lifecycle. A new library should iterate rapidly and be very clear about that. Big warnings, obvious version numbering, and good communications. You look for an audience of actual users that can keep up with that. That's how you figure out what the right library is, both in terms of the big abstractions and the small details. But then once you hit a 1.0 release, things change. You're shifting from an audience of fellow explorers to an audience who wants stability. You can still do exploratory work, but it has to be additive in the 1.x series, and eventually you need to start a 2.x series as you learn more about what needs to change. So the iteration still happens, it just happens away from the people who just want the basic thing you got right via the initial burst of iteration.
eloisant 4y ago
They're not opposite. "If it ain't broke don't fix it" => don't try to "optimize" something that works well. Maybe it will be able to handle more load after your big refactor, but if it already handle the load nicely and there is no indication load will go up, don't bother. "Release early and often" => release after each change, don't bundle everything into a big release that happen every 6 months. The first one is about deciding what to do, the second one is how you release the work you did.
- JohnFen 4y ago
  > "Release early and often" => release after each change, don't bundle everything into a big release that happen every 6 months After years of dealing with software that does this, I've soured on it completely. It has resulted in greater difficulty using software, and the software being less solid.
  - mdtusz 4y ago
    I've seen this happen as well, but I think it's more of a problem of bundling too small a changeset in a release. A partial implementation of a feature shouldn't warrant a release (without feature flags to disable it), but bundling a year's worth of work in a single release is, in my opinion, a recipe for disaster (and this is sadly something that happens all too often in enterprise and non-software engineering businesses).
  - michaelcampbell 4y ago
    I've seen this happen as well; perhaps for different reasons. My early career was mostly fin-tech and when installing into banks or other financial institutions, they _very much_ want the every year or 6 mo release cycle. And not just because of "old stodgy", but rather they want a good bit of ramp-up time to ensure things are running right. In a Wall St trading firm I worked at, a release ran in parallel with real data for MONTHS before the old version was retired. So, I'm used to that model. From the customer's standpoint, they could "gear up" for the next release, get their ducks in a line, get the right people to assess things. I get that some of that is because the infrequent releases are "big", but also as a human customer it's fun to see a shiny new truck toy rather than a coat of anti-rust paint on its exhaust pipe, then a replacement for the broken window crank, then a new color on the odometer numbers, then ... As a developer I also miss the Big Project Ramp Up, then a period of a lot of work (which can be Waterfall, or Agile, or bespoke artisinal project management, which is pretty much 99% of the time, just CALLED "Agile"), then a Big Project Release (+ party, some down time, and prepare for the next one). SAAS release early/often is like death by 1000 boring dull cuts. They cut out the spikiness _so much_ that there is so little novelty that it's just not fun to work on it.
AaronM 4y ago
Error Budgets are your friend
chuso 4y ago
Break it often and don't fix it.
jjslocum3 4y ago
I would love to read a Malcom Gladwell book on this topic. It touches on a core truth, not only in engineering but in society as a whole. Three examples from different spheres: In software, I've seen repeatedly how new hires with big ideas (and youthful confidence that they know a better way of doing things) will come into a company and want to rewrite things "properly." In one management position, much of my role involved the politics of defending a well-working, well-maintained, big-revenue-driving application from a steady onslaught of such ambitious exuberance. At the time, I thought "this must be the same phenomenon that drives shrinkflation": new MBA arrives at pickle-making company, seeks to bolster their career by demonstrably saving the company millions, convinces execs to put one less pickle in each pickle jar, consumers won't notice; step 3: profit!! In 2022, when you buy a cereal box, it's half air on the inside. In 1980, it was only 20% air. I also shake my head when media pundits equate the success of a particular congressional session with the amount of new legislation they pass. I can imagine no simpler way to cruft-paralyze a democracy (while following the rules) than "releasing laws early and often."
- SQueeeeeL 4y ago
  Shrink-flation is driven by short term quarter earnings thinking. Reducing the amount of your product (and therefore it's overall quality compared to competitors), is an act of cannibalization of a brand. They may increase profits for a short time, but as consumers realize what's happened, ultimately they see the brand as not having standards and lead to it becoming a commodity. A small group of long term invested owners would never do such a thing, because that would be 'killing the goose who laid the golden egg', but modern shareholding has no loyalty to quality.
- klodolph 4y ago
  > In software, I've seen repeatedly how new hires with big ideas (and youthful confidence that they know a better way of doing things) will come into a company and want to rewrite things "properly." In my experience, these new hires are right a certain percentage of the time, and wrong a certain percentage of the time. I can take no default position here. I tell new hires that they're gonna see something that offends their sensibilities about the way code should be written. That puts them in the center of a conflict. On the one hand, they believe the code should be written differently. On the other hand, someone else wrote the code that way for a reason. Their first step is to dig in and figure out the reasons on both sides--why it was written one way, why they think it should be written another. The second step is to figure out what the consequences and risks are of making changes or not making changes. The way I think about it, I want to simultaneously keep the software working, and protect the "flame" that new engineers have to improve things--something which can be all too easily extinguished.
gdsdfe 4y ago
“If it ain’t broke, don’t fix it” is the absolute worst mindset to have! I've seen systems that are literally garbage because of it.
scombridae 4y ago
tl;dr continually test, continually deploy.
onetokeoverthe 4y ago
ChrisMarshallNY 4y ago
Man, I feel this. I worked for a corporation that was so change-averse, that we needed to buy our development systems on eBay. They actually had a point, and it was hard to argue against, but I found it absolutely infuriating, as the solution was to plan for change, and establish a process to be constantly evaluating and refining for new developments. Instead, it was "Wait until we can't bear it any longer, then have a huge screaming match meeting." A new budget would be approved for new machines (and, thus, new operating systems and development tools), processes would be updated, and that would sit, until it was no longer new. As I said, it was hard to argue against, because it was a 100-year-old company that had been successfully delivering really high-end stuff, since Day One. They did that by being so conservative that they hadn't discovered fire, yet. Measure 300 times, cut once, etc. When there was a problem, it escalated quickly (and was used as fuel to tighten things down even more), as the company was held to standards that are probably up there with NASA. Speaking of NASA, I can't help but notice that this rather plucky little outfit, called SpaceX, seems to be running circles around them. They seem to have figured out how to "fail fast," yet also deliver insanely high Quality stuff. Might be worth ignoring their CEO's tweets, and look at what they are doing...
- pclmulqdq 4y ago
  SpaceX shows the success of using modern, commodity hardware rather than purchased-by-government contract obsolete components for space applications. The same is true for the cubesat revolution. I honestly think the idea of "if it ain't broke, don't fix it" is pretty silly as applied to large, engineered systems. If your system is more than 1000 lines of code, I guarantee you have a latent bug somewhere. Even if you don't have any bugs, the ecosystem around you changes so fast that you eventually will.
  - Gibbon1 4y ago
    SpaceX shows the advantage of not allowing aerospace lobbyists and congress to design things. That's how you get the SLS with it's inane H2/LOX first stage and solid propellant boosters.
- mgbmtl 4y ago
unity1001 4y ago
Just keep things backwards compatible, then do whatever you want.
PaulDavisThe1st 4y ago
TFA seems to try to pin DEC's collapse & failure on their committment to back compatibility. Yet Microsoft has essentially done the same thing with Windows, and has not suffered the way DEC did. Seems like a weak argument.
esaym 4y ago
And if it ain't broke, fix it 'til it is!
temporallobe 4y ago
We have relevant case of this. We’re on Angular 8 and the app is working just fine - customer is happy, it’s pretty easy to maintain, well-tested and hardened, etc., but we are now facing having to update to 14.x “or else”. It’s gonna be a long and difficult migration.
- kmitz 4y ago
  Well depending on the size of your codebase, npm dependencies and Angular APIs used, it might or might not be a big deal. You can have a rough estimate of the complexity by running 'ng update'. Yes major releases come at a rapid pace and it is hard to always be on the latest version. We update regularly once we are 2-3 major versions behind and usually it's not so awful.
pdpi 4y ago
Lord Vetinari in the Discworld lives by the mantra that "If it ain't broke, don't fix it". Other characters noted that, under his governance, "things that didn't work ... got broken."
exabrial 4y ago
The problem is a bunch of overpaid devs conflate anything that isn't absolute bleeding edge as a "bug", and we get into and endless cycle of UI refreshes and updates that exist solely to exist.
bradwood 4y ago
These are not mutually exclusive. It seems like the whole premise here just lacking nuance.
jmartrican 4y ago
More often than not its "if it ain't broke don't fix it because we are too busy releasing early and often".
arminiusreturns 4y ago
Release to prod so you can fix stuff in lower envs.
GnarfGnarf 4y ago
"If it ain't broke don't fix it" Would you fly on an airline that followed this principle? Wait till the engine fails before fixing it. In the air.
jillesvangurp 4y ago
Reducing cycle times for changes minimizes integration testing effort. Effort and risk increases non linearly with the amount of change and time. So, release as often as you can to minimize the risk and effort while balancing the cost of doing so. Some release processes just involve a certain amount of heavy process that takes time and money. E.g. app store releases are a PITA so doing that multiple times per day is not worth it. Server updates on the other hand are fine. Create a PR, tests pass, merge it and it goes live right away. We automated that process. A decision to merge implies rolling out the change. The minimum cycle time is the time it takes to build and deploy (about 7 minutes for the PR and another 10 to deploy) plus whatever time we need to do a change. Some changes are as small as 1 character. We have a production branch that we merge to from our master branch. We test the and use the master branch intensively and merge to production multiple times per week. No point in sitting on changes that work fine. Get it out and create some value for your customers and company and keep feedback loops short as well. Short cycle times is also why I use a rolling release linux distribution (Manjaro) and browser (Firefox). Always fresh and up to date. And even though I'm on the Firefox Beta channel I never have to deal with it breaking or being unstable. It's stable because they have frequent nightly builds. By the time builds hit the beta channel they are already rock solid. I was on the nightly channel for a while and never experienced many issues there either. Great example of short cycle times. With the Beta channel I'm a few weeks separated from changes happening and me seeing the feature. With the release channel it's another few weeks. Not updating because it aint broken is very valid until the time comes when you finally have to upgrade and all hell breaks loose because you are two years behind on dealing with breaking changes and have to do a massive project to make it happen. It was getting increasingly more broken while you were doing nothing; you just did not know about it. It's still technical debt. And now you get to deal with the non linear effort to fix it and pay the price. So, on all projects where I'm in charge we update everything very frequently. If something doesn't work I want to know ASAP and mitigate now instead of not even knowing stuff is not going to work for another few years. If you stay on top of changes like that, the effort for this is very low. Mostly stuff just works. Occasionally some library has an issue. And then we fix it, work around it or wait for the next version (and document why we can't update). Easy stuff. Basic project hygiene. The first thing I do when working on a project I haven't touched in a while is update dependencies. If I'm working on it all dependencies have to be current. I get annoyed with being a few minor versions behind. I might wait a few dot releases with major releases. But generally, I want to get that over with ASAP. If it breaks, I'll at least know that I need to deal with that. Rolling back is always an option.
m463 4y ago
I think there is a continuum... server folks want nothing to change, ever. desktop folks want the latest and greatest. mobile folks get spurious changes forced down their neck and want... well who cares what they want, this is what they are getting.
teawrecks 4y ago
The problem with "if it ain't broke, don't fix it" is that it's always broken for someone.
cratermoon 4y ago
I bet Equifax wishes they hadn't looked at their Apache Struts-based website and though "It ain't broke".
sedatk 4y ago
I mention this problem in my book Street Coder in the section titled "Fix it even if it ain't broken". There's value in changing code, but there's also risk in doing it. One of my suggestions is that any component that you avoid breaking, deserves to be broken the earliest. The rigidity (the resistance to change and the inclination to break) of code is a sign of greater trouble in the future, so should be addressed the soonest. That means, I don't agree with the "don't fix it if it ain't broken" sentiment fully. If that component's so valuable, it must be covered by hell a lot of test coverage to make it impossible to break it with changes in the first place. If you can break it without knowing, that's a problem that needs to be addressed immediately. I say, break it, identify breaking points, add tests to those points, so the component becomes flexible. My second suggestion is to make the changes, but discarding them after. I also emphasize this in the section titled "Write it from scratch". By doing that, you gain certain level of insight into that rigid design, and improve your understanding of the code base, which eventually leads you to the phase where you can finally code changes that you're comfortable sharing with your colleagues. You may think of the writing code only to throw it away a wasteful exercise, but I argue that it's not as great a loss as you may anticipate.
kazinator 4y ago
Release early and often, but make sure you're fixing something broken each time?
pjbster 4y ago
Odd that Kaizen doesn't get a mention here. The article draws battle lines between the two ends of the spectrum but there could be a middle way. In Kaizen, "Fail Fast, Fail Early" would never get ouf of the gate. The game here is all about making other, smaller bets which gradually ratchet the company to a better place without ever having to come off the rope. Perhaps this organisational conflict is rooted in the realities of burning through capital whilst still searching for a sustainable revenue model. In many ways the idea of Scrum is very Kaizen-like. It mandates the idea of experimentation and measurement. With trying stuff even when what you have appears to be working. And with documenting failed experiments and moving on. Nothing risky about any of it but you wouldn't know it when you see the typical reaction to "how about we move the standups to the afternoon instead of first thing in the morning?" The innovation pipeline described is encouraging but it's already too dogmatic in its proposed implementation. Kaizen is much more organic and, dare I say it, cultural. One of the key principles of Kaizen is empowerment and the innovation pipeline doesn't offer that at all.

128 points by yarapavan 4y ago

> Customers often ask me, why pour time into updates when what I have runs just fine? The answer is that a bug in your old version might assert itself at the worst possible time and force you to forego sleep for the next five days to recover Why not do proper testing ? I know it's expensive. And when you have an OS as an init system is even difficult. It is sad the the UNIX philosophy is dying being replaced with the Windows philosophy.

codegeek 4y ago
You cannot test everything to perfection. Shit will break in production. Question is: how to minimize shit breaking in production and hopefully eliminating critical bugs that can cause revenue/data loss.

peteradio 4y ago

How do people disentangle "ship the incorrect thing and let customers let us know what they really want" vs "ship the correct thing but built incorrectly and let customers know why its wrong". I hate the second with a burning passion because it seems like its bound to give the impression that you really don't know what you are doing. I've been put into situations too many times where I just could not get the point across that "yes I understand that we may not be giving the customers what they want, but what you are telling me to build is clearly not correct" its maddening. Sorry if its off topic but it seems at least adjacent to the articles balance. This seems to happen with PMs who do not have a deep enough background in the technical detail and yet reject the feedback given because it will reflect that poor understanding. "Agile" environments seem to enable this kind of behavior. I'm beginning to think I need to pivot to PM in my career so I can avoid the madness as a developer.

sumtechguy 4y ago
That depends on your customer. Some are cool with 'something is better than nothing'. Some are not. Some also are very 'I put it in the contract just do it that way' others are 'I just want the right thing'. Some just do not know what they want at all and just want you to do it and put all the liability off onto you but do not really want to spell out what that means at all. Being a PM is that turned up to 11 but now you do not control the code at all.
- peteradio 4y ago
  I guess what I'm talking about is this: 1) PO wants feature X for customer P. 2) Architect in meeting with PM and PO indicates that feature X is a simple addition to Y 3) Engineer is assigned X and says no this isn't a simple addition to Y because on deeper inspection it will fail for cases 1,2,3, we would need a different architecture. 4) PM says build it the naive way, let the customers find case 1,2,3 before we fix, we are an Agile team after all. 5) I quit.
  - antupis 4y ago
    I would do 4.5) escalate to PM boss everytime 1,2,3 happens.
  - helge9210 4y ago
    > PM says build it the naive way Why PM has an option to choose here? > will fail for cases 1,2,3 a) tip QA to test these cases b) add cases 1,2,3 to "Known issues" After all that is the reason they call it "beta": "Cause it beta then nothing"
  - skellera 4y ago
    Seems like you’re working with a PM who thinks agile is an excuse to release bad software. MVP should still solve a customer pain point. To play devils advocate though, maybe case 1,2,3 are low enough risk to release. Having metrics set to watch if these are actually big problems could be “good enough.”
  - 4y ago

henriquecm8 4y ago

Release early, don't fix it

_8j50 4y ago
This is the way.
tomalaci 4y ago
Creating bugs? No, I am creating learning opportunities for more junior developers.
- devteambravo 4y ago
  Are you my manager?
BurningFrog 4y ago
"Move Fast and Break Things" at maximum speed!

jbverschoor 4y ago

Those are two different things.. Nothing to balance about. But put it in a graph or drawing, and it means it's true and supported by 'science'

silentsea90 4y ago

How about use Feature flags and ship daily? Decouple your release-schedule/deployments from your product iteration/launch timeline.

allendoerfer 4y ago
Never miss an opportunity to add complexity!
- generalizations 4y ago
  Unofficial RedHat motto.
- int0x2e 4y ago
  Feature flags or another way to release bleeding edge features to only some of your users can sometimes be the best way to iterate over new features quickly when your most user base is risk-averse
- silentsea90 4y ago
  Centralized coordination complexity v/s decentralized FF rollout complexity. The bigger the team contributing to a deploy, the better the latter gets because engineers rolling out their FF have all the context they need.
alldayeveryday 4y ago
I think I know your meaning, though I would rephrase it as decoupling your deployments from your releases/launches.
- silentsea90 4y ago
  Edited.

sylware 4y ago

Grotesque generalities. The devil hides in the details: planned obsolescence and redhad (IBM) with its glibc symbol versionning frenzy and not a clean ELF ABI set of libs (as an example). Yep. Open source is not from enough, lean and stable in time open source is required.

torstenvl 4y ago

Simple: Release Early. Release Often until it Ain't Broke. Don't change things that Ain't Broke. I don't think there's much consternation over actual bugfixes. There's consternation over constantly-shifting UIs and APIs. The only time there's any tension between those two things is when people conflate bugs with poor design. The messy but correct thing to do when you've correctly implemented a poor design is to maintain backward compatibility while also offering the option to use the newer, more correct behavior. See, e.g., strcat() → strncat() → strlcat()

pabs3 4y ago
Preferably also provide assistance to people using the old things, in the form of making the old API a wrapper for the new API where possible, adding warnings for the old API, adding migration tools etc.
klodolph 4y ago
(I think it would be strcat() → strlcat(), since strncat() is not really safer / newer.)
- david2ndaccount 4y ago
  Just always use memcpy, you should always know the size of your buffers and if you don’t you will get a buffer overflow at some point.
  - klodolph 4y ago
    I've heard this advice before, and the problem with this advice is that people who just use memcpy in practice often write code with bugs in it. So the advice is bad advice. Good advice will result in code that has fewer bugs, or bugs with lower severity.
    - kevin_thibedeau 4y ago
      It can also lead to accidentally quadratic behavior which is why the modernized string ops that get so much hate these days rely on one iteration over the input.

jrochkind1 4y ago

I think libraries (used as dependencies by other software) and "top-level" software/applications (used by users directly, whether that user is a developer or not) -- have different pressures and concerns. Libraries should be much more conservative. I don't think "release early and often" was said about libraries. And yes, it's tricky that it's not always a clear line between these two categories. A unix command line utility is sort of top-level software, but also likely to be use in a scripted/automated fashion. So it's not always cut and dry, but the more you are aware of software depending on your software, the more careful you should be with releases.

wongarsu 4y ago
A great example of an end-user application that is like a library is Microsoft Excel. Sure, the UX can change, but the formula engine has to be extremely conservative, staying bug-compatible with basically every version, or all hell breaks lose.
JohnFen 4y ago
I agree. As a developer using libraries and component, I really hate "release early and often". As a user, I also hate "release early and often". In both roles, releasing early and often effectively means I'm always using beta software, with all the headaches using such software brings. However, as part of an internal development process -- that is, before customers see the release, "fail fast" is a totally legitimate and decent approach.
wpietri 4y ago
I think it depends a lot on where the library is in its lifecycle. A new library should iterate rapidly and be very clear about that. Big warnings, obvious version numbering, and good communications. You look for an audience of actual users that can keep up with that. That's how you figure out what the right library is, both in terms of the big abstractions and the small details. But then once you hit a 1.0 release, things change. You're shifting from an audience of fellow explorers to an audience who wants stability. You can still do exploratory work, but it has to be additive in the 1.x series, and eventually you need to start a 2.x series as you learn more about what needs to change. So the iteration still happens, it just happens away from the people who just want the basic thing you got right via the initial burst of iteration.

eloisant 4y ago

They're not opposite. "If it ain't broke don't fix it" => don't try to "optimize" something that works well. Maybe it will be able to handle more load after your big refactor, but if it already handle the load nicely and there is no indication load will go up, don't bother. "Release early and often" => release after each change, don't bundle everything into a big release that happen every 6 months. The first one is about deciding what to do, the second one is how you release the work you did.

JohnFen 4y ago
> "Release early and often" => release after each change, don't bundle everything into a big release that happen every 6 months After years of dealing with software that does this, I've soured on it completely. It has resulted in greater difficulty using software, and the software being less solid.
- mdtusz 4y ago
  I've seen this happen as well, but I think it's more of a problem of bundling too small a changeset in a release. A partial implementation of a feature shouldn't warrant a release (without feature flags to disable it), but bundling a year's worth of work in a single release is, in my opinion, a recipe for disaster (and this is sadly something that happens all too often in enterprise and non-software engineering businesses).
- michaelcampbell 4y ago
  I've seen this happen as well; perhaps for different reasons. My early career was mostly fin-tech and when installing into banks or other financial institutions, they _very much_ want the every year or 6 mo release cycle. And not just because of "old stodgy", but rather they want a good bit of ramp-up time to ensure things are running right. In a Wall St trading firm I worked at, a release ran in parallel with real data for MONTHS before the old version was retired. So, I'm used to that model. From the customer's standpoint, they could "gear up" for the next release, get their ducks in a line, get the right people to assess things. I get that some of that is because the infrequent releases are "big", but also as a human customer it's fun to see a shiny new truck toy rather than a coat of anti-rust paint on its exhaust pipe, then a replacement for the broken window crank, then a new color on the odometer numbers, then ... As a developer I also miss the Big Project Ramp Up, then a period of a lot of work (which can be Waterfall, or Agile, or bespoke artisinal project management, which is pretty much 99% of the time, just CALLED "Agile"), then a Big Project Release (+ party, some down time, and prepare for the next one). SAAS release early/often is like death by 1000 boring dull cuts. They cut out the spikiness _so much_ that there is so little novelty that it's just not fun to work on it.

AaronM 4y ago

Error Budgets are your friend

chuso 4y ago

Break it often and don't fix it.

jjslocum3 4y ago

I would love to read a Malcom Gladwell book on this topic. It touches on a core truth, not only in engineering but in society as a whole. Three examples from different spheres: In software, I've seen repeatedly how new hires with big ideas (and youthful confidence that they know a better way of doing things) will come into a company and want to rewrite things "properly." In one management position, much of my role involved the politics of defending a well-working, well-maintained, big-revenue-driving application from a steady onslaught of such ambitious exuberance. At the time, I thought "this must be the same phenomenon that drives shrinkflation": new MBA arrives at pickle-making company, seeks to bolster their career by demonstrably saving the company millions, convinces execs to put one less pickle in each pickle jar, consumers won't notice; step 3: profit!! In 2022, when you buy a cereal box, it's half air on the inside. In 1980, it was only 20% air. I also shake my head when media pundits equate the success of a particular congressional session with the amount of new legislation they pass. I can imagine no simpler way to cruft-paralyze a democracy (while following the rules) than "releasing laws early and often."

SQueeeeeL 4y ago
Shrink-flation is driven by short term quarter earnings thinking. Reducing the amount of your product (and therefore it's overall quality compared to competitors), is an act of cannibalization of a brand. They may increase profits for a short time, but as consumers realize what's happened, ultimately they see the brand as not having standards and lead to it becoming a commodity. A small group of long term invested owners would never do such a thing, because that would be 'killing the goose who laid the golden egg', but modern shareholding has no loyalty to quality.
klodolph 4y ago
> In software, I've seen repeatedly how new hires with big ideas (and youthful confidence that they know a better way of doing things) will come into a company and want to rewrite things "properly." In my experience, these new hires are right a certain percentage of the time, and wrong a certain percentage of the time. I can take no default position here. I tell new hires that they're gonna see something that offends their sensibilities about the way code should be written. That puts them in the center of a conflict. On the one hand, they believe the code should be written differently. On the other hand, someone else wrote the code that way for a reason. Their first step is to dig in and figure out the reasons on both sides--why it was written one way, why they think it should be written another. The second step is to figure out what the consequences and risks are of making changes or not making changes. The way I think about it, I want to simultaneously keep the software working, and protect the "flame" that new engineers have to improve things--something which can be all too easily extinguished.

gdsdfe 4y ago

“If it ain’t broke, don’t fix it” is the absolute worst mindset to have! I've seen systems that are literally garbage because of it.

scombridae 4y ago

tl;dr continually test, continually deploy.

onetokeoverthe 4y ago

ChrisMarshallNY 4y ago

Man, I feel this. I worked for a corporation that was so change-averse, that we needed to buy our development systems on eBay. They actually had a point, and it was hard to argue against, but I found it absolutely infuriating, as the solution was to plan for change, and establish a process to be constantly evaluating and refining for new developments. Instead, it was "Wait until we can't bear it any longer, then have a huge screaming match meeting." A new budget would be approved for new machines (and, thus, new operating systems and development tools), processes would be updated, and that would sit, until it was no longer new. As I said, it was hard to argue against, because it was a 100-year-old company that had been successfully delivering really high-end stuff, since Day One. They did that by being so conservative that they hadn't discovered fire, yet. Measure 300 times, cut once, etc. When there was a problem, it escalated quickly (and was used as fuel to tighten things down even more), as the company was held to standards that are probably up there with NASA. Speaking of NASA, I can't help but notice that this rather plucky little outfit, called SpaceX, seems to be running circles around them. They seem to have figured out how to "fail fast," yet also deliver insanely high Quality stuff. Might be worth ignoring their CEO's tweets, and look at what they are doing...

pclmulqdq 4y ago
SpaceX shows the success of using modern, commodity hardware rather than purchased-by-government contract obsolete components for space applications. The same is true for the cubesat revolution. I honestly think the idea of "if it ain't broke, don't fix it" is pretty silly as applied to large, engineered systems. If your system is more than 1000 lines of code, I guarantee you have a latent bug somewhere. Even if you don't have any bugs, the ecosystem around you changes so fast that you eventually will.
- Gibbon1 4y ago
  SpaceX shows the advantage of not allowing aerospace lobbyists and congress to design things. That's how you get the SLS with it's inane H2/LOX first stage and solid propellant boosters.
mgbmtl 4y ago

unity1001 4y ago

Just keep things backwards compatible, then do whatever you want.

PaulDavisThe1st 4y ago

TFA seems to try to pin DEC's collapse & failure on their committment to back compatibility. Yet Microsoft has essentially done the same thing with Windows, and has not suffered the way DEC did. Seems like a weak argument.

esaym 4y ago

And if it ain't broke, fix it 'til it is!

temporallobe 4y ago

We have relevant case of this. We’re on Angular 8 and the app is working just fine - customer is happy, it’s pretty easy to maintain, well-tested and hardened, etc., but we are now facing having to update to 14.x “or else”. It’s gonna be a long and difficult migration.

kmitz 4y ago
Well depending on the size of your codebase, npm dependencies and Angular APIs used, it might or might not be a big deal. You can have a rough estimate of the complexity by running 'ng update'. Yes major releases come at a rapid pace and it is hard to always be on the latest version. We update regularly once we are 2-3 major versions behind and usually it's not so awful.

pdpi 4y ago

Lord Vetinari in the Discworld lives by the mantra that "If it ain't broke, don't fix it". Other characters noted that, under his governance, "things that didn't work ... got broken."

exabrial 4y ago

The problem is a bunch of overpaid devs conflate anything that isn't absolute bleeding edge as a "bug", and we get into and endless cycle of UI refreshes and updates that exist solely to exist.

bradwood 4y ago

These are not mutually exclusive. It seems like the whole premise here just lacking nuance.

jmartrican 4y ago

More often than not its "if it ain't broke don't fix it because we are too busy releasing early and often".

arminiusreturns 4y ago

Release to prod so you can fix stuff in lower envs.

GnarfGnarf 4y ago

"If it ain't broke don't fix it" Would you fly on an airline that followed this principle? Wait till the engine fails before fixing it. In the air.

jillesvangurp 4y ago

Reducing cycle times for changes minimizes integration testing effort. Effort and risk increases non linearly with the amount of change and time. So, release as often as you can to minimize the risk and effort while balancing the cost of doing so. Some release processes just involve a certain amount of heavy process that takes time and money. E.g. app store releases are a PITA so doing that multiple times per day is not worth it. Server updates on the other hand are fine. Create a PR, tests pass, merge it and it goes live right away. We automated that process. A decision to merge implies rolling out the change. The minimum cycle time is the time it takes to build and deploy (about 7 minutes for the PR and another 10 to deploy) plus whatever time we need to do a change. Some changes are as small as 1 character. We have a production branch that we merge to from our master branch. We test the and use the master branch intensively and merge to production multiple times per week. No point in sitting on changes that work fine. Get it out and create some value for your customers and company and keep feedback loops short as well. Short cycle times is also why I use a rolling release linux distribution (Manjaro) and browser (Firefox). Always fresh and up to date. And even though I'm on the Firefox Beta channel I never have to deal with it breaking or being unstable. It's stable because they have frequent nightly builds. By the time builds hit the beta channel they are already rock solid. I was on the nightly channel for a while and never experienced many issues there either. Great example of short cycle times. With the Beta channel I'm a few weeks separated from changes happening and me seeing the feature. With the release channel it's another few weeks. Not updating because it aint broken is very valid until the time comes when you finally have to upgrade and all hell breaks loose because you are two years behind on dealing with breaking changes and have to do a massive project to make it happen. It was getting increasingly more broken while you were doing nothing; you just did not know about it. It's still technical debt. And now you get to deal with the non linear effort to fix it and pay the price. So, on all projects where I'm in charge we update everything very frequently. If something doesn't work I want to know ASAP and mitigate now instead of not even knowing stuff is not going to work for another few years. If you stay on top of changes like that, the effort for this is very low. Mostly stuff just works. Occasionally some library has an issue. And then we fix it, work around it or wait for the next version (and document why we can't update). Easy stuff. Basic project hygiene. The first thing I do when working on a project I haven't touched in a while is update dependencies. If I'm working on it all dependencies have to be current. I get annoyed with being a few minor versions behind. I might wait a few dot releases with major releases. But generally, I want to get that over with ASAP. If it breaks, I'll at least know that I need to deal with that. Rolling back is always an option.

m463 4y ago

I think there is a continuum... server folks want nothing to change, ever. desktop folks want the latest and greatest. mobile folks get spurious changes forced down their neck and want... well who cares what they want, this is what they are getting.

teawrecks 4y ago

The problem with "if it ain't broke, don't fix it" is that it's always broken for someone.

cratermoon 4y ago

I bet Equifax wishes they hadn't looked at their Apache Struts-based website and though "It ain't broke".

sedatk 4y ago

I mention this problem in my book Street Coder in the section titled "Fix it even if it ain't broken". There's value in changing code, but there's also risk in doing it. One of my suggestions is that any component that you avoid breaking, deserves to be broken the earliest. The rigidity (the resistance to change and the inclination to break) of code is a sign of greater trouble in the future, so should be addressed the soonest. That means, I don't agree with the "don't fix it if it ain't broken" sentiment fully. If that component's so valuable, it must be covered by hell a lot of test coverage to make it impossible to break it with changes in the first place. If you can break it without knowing, that's a problem that needs to be addressed immediately. I say, break it, identify breaking points, add tests to those points, so the component becomes flexible. My second suggestion is to make the changes, but discarding them after. I also emphasize this in the section titled "Write it from scratch". By doing that, you gain certain level of insight into that rigid design, and improve your understanding of the code base, which eventually leads you to the phase where you can finally code changes that you're comfortable sharing with your colleagues. You may think of the writing code only to throw it away a wasteful exercise, but I argue that it's not as great a loss as you may anticipate.

kazinator 4y ago

Release early and often, but make sure you're fixing something broken each time?

pjbster 4y ago

Odd that Kaizen doesn't get a mention here. The article draws battle lines between the two ends of the spectrum but there could be a middle way. In Kaizen, "Fail Fast, Fail Early" would never get ouf of the gate. The game here is all about making other, smaller bets which gradually ratchet the company to a better place without ever having to come off the rope. Perhaps this organisational conflict is rooted in the realities of burning through capital whilst still searching for a sustainable revenue model. In many ways the idea of Scrum is very Kaizen-like. It mandates the idea of experimentation and measurement. With trying stuff even when what you have appears to be working. And with documenting failed experiments and moving on. Nothing risky about any of it but you wouldn't know it when you see the typical reaction to "how about we move the standups to the afternoon instead of first thing in the morning?" The innovation pipeline described is encouraging but it's already too dogmatic in its proposed implementation. Kaizen is much more organic and, dare I say it, cultural. One of the key principles of Kaizen is empowerment and the innovation pipeline doesn't offer that at all.