4 ms·
> why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point This seems to be exactly the point! For 8 y
by lolinder 3y ago
> why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point
This seems to be exactly the point! For 8 years they left unused code in place, seemingly only bothering to remove it because they wanted to repurpose a flag. If they'd done the right thing 8 years prior and removed code they weren't using, this story plays out very differently. No ancient routines get resurrected, no rogue server.
Maybe Knight Capital wasn't using version control and held onto this code "just in case", but I've seen this same resistance to deleting code in programmers working in repos that are completely under VCS, and it's flabbergasting. If you need it again, you can always bring it back from version control. If you need it again but forget it's there, you'd do the same with the dead code path. Leaving it in the source tree is pure liability.
EDIT: Kevlin Henney gave an excellent talk at GOTO about software reliability and he touches on this, using Knight Capital as the example—he actually cites this very blog post [0]. The whole talk is excellent, but I've linked the three minutes where he talks about Knight Capital.
> The problem is there is no code that is truly dead. It turns out all you need to do is make a small assumption, a change of an assumption and then suddenly it's no longer dead, it's zombie code. It has come back to life and the zombie apocalypse costs money.
[0] https://youtu.be/IiGXq3yY70o?si=hZ9HB2dlfj0vHvNK&t=463 https://youtu.be/IiGXq3yY70o?si=hZ9HB2dlfj0vHvNK&t=463
- datadrivenangel 3y agoMaybe they had engineering leadership mandates where creating new feature flags was more paperwork? I've seen that happen, where 'new' features took a lot of justification, but 'bugfixes' were free.
- SamuelAdams 3y agoVersion control isn’t bulletproof either. All your code history is one git rebase away from being abolished forever. I hope most orgs have processes around their main branches so this does not occur, but I’ve also been in smaller orgs and accidentally screwed up prod database tables, so the accidental git rebase isn’t impossible to consider…
- chii 3y ago> All your code history is one git rebase away from being abolished forever unless you also GC right away, it's not gone.
- Terr_ 3y agoYep, and also: 1. You shouldn't be allowing anybody to force-push rebased stuff onto major branches in your main repo (the one builds come from) anyway. This is especially important to support auditing and trace-ability. 2. Just because the folder is a git-repo isn't an excuse not to have it part of your regular offsite backup set.
- gorgoiler 3y agoThere are reasons to want branch protection but audit isn’t one of them. An auditable system would be keeping an immutable log of the git repo actions in a separate, append-only location. It wouldn’t rely on the thing being written to all the time, by users, to never get broken. In your model, your audit history only needs one bug in the branch-protection tool for it to be destroyed. Think of it like syslog. It’s good to keep a log of events but it’s bad to rely on the /var/log/syslog on your web server. You should be logging to a remote, append-only system. (However I would concede that if we are talking about lawyer-proof-audit — SOC2, ISO etc — rather than actual security auditing, then branch protection is probably just fine.)
- lolinder 3y agoYes, you should definitely have branch protection turned on on main, I kind of assumed that went without saying. But to actually lose your entire git history would require both having no branch protection and having every single developer on your staff be in the regular habit of force rebasing their own branches on main. If a single developer does a double take when they're told that their branch has a different history than origin/main, then you probably didn't lose more than a week of work. Also, it's worth noting that this dead code will almost certainly not get reused wherever you decide to store it, so it's best to keep it as far away from zombification as possible, even if it's not the safest place.
- SoftTalker 3y ago"If it works, don't touch it" is something I've heard a lot, especially said by managers who don't understand what they are talking about. An update to simply "remove old code" might be difficult if someone sees any change as creating a risk of something going wrong. And to be fair, any change is a risk, but so is leaving old code around. At least now we have this case to point to as a clear example of the risk.
- asalahli 3y agoI'm less surprised by them leaving dead code around, honestly. I've seen that happen at virtually every company I've worked for. What completely baffles me everytime I come across their story is that they repurposed an existing flag instead of making a new one! Why?
- TheDong 3y ago> I've seen this same resistance to deleting code in programmers working in repos that are completely under VCS, and it's flabbergasting I think a lot of developers only know the basics of git. They can check in changes, they can look at history with git log, and maybe they know how to use git blame. They often don't know how to filter git history. They often don't know about the git pickaxe, or about exclude patterns, and don't even think to question if you can do something like "git log -G'int.*foo\(' -- ':(exclude)directory'" to search for 'foo' in the git log, excluding some directory. They know how to "grep" within the existing code tree though, so they know if it's not deleted they can find it again with the right grep. If it's deleted, they might not know how to find it in git history. I sympathize with this to a degree actually. Code in the git log is invisible to a lot of tooling, so for example it won't show up in autocomplete if your text editor might have otherwise suggested it, it won't show up in your library documentation, etc. If you truly think the code will be used again, I think it's at least defensible to leave it in the tree so that it comes along for refactors, and ends up being found when it's needed. For cases like Knight capital, where it's obviously never going to be useful again, it's not defensible of course.
- lolinder 3y agoI can understand that argument for small, standalone functions. Where it really gets at me is when people insist on leaving whole use cases or subsystems in place, which seems to be what happened here. You don't want these to be visible to autocomplete, because they're outdated and would need major modification to be correct again. If you do need to resurrect them, they're trivial to find in the git history—just search for the commit named "remove foo"—and they should pass through code review as if they were brand new code, because a lot of stuff will have changed around them in the intervening time.
- bhasi 3y ago> git pickaxe, or about exclude patterns, and don't even think to question if you can do something like "git log -G'int.*foo\(' -- ':(exclude)directory'" to search for 'foo' in the git log, excluding some directory Learned something new today, thank you! Will find ways to use these in my daily workflow.
- 3y ago
- mirekrusin 3y agoThe morale should be "don't reuse feature flags kids".
- philjohn 3y agoI always remember the Codeless Code on comments when I read about this: http://thecodelesscode.com/case/41?topic=comments http://thecodelesscode.com/case/41?topic=comments
- pnt12 3y agonitpick: git is enough to keep the history of old code, but it's not great at letting you find it.
- alexpotato 3y agoSee my sister thread where I worked at Knight just after the outage. > For 8 years they left unused code in place, seemingly only bothering to remove it because they wanted to repurpose a flag There was another issue where they were using a database with only 256 columns. Sometimes, they needed a new column so they would just reuse an old columns that "wasn't being used at the time". IIRC, this was generally acknowledged internally to be "a bad idea" but no one had prioritized cleaning up the old code and/or coming up with a better best practice.
- nijave 3y agoImo it's easier to quantify the value of adding new code than removing old code (in terms of dollars). Things that are more easily to quantify the value of tend to get higher priority. Same applies to other areas like performance optimization, testing, security, patching (some of these have high level data like the cost of incidents/downtime or the cost of a breach) Some companies have processes in place to try to balance this like 20% of time spent on tech debt reduction