21 ms·

It Takes 6 Days to Change 1 Line of Code (2015)

568 points by Mz 10y ago

smegel 10y ago
Become a $$$/hour contractor and learn to love the bureaucracy. Then wait for those all-too-often moments where a manager's ass is on the line if feature X doesn't ship by tomorrow and watch all the rules and regulations get chucked wholesale out the window. Then chuckle.
- eplanit 10y ago
  Exactly. The mantra I was taught when I first began consulting, when faced with frustrations like these was (adapted from the old school Hawaii Five-O): "Bill 'em, Danno!" I've been at it for 20+ years and it truly works.
ztratar 10y ago
This is why startups kill big companies. Lots of stupid little processes that add up and make the managers feel power, but end up killing the performance of the business. Brings up the interesting differences between leadership and management. Managing is what gets people into situations like this, whereas leadership (in late stage companies) is what gets people questioning these processes instead of moving like cattle into a slaughter. I saw this exact type of behavior at General Electric back in 2011. People wanted to argue about acronyms and process design all day, whilst Samsung ate up 20% market share in 1 year. That business of GE no longer exists and was sold to a foreign buyer. Move fast or die -- this rule does not only apply to startups!
- maxxxxx 10y ago
  Considering how inefficient big companies are it's surprising that they don't get killed in the market immediately. You would expect that there are plenty of nimble newcomers who kick their asses but that's not really happening.
  - Afforess 10y ago
    Two words: Regulatory Capture. See Also: Boeing, IBM, CGI Group, AT&T, NRC, etc.
    - grzm 10y ago
      I don't read that as being the point of the article. Regulation is not even mentioned or alluded to. Also, my understanding of regulatory capture has little to do with regulation slowing down business. If anything, rather the opposite. But that's off-topic anyway, and definitely not appropriate for this week. Edited to update comment on regulatory capture.
      - Afforess 10y ago
        I don't know what you're talking about, it's in there plain as day. The company has a department called IT Security. Not IT. Not Security. IT Security. Companies that legitimately care about security integrate security personnel with all of their operations, into every department. Top-down security departments exist to check boxes on regulatory forms. So I will be shocked if "IT Security" is not a made-to-order department whose existence is solely to satisfy things like PCI, HIPAA, etc.
        cortesoft 10y ago
        What? In any large company, you are going to have access controls - not everyone is going to be allowed access to every system. In the example given, the guy was requesting access to a new system he didn't previously have access to; having a security team that is in charge of that access doesn't seem to have anything to do with regulation.
        trdtaylor1 10y ago
        True, but the greater danger of integrating security into every department is discounting security in favor of the main requirement of your department. Your department is viewed for political and performance reasons, budgeting time away from the main goal of your area towards security is technically ... waste. Additionally, you ignore the business logic cross-departmental issues when you do not have a centralized security department.
- robotsonic 10y ago
  Reminds me of the bike shed effect: https://en.wikipedia.org/wiki/Law_of_triviality https://en.wikipedia.org/wiki/Law_of_triviality
- lurkinggrue 10y ago
  Startups can afford to have their critical systems explode.
- vacri 10y ago
  So... one of the places I work is one of those 'nimble little startups', and it gets more and more bogged-down by the day due to some poor technical decisions taken early. There are seven different kinds of timestamp in the database, for example. That's pure dev overhead that gets applied for everything. And because the nimble little startups are pressed to work on new features rather than refactoring, more tech debt just gets added to the pile... Also, don't forget selection bias - most small companies, startups included, die in the first few years. Only the success stories make the papers.
- jlarocco 10y ago
  It's easy to criticize this type of thing as wasteful and inefficient for the one time in a thousand where it really is just a single line code change that doesn't break anything, but the reason things like mandatory code review, required tech debt fixes, and all of the rest end up existing is because developers are notorious for saying, "Oh, it's just a one line change!" and then breaking everything when they slam it through to production without thinking about it.
- mevile 10y ago
  > whilst Samsung ate up 20% market share in 1 year Right a 78 year old company with hundreds of billions of dollars of revenue is a startup killing big companies. I'll just note that while Samsung is about half of G&E's age it makes twice as much in revenue.
  - gamegoblin 10y ago
    The comment you are replying to included: Move fast or die -- this rule does not only apply to startups! So I think they are aware that Samsung is not a startup.
  - taneq 10y ago
    Apparently "startup" doesn't mean "company which has just started running" these days, it means "company which is growing rapidly."
- deleted 10y ago
  [deleted]
- Johnny555 10y ago
  It's easy for startups to be nimble because they have no legacy processes to maintain. The startup may have 5 guys in a warehouse building their product, while the big company has 100 people on a $10M assembly line. When the startup wants to scale by 20%, they work longer hours or add another worker. If they didn't plan properly and find that they don't have another workstation for that worker, they buy a table at OfficeDepot and there's their new workstation. The assembly line might need an entire process flow redesign to add more workstations, plus a physical reconfiguration, software changes, etc. If a developer just blindly adds a parameter that adds 10% to the workday without going through reviews and tests, he may find out from the physical plant engineers that they no longer have enough time to do preventative maintenance so they need to scheduled an outage to do deferred maintenance. There's a reason why process exists in large companies, and it's not just for the sake of process. That small startup might be able to usurp the large corporation, but the large company still has an advantage in throughput and/or efficiency. The corporation can promise Walmart that they'll have 100,000 units delivered by Dec 1st... to meet the same goal, the small startup might have to scale their workflow 100X, and they can't do that by scaling from 5 guys in a warehouse to 500 guys in a warehouse.
  - tonyedgecombe 10y ago
    I don't know, for every justifiable decision I've seen in corporates there have been ten where it was all about politics, needless bureaucracy, incompetence and power and nothing to do with getting things done.
- ap22213 10y ago
  I don't know. It seems more like 99% of startups die, and 99% of large companies stick around. Not that I enjoy working for large companies, but just saying...
  - ryhamz 10y ago
    Right. For every startup that skips process and gets lucky there is a pile that rolled snake eyes and died. Big companies aren't taking the chance.
    - flukus 10y ago
      Big companies tend to not take that chance even when it's worth taking though. Not everything is mission critical but the procedures around them pretend they are.
  - TheOtherHobbes 10y ago
    Large companies stick around because they've solved the one problem all startups have - finding and keeping customers and turning them into revenue through sales and product pipelines. Code is pretty much a footnote to that process. It's exactly one of very many tools a business can use. It's true that technical debt can cause business problems. But it doesn't cause nearly as many problems as having an ineffective sales team, getting killed by regulation, or being eaten alive by a competitor.
  - B1FF_PSUVM 10y ago
    > 99% of startups die And no one bothers suing into the ground a muppet with two packs of ramen to their name ...
renownedmedia 10y ago
[2015]
LargeCompanies 10y ago
Didn't read this but is this about a government job... lol
hacker_9 10y ago
I want to fault this but I can't. Code reviewer who's not afraid to reject code? Check. Proper testing of code changes? Check. The process probably looks silly in this case because nothing went wrong. But with a hard-coded variable being changed, who is to say it won't break some other system by being changed? And if the variable was then moved to a config file, who's to say again that it might not break in some obscure way? Such as if the config file is not found in the uat/test environment? Additionally after all this, the manager can change the variable in the config file as they please without this process need going through again.
- zzalpha 10y ago
  My reaction exactly. And 2 days were lost to escalation while 2 were spent in test. Everything else didn't look crazy to me if the change is small but has a high system impact. For an industry that is implicated in downed spacecraft, massive security vulnerabilities, and a reputation for unstable products, it's amazing how happy we are to optimize for developer satisfaction over all other metrics (I'm looking at you, Agile). Edit: And before anyone says it, I'm not claiming this is a good process. But coding velocity is just one metric by which we should be measuring our success.
  - dreamfactory2 10y ago
    we should measure value, that is all. and that's what agile is supposed to aim for, not developer comfort. these processes offer negative value (prohibitive costs of change and inability to execute in time which massively outweigh any perceived security). even a vaguely leaner organisation will eat their lunch. and that error-proofing is entirely imaginary - they have added a ton of moving parts, handoffs, and manual processes and so will have way more screw ups than a small tight team practising continuous delivery, as anyone who has worked in such environments will know.
    - delinka 10y ago
      "value." I'm getting this BS from a PM right now over a refactor job that needs to happen. Sustainability and tech debt be damned. Every time they want to support a new file type, it's editing several files, most of which are source, recompile, put QA resources on it, get IT to sign the binaries ... If you'll get off my back, let me refactor and get my "developer comfort" on, you'll never have to involve me or QA or IT in supporting Just One More File Type ever again - you can push that from policy on the server side. But "developer comfort" isn't "value for the customer." _sigh_
manarth 10y ago
This sounds like perfect is the enemy of good.
kazinator 10y ago
That can be okay if the same 6 days can be used in parallel to change 999 other lines of code, too. I.e. "1 line in 6 days" is only a measure of latency, and not of throughput.
- vacri 10y ago
  > a measure of latency, and not of throughput. You could drive a station wagon full of backup tapes quite far in 6 days :)
  - kazinator 10y ago
    It's not about how far, but how fast it is going. The bandwidth of that wagon is about how fast those tapes pass a given point of travel. How they are arranged inside the vehicle is critical. Tapes moving at 100 km/h side by side have more instantaneous or "burst" bandwidth than tapes moving at 100 km/h in a single file.
perilunar 10y ago
I once waited 6 weeks for a backend team to fix a prematurely closed div tag that was screwing up the layout of a news site. The templates they were given were good; they screwed up the implementation. In desperation I had to hack the layout with JS (which I was able to update at will).
- rewrew 10y ago
  That kills me to hear that. I'm in charge of Web where I am and we make 95 percent of our changes instantly. There is no reason to make people wait; it honestly takes longer to project manage most changes than to just make them, so we just make them, but of course Web sites/front end is a different beast than application or other code.
nickthemagicman 10y ago
Thats a dream orocess right there right? The solution imo is to tell the higher ups to fast track it if peoples jobs arecat stake.
Wellshit 10y ago
Well, I've seen 5 line patch take 6 months (and counting), so...
- bbcbasic 10y ago
  Yeah it's sad I was thinking '6 days? that's pretty quick'
- weavie 10y ago
  I've had 6 months of work thrown out because management changed its mind over which approach to take. So 0 lines in 6 months..
sthatipamala 10y ago
Are there really factories whose policies and parameters are encoded in software vs. under the guidance of humans? That in itself is amazing to me considering that the tech company I work at, lots of things are just done by fiat.
nsoldiac 10y ago
I'm surprised no one has raised the problem of senior managers and VPs underestimating and oversimplifying the complexity and LOE of work. It causes unrealistic timelines for others who do have to adhere to a release process (which is never perfect but certainly necessary). "Yes CEO, of course we can get that in by tomorrow, no problem." When the work misses the impossible deadline the feedback usually points to an underperforming organization and rarely to unrealistic estimation.
joslin01 10y ago
I don't think I would eat 24 excedrin over something like this.
patmcguire 10y ago
Favorite line: "Ed: Fuck that shit." "Shirley: That may very well be true."
partycoder 10y ago
If technical debt accumulates, you slow down until the point you need a lot of time to understand the consequences of changing code. Pretty much like the end game in Jenga. One good approach is SOLID: https://en.wikipedia.org/wiki/SOLID_(object-oriented_design) https://en.wikipedia.org/wiki/SOLID_(object-oriented_design)
valine 10y ago
Related anecdote: I interned for a medical device company over summer of 2015 and worked on a team that was in charge of maintaining several implantable class 3 medical devices. The devices used a short range proprietary wireless standard to talk with a intermediary device which could be connected to via Bluetooth. One product had a pretty serious usability issue where connecting to the intermediary device could take upwards of several minutes. After the issue was discovered it took roughly an afternoon and several lines of code to fix. However, in order to provide software updates for a class 3 medical device each update needs to be individually approved by the FDA. It took 12 months from the time the fix was implemented to when the FDA allowed it to be pushed out.
- hughes 10y ago
  I think the word you want is "anecdote".
  - valine 10y ago
    thanks
- joshvm 10y ago
  What is the approval process? Do the FDA manually check the device (similar to FCC certification for wireless things)?
  - valine 10y ago
    I never had the opportunity to be involved in the FDA submission process, but from what I understand the FDA does not do any testing of their own. It's the companies responsibility to prove to the FDA that their device is safe and effective. All the testing is done by the device company, carefully documented, and then reviewed by the FDA. I've heard stories that the documentation for a class 3 medical device can be upwards of 20,000 pages which are all printed and handed to the FDA in 3 ring binders.
joeld42 10y ago
This is why it's important to have a "hotfix" process where you can make a change fast and then still follow up with all the QA and coding standards.
deleted 10y ago
[deleted]
patmcguire 10y ago
It seems like buried in all the other policies in one meta-policy: nothing gets done until someone with sufficient authority overrides all the rules. How often does the president have to do this? Has anything ever been done the correct way?
- spydum 10y ago
  Reality is in a large org, process can usually be sidestepped with emergency change requests and executive sponsor. Which is basically what happened here, it just took extra days because it wasn't started that way.
ipsin 10y ago
In Case of Emergency Break Glass There really needs to be a procedure for violating procedure, so that when higher-ups say "do this immediately", those same higher-ups can own the consequences of violating procedure, and so that they know the business implications of the rollback plan, etc. If the bug is "people are getting laid off" or "our customer database is being siphoned by an SQL injection", it probably should not wait.
- Falkon1313 10y ago
  The process sounds fine, no need to violate it. The problem was that the IT Admin said "I'll put this on the fast track" but didn't, and no one else saw the urgency. It was just dumped at the back of the queue behind a lot of other things and mistakenly labeled an 'Enhancement'. Proper handling would have had a dev working on it within the hour. The response to the code review would have been "other changes are out of scope, open another ticket for enhancements". If the developer didn't already have access to the test environment, at least IT would know that they're authorized for it. (Critical fixes shouldn't be assigned to a new hire who doesn't know the code base and doesn't have access to commit code yet.) The ticket would be edited (Test Case: view WorkOrdersHours report, Expected Result: ~10% higher total) and tester would be told to reread it and go ahead. And of course documenting the test runs is the tester's responsibility (they're the one doing the test runs), not the developer's, as is notifying the IT Admin and Operations Manager when the testing is done and it's ready for sign off. Exact same process, same quality checking along the way, timeframe reduced from 6 days to 4 hours or less; likely less than 1 if the team works together well and their tools are decent. A good process can handle emergencies without sacrificing quality, as long as the people work together and the priority is communicated.
cthulhuology 10y ago
I this had happened at any of the companies I started, I would have fired several people in the aftermath. The problem is the policy is broken by design. Let's look at what is wrong here: 1.) priority is not just high, it is critical, communicating this is lost at each layer (executive, planning, execution, process control, quality control) 2.) leadership is lax, the chain of command doesn't designate a clear single responsible individual 3.) policy enforcement in this example actually increases the risk of an unsatisfactory outcome, by increasing the complexity of the solution vs. what is in production 4.) quality control is adversarial and ass backwards, code review is supposed to be a sanity check "does this code do what the developer thinks it does" aka. "can some other person understand it" 5.) test planning should not be the developer's responsibility, quite frankly if QA can't figure out if it is working or not you should fire your QA department. 6.) Ultimately, it is a total failure of policy and management as it requires the President of the company to micromanage the situation. If you think any of this is fine, I'm sorry but your company is doomed to fail (unless it is already so big it is too big to fail).
- kevinstubbs 10y ago
  You would fire them, instead of trying to build up their skills? I'm not even sure who you would fire in this situation, the IT director for not shepherding the ticket all the way through, or Shirley for closely following company policy? Either would be overreacting. Slow/inflexible company policy has been built up, presumably under the IT Director's watch. The president didn't communicate that it was extremely urgent (and maybe it wasn't - the article doesn't say when the change actually needed to be made by to make a difference). Ideally, management can recognize that this change took a week because of their accumulated decisions, and lack of urgency. They should figure out exactly how their company policy is slowing them down instead of letting heads roll.
  - user5994461 10y ago
    Don't fire. Don't touch anything. Just remember the chapter 12 of Peopleware. Between two organizations, the productivity can vary by 10 times. It's not about you, it's a social characteristic of the organisation. Just leave and find one whom speed suits you.
deleted 10y ago
[deleted]
bbcbasic 10y ago
TRWTF is that it is considered a problem that it took 6 days. Sounds like there was no harm done and 6 days was OK.
xarope 10y ago
I don't get this, whatever happened to a hot fix in minutes, and then a proper fix 6 days later? Have we forgotten how to do this? Together with some other reasonable comments, I can't fault the process in the story, but I just can't get why they didn't allow a hot fix? Then it would be a different title; It takes 5 minutes to approve the change to 1 line of code and rolled into production, and then 6 days to get an official patch - done in accordance with proper IT policy.
- user5994461 10y ago
  What makes you think that they have CI and CD? It's entirely possible that just building & shipping is a 1 full day procedure.
dosethree 10y ago
Agree with some other comments here: Good: Code review? Great. Boy scout rules enforced in legacy code? awesome. The QE analysis seems great! Bad: Failure to expedite? Fail. And without that there's no story. Also at a high level, the fact that it is only 1 line of code is irrelevant. It's not closely correlated to the riskiness of the change.
timwaagh 10y ago
these processes are beyond fixing. if you absorbed into a corporate monolith that values 'best practise' you'll just have to put up with it. however this really would not happen at any shop that is concerned with saving money.
cdevs 10y ago
It's hilarious to read but part of the events that lead up to this are compliance , audit trails, security, investors and customers want to know every 5 minutes some dev isn't jumping into vim and saving new code on production and taking things down with a syntax error.
makecheck 10y ago
A risk of allowing too much bureaucracy in commit processes is that engineers try to avoid the slowdown as much as possible. For example, several unrelated changes may bleed together into “super commits” so as to clear bureaucratic hurdles only once. Next thing you know, it’s harder than ever to identify the exact cause of any one issue: you have more changes in each commit, fewer comments than ever, and a high probability that no thorough review was really conducted. Large blobs make the review process completely fall apart. In the past I dug up too many cases of people allowing 50-file steamrollers into a repository when the entire “review” was literally two words (“looks good”). While this probably happened due to time constraints, it makes the entire thing pointless. With a simpler commit process, engineers might submit smaller change requests at a time and people asked to review “7 files” might do a thorough job instead of balking at requests to review “50 files”.
- KKKKkkkk1 10y ago
  One of the things that happens as a result of anal code reviews is that at some point you learn to commit as little as you can get away with (and this includes things like comments and tests).
  - dom0 10y ago
    Avoiding comments or diagnostics/logging, because these two will be picked on every single time. "Don't have it debug log 'Foo'd 5 bars', make it 'Foodulated five bar things'"
awqrre 10y ago
They should also limit the number of lines that are allowed to be changed/added/removed every 6 days... and maybe it should be a law for popular products.
edblarney 10y ago
Surely there could be some optimizations, but there are no gross failures here. At every step there were reasonable policies it seems. It's not wise to frame this issue as '1 line of code' because '1 line of code' could ruin a company. It's frustrating ... but it's not that bad.
siliconc0w 10y ago
Change control needs to be structured around risk. I.e how much $ are you losing a second given an outage. Also, not all changes have the same risk. Your change control process should be malleable to these realities. If a change is low risk and the reward is high then it shouldn't need the same controls as a high risk change. Also, there are more ways to mitigate risk other than additional steps in the release process. TLDR absolutists make poor release engineers.
yason 10y ago
The bad thing with bureaucracy and slow processes is that eventually you'll find yourself in a situation where you either 1) start working on something else while waiting on some process to complete and get your changes merged, and then you gradually forget to play your turns in the process, delaying it even further, or 2) waste days sitting on the important change, pushing it forward, being active and responsive, and basically just avoiding any other non-trivial task because of #1 happens. So you need to decide whether you want to get real work done or if you want to get the one task done, for which coding took a matter of minutes or hours, but the process eats up days.
Davidbrcz 10y ago
This is true beauty
suls 10y ago
To change the frame of reference a bit, how would this story work out at Google? I just have an outsider perspective on this, but I really think that monorepos & automation that comes with it will have an amplifying effect in the years to come ..
eykanal 10y ago
The only difference between this and a large modern tech company is that the latter has this process automated. If anything, large tech companies have more respect for the need for process, as they understand how critical it is for everything to run smoothly.
mnarayan01 10y ago
Requiring all your constants to be configurable seems like a great way to bypass (potentially critical) code review and QA. I can totally see changing something like "number of months" causing extremely non-obvious issues (e.g. integer overflow in something 20 stack levels away)...once you make it a configurable constant though, changing it won't be QA'd at all.
lucasnemeth 10y ago
For me the absurd part was this one: "Philip (President): Our factory is underutilized by 10%. Either we start building more of our backlog or we lay people off. I'd rather keep everyone busy, build inventory, and get ahead of the curve before the busy season. How can we do that? Lee (Operations Manager): Company policy restricts us from building more than 3 months of backlog. If you just change that to 4 months, we'll have plenty of work." You can't have one week of 10% less productivity if not you will immediately fire everyone? And you company policies only work if you change a line of code? And this line of code is from some old legacy code that no one touch and your whole process depends on it? No, 6 days of development is the least of the problems here. This company is absurd. 6 days for the line of code wasn't optimal, but wasn't that bad. What this story shows to me how bad management pretend things are IT fault.
- rpeden 10y ago
  It didn't mention firing anyone. When I see 'layoff' used in the context of manufacturing, it usually means something more like an furlough (i.e. the laid off employees stay home and don't get paid until the company has enough work for them to do). In this case, it seems they'd rather not do it, but if the software change drags on long enough, they'd have to start layoffs. As for requiring a code change to implement new policy; maybe not ideal, but it looks like they're trying to move away from that with the requirement that hard coded constants be moved to a parameters file. And in general, using the company's production software to enforce business policies is probably an important management control. Often when you see rules like that, it's because the company got burned by something in the past and wants to enforce a process that prevents it from happening again. In this case, maybe in the past an operations manager was evaluated based on how busy the factory does, and so just kept building up more and more inventory to game the metrics he/she was being judged by. I think it was a tad inaccurate of the manager to call it 'legacy' software. It might be an old system, but it appears to be critical to the company's operations, and it seems to be something that is worked on fairly regularly. Even if it's written in Cobol and JCL, I'm not sure it would be fair to call it legacy software unless the company is actively working on a replacement and making only necessary changes to the old system in the interim. So, while I understand your concerns, I'm not sure it's fair to pin this on bad management. It sounds to me like a company that's probably been in operation for a long time, and has processes in place to make sure nobody pushes any changes to a production-critical system without proper approval and testing. They could possibly streamline the process, but it doesn't seem (necessarily) crazy based on the information we were given.
dsego 10y ago
Policies are organizational scar tissue. https://m.signalvnoise.com/dont-scar-on-the-first-cut-e9afd256afdf#.f1od1i2ax https://m.signalvnoise.com/dont-scar-on-the-first-cut-e9afd2...
shakencrew 10y ago
This has been discussed on Hacker News before (https://news.ycombinator.com/item?id=3989136 https://news.ycombinator.com/item?id=3989136). The cited article was published in 2012 at a different domain that is no longer alive. Thanks to the Wayback Machine, we have copies. Here's one: https://web.archive.org/web/20120623032243/http://edweissman.com/it-takes-6-days-to-change-1-line-of-code https://web.archive.org/web/20120623032243/http://edweissman...
daveheq 10y ago
This sounds like bureaucratic nonsense waiting for signatures and approvals. Maybe the task flow needs to improve.
wwarneck 10y ago
Since this was a critical issue I think "David" should've brought key stakeholders (or leads) together and set expectations for who was doing what parts and when it would be complete.
HeavyStorm 10y ago
I work on a consulting company. We have extremely aggressive deadlines, and work with less than capable people on both sides of the table. This means sometimes that takes a usual programmer a few minutes to change a line of code and promote it to UAT. Forget the number of bugs that happens after this. Yeah, the number is VERY high, but is subjective. Think about the overall state of the program. When all that matters is velocity, people takes all forms of shortcuts. OO is out of the window. Hardcode? What about things that should've been parameters and are not only hard-coded, but spread along many files? TL;DR OP case seems contrived, or at least the other extreme. Processes should be followed. If the factory stopped because of that single line - or any other, someone would ask: all this to save 6 man days?
keithnz 10y ago
there's a saying, "if it hurts, do it more often" ( https://www.martinfowler.com/bliki/FrequencyReducesDifficulty.html https://www.martinfowler.com/bliki/FrequencyReducesDifficult... ) these guys should do all their PRs as one line (or a few lines ) changes and put it through the system and improve the system till they can get things deployed within an hour ( or some reasonable time period )
manigandham 10y ago
Why did they hard code a variable in the first place? Hundreds of comments about technical debt when it's just shitty engineering.

568 points by Mz 10y ago

Become a $$$/hour contractor and learn to love the bureaucracy. Then wait for those all-too-often moments where a manager's ass is on the line if feature X doesn't ship by tomorrow and watch all the rules and regulations get chucked wholesale out the window. Then chuckle.

eplanit 10y ago
Exactly. The mantra I was taught when I first began consulting, when faced with frustrations like these was (adapted from the old school Hawaii Five-O): "Bill 'em, Danno!" I've been at it for 20+ years and it truly works.

ztratar 10y ago

This is why startups kill big companies. Lots of stupid little processes that add up and make the managers feel power, but end up killing the performance of the business. Brings up the interesting differences between leadership and management. Managing is what gets people into situations like this, whereas leadership (in late stage companies) is what gets people questioning these processes instead of moving like cattle into a slaughter. I saw this exact type of behavior at General Electric back in 2011. People wanted to argue about acronyms and process design all day, whilst Samsung ate up 20% market share in 1 year. That business of GE no longer exists and was sold to a foreign buyer. Move fast or die -- this rule does not only apply to startups!

maxxxxx 10y ago
Considering how inefficient big companies are it's surprising that they don't get killed in the market immediately. You would expect that there are plenty of nimble newcomers who kick their asses but that's not really happening.
- Afforess 10y ago
  Two words: Regulatory Capture. See Also: Boeing, IBM, CGI Group, AT&T, NRC, etc.
  - grzm 10y ago
    I don't read that as being the point of the article. Regulation is not even mentioned or alluded to. Also, my understanding of regulatory capture has little to do with regulation slowing down business. If anything, rather the opposite. But that's off-topic anyway, and definitely not appropriate for this week. Edited to update comment on regulatory capture.
    - Afforess 10y ago
      I don't know what you're talking about, it's in there plain as day. The company has a department called IT Security. Not IT. Not Security. IT Security. Companies that legitimately care about security integrate security personnel with all of their operations, into every department. Top-down security departments exist to check boxes on regulatory forms. So I will be shocked if "IT Security" is not a made-to-order department whose existence is solely to satisfy things like PCI, HIPAA, etc.
      - cortesoft 10y ago
        What? In any large company, you are going to have access controls - not everyone is going to be allowed access to every system. In the example given, the guy was requesting access to a new system he didn't previously have access to; having a security team that is in charge of that access doesn't seem to have anything to do with regulation.
      - trdtaylor1 10y ago
        True, but the greater danger of integrating security into every department is discounting security in favor of the main requirement of your department. Your department is viewed for political and performance reasons, budgeting time away from the main goal of your area towards security is technically ... waste. Additionally, you ignore the business logic cross-departmental issues when you do not have a centralized security department.
robotsonic 10y ago
Reminds me of the bike shed effect: https://en.wikipedia.org/wiki/Law_of_triviality https://en.wikipedia.org/wiki/Law_of_triviality
lurkinggrue 10y ago
Startups can afford to have their critical systems explode.
vacri 10y ago
So... one of the places I work is one of those 'nimble little startups', and it gets more and more bogged-down by the day due to some poor technical decisions taken early. There are seven different kinds of timestamp in the database, for example. That's pure dev overhead that gets applied for everything. And because the nimble little startups are pressed to work on new features rather than refactoring, more tech debt just gets added to the pile... Also, don't forget selection bias - most small companies, startups included, die in the first few years. Only the success stories make the papers.
jlarocco 10y ago
It's easy to criticize this type of thing as wasteful and inefficient for the one time in a thousand where it really is just a single line code change that doesn't break anything, but the reason things like mandatory code review, required tech debt fixes, and all of the rest end up existing is because developers are notorious for saying, "Oh, it's just a one line change!" and then breaking everything when they slam it through to production without thinking about it.
mevile 10y ago
> whilst Samsung ate up 20% market share in 1 year Right a 78 year old company with hundreds of billions of dollars of revenue is a startup killing big companies. I'll just note that while Samsung is about half of G&E's age it makes twice as much in revenue.
- gamegoblin 10y ago
  The comment you are replying to included: Move fast or die -- this rule does not only apply to startups! So I think they are aware that Samsung is not a startup.
- taneq 10y ago
  Apparently "startup" doesn't mean "company which has just started running" these days, it means "company which is growing rapidly."
deleted 10y ago
[deleted]
Johnny555 10y ago
It's easy for startups to be nimble because they have no legacy processes to maintain. The startup may have 5 guys in a warehouse building their product, while the big company has 100 people on a $10M assembly line. When the startup wants to scale by 20%, they work longer hours or add another worker. If they didn't plan properly and find that they don't have another workstation for that worker, they buy a table at OfficeDepot and there's their new workstation. The assembly line might need an entire process flow redesign to add more workstations, plus a physical reconfiguration, software changes, etc. If a developer just blindly adds a parameter that adds 10% to the workday without going through reviews and tests, he may find out from the physical plant engineers that they no longer have enough time to do preventative maintenance so they need to scheduled an outage to do deferred maintenance. There's a reason why process exists in large companies, and it's not just for the sake of process. That small startup might be able to usurp the large corporation, but the large company still has an advantage in throughput and/or efficiency. The corporation can promise Walmart that they'll have 100,000 units delivered by Dec 1st... to meet the same goal, the small startup might have to scale their workflow 100X, and they can't do that by scaling from 5 guys in a warehouse to 500 guys in a warehouse.
- tonyedgecombe 10y ago
  I don't know, for every justifiable decision I've seen in corporates there have been ten where it was all about politics, needless bureaucracy, incompetence and power and nothing to do with getting things done.
ap22213 10y ago
I don't know. It seems more like 99% of startups die, and 99% of large companies stick around. Not that I enjoy working for large companies, but just saying...
- ryhamz 10y ago
  Right. For every startup that skips process and gets lucky there is a pile that rolled snake eyes and died. Big companies aren't taking the chance.
  - flukus 10y ago
    Big companies tend to not take that chance even when it's worth taking though. Not everything is mission critical but the procedures around them pretend they are.
- TheOtherHobbes 10y ago
  Large companies stick around because they've solved the one problem all startups have - finding and keeping customers and turning them into revenue through sales and product pipelines. Code is pretty much a footnote to that process. It's exactly one of very many tools a business can use. It's true that technical debt can cause business problems. But it doesn't cause nearly as many problems as having an ineffective sales team, getting killed by regulation, or being eaten alive by a competitor.
- B1FF_PSUVM 10y ago
  > 99% of startups die And no one bothers suing into the ground a muppet with two packs of ramen to their name ...

renownedmedia 10y ago

[2015]

LargeCompanies 10y ago

Didn't read this but is this about a government job... lol

hacker_9 10y ago

I want to fault this but I can't. Code reviewer who's not afraid to reject code? Check. Proper testing of code changes? Check. The process probably looks silly in this case because nothing went wrong. But with a hard-coded variable being changed, who is to say it won't break some other system by being changed? And if the variable was then moved to a config file, who's to say again that it might not break in some obscure way? Such as if the config file is not found in the uat/test environment? Additionally after all this, the manager can change the variable in the config file as they please without this process need going through again.

zzalpha 10y ago
My reaction exactly. And 2 days were lost to escalation while 2 were spent in test. Everything else didn't look crazy to me if the change is small but has a high system impact. For an industry that is implicated in downed spacecraft, massive security vulnerabilities, and a reputation for unstable products, it's amazing how happy we are to optimize for developer satisfaction over all other metrics (I'm looking at you, Agile). Edit: And before anyone says it, I'm not claiming this is a good process. But coding velocity is just one metric by which we should be measuring our success.
- dreamfactory2 10y ago
  we should measure value, that is all. and that's what agile is supposed to aim for, not developer comfort. these processes offer negative value (prohibitive costs of change and inability to execute in time which massively outweigh any perceived security). even a vaguely leaner organisation will eat their lunch. and that error-proofing is entirely imaginary - they have added a ton of moving parts, handoffs, and manual processes and so will have way more screw ups than a small tight team practising continuous delivery, as anyone who has worked in such environments will know.
  - delinka 10y ago
    "value." I'm getting this BS from a PM right now over a refactor job that needs to happen. Sustainability and tech debt be damned. Every time they want to support a new file type, it's editing several files, most of which are source, recompile, put QA resources on it, get IT to sign the binaries ... If you'll get off my back, let me refactor and get my "developer comfort" on, you'll never have to involve me or QA or IT in supporting Just One More File Type ever again - you can push that from policy on the server side. But "developer comfort" isn't "value for the customer." _sigh_

manarth 10y ago

This sounds like perfect is the enemy of good.

kazinator 10y ago

That can be okay if the same 6 days can be used in parallel to change 999 other lines of code, too. I.e. "1 line in 6 days" is only a measure of latency, and not of throughput.

vacri 10y ago
> a measure of latency, and not of throughput. You could drive a station wagon full of backup tapes quite far in 6 days :)
- kazinator 10y ago
  It's not about how far, but how fast it is going. The bandwidth of that wagon is about how fast those tapes pass a given point of travel. How they are arranged inside the vehicle is critical. Tapes moving at 100 km/h side by side have more instantaneous or "burst" bandwidth than tapes moving at 100 km/h in a single file.

perilunar 10y ago

I once waited 6 weeks for a backend team to fix a prematurely closed div tag that was screwing up the layout of a news site. The templates they were given were good; they screwed up the implementation. In desperation I had to hack the layout with JS (which I was able to update at will).

rewrew 10y ago
That kills me to hear that. I'm in charge of Web where I am and we make 95 percent of our changes instantly. There is no reason to make people wait; it honestly takes longer to project manage most changes than to just make them, so we just make them, but of course Web sites/front end is a different beast than application or other code.

nickthemagicman 10y ago

Thats a dream orocess right there right? The solution imo is to tell the higher ups to fast track it if peoples jobs arecat stake.

Wellshit 10y ago

Well, I've seen 5 line patch take 6 months (and counting), so...

bbcbasic 10y ago
Yeah it's sad I was thinking '6 days? that's pretty quick'
weavie 10y ago
I've had 6 months of work thrown out because management changed its mind over which approach to take. So 0 lines in 6 months..

sthatipamala 10y ago

Are there really factories whose policies and parameters are encoded in software vs. under the guidance of humans? That in itself is amazing to me considering that the tech company I work at, lots of things are just done by fiat.

nsoldiac 10y ago

I'm surprised no one has raised the problem of senior managers and VPs underestimating and oversimplifying the complexity and LOE of work. It causes unrealistic timelines for others who do have to adhere to a release process (which is never perfect but certainly necessary). "Yes CEO, of course we can get that in by tomorrow, no problem." When the work misses the impossible deadline the feedback usually points to an underperforming organization and rarely to unrealistic estimation.

joslin01 10y ago

I don't think I would eat 24 excedrin over something like this.

patmcguire 10y ago

Favorite line: "Ed: Fuck that shit." "Shirley: That may very well be true."

partycoder 10y ago

If technical debt accumulates, you slow down until the point you need a lot of time to understand the consequences of changing code. Pretty much like the end game in Jenga. One good approach is SOLID: https://en.wikipedia.org/wiki/SOLID_(object-oriented_design) https://en.wikipedia.org/wiki/SOLID_(object-oriented_design)

valine 10y ago

Related anecdote: I interned for a medical device company over summer of 2015 and worked on a team that was in charge of maintaining several implantable class 3 medical devices. The devices used a short range proprietary wireless standard to talk with a intermediary device which could be connected to via Bluetooth. One product had a pretty serious usability issue where connecting to the intermediary device could take upwards of several minutes. After the issue was discovered it took roughly an afternoon and several lines of code to fix. However, in order to provide software updates for a class 3 medical device each update needs to be individually approved by the FDA. It took 12 months from the time the fix was implemented to when the FDA allowed it to be pushed out.

hughes 10y ago
I think the word you want is "anecdote".
- valine 10y ago
  thanks
joshvm 10y ago
What is the approval process? Do the FDA manually check the device (similar to FCC certification for wireless things)?
- valine 10y ago
  I never had the opportunity to be involved in the FDA submission process, but from what I understand the FDA does not do any testing of their own. It's the companies responsibility to prove to the FDA that their device is safe and effective. All the testing is done by the device company, carefully documented, and then reviewed by the FDA. I've heard stories that the documentation for a class 3 medical device can be upwards of 20,000 pages which are all printed and handed to the FDA in 3 ring binders.

joeld42 10y ago

This is why it's important to have a "hotfix" process where you can make a change fast and then still follow up with all the QA and coding standards.

deleted 10y ago

[deleted]

patmcguire 10y ago

It seems like buried in all the other policies in one meta-policy: nothing gets done until someone with sufficient authority overrides all the rules. How often does the president have to do this? Has anything ever been done the correct way?

spydum 10y ago
Reality is in a large org, process can usually be sidestepped with emergency change requests and executive sponsor. Which is basically what happened here, it just took extra days because it wasn't started that way.

ipsin 10y ago

In Case of Emergency Break Glass There really needs to be a procedure for violating procedure, so that when higher-ups say "do this immediately", those same higher-ups can own the consequences of violating procedure, and so that they know the business implications of the rollback plan, etc. If the bug is "people are getting laid off" or "our customer database is being siphoned by an SQL injection", it probably should not wait.

Falkon1313 10y ago
The process sounds fine, no need to violate it. The problem was that the IT Admin said "I'll put this on the fast track" but didn't, and no one else saw the urgency. It was just dumped at the back of the queue behind a lot of other things and mistakenly labeled an 'Enhancement'. Proper handling would have had a dev working on it within the hour. The response to the code review would have been "other changes are out of scope, open another ticket for enhancements". If the developer didn't already have access to the test environment, at least IT would know that they're authorized for it. (Critical fixes shouldn't be assigned to a new hire who doesn't know the code base and doesn't have access to commit code yet.) The ticket would be edited (Test Case: view WorkOrdersHours report, Expected Result: ~10% higher total) and tester would be told to reread it and go ahead. And of course documenting the test runs is the tester's responsibility (they're the one doing the test runs), not the developer's, as is notifying the IT Admin and Operations Manager when the testing is done and it's ready for sign off. Exact same process, same quality checking along the way, timeframe reduced from 6 days to 4 hours or less; likely less than 1 if the team works together well and their tools are decent. A good process can handle emergencies without sacrificing quality, as long as the people work together and the priority is communicated.

cthulhuology 10y ago

I this had happened at any of the companies I started, I would have fired several people in the aftermath. The problem is the policy is broken by design. Let's look at what is wrong here: 1.) priority is not just high, it is critical, communicating this is lost at each layer (executive, planning, execution, process control, quality control) 2.) leadership is lax, the chain of command doesn't designate a clear single responsible individual 3.) policy enforcement in this example actually increases the risk of an unsatisfactory outcome, by increasing the complexity of the solution vs. what is in production 4.) quality control is adversarial and ass backwards, code review is supposed to be a sanity check "does this code do what the developer thinks it does" aka. "can some other person understand it" 5.) test planning should not be the developer's responsibility, quite frankly if QA can't figure out if it is working or not you should fire your QA department. 6.) Ultimately, it is a total failure of policy and management as it requires the President of the company to micromanage the situation. If you think any of this is fine, I'm sorry but your company is doomed to fail (unless it is already so big it is too big to fail).

kevinstubbs 10y ago
You would fire them, instead of trying to build up their skills? I'm not even sure who you would fire in this situation, the IT director for not shepherding the ticket all the way through, or Shirley for closely following company policy? Either would be overreacting. Slow/inflexible company policy has been built up, presumably under the IT Director's watch. The president didn't communicate that it was extremely urgent (and maybe it wasn't - the article doesn't say when the change actually needed to be made by to make a difference). Ideally, management can recognize that this change took a week because of their accumulated decisions, and lack of urgency. They should figure out exactly how their company policy is slowing them down instead of letting heads roll.
- user5994461 10y ago
  Don't fire. Don't touch anything. Just remember the chapter 12 of Peopleware. Between two organizations, the productivity can vary by 10 times. It's not about you, it's a social characteristic of the organisation. Just leave and find one whom speed suits you.

deleted 10y ago

[deleted]

bbcbasic 10y ago

TRWTF is that it is considered a problem that it took 6 days. Sounds like there was no harm done and 6 days was OK.

xarope 10y ago

I don't get this, whatever happened to a hot fix in minutes, and then a proper fix 6 days later? Have we forgotten how to do this? Together with some other reasonable comments, I can't fault the process in the story, but I just can't get why they didn't allow a hot fix? Then it would be a different title; It takes 5 minutes to approve the change to 1 line of code and rolled into production, and then 6 days to get an official patch - done in accordance with proper IT policy.

user5994461 10y ago
What makes you think that they have CI and CD? It's entirely possible that just building & shipping is a 1 full day procedure.

dosethree 10y ago

Agree with some other comments here: Good: Code review? Great. Boy scout rules enforced in legacy code? awesome. The QE analysis seems great! Bad: Failure to expedite? Fail. And without that there's no story. Also at a high level, the fact that it is only 1 line of code is irrelevant. It's not closely correlated to the riskiness of the change.

timwaagh 10y ago

these processes are beyond fixing. if you absorbed into a corporate monolith that values 'best practise' you'll just have to put up with it. however this really would not happen at any shop that is concerned with saving money.

cdevs 10y ago

It's hilarious to read but part of the events that lead up to this are compliance , audit trails, security, investors and customers want to know every 5 minutes some dev isn't jumping into vim and saving new code on production and taking things down with a syntax error.

makecheck 10y ago

A risk of allowing too much bureaucracy in commit processes is that engineers try to avoid the slowdown as much as possible. For example, several unrelated changes may bleed together into “super commits” so as to clear bureaucratic hurdles only once. Next thing you know, it’s harder than ever to identify the exact cause of any one issue: you have more changes in each commit, fewer comments than ever, and a high probability that no thorough review was really conducted. Large blobs make the review process completely fall apart. In the past I dug up too many cases of people allowing 50-file steamrollers into a repository when the entire “review” was literally two words (“looks good”). While this probably happened due to time constraints, it makes the entire thing pointless. With a simpler commit process, engineers might submit smaller change requests at a time and people asked to review “7 files” might do a thorough job instead of balking at requests to review “50 files”.

KKKKkkkk1 10y ago
One of the things that happens as a result of anal code reviews is that at some point you learn to commit as little as you can get away with (and this includes things like comments and tests).
- dom0 10y ago
  Avoiding comments or diagnostics/logging, because these two will be picked on every single time. "Don't have it debug log 'Foo'd 5 bars', make it 'Foodulated five bar things'"

awqrre 10y ago

They should also limit the number of lines that are allowed to be changed/added/removed every 6 days... and maybe it should be a law for popular products.

edblarney 10y ago

Surely there could be some optimizations, but there are no gross failures here. At every step there were reasonable policies it seems. It's not wise to frame this issue as '1 line of code' because '1 line of code' could ruin a company. It's frustrating ... but it's not that bad.

siliconc0w 10y ago

Change control needs to be structured around risk. I.e how much $ are you losing a second given an outage. Also, not all changes have the same risk. Your change control process should be malleable to these realities. If a change is low risk and the reward is high then it shouldn't need the same controls as a high risk change. Also, there are more ways to mitigate risk other than additional steps in the release process. TLDR absolutists make poor release engineers.

yason 10y ago

The bad thing with bureaucracy and slow processes is that eventually you'll find yourself in a situation where you either 1) start working on something else while waiting on some process to complete and get your changes merged, and then you gradually forget to play your turns in the process, delaying it even further, or 2) waste days sitting on the important change, pushing it forward, being active and responsive, and basically just avoiding any other non-trivial task because of #1 happens. So you need to decide whether you want to get real work done or if you want to get the one task done, for which coding took a matter of minutes or hours, but the process eats up days.

Davidbrcz 10y ago

This is true beauty

suls 10y ago

To change the frame of reference a bit, how would this story work out at Google? I just have an outsider perspective on this, but I really think that monorepos & automation that comes with it will have an amplifying effect in the years to come ..

eykanal 10y ago

The only difference between this and a large modern tech company is that the latter has this process automated. If anything, large tech companies have more respect for the need for process, as they understand how critical it is for everything to run smoothly.

mnarayan01 10y ago

Requiring all your constants to be configurable seems like a great way to bypass (potentially critical) code review and QA. I can totally see changing something like "number of months" causing extremely non-obvious issues (e.g. integer overflow in something 20 stack levels away)...once you make it a configurable constant though, changing it won't be QA'd at all.

lucasnemeth 10y ago

For me the absurd part was this one: "Philip (President): Our factory is underutilized by 10%. Either we start building more of our backlog or we lay people off. I'd rather keep everyone busy, build inventory, and get ahead of the curve before the busy season. How can we do that? Lee (Operations Manager): Company policy restricts us from building more than 3 months of backlog. If you just change that to 4 months, we'll have plenty of work." You can't have one week of 10% less productivity if not you will immediately fire everyone? And you company policies only work if you change a line of code? And this line of code is from some old legacy code that no one touch and your whole process depends on it? No, 6 days of development is the least of the problems here. This company is absurd. 6 days for the line of code wasn't optimal, but wasn't that bad. What this story shows to me how bad management pretend things are IT fault.

rpeden 10y ago
It didn't mention firing anyone. When I see 'layoff' used in the context of manufacturing, it usually means something more like an furlough (i.e. the laid off employees stay home and don't get paid until the company has enough work for them to do). In this case, it seems they'd rather not do it, but if the software change drags on long enough, they'd have to start layoffs. As for requiring a code change to implement new policy; maybe not ideal, but it looks like they're trying to move away from that with the requirement that hard coded constants be moved to a parameters file. And in general, using the company's production software to enforce business policies is probably an important management control. Often when you see rules like that, it's because the company got burned by something in the past and wants to enforce a process that prevents it from happening again. In this case, maybe in the past an operations manager was evaluated based on how busy the factory does, and so just kept building up more and more inventory to game the metrics he/she was being judged by. I think it was a tad inaccurate of the manager to call it 'legacy' software. It might be an old system, but it appears to be critical to the company's operations, and it seems to be something that is worked on fairly regularly. Even if it's written in Cobol and JCL, I'm not sure it would be fair to call it legacy software unless the company is actively working on a replacement and making only necessary changes to the old system in the interim. So, while I understand your concerns, I'm not sure it's fair to pin this on bad management. It sounds to me like a company that's probably been in operation for a long time, and has processes in place to make sure nobody pushes any changes to a production-critical system without proper approval and testing. They could possibly streamline the process, but it doesn't seem (necessarily) crazy based on the information we were given.

dsego 10y ago

Policies are organizational scar tissue. https://m.signalvnoise.com/dont-scar-on-the-first-cut-e9afd256afdf#.f1od1i2ax https://m.signalvnoise.com/dont-scar-on-the-first-cut-e9afd2...

shakencrew 10y ago

This has been discussed on Hacker News before (https://news.ycombinator.com/item?id=3989136 https://news.ycombinator.com/item?id=3989136). The cited article was published in 2012 at a different domain that is no longer alive. Thanks to the Wayback Machine, we have copies. Here's one: https://web.archive.org/web/20120623032243/http://edweissman.com/it-takes-6-days-to-change-1-line-of-code https://web.archive.org/web/20120623032243/http://edweissman...

daveheq 10y ago

This sounds like bureaucratic nonsense waiting for signatures and approvals. Maybe the task flow needs to improve.

wwarneck 10y ago

Since this was a critical issue I think "David" should've brought key stakeholders (or leads) together and set expectations for who was doing what parts and when it would be complete.

HeavyStorm 10y ago

I work on a consulting company. We have extremely aggressive deadlines, and work with less than capable people on both sides of the table. This means sometimes that takes a usual programmer a few minutes to change a line of code and promote it to UAT. Forget the number of bugs that happens after this. Yeah, the number is VERY high, but is subjective. Think about the overall state of the program. When all that matters is velocity, people takes all forms of shortcuts. OO is out of the window. Hardcode? What about things that should've been parameters and are not only hard-coded, but spread along many files? TL;DR OP case seems contrived, or at least the other extreme. Processes should be followed. If the factory stopped because of that single line - or any other, someone would ask: all this to save 6 man days?

keithnz 10y ago

there's a saying, "if it hurts, do it more often" ( https://www.martinfowler.com/bliki/FrequencyReducesDifficulty.html https://www.martinfowler.com/bliki/FrequencyReducesDifficult... ) these guys should do all their PRs as one line (or a few lines ) changes and put it through the system and improve the system till they can get things deployed within an hour ( or some reasonable time period )

manigandham 10y ago

Why did they hard code a variable in the first place? Hundreds of comments about technical debt when it's just shitty engineering.