6 ms·

In Case of Emergency Break Glass There really needs to be a procedure for violating procedure, so that when higher-ups say "do this immediately", those same hi

by ipsin 10y ago

In Case of Emergency Break Glass There really needs to be a procedure for violating procedure, so that when higher-ups say "do this immediately", those same higher-ups can own the consequences of violating procedure, and so that they know the business implications of the rollback plan, etc. If the bug is "people are getting laid off" or "our customer database is being siphoned by an SQL injection", it probably should not wait.

Falkon1313 10y ago
The process sounds fine, no need to violate it. The problem was that the IT Admin said "I'll put this on the fast track" but didn't, and no one else saw the urgency. It was just dumped at the back of the queue behind a lot of other things and mistakenly labeled an 'Enhancement'. Proper handling would have had a dev working on it within the hour. The response to the code review would have been "other changes are out of scope, open another ticket for enhancements". If the developer didn't already have access to the test environment, at least IT would know that they're authorized for it. (Critical fixes shouldn't be assigned to a new hire who doesn't know the code base and doesn't have access to commit code yet.) The ticket would be edited (Test Case: view WorkOrdersHours report, Expected Result: ~10% higher total) and tester would be told to reread it and go ahead. And of course documenting the test runs is the tester's responsibility (they're the one doing the test runs), not the developer's, as is notifying the IT Admin and Operations Manager when the testing is done and it's ready for sign off. Exact same process, same quality checking along the way, timeframe reduced from 6 days to 4 hours or less; likely less than 1 if the team works together well and their tools are decent. A good process can handle emergencies without sacrificing quality, as long as the people work together and the priority is communicated.

by ipsin 10y ago

Falkon1313 10y ago

The process sounds fine, no need to violate it. The problem was that the IT Admin said "I'll put this on the fast track" but didn't, and no one else saw the urgency. It was just dumped at the back of the queue behind a lot of other things and mistakenly labeled an 'Enhancement'. Proper handling would have had a dev working on it within the hour. The response to the code review would have been "other changes are out of scope, open another ticket for enhancements". If the developer didn't already have access to the test environment, at least IT would know that they're authorized for it. (Critical fixes shouldn't be assigned to a new hire who doesn't know the code base and doesn't have access to commit code yet.) The ticket would be edited (Test Case: view WorkOrdersHours report, Expected Result: ~10% higher total) and tester would be told to reread it and go ahead. And of course documenting the test runs is the tester's responsibility (they're the one doing the test runs), not the developer's, as is notifying the IT Admin and Operations Manager when the testing is done and it's ready for sign off. Exact same process, same quality checking along the way, timeframe reduced from 6 days to 4 hours or less; likely less than 1 if the team works together well and their tools are decent. A good process can handle emergencies without sacrificing quality, as long as the people work together and the priority is communicated.