2 ms·
You can also work on damage control. It's ideal to tell everyone to test thoroughly and never release bugs. But if you're stuck in a high bug environment, it c
by muzani 3y ago
You can also work on damage control.
It's ideal to tell everyone to test thoroughly and never release bugs. But if you're stuck in a high bug environment, it can be useful to set up a system where you can fast track bug fixes to production in minutes, sometimes even before the customer realizes you were down. Especially in a startup environment, you can take advantage of the high speed, low quality culture to just release fixes faster.
Sometimes a lot of damage comes in when a fix takes too long to approve - billing being down for hours, for example. We have people on a rotated shift every week, aka fire wardens as well as a manager on emergency duty. There should be channels to contact them for emergencies, personal numbers, etc. The fire warden's job isn't necessarily to fix bugs but rather to direct them to the best person. The release engineer is also the fire warden, to prevent being blocked by each other.
There should also be a clear process of to release emergency fixes, e.g. CS talks to manager. Manager talks to engineer. Engineer releases fix. Engineer pushes fix to staging. QA runs the bare minimum of testing. Engineer releases it to prod. If you can't do that quickly, find out what's blocking it. Figure out what the bare minimum of testing is. Maybe consider CI/CD if it works for you.
We've realized that Git Flow got in the way too and used trunk based development helped a lot in releasing faster: https://trunkbaseddevelopment.com/ https://trunkbaseddevelopment.com/