4 ms·

One DBA was responsible for all backups for an airline? Why not make it a team effort?

• rawgabbit 3y ago
  Many years ago, I worked for a fortune 500 company that did billions in revenue. I was literally one of two DBAs for the entire company. Yes, we did have a revolving door of contractors and a team of "DBAs" in India. Did I ever let them touch production? No. I tasked them with monitoring and dealing with all the non prod environments. You have to realize in a big corp there are an infinite number of ways for things to fall apart. Much of it beyond your control. My goal was to minimize the blast radius. One time I took vacation for the first time in years. When I came back I was blamed for an outage while I was on holiday. Their reasoning was that they didn’t push any new code into production so it must be the database’s fault. After a day of troubleshooting, I learned the application code was not using a framework like Microsoft Linq to read the SQL output but was reading it raw. I also learned that the Windows admin had been pushing Windows updates into the servers. The issue was that a Windows update (not a SQL Server update) changed how timestamps were recorded. It was beyond stupid. Instead of yyyymmdd hhmmss like the application expected. It was now yyyymmdd hhmmss timezoneoffset +- hh. They now had some weird hour offset and the actual values changed. To fix the issue the windows update was rolled back. And the Windows admins were told never to touch the database servers without my permission. And the application was rewritten.