3 ms·
Assuming you are in a very large organization. 1) When the system is down and you're asked for an explanation. You deliver a post mortem putting the responsibi
by user5994461 9y ago
Assuming you are in a very large organization.
1) When the system is down and you're asked for an explanation. You deliver a post mortem putting the responsibility on the database. The database is down. The database team is working on it.
2) If you are facing repeated major downtime. You keep track of failures caused by the database. You make a chart with a health check to the database every minute.
3) You report to your management that the database service doesn't meet expectations and it's affecting your services, supported by the previous reports you just created. It's the responsibility of the database and the management to run their services and decide where to allocate resources.
4) Make it clear that your service depends on its database. Any question about downtime or better SLA should be replied with the postmortem from 1 and/or the improvement plan from 3.
All you can do is report. You probably don't have a say. Your manager probably doesn't have a say. Maybe the database team doesn't have the power to fix anything either.
- xendo 9y agoI don't think end customer cares about why is his experience broken. Following a process that accepts things being broken, but shifts the responsibility doesn't fix anything. Op's question is a great sign of ownership and customer obsession on his side. The actuall solution is tough, but I think something can be improved. First of all I would try to measure and root cause the problem. It's hard for me to believe that the DB has 90% availability, I'm not aware of any commercial DB that's this bad. I would avoid creating shards of db, it will lead to hard to debug data inconsistencies. In this regard, cache with reasonable TTL is a better choice wince it guarantees eventual consistency of the data. I would also take a deep look into retry approaches you can take, something like exponential backoff with jitter and properly adjusted number of retries can greatly increase your availability.
- user5994461 9y agoIf it's impacting a public service with external clients, you should add that to the report, with an estimation of monetary and reputation loss. Customer issues are more likely to be noticed. The OP is the customer and the database team is the service provider. It's their role to provide a service working within specifications. That's how it works in large organization. Most of everything is internal services providing services to other internal services.
- beckler 9y agoComing from the small business/startup world, to a giant enterprise, this is what I struggle with the most. I'm used to digging in the dirt to figure out an issue, and get it done as quickly as possible. Having zero control over the infrastructure and having to work with various teams can be a pain whenever you want to make a change, but they have a silver lining in that production issues are usually not your responsibility. But with their response times, you would think it's not their responsibility either...
- deleted 9y ago[deleted]