5 ms·
> I would be grateful if they monitor the database for some horrible queries; and, use their knowledge of which deployments made that bad query, to file a ticke
by solatic 4y ago
> I would be grateful if they monitor the database for some horrible queries; and, use their knowledge of which deployments made that bad query, to file a ticket to the right team so they fix their code or add an index or whatever is necessary.
What you're asking for is:
a) SRE wrote an alert for slow queries, because slow queries affect all shared infrastructure users
b) SRE gets woken up at 2 AM by his alert
c) SRE sees Dev wrote a bad query
d) SRE files a ticket for Dev
e) SRE goes back to sleep
f) SRE gets woken up at 2:30 AM by his alert, because the query fired again
g) SRE has a restless night
h) SRE goes into work, asks Dev to prioritize the ticket to fix the query because sleep
i) Dev tells SRE that maybe it'll get into the next sprint, in the meantime the alerts are SRE's problem
No thanks.
- t-writescode 4y agoThere are ticketing paradigms that don't require alerting in the middle of the night. All you need to repo and fix a bad query is the query itself and/or an explain of the query itself, and you can work on it during business hours. I'm sorry you've had bad experiences with alert-all-the-things ticketing paradigms. addendum: in fact, you can automate all of this so it just shows up in a team's "known issues" query that they may manage in standup or sprint planning.
- amne 4y agoI still have the proverbial nightmare from hearing at 2 am: "Hello, this is Nagios" in a creepy robotic voice (we had it setup to go through Asterisk with a TTS spelling out what the alert was about. but it always started with that prompt. Free (as in beer) TTS voice packs in 2010 were .. interesting)