10 ms·
Grafana OnCall: an easy-to-use on-call management tool
- markbnj 5y agoI'm a grafana fan and a current user of PagerDuty. Maybe there's more to the story but after reading the post I feel like using a calendar integration to manage on-call schedules is the wrong approach. Calendar events are a result of overlaying a rotation on a date range: they're the output, not the input. I'm sure the designers here have looked at how PD enables creating and editing rotations. Curious to know their views on it.
- halfmatthalfcat 5y agoIs there really anybody else in the "Pager" category of SaaS products other than PagerDuty that have any traction?
- abhishekjha 5y agoAlso what happens if pagerduty goes down?
- julianlam 5y agoIdeally, the services you use should handle that (detect a non-200 and fire off a backup method like a slack webhook or email.) In reality, probably a lot of missed downtime events, and ops sleeping peacefully I guess.
- vorpalhex 5y agoIt's very rare for them to go down. I think I can remember one major outage during business hours in the last few years at which point we just switched to manual monitoring for the few hours. If that is within your outage model, you'd probably want a redundant on-call service I suppose, even if it's just escalating to a single known email or sms group.
- jq-r 5y agoYour service(s) going down and pagerduty going fully down is very unlikely to happen. Even if it does, you're probably going to get called by customer support because users never go down;)
- kevindong 5y agoIn the year I used it, I never personally noticed it going down. Although that being said, their SLA is only 99.9% delivery in any calendar month within 5 minutes. The penalty for missing that SLA is only 10% of that month's bill. > Once an Incident is triggered, PagerDuty will deliver the First Responder Alert within the Notification Delivery Period for 99.9% of the notifications sent by PagerDuty for the Customer during any calendar month. The “Notification Delivery Period” is five (5) minutes and it is measured as the time it takes PagerDuty to deliver a First Responder Alert to telecommunication providers in accordance with the Service configuration and Contact Information. > ... > If PagerDuty fails to meet the SLA set forth herein, Customer may receive a service credit. Customer will be eligible for a credit toward future fees owed to PagerDuty for the PagerDuty Service. The Service Credit is calculated as ten percent (10%) of the fees paid for or attributable to the month when the alleged SLA breach occurred. https://www.pagerduty.com/standard-service-level-agreement/ https://www.pagerduty.com/standard-service-level-agreement/
- haliskerbas 5y agoTechnically Splunk On-call. But I have a few pain points with it, and I miss pagerduty. If you want to see what teams you are on as the current logged in user, the only way to do it as far as what support told me, is to search for yourself and then check that result.
- rconti 5y agoI see my teams listed under my user profile. Or if I go to the left side bar and click on my name, it says when I'm next on-call for various teams. But the UI looks different than last time I logged in a few weeks ago, so maybe something has changed. Disclaimer: Am an employee.
- bboreham 5y agoOpsgenie?
- bgm1975 5y agoThere’s Splunk OnCall (formerly known as VictorOps). It’s a very decent solution.
- therealdrag0 5y agoWe use OpsGenie. not sure how widely it’s used but given its Atlasian I’d guess a non-trivial amount.
- armiiller 5y agoPagerTree - https://pagertree.com https://pagertree.com
- coderchix 5y agoMy team uses PagerTree. Easy to get started with, has the tools you need without being overcomplicated.
- saminzadeh 5y agoDataDog also launched their own Incident Management tool, not sure how widely it's used: https://www.datadoghq.com/blog/incident-response-with-datadog/ https://www.datadoghq.com/blog/incident-response-with-datado...
- dvtrn 5y agoI've been seeing them recommended more and more, and myself have been keeping a passive eye on BetterUptime (which has an on-call feature): https://betteruptime.com/incident-management https://betteruptime.com/incident-management
- fredman 5y agoThere is xMatters: https://www.xmatters.com/ https://www.xmatters.com/ Disclaimer: I work at xMatters.
- Forfold 5y agoI work on/for an open source solution that we based off of PagerDuty, called GoAlert: https://github.com/target/goalert https://github.com/target/goalert
- craigching 5y agoTarget uses go alert across the enterprise for all on call. Definitely enterprise capable!
- toomuchtodo 5y agoTarget having a GitHub org has bolstered my future use of Target pickup orders. Didn’t know Target open sourced tech, love to see it.
- craigching 5y agoThanks for your comment! We are working on doing more in open source to help pay back our use of open source. We can and will do more!
- awestman 5y agoYep. This is a great product. Has the features you need, is super reliable and easy to manage.
- bilalq 5y agoWe started using Squadcast: https://squadcast.com https://squadcast.com Their free and lower prices tiers offer a lot of what others have on their top/most expensive tiers. Also, integrations with various alert sources are just easier in most cases. I spent I don't know how long trying to get OpsGenie to work before I gave up.
- aiisjustanif 5y agoxMatters
- kenrose 5y agoPD does two big things: 1. Alerting: Phones you when your servers are down. 2. Incident Management: Help coordinate a response across multiple people. For the first, there's also: - OpsGenie (owned by Atlassian) - Squadcast - VictorOps (now Splunk On Call) - xMatters - PagerTree For the second, there's a bunch of new contenders: - Datadog now has an IM product - Blameless - Rootly - Incident.io - FireHydrant
- miller_joe 5y agohttps://allma.io/ https://allma.io/ is also in the 2nd category and is a pretty awesome product so far, IMO.
- amozoss 5y agoCan also checkout https://tellspin.app https://tellspin.app for a directly in Slack solution
- itsjloh 5y agoI use VictorOps (Now Splunk On-Call) currently and it does the job. Its shift override functionality is quite confusing to get your head around at first but makes sense after the first few times. I've also used OpsGenie (Atlassian now) and really enjoyed it. The amount of integrations they have is staggering.
- varlogix 5y agoSpike.sh - https://spike.sh https://spike.sh I may be biased as a co-founder of Spike.sh, but I think we have one of the best designed incident management products out there. We've focused on making it easy to create on-call schedule and overrides, and added templates for escalation, on-call and alert rules.
- moepstar 5y agoA few more screenshots of the "Scheduling" options would've been great... We're (more or less) using OpsGenie's free tier, however their scheduling never really "clicked" with me... not sure if i'm special in that regard, however i find the UI/UX pretty... weird...
- motakuk 5y agoHey everyone, Matvey, ex-CEO of Amixr is here. Me and Ildar Iskhakov started this project three years ago because we used to be on-call ourselves and needed better tools. It was an amazing journey from 0 to 1. Tons of coding, first customers, fundraising, iterating, and finally the honor to join Grafana Labs and build Grafana OnCall! I'll be happy to answer your questions if you have any.
- SeriousM 5y agoHi! Thanks for sharing this news. Will this be available for on-premise installations, and when?
- motakuk 5y agoFor now, we are focusing on rolling Grafana OnCall in the Grafana Cloud. It's a very common use case to have such a system outside of your infrastructure so it won't be affected by probable issues. It should be alive even when everything goes wrong. We've already received multiple questions about OSS and on-premises. Will roll cloud version first, see how it works, collect feedback and build (and share) future plans!
- bilalq 5y agoThis looks really neat. We don't use Grafana today. We're running CloudWatch/insights and Squadcast for alerting, but deep integration with the monitoring tool looks cool. Is this usable with self-hosted or AWS managed Grafana?
- motakuk 5y agoYep! The idea of Grafana OnCall is to help you to group, deduplicate, route & deliver to Slack/SMS/Phone alerts from any sources. It could be a CloudWatch, DataDog, self-hosted Alertmanager, or Grafana of course. The only requirement for the alert source is to be able to generate a webhook and send it to us.
- bilalq 5y ago
- CSDude 5y ago> Alerts from each integration 300 5 minutes > Alerts from the whole team 500 5 minutes > API requests per API key 300 5 minutes Product looks great but those API request limits are too low, because alerts rain when you are having incidents and rate limiting all of them is harmful. That's why other products have deduplication keys / aliases so you don't miss important ones. https://grafana.com/docs/grafana-cloud/oncall/oncall-api-reference/ https://grafana.com/docs/grafana-cloud/oncall/oncall-api-ref...
- deeblering4 5y agoI'd think that receiving even 1/5th the rate limit in a 5 minute window would be disorienting enough to render alerting effectively useless. I'd question the configuration which fires that many alerts in that time frame, and suggest improving alert aggregations and dependencies to get the number down to one or a handful of meaningful alerts.
- curryst 5y agoThe overhead of maintaining those configurations all the time is usually too high to be worth it considering the benefit and likelihood of reaping it. Also, in my experience with those systems, they only make sense to use very sparingly. Your monitoring becomes extremely fragile when your aggregations and dependencies get complicated enough that "what will our alerting system do when X happens?" results in a flow chart with 18 steps. If you aren't careful, you can end up making your aggregations less useful than the raw alerts would be.
- rmetzler 5y agoIt would be great to have a dependency graph or labels in the alerts, so they are easily mapped to the things that can break and are important enough to be monitored. We just had a short outage where an editor removed the index page in the cms which is central to the site. It's stupid that this is possible but we just operate the cms while we build and operate everything around it for our customer. I think a large part of our alerts where triggered all at once but the one thing they had in common was that the alerts all pointed to the index page in the cms. E.g. the public www alert for index, the public api alert for index, the preview www alert for index, the preview api alert for index....
- steveBK123 5y agoFor a product that's been around 12 years, I've been surprised at how minimally featured PagerDuty is. Stuff like national holiday awareness, integration to vacation calendars, a better UI for swapping days/overrides, etc. PD schedule checking and trade negotiation becomes yet another thing in the long list of things I need to do when taking a day off. HR system request off, Department Outlook calendar update, PagerDuty coverage check, Outlook out-of-office status & auto-replies, Slack set away, update status AND pause notifications. I suppose that's because as an on-call developer I am not the user. The user, management who bought the product, gets KPIs & pretty graphs, so they are happy.
- ethbr0 5y agoEvery delightful, successful developer product is eventually doomed to become JIRA.
- rvnx 5y agoA multi-billion USD success story ?
- ethbr0 5y agoThat's one way to look at it.
- jrockway 5y agoJIRA is one of those pieces of software that creates more problems than it solves. That's good for the industry -- it means to get an equivalent amount of work done, you need more people. More jobs competing for the same pool of workers means that you have to pay the workers more. That's good for everyone. (If I were starting a company tomorrow, though, I'd use Linear. Nicest issue tracking tool I have ever seen. It has all the "big business" features like roadmaps and story points, with a lot of friendliness for the individual contributors -- dark mode, keyboard shortcuts, a dedicated triage UI. It's so nice to see someone finally get it right!)
- marcoboffi 5y agobut is it possible to send sms/phone call directly from grafana oncall ? If yes, is there a pricing ?
- kungfufrog 5y agoI'm not sure what this is competing with in it's current incarnation. I need corresponding mobile phone applications for any alert product I intend to use that can override DND/volume etc. on my phone so I can get woken up at night and respond to problems.
- vvoyer 5y agoShameless plug, if you're looking for a simple shift scheduling calendar connected to Slack, I built this: https://turnshift.app https://turnshift.app. It's a team calendar to share recurring tasks as a team. Things like PR reviews, who's on support, or who's qualifying leads. It has far less features than PagerDuty or Grafana OnCall but it serves well a bunch of customers looking for a simple tool to manage team schedules.