4 ms·

At Uber, we are also adopting a lot more SLOs and SLIs. It's a long and painful process. We started implementing error budgets as well, on an area-by-area basis

At Uber, we are also adopting a lot more SLOs and SLIs. It's a long and painful process. We started implementing error budgets as well, on an area-by-area basis. There's a lot of learning, and a lot of trial-and-error. As someone who is working in this space, I found this article to be quite vague, and couldn't take too much out of it. It would be great to see specifics on one SLO, the associated error budget and what graphs Twitter uses to track this. And while SLOs are a good start - I assume this will be SLOs across services on latency and availability - the next step is measuring key user journeys. "What % of users get stuck on a page, getting the same error message over and over?". "Where can we see a sharp increase from p95 to p99 on success or latency on a user journey, that we can click into?" I have a feeling Twitter is still at the first level, as most companies I know are. Clearing this one, the next step is what is even more interesting.

athenot 6y ago
Same at Webex. And yes, coming up with meaningful indicators and objectives is the tricky part. But it's a work of refinement, and over time (with enough persistence), the measurements can converge to cover what really represents the experience of users.
- vosper 6y ago
  You and GP are lucky you work at a company that's moved past an "SLA" being something that's made up on the spot by some manager who has just decided they're not happy about something, and has heard of that acronym. I recently discovered from the QA manager that I have an "SLA" with him on how long I can take to tell him there's some new stuff to test (yes, we're still at the stage of "send me an email to tell me what to test, because I don't believe in automating this stuff"). Must be nice living in your world :) EDIT: I realised my comment contributes nothing to this discussion, and is just noise. I'd be interested if you could expand on what you've learned at Webex.
tdumitrescu 6y ago
Yeah, aggregate service stats are a good start but can get very noisy (e.g. highly skewed by errors uncorrelated with broad user impact). This is an article that provides a nice intro to user-based monitoring approaches, which i'll bet are going to become more and more widespread: https://blog.acolyer.org/2020/02/26/meaningful-availability/ https://blog.acolyer.org/2020/02/26/meaningful-availability/

athenot 6y ago

Same at Webex. And yes, coming up with meaningful indicators and objectives is the tricky part. But it's a work of refinement, and over time (with enough persistence), the measurements can converge to cover what really represents the experience of users.

vosper 6y ago
You and GP are lucky you work at a company that's moved past an "SLA" being something that's made up on the spot by some manager who has just decided they're not happy about something, and has heard of that acronym. I recently discovered from the QA manager that I have an "SLA" with him on how long I can take to tell him there's some new stuff to test (yes, we're still at the stage of "send me an email to tell me what to test, because I don't believe in automating this stuff"). Must be nice living in your world :) EDIT: I realised my comment contributes nothing to this discussion, and is just noise. I'd be interested if you could expand on what you've learned at Webex.

tdumitrescu 6y ago

Yeah, aggregate service stats are a good start but can get very noisy (e.g. highly skewed by errors uncorrelated with broad user impact). This is an article that provides a nice intro to user-based monitoring approaches, which i'll bet are going to become more and more widespread: https://blog.acolyer.org/2020/02/26/meaningful-availability/ https://blog.acolyer.org/2020/02/26/meaningful-availability/