3 ms·

Question for folks who work at large technology companies: In the peasant world, some of us use various combinations of software to go through our logs and mon

Question for folks who work at large technology companies: In the peasant world, some of us use various combinations of software to go through our logs and monitor errors - 500 and others, as part of our day-to-day. There are products built around this like rollbar, sentry, logrocket etc. Typically we'd encounter these routinely and fix them, or reach out to the users independently if we're unsure of the specific use-case that's leading to the crash. How does this kind of thing work at large corporations? I imagine there are hundreds of different web applications, and there's a separate devops team, possibly for each of them. How do the developers in places like FAANG, Microsoft, etc. stay notified about these kinds of crashes? What is the average volume of 500/internal server errors at scale? Is it considered normal to have hundreds/thousands of these a day, thereby creating a needle in a haystack kind of problem?

munchbunny 5y ago
Speaking for my own experience, the sheer scale means that there are always errors in varying amounts and you are just dealing with the most important ones at any given time. Running with your needle in a haystack analogy: you will not go looking for a needle in a haystack, nor will that needle be relevant, if right next to it you have a whole bale of needles. And you often do.
lnanek2 5y ago
Not even just that. Often we engineers know about the issues, but they are so rare the PMs aren't willing to let us work on them. PMs get bonuses for shipping features, not letting their engineers do KLO work and keep all the rare edge cases functioning. My team once had a bug on a client app that caused the phone to vibrate randomly, and the PM still wanted us to spend a week to rewrite an Android demo to iOS so he could look better for a presentation to the CEO next week instead of fix it. Sometimes the release team will put their foot down on a really bad error rate and force a team to fix it or not get into the next release. You pretty much need hundreds of crashes an hour for that, though. One guy paranoid about people trying to login with his phone number and flipping a weird option would just get set to priority normal, backlog status in the issue tracker and never looked at.
Kilenaitor 5y ago
I work at a big tech company (not MS, though). The short answer is it really all comes down to prioritization. In my experience, we do have the same kinds of tools for surfacing those issues. Granted, they do tend to differ per function or even per org, although there are some standardized ones you can plug into to get e.g. issue aggregation, stack traces + metadata, etc. Granted... a lot of the time that requires a team to 1. actually plug into this infrastructure (otherwise it can either bubble up as an unknown exception or, worst case, get swallowed by a try/catch) and 2. actually pay attention to things once they do plug in. The other problem is not just the high scale of traffic (or 5XXs or errors in general) but that high traffic is usually due to a myriad of products, features, etc. And there's just simply too much work for too few people. So, we prioritize based on severity of the inherent issue (e.g. privacy/security issues go to the top) or, if that's equivalent, number of affected users take priority. Which just means low-severity, low-impact bugs almost never get fixed. Especially if they have a workaround that can be surfaced e.g. when someone reports the issue to support. Happy to elaborate further if that doesn't make sense. I also want to clarify that this explanation isn't meant to be some perfectly valid reason as to why this kind of work doesn't get prioritized. I agree a lot of times big companies fall short here, including where I work. This is just one of the many tradeoff that get made when there is more work than people.
theshadowknows 5y ago
We just launched a new product. It wasn’t really new, though. It’s a copy of a product that we used to buy but for some reason decided to build in house. It got rushed forward by a month (the announcement internally was “the new prod date is X is your team ready?”). It’s been awful for users since the second it went live. I’ve personally had to spend dozens of hours on it so far and I’m sure some fresh hell will show up tomorrow. But everyone is excited because it’s one more dot on the PowerPoint of stuff team X has achieved this quarter. And tracking down errors? Logs? Ppfftttt…guess what got descoped as low priority when everyone had to scramble. So chasing down even seemingly simple bugs is a nightmare.
tallanvor 5y ago
Honestly, at large companies things work similarly to what you're used to, just at a different scale. But that means that you're usually focusing on issues that are affecting thousands of users - something like what is described in the blog post probably doesn't happen very frequently, so it might not be noticed. One thing that the author never mentioned is whether or not he used the "Enable Flagging" option that is in one of the screenshots - that would actually be one of the best options, especially if he could then provide the specific correlation IDs in support requests.
shepherdjerred 5y ago
Lots of alarms on metrics, with important ones connected to a pager. Issues prioritization comes down to someone forcing the team to fix an issue, whether that be customer support, another internal team, or a org wide mandate. My team had pretty high thresholds for errors. We’d review dashboards weekly to see any trends or changes.
lozenge 5y ago
Remember Microsoft had an issue where a Windows upgrade deleted the contents of your Documents folder as part of the OneDrive migration. Users reported it but it was ignored. Since then, they added a field on all feedback forms to fast track issues that result in data loss. However, they clearly didn't apply the same logic to login issues, perhaps fearing social engineering.

munchbunny 5y ago

Speaking for my own experience, the sheer scale means that there are always errors in varying amounts and you are just dealing with the most important ones at any given time. Running with your needle in a haystack analogy: you will not go looking for a needle in a haystack, nor will that needle be relevant, if right next to it you have a whole bale of needles. And you often do.

lnanek2 5y ago

Not even just that. Often we engineers know about the issues, but they are so rare the PMs aren't willing to let us work on them. PMs get bonuses for shipping features, not letting their engineers do KLO work and keep all the rare edge cases functioning. My team once had a bug on a client app that caused the phone to vibrate randomly, and the PM still wanted us to spend a week to rewrite an Android demo to iOS so he could look better for a presentation to the CEO next week instead of fix it. Sometimes the release team will put their foot down on a really bad error rate and force a team to fix it or not get into the next release. You pretty much need hundreds of crashes an hour for that, though. One guy paranoid about people trying to login with his phone number and flipping a weird option would just get set to priority normal, backlog status in the issue tracker and never looked at.

Kilenaitor 5y ago

I work at a big tech company (not MS, though). The short answer is it really all comes down to prioritization. In my experience, we do have the same kinds of tools for surfacing those issues. Granted, they do tend to differ per function or even per org, although there are some standardized ones you can plug into to get e.g. issue aggregation, stack traces + metadata, etc. Granted... a lot of the time that requires a team to 1. actually plug into this infrastructure (otherwise it can either bubble up as an unknown exception or, worst case, get swallowed by a try/catch) and 2. actually pay attention to things once they do plug in. The other problem is not just the high scale of traffic (or 5XXs or errors in general) but that high traffic is usually due to a myriad of products, features, etc. And there's just simply too much work for too few people. So, we prioritize based on severity of the inherent issue (e.g. privacy/security issues go to the top) or, if that's equivalent, number of affected users take priority. Which just means low-severity, low-impact bugs almost never get fixed. Especially if they have a workaround that can be surfaced e.g. when someone reports the issue to support. Happy to elaborate further if that doesn't make sense. I also want to clarify that this explanation isn't meant to be some perfectly valid reason as to why this kind of work doesn't get prioritized. I agree a lot of times big companies fall short here, including where I work. This is just one of the many tradeoff that get made when there is more work than people.

theshadowknows 5y ago

We just launched a new product. It wasn’t really new, though. It’s a copy of a product that we used to buy but for some reason decided to build in house. It got rushed forward by a month (the announcement internally was “the new prod date is X is your team ready?”). It’s been awful for users since the second it went live. I’ve personally had to spend dozens of hours on it so far and I’m sure some fresh hell will show up tomorrow. But everyone is excited because it’s one more dot on the PowerPoint of stuff team X has achieved this quarter. And tracking down errors? Logs? Ppfftttt…guess what got descoped as low priority when everyone had to scramble. So chasing down even seemingly simple bugs is a nightmare.

tallanvor 5y ago

Honestly, at large companies things work similarly to what you're used to, just at a different scale. But that means that you're usually focusing on issues that are affecting thousands of users - something like what is described in the blog post probably doesn't happen very frequently, so it might not be noticed. One thing that the author never mentioned is whether or not he used the "Enable Flagging" option that is in one of the screenshots - that would actually be one of the best options, especially if he could then provide the specific correlation IDs in support requests.

shepherdjerred 5y ago

Lots of alarms on metrics, with important ones connected to a pager. Issues prioritization comes down to someone forcing the team to fix an issue, whether that be customer support, another internal team, or a org wide mandate. My team had pretty high thresholds for errors. We’d review dashboards weekly to see any trends or changes.

lozenge 5y ago

Remember Microsoft had an issue where a Windows upgrade deleted the contents of your Documents folder as part of the OneDrive migration. Users reported it but it was ignored. Since then, they added a field on all feedback forms to fast track issues that result in data loss. However, they clearly didn't apply the same logic to login issues, perhaps fearing social engineering.