4 ms·

Meta Uses LLMs to Improve Incident Response

138 points by wilson090 2y ago

tkgally 2y ago
Previous discussion: https://news.ycombinator.com/item?id=41326039 https://news.ycombinator.com/item?id=41326039
jedberg 2y ago
Wild Moose (W23) does this for everyone. https://www.wildmoose.ai https://www.wildmoose.ai Edit: Lol at myself, I thought this was a blog post from Meta and I was pointing out that there is a YC company that does this for everyone. Now I realize that this was an ad for a different YC company that also does (although WM is a year older).
- netik 2y ago
  Buy our product !
  - lmm 2y ago
    When the post is "buy our product", "buy our product" is a reasonable comment.
    - whiplash451 2y ago
      And "flag the post" is even more reasonable.
- Terretta 2y ago
  Warning, pricing is: free, call us, call us.
netik 2y ago
Great idea but yet another blog post, which is actually marketing, which ends with “they did it buy our product so you can too”, which is probably not what Meta did.
beoberha 2y ago
You’re better off reading the original post instead of this ad: https://news.ycombinator.com/item?id=41326039 https://news.ycombinator.com/item?id=41326039 On the general topic - I’m interested in people sharing their experiences using LLMs in livesite scenarios.
bagels 2y ago
I found something important was broken at Facebook, created a SEV, but the hardest part was figuring out what team to page. I unfortunately roped in one non-responsible team's oncall who was making dinner at home, but thankfully, he did know what team to reach out to, based on the description of the problem. Would be nice if there was better tooling for going from observed problem to responsible team.
- chronid 2y ago
  Some FAANGs at least (though they may not cover everything) have a "help something is broken but I don't know what to do" team and/or rotation for incident response, staffed on multiple continents to "follow the sun". But you need to know they exist. :)
  - fma 2y ago
    I never worked at a FAANG...but a Fortune 20 company the last 9 years. There is no system of record of applications? I can go to a website and type in search terms, URLs and pull up exactly who to contact. Even our generic "help something is broken" group relies on this. There are many names listed so even if the on call person listed is "making dinner", you have their backup, their manager, etc. I can tag my system as dependent on another and if they have issues I get alerted.
    - chronid 2y ago
      I am fairly simplifying, but you are expected to know your direct dependencies (and normally wil), pagers have embedded escalation rules with prinaries and secondaries, etc. The tooling once you know what to do is better than anything outside of FAANGs I've seen in terms of integration and reliability. Escalation teams are usually reserved for the "oh fuck" situations, like "I don't work on this site but I found it broken" or "hey I think we are going to lose soon this availability zone" or "I am panicking and have no idea how to manage this incident, please help me". They're a glue mechanism to prevent silos and paralysis during an event, usually pretty good engineers too.
  - 2y ago
nyellin 2y ago
I know there are already a number of comments here about proprietary solutions. If you're looking for something open source: https://github.com/robusta-dev/holmesgpt/ https://github.com/robusta-dev/holmesgpt/
evnsio 2y ago
> Humans aren't great at incident response, and we all hate waking up at 2am to resolve an issue. Agree that most people hate being woken at 2am, but disagree that humans aren't great at incident response. Speaking generally, I think we're about as good as it gets when it comes to adaptability and the kind reasoning that's necessary to investigate complex issues. That said, I also think AI can play a massive role aiding humans, especially in undifferentiated tasks like checking deployments, code changes, past incidents, and when it comes to spotting patterns. IMO the sweet spot is going to come from highly ergonomic AI products that enable collaborative incident response, rather AI incident management or any other marketing BS.
- serious_angel 2y ago
  Frankly, the whole phrase quoted sounds like a personal issue. If it is, let's hope they'll manage it, and get a proper sleep.
- wkat4242 2y ago
  The thing is, nobody needs to be woken up at 2am at Meta. They're big enough to have a follow the sun model where they rotate support around the world. They have a footprint almost everywhere.
whiplash451 2y ago
I would expect HN to flag product marketing posts when the original post is available instead
jolarkin 2y ago
of course they do, why would this be interesting at all???
- trod1234 2y ago
  The interest is in the broad trend. This clearly telegraphs a few things for those that are observant enough to notice. Anyone interested in going into that field who has a brain would note that they are removing the entry level jobs using this since those are the simpler jobs. With less demand for said labor, the more cutthroat the competition to get said jobs will be. If you double the competition, wages must naturally drop and with that so does competency and skill. When proficiency isn't rewarded, those with options go elsewhere. When any other job with 1/10th the responsibility gets paid the same, its a simple no brainer choice. When industry with large marketshare telegraphs this, it is a strong indicator that IT operations (in this case) will soon be a dead profession (soon being a relative generation), and anyone investing in education for it will get a negative ROI. Upfront, they companies may get some profits off the bottom-line, but long-term they won't be able to find or keep talent. Said talent will have left and you get Atlas Shrugged dynamics. Aside from that, that type of strategy also suggests other cascading dynamics that lead to grid/societal collapse following a burning the bridges approach. Those in their ivory towers may not see the consequences of their choices until it is too late for anything except coffin nails. The fundamental issue of market sector concentration where few parties cooperate is that counter-party systemic risk becomes unmanageable and chaotic, destructive decisions cannot be softened. In chaotic feedback systems, the general rule of thumb is chaos increases until the underlying imbalance is equalized. For some, recognizing the inevitable conclusion of a series of events may dictate a radically altered approach towards life planning/survival.
pzh 2y ago
So if you look at the flip side, the on call engineer is being misled by AI ~60% of the time. The question is does this slow them down more or less than the speedup they get when the AI is right the other 40% of the time.
trod1234 2y ago
Yeah this is just brilliant. /s Reading between the lines/Translation: We need ~50% less IT operations staff, we can have LLMs do this instead. The LLMs are coming for your white-collar work, and this is likely driving the white-collar recession. Here's a thought, incident response is one of the areas where entry-level SA's sharpen their teeth and skills to become able to complete intermediate and senior level roles. What happens when all the low-hanging fruit, the entry-level jobs are now replaced by LLMs and there is no short-term business need to hire such people. No jobs means labor pool finds something else outside their profession, or they sit around on food stamps homeless, agitating with the homies, until a critical mass occurs. Your intermediate and senior people naturally age and die, so how do you find replacements for them? ... When there is no economic advantage for developing a skill set, no one goes into the field, it acts like a sieve, and eventually the skills involved becomes lost knowledge. Those that have the skills that are unable to find work seek work elsewhere, and rarely return. They were burned severely, enough times that it becomes a bad bet to try again in that field. These things aren't rocket science, and yet people seem to be so slothful or greedy, that they are unable to see or act to prevent what naturally happens next. When people can't find jobs to feed themselves or loved ones, where the only future which has been imposed on them is slavery or death, these people will organize and do the only thing they can once they are desperate enough; and that is violence. These same dynamics were present in the decades leading up to 1776, according to historic record. It is so extremely short-sighted.

138 points by wilson090 2y ago

Previous discussion: https://news.ycombinator.com/item?id=41326039 https://news.ycombinator.com/item?id=41326039

Wild Moose (W23) does this for everyone. https://www.wildmoose.ai https://www.wildmoose.ai Edit: Lol at myself, I thought this was a blog post from Meta and I was pointing out that there is a YC company that does this for everyone. Now I realize that this was an ad for a different YC company that also does (although WM is a year older).

netik 2y ago
Buy our product !
- lmm 2y ago
  When the post is "buy our product", "buy our product" is a reasonable comment.
  - whiplash451 2y ago
    And "flag the post" is even more reasonable.
Terretta 2y ago
Warning, pricing is: free, call us, call us.

netik 2y ago

Great idea but yet another blog post, which is actually marketing, which ends with “they did it buy our product so you can too”, which is probably not what Meta did.

beoberha 2y ago

You’re better off reading the original post instead of this ad: https://news.ycombinator.com/item?id=41326039 https://news.ycombinator.com/item?id=41326039 On the general topic - I’m interested in people sharing their experiences using LLMs in livesite scenarios.

bagels 2y ago

I found something important was broken at Facebook, created a SEV, but the hardest part was figuring out what team to page. I unfortunately roped in one non-responsible team's oncall who was making dinner at home, but thankfully, he did know what team to reach out to, based on the description of the problem. Would be nice if there was better tooling for going from observed problem to responsible team.

chronid 2y ago
Some FAANGs at least (though they may not cover everything) have a "help something is broken but I don't know what to do" team and/or rotation for incident response, staffed on multiple continents to "follow the sun". But you need to know they exist. :)
- fma 2y ago
  I never worked at a FAANG...but a Fortune 20 company the last 9 years. There is no system of record of applications? I can go to a website and type in search terms, URLs and pull up exactly who to contact. Even our generic "help something is broken" group relies on this. There are many names listed so even if the on call person listed is "making dinner", you have their backup, their manager, etc. I can tag my system as dependent on another and if they have issues I get alerted.
  - chronid 2y ago
    I am fairly simplifying, but you are expected to know your direct dependencies (and normally wil), pagers have embedded escalation rules with prinaries and secondaries, etc. The tooling once you know what to do is better than anything outside of FAANGs I've seen in terms of integration and reliability. Escalation teams are usually reserved for the "oh fuck" situations, like "I don't work on this site but I found it broken" or "hey I think we are going to lose soon this availability zone" or "I am panicking and have no idea how to manage this incident, please help me". They're a glue mechanism to prevent silos and paralysis during an event, usually pretty good engineers too.
- 2y ago

nyellin 2y ago

I know there are already a number of comments here about proprietary solutions. If you're looking for something open source: https://github.com/robusta-dev/holmesgpt/ https://github.com/robusta-dev/holmesgpt/

evnsio 2y ago

> Humans aren't great at incident response, and we all hate waking up at 2am to resolve an issue. Agree that most people hate being woken at 2am, but disagree that humans aren't great at incident response. Speaking generally, I think we're about as good as it gets when it comes to adaptability and the kind reasoning that's necessary to investigate complex issues. That said, I also think AI can play a massive role aiding humans, especially in undifferentiated tasks like checking deployments, code changes, past incidents, and when it comes to spotting patterns. IMO the sweet spot is going to come from highly ergonomic AI products that enable collaborative incident response, rather AI incident management or any other marketing BS.

serious_angel 2y ago
Frankly, the whole phrase quoted sounds like a personal issue. If it is, let's hope they'll manage it, and get a proper sleep.
wkat4242 2y ago
The thing is, nobody needs to be woken up at 2am at Meta. They're big enough to have a follow the sun model where they rotate support around the world. They have a footprint almost everywhere.

whiplash451 2y ago

I would expect HN to flag product marketing posts when the original post is available instead

jolarkin 2y ago

of course they do, why would this be interesting at all???

trod1234 2y ago
The interest is in the broad trend. This clearly telegraphs a few things for those that are observant enough to notice. Anyone interested in going into that field who has a brain would note that they are removing the entry level jobs using this since those are the simpler jobs. With less demand for said labor, the more cutthroat the competition to get said jobs will be. If you double the competition, wages must naturally drop and with that so does competency and skill. When proficiency isn't rewarded, those with options go elsewhere. When any other job with 1/10th the responsibility gets paid the same, its a simple no brainer choice. When industry with large marketshare telegraphs this, it is a strong indicator that IT operations (in this case) will soon be a dead profession (soon being a relative generation), and anyone investing in education for it will get a negative ROI. Upfront, they companies may get some profits off the bottom-line, but long-term they won't be able to find or keep talent. Said talent will have left and you get Atlas Shrugged dynamics. Aside from that, that type of strategy also suggests other cascading dynamics that lead to grid/societal collapse following a burning the bridges approach. Those in their ivory towers may not see the consequences of their choices until it is too late for anything except coffin nails. The fundamental issue of market sector concentration where few parties cooperate is that counter-party systemic risk becomes unmanageable and chaotic, destructive decisions cannot be softened. In chaotic feedback systems, the general rule of thumb is chaos increases until the underlying imbalance is equalized. For some, recognizing the inevitable conclusion of a series of events may dictate a radically altered approach towards life planning/survival.

pzh 2y ago

So if you look at the flip side, the on call engineer is being misled by AI ~60% of the time. The question is does this slow them down more or less than the speedup they get when the AI is right the other 40% of the time.

trod1234 2y ago

Yeah this is just brilliant. /s Reading between the lines/Translation: We need ~50% less IT operations staff, we can have LLMs do this instead. The LLMs are coming for your white-collar work, and this is likely driving the white-collar recession. Here's a thought, incident response is one of the areas where entry-level SA's sharpen their teeth and skills to become able to complete intermediate and senior level roles. What happens when all the low-hanging fruit, the entry-level jobs are now replaced by LLMs and there is no short-term business need to hire such people. No jobs means labor pool finds something else outside their profession, or they sit around on food stamps homeless, agitating with the homies, until a critical mass occurs. Your intermediate and senior people naturally age and die, so how do you find replacements for them? ... When there is no economic advantage for developing a skill set, no one goes into the field, it acts like a sieve, and eventually the skills involved becomes lost knowledge. Those that have the skills that are unable to find work seek work elsewhere, and rarely return. They were burned severely, enough times that it becomes a bad bet to try again in that field. These things aren't rocket science, and yet people seem to be so slothful or greedy, that they are unable to see or act to prevent what naturally happens next. When people can't find jobs to feed themselves or loved ones, where the only future which has been imposed on them is slavery or death, these people will organize and do the only thing they can once they are desperate enough; and that is violence. These same dynamics were present in the decades leading up to 1776, according to historic record. It is so extremely short-sighted.