4 ms·

> It took 10 minutes to identify the cause. I'm impressed they were able to do this so quickly.

bluedino 10y ago
I'm guessing they have the right tools to identify the problem. Wish they went into that a little bit more. Probably went like this: 'web server crashed. so did another. what page did they crash on? ok, let's take a look at the post on that page. what in the....'
ww520 10y ago
System not responsive. Look at the CPU load. Look at the process peaking at 100%. Force dump the stack track of the process couple times. Hmm. All of them stuck in the regex engine. Look back up the stack track to see who calls it. Oh, it's on the home page's text cleansing code. Something like that.
- KyleBrandt 10y ago
  Yes, we took a stackdump and saw the traces leading to that regex.
  - skeletonjelly 10y ago
    How large was the stack dump? I'm impressed you identified it this quickly considering the time it would take to get this to disk.
    - Nick-Craver 10y ago
      We have an internal tool where we can dump stack traces almost instantly by attaching to a running process. We can't open source it due to using Microsoft lab code which was never licensed itself. However, clrmd (https://github.com/Microsoft/clrmd https://github.com/Microsoft/clrmd and https://www.nuget.org/packages/Microsoft.Diagnostics.Runtime https://www.nuget.org/packages/Microsoft.Diagnostics.Runtime) means an open source version is hopefully on the horizon. As an example, here's the result of me piddling in 2 hours with LinqPad: https://gist.github.com/NickCraver/d6292c0c7f93767686e8c5c89ed45c54 https://gist.github.com/NickCraver/d6292c0c7f93767686e8c5c89... Process attachment is in the clrmd roadmap, but isn't stable yet.
      - skeletonjelly 10y ago
        Awesome. Thanks for the response, love your work Nick.
- df07 10y ago
  This is exactly what we did to diagnose (source: I was on the call). The only tricky part was figuring out which post it was, since it wasn't in the stacktrace. To do that, we grabbed the 3000 most recent posts and ran the regex against them. By that point we already had the code fix (another dev working on it in parallel), but if we hadn't we also could have gotten back up by just deleting the post.
  - JoelBennett 10y ago
    Any idea if it was a malicious attempt? It kind of sounds like it was.
    - ars 10y ago
      Nah, https://en.wikipedia.org/wiki/Hanlons_razor https://en.wikipedia.org/wiki/Hanlons_razor If it was malicious they would have made a bunch of them, not just one. I personally have seen many files with unreal amounts of whitespace at the end.
      - aalear 10y ago
        Yeah, that's kinda what we're thinking. If it were malicious, the question itself would be unlikely to actually look like a real honest question, too. It'd just be a bunch of garbage input.
  - ww520 10y ago
    Kudos for thinking clearly under tremendous pressure. A production server down is always a highly stressful situation. If time pressure is not a factor and the production server has the VStudio or WinDbg installed, attaching the debugger to the process can see the data related to the post. But the symbol file and source files might be needed; it's just more hassle to set up. For stressful situation, simple steps and simple tools are more useful. Whatever works.
- teh_klev 10y ago
  ADPlus + WinDBG + SOS possibly.
deleted 10y ago
[deleted]
oddx 10y ago
I guess it was just: - login to server - make dump of all threads stack traces - see that something like Regexp.match present in all stacks - find function that called this regexp.
gibrown 10y ago
Agreed that's impressive debugging for this issue. But... > 10 minutes to roll out the fix That seems very slow to me. 30% of their down time was because their deploy process is slow.
- skeletonjelly 10y ago
  http://nickcraver.com/blog/2016/05/03/stack-overflow-how-we-do-deployment-2016-edition/ http://nickcraver.com/blog/2016/05/03/stack-overflow-how-we-... FWIW here's a write up on their process Also I imagine that 10 minutes included dev and testing, not just the deployment part of "rolling out"
  - gibrown 10y ago
    Seemed to say that 14 minutes were spent writing the code to fix it which I assumed meant testing it, but ya not entirely clear. Maybe "deploy" means the "Deploy" section of this article: http://highscalability.com/blog/2014/7/21/stackoverflow-update-560m-pageviews-a-month-25-servers-and-i.html http://highscalability.com/blog/2014/7/21/stackoverflow-upda... Seems to target only being able to "deploy 5 times a day". I guess maybe the build time is the limiting factor.
  - Nick-Craver 10y ago
    Correct. 10 minutes was from checkin to all servers built out, including a dev and staging tier.
- ww520 10y ago
  10 minutes roll out to production is insanely fast. Roll out usually goes through build, test, staging, and to farms of production servers, with smoke tests in each stage along the way.
  - gibrown 10y ago
    Rollout to thousands of servers on WordPress.com is typically less than 60 seconds. We optimize for reverting fast. Its just interesting to me the implications of what folks optimize for and that this is considered fast. We have very minimal deploy testing and optimize to be able to revert quickly when there are problems because performance issues like this are very hard to predict. Probably means we create many smaller short hiccups though (that generally are not a full site crash).

I'm guessing they have the right tools to identify the problem. Wish they went into that a little bit more. Probably went like this: 'web server crashed. so did another. what page did they crash on? ok, let's take a look at the post on that page. what in the....'

ww520 10y ago

System not responsive. Look at the CPU load. Look at the process peaking at 100%. Force dump the stack track of the process couple times. Hmm. All of them stuck in the regex engine. Look back up the stack track to see who calls it. Oh, it's on the home page's text cleansing code. Something like that.

KyleBrandt 10y ago
Yes, we took a stackdump and saw the traces leading to that regex.
- skeletonjelly 10y ago
  How large was the stack dump? I'm impressed you identified it this quickly considering the time it would take to get this to disk.
  - Nick-Craver 10y ago
    We have an internal tool where we can dump stack traces almost instantly by attaching to a running process. We can't open source it due to using Microsoft lab code which was never licensed itself. However, clrmd (https://github.com/Microsoft/clrmd https://github.com/Microsoft/clrmd and https://www.nuget.org/packages/Microsoft.Diagnostics.Runtime https://www.nuget.org/packages/Microsoft.Diagnostics.Runtime) means an open source version is hopefully on the horizon. As an example, here's the result of me piddling in 2 hours with LinqPad: https://gist.github.com/NickCraver/d6292c0c7f93767686e8c5c89ed45c54 https://gist.github.com/NickCraver/d6292c0c7f93767686e8c5c89... Process attachment is in the clrmd roadmap, but isn't stable yet.
    - skeletonjelly 10y ago
      Awesome. Thanks for the response, love your work Nick.
df07 10y ago
This is exactly what we did to diagnose (source: I was on the call). The only tricky part was figuring out which post it was, since it wasn't in the stacktrace. To do that, we grabbed the 3000 most recent posts and ran the regex against them. By that point we already had the code fix (another dev working on it in parallel), but if we hadn't we also could have gotten back up by just deleting the post.
- JoelBennett 10y ago
  Any idea if it was a malicious attempt? It kind of sounds like it was.
  - ars 10y ago
    Nah, https://en.wikipedia.org/wiki/Hanlons_razor https://en.wikipedia.org/wiki/Hanlons_razor If it was malicious they would have made a bunch of them, not just one. I personally have seen many files with unreal amounts of whitespace at the end.
    - aalear 10y ago
      Yeah, that's kinda what we're thinking. If it were malicious, the question itself would be unlikely to actually look like a real honest question, too. It'd just be a bunch of garbage input.
- ww520 10y ago
  Kudos for thinking clearly under tremendous pressure. A production server down is always a highly stressful situation. If time pressure is not a factor and the production server has the VStudio or WinDbg installed, attaching the debugger to the process can see the data related to the post. But the symbol file and source files might be needed; it's just more hassle to set up. For stressful situation, simple steps and simple tools are more useful. Whatever works.
teh_klev 10y ago
ADPlus + WinDBG + SOS possibly.

deleted 10y ago

[deleted]

oddx 10y ago

I guess it was just: - login to server - make dump of all threads stack traces - see that something like Regexp.match present in all stacks - find function that called this regexp.

gibrown 10y ago

Agreed that's impressive debugging for this issue. But... > 10 minutes to roll out the fix That seems very slow to me. 30% of their down time was because their deploy process is slow.

skeletonjelly 10y ago
http://nickcraver.com/blog/2016/05/03/stack-overflow-how-we-do-deployment-2016-edition/ http://nickcraver.com/blog/2016/05/03/stack-overflow-how-we-... FWIW here's a write up on their process Also I imagine that 10 minutes included dev and testing, not just the deployment part of "rolling out"
- gibrown 10y ago
  Seemed to say that 14 minutes were spent writing the code to fix it which I assumed meant testing it, but ya not entirely clear. Maybe "deploy" means the "Deploy" section of this article: http://highscalability.com/blog/2014/7/21/stackoverflow-update-560m-pageviews-a-month-25-servers-and-i.html http://highscalability.com/blog/2014/7/21/stackoverflow-upda... Seems to target only being able to "deploy 5 times a day". I guess maybe the build time is the limiting factor.
- Nick-Craver 10y ago
  Correct. 10 minutes was from checkin to all servers built out, including a dev and staging tier.
ww520 10y ago
10 minutes roll out to production is insanely fast. Roll out usually goes through build, test, staging, and to farms of production servers, with smoke tests in each stage along the way.
- gibrown 10y ago
  Rollout to thousands of servers on WordPress.com is typically less than 60 seconds. We optimize for reverting fast. Its just interesting to me the implications of what folks optimize for and that this is considered fast. We have very minimal deploy testing and optimize to be able to revert quickly when there are problems because performance issues like this are very hard to predict. Probably means we create many smaller short hiccups though (that generally are not a full site crash).