5 ms·

> We learned that accidentally sticking a server with nothing but naked IIS into rotation is really bad. Sorry about that one. Wait, does that mean what I thin

> We learned that accidentally sticking a server with nothing but naked IIS into rotation is really bad. Sorry about that one. Wait, does that mean what I think it means? Did someone get an IIS splash page when visiting SO? I was going to say they should live-blog during their next upgrade but it they did it on twitter which is awesome.

aalear 12y ago
Basically, yes. This happened: https://twitter.com/atimmer10/status/560568418945236995 https://twitter.com/atimmer10/status/560568418945236995.
- xtrumanx 12y ago
  How long did it take to figure out it occurred? Do you have any stats on how many people hit that machine? Would be hilarious if someone who went to SO to figure out why the IIS splash was showing instead of his app happened to hit that page.
  - GeorgeBeech 12y ago
    It took about 20 minutes to notice and fix. The Twitter alerting system let us know really quick.
GeorgeBeech 12y ago
We generally pick a hashtag on twitter and do it that way. We've also done Hangouts On Air for non-hardware in the past which have been pretty fun as well.
- gbrayut 12y ago
  If by fun you mean a 4 hour maintenance window turning into 8 hours ending well after midnight :-P
  - GeorgeBeech 12y ago
    I never claimed to be sane
bagels 12y ago
Why is their load balancer not configured to take servers like that out of rotation?
- Nick-Craver 12y ago
  It normally would, except the base IIS page is a proper 200 to all requests. The load balancer removes all servers having issues - from HAProxy's point of view things were dandy. As always, we are looking how to adjust our deployment process to prevent it from happening in the future.
  - clogston 12y ago
    Everyone I know big enough to run a load balancer has had this happen. The next step in evolution is to have your script look for specific text on a page, which will change 24 months later and have all the perfectly good servers pulled out of rotation.
    - deleted 12y ago
      [deleted]
  - xxpor 12y ago
    Does bare IIS respond with a 200 to /ping or /sping?

aalear 12y ago

Basically, yes. This happened: https://twitter.com/atimmer10/status/560568418945236995 https://twitter.com/atimmer10/status/560568418945236995.

xtrumanx 12y ago
How long did it take to figure out it occurred? Do you have any stats on how many people hit that machine? Would be hilarious if someone who went to SO to figure out why the IIS splash was showing instead of his app happened to hit that page.
- GeorgeBeech 12y ago
  It took about 20 minutes to notice and fix. The Twitter alerting system let us know really quick.

GeorgeBeech 12y ago

We generally pick a hashtag on twitter and do it that way. We've also done Hangouts On Air for non-hardware in the past which have been pretty fun as well.

gbrayut 12y ago
If by fun you mean a 4 hour maintenance window turning into 8 hours ending well after midnight :-P
- GeorgeBeech 12y ago
  I never claimed to be sane

bagels 12y ago

Why is their load balancer not configured to take servers like that out of rotation?

Nick-Craver 12y ago
It normally would, except the base IIS page is a proper 200 to all requests. The load balancer removes all servers having issues - from HAProxy's point of view things were dandy. As always, we are looking how to adjust our deployment process to prevent it from happening in the future.
- clogston 12y ago
  Everyone I know big enough to run a load balancer has had this happen. The next step in evolution is to have your script look for specific text on a page, which will change 24 months later and have all the perfectly good servers pulled out of rotation.
  - deleted 12y ago
    [deleted]
- xxpor 12y ago
  Does bare IIS respond with a 200 to /ping or /sping?