9 ms·
Stack Overflow: How we upgrade a live data center
- 3princip 12y agoI love reading about StackOverflow, particularly their infrastructure. The site has been a useful resource for so many years, and it works so well. It was a joy to discover that it all ran on like two racks worth of servers, and still does. Having seen corporate intranet portals, with maybe a thousand daily active users, running on excessive* hardware (needlessly, of course), it's like a breath of fresh air. *EDIT: Removed hyperbole. Not more hardware, but too much nonetheless.
- GeorgeBeech 12y agoWe take a lot of pride in how much we get out of each piece of hardware. You don't need to have 1000 servers to run a large site, just the mindset of performance first.
- deleted 12y ago[deleted]
- _cudgel 12y agoLikewise, I always find these write-ups fascinating, well written, well planned, etc. I really appreciate the extra effort involved in making all this public!
- xtrumanx 12y ago> We learned that accidentally sticking a server with nothing but naked IIS into rotation is really bad. Sorry about that one. Wait, does that mean what I think it means? Did someone get an IIS splash page when visiting SO? I was going to say they should live-blog during their next upgrade but it they did it on twitter which is awesome.
- aalear 12y agoBasically, yes. This happened: https://twitter.com/atimmer10/status/560568418945236995 https://twitter.com/atimmer10/status/560568418945236995.
- xtrumanx 12y agoHow long did it take to figure out it occurred? Do you have any stats on how many people hit that machine? Would be hilarious if someone who went to SO to figure out why the IIS splash was showing instead of his app happened to hit that page.
- GeorgeBeech 12y agoIt took about 20 minutes to notice and fix. The Twitter alerting system let us know really quick.
- GeorgeBeech 12y agoWe generally pick a hashtag on twitter and do it that way. We've also done Hangouts On Air for non-hardware in the past which have been pretty fun as well.
- gbrayut 12y agoIf by fun you mean a 4 hour maintenance window turning into 8 hours ending well after midnight :-P
- GeorgeBeech 12y agoI never claimed to be sane
- bagels 12y agoWhy is their load balancer not configured to take servers like that out of rotation?
- import 12y agoI'm really wondering how SO uses Redis. For example, how failure scenarios handled and what is the role of slaves.
- daigoba66 12y agoI've asked, and gotten an answer: http://meta.stackexchange.com/questions/248446/how-does-stack-exchange-achieve-high-availability-with-redis http://meta.stackexchange.com/questions/248446/how-does-stac... Other discussions: http://meta.stackexchange.com/questions/69164/does-stack-overflow-use-caching-and-if-so-how/69172#69172 http://meta.stackexchange.com/questions/69164/does-stack-ove... http://meta.stackexchange.com/questions/110320/stack-overflow-db-performance-and-redis-cache http://meta.stackexchange.com/questions/110320/stack-overflo...
- Nick-Craver 12y agoWe have a hot slave at all times for all instances. In the event of a failure the second slave will kick in. All applications are already connected to both servers via the StackExchange.Redis (https://github.com/StackExchange/StackExchange.Redis https://github.com/StackExchange/StackExchange.Redis) library. The mechanisms for failover, etc. are built in there. We use this library via the dashboard (pictured at the bottom of the post) in Opserver (https://github.com/opserver/Opserver https://github.com/opserver/Opserver) to do quick swapping of master/slave, etc. We can do this during the day without anyone noticing. Slaves are not just for backups - they also serve as pub/sub mechanisms. Every publish propagates to slaves. Since we use pub/sub for things like web sockets, we can easily move that entire concurrent connection load to another data center with a simple DNS change. Yep, we've tested this - it worked well. It is of course noting: redis just doesn't fail. We had one out of memory fail for the Q&A web sites when forking and that's it. It has been rock solid here.
- cagenut 12y agoWoah, the datacenter you guys moved to is across the street from my apartment. Just a heads up, I considered it for a project myself but ruled it out because... well this is the back corner of that building: http://40.media.tumblr.com/tumblr_lqnh988WRS1qzpdb2o4_1280.jpg http://40.media.tumblr.com/tumblr_lqnh988WRS1qzpdb2o4_1280.j...
- GeorgeBeech 12y agoThat's nothing: https://gigaom.com/2012/10/31/how-good-prep-and-a-bucket-brigade-kept-peer-1-online-during-hurricane-sandy/ https://gigaom.com/2012/10/31/how-good-prep-and-a-bucket-bri... We where hosted at that Peer 1 facility during Sandy. Also, if the water makes it up to the 16th floor it's time to give up anyway ... no one will care at that point :-D
- cagenut 12y agoyea we got hit hard by it too (33 whitehall), thats why we went shopping for a second location in jersey. we wound up in newark at 165halsey.
- jschwartzi 12y agoIt's odd to see bollards and speed bumps in a swimming pool that big.
- cagenut 12y agohere's the reverse angle in street view: https://www.google.com/maps/@40.718382,-74.042041,3a,75y,355.48h,87.73t/data=!3m4!1e1!3m2!1sIf6OyVdXpkXv2ZrlSx4mhg!2e0 https://www.google.com/maps/@40.718382,-74.042041,3a,75y,355...
- jmccree 12y agoPerhaps off topic, but this post just reminds me so much of why I love AWS EC2, etc. Not ever having to think about hardware again is wonderful.
- Nick-Craver 12y agoThe cloud is awesome if you have zero interest whatsoever in hardware. It's not without trade-offs (nor is the other direction), and too many for us - but if it works for you then great I say. We obviously feel very different, and are just doing what works best for us.
- balls187 12y ago> We obviously feel very different, and are just doing what works best for us. And doing it very well, if I may say so.
- beachstartup 12y agomost of the public cloud cheer-leading is just people rationalizing to themselves what an awesome decision they made deploying on aws onto a billion tiny instances or whatever. and for lots of folks, it probably is pretty awesome. however, i've seen very, very few people compare actual before/after $ figures on hn. when it comes time to show your cards everyone gets cold feet, either because they 1. don't have any idea because they aren't the ones paying for it or 2. don't have a baseline for comparison and are just paying whatever amazon asks or 3. it ended up costing 2-4x as much on amazon when your bill is $5k/month 2-4x isn't that big of a deal. when it's $100k+ a month, it becomes a really big deal.
- rdoherty 12y agoI agree. I read this post and was shocked at the amount of planning, process, man-hours, hardware issues and other problems that come with hardware. I've worked at places with ~500 EC2 machines in a dozen autoscaling groups across 3 AZs with many ELBs, databases, SQS queues and other AWS infrastructure and never had to deal with anything like this when upgrading. Upgrading hardware in EC2 is as simple as changing a launch configuration and updating an auto-scaling group. Maybe an hour of my time to update configs, verify and deploy. Updating something like a database or caching servers is more work for sure, but with 0 time needed to get to the DC, unpack, rack and configure servers you do save time with 'the cloud'. I get that you do pay more for EC2 instances, especially if you keep hardware for 4 years. But AWS prices drop every year or two along with (generally) faster versions of software so your overall costs do drop. How many ops employees would you need for a fleet of 500 servers in a datacenter? We managed it all with 4 people with AWS.
- rasur 12y agoThat's a really nice cabling job. I've inherited 2 DC's with just-awful-enough cabling that I'd be exceptionally happy to set up new racks again and migrate, just to sort the bloody cables out Sigh, don't think it's gonna happen. As a customer of Dell, it was interesting to see someone elses insight on managing a dell farm. So neat! I am mildly envious! ;)
- Nick-Craver 12y agoThanks! We honestly do this for us and my mild cabling OCD, but if we can help anyone by sharing the details then it's worth it.
- rasur 12y agoI'm curious to know what your perceptions of the FX2's are, since it had been racked up. (EDIT: Over and above what's in the article.. Because they look quite nice, is why I'm asking - we haven't got any.. yet! Still performing well?)
- Nick-Craver 12y agoI haven't had nearly as much experience as George and Shane have with them after initial spin up. I'll get one of those guys to chime in here.
- GeorgeBeech 12y agoThey are nice, we honestly haven't had to do much with them after we got them setup since they are VM hosts. The most annoying part was that the Force10 OS that the IOA's run is slightly annoying to work with when you are used to working on Cisco gear. I'd almost rather people stopped doing "it's close to Cicso" and did their own thing because the connotative dissonance is jarring when something is close but not quite what you expect.
- rasur 12y agoWe've got 2 M1000e's, but they're slowly being retired and TBH they're in a 'do-less not do-more' situation (mostly!), so I haven't honestly had much experience past run-of-the-mill blade config, so that particular issue happily hasn't really bitten me :) But thanks (to both you and Nick) for responding, the FX2s look interesting enough that I might have to see about an eval unit :)
- chippy 12y agoIf anything you can see the enthusiasm and passion they have for their job. It's inspiring!
- kfcm 12y agoReading this brought back fond memories. Or nightmares. Sometimes it's difficult to distinguish between the two.
- kfcm 12y agoReading this brought back fond memories. Or nightmares. Sometimes it's difficult to distinguish between the two.
- zaroth 12y agoI think what impressed me most was the power density that data center is supporting. That looks like nearly 10kW of power in a single rack! Many providers top out at 4kW and HE2 in Fremont was just 1.8kW per rack when I was there.
- aristotle 12y agoThanks for sharing! Happy to see some of my past work in action.
- stinos 12y agoMost companies depreciate hardware across 3 years Can somebody explain why exactly? Was there ever some large scale research with the outcome this was somehow the cheapest? Or is it a mere byproduct of the consumption-based society? I'm asking because I'ved used lots of different types of hardware (both computing and non-computing) and >90% lasted well over 3 years. It seems a waste of time/money to get rid of ot after 3 years? Which matches with SE's findings: they choose 4 years. Still not that much, but already 'better'