6 ms·
I know you probably aren't trying to address all cases, but just because you can't re-build your prod infrastructure in minutes or hours doesn't mean you aren't
by cortesoft 10y ago
I know you probably aren't trying to address all cases, but just because you can't re-build your prod infrastructure in minutes or hours doesn't mean you aren't doing devops right.
Many larger companies can't do this; my company has 70+ datacenters with tens of thousands of servers. We can't re-build our prod infra in minutes or hours. We are still doing devops right :D
Like I said, I know you aren't talking about my situation when you made your statement... I just get frustrated when people act like there are hard and fast rules for everyone.
- StreamBright 10y agoWell I am not talking about a 1M node outage. That you cannot fix with anything. I am talking about a maximum datacenter wide outage, that actually happens pretty often. Amazon has game days, Netflix has chaos monkeys for the same reason. Make sure that you can rebuild parts of your infra pretty quick.
- cortesoft 10y agoOh, of course. Our datacenters often go offline (both planned and unplanned), and we are always ready to handle that. We are pretty much constantly re-provisioning servers... with so many physical machines, hard drive and other hardware failures are a daily occurrence.
- toomuchtodo 10y agoAs long as your VMs are prebaked and your cloud provider supports ASG-esq primatives (i.e. "I need X instances running at a time, instantiate with such and such meta data") anyone can rebuild their prod infra quickly. You don't need Docker or containers to do that.
- cortesoft 10y agoSome of us work for those cloud providers :)
- toomuchtodo 10y agoI have no doubt ;) it's why I always strive for accuracy here!
- yo-code-sucks 10y agoSymantec does this with 10's of thousands of nodes. Your point is moot.
- ckdarby 10y agoI'm going to have to call bullshit, Where do you work or what company do you own? I work at MindGeek, depending on the time of the year it would be fair to say we rank within the top 25 bandwidth users in the world. We are not even close to that amount of servers and we deal with some of the largest traffic in the world. What company is running in 70+ datacenters!? World's largest VPN provider? Security company providing all the data to the NSA? Maybe it is just my broad assumptions but I would hope that the major big 10 that come to mind such as Google, Amazon, Microsoft, etc would be able to rebuild their production regions in hours.
- Sanddancer 10y agoNot the OP, but first thing that comes to mind as to why you'd need a lot of datacenters are CDNs like Cloudflare or Akamai. Stuff like that, you need lots of servers, lots of storage, and low latency. You'd also need a good amount of configurations because things like video streaming would require different server settings than, say, protecting a site that's being DDOSed.
- cortesoft 10y agoDing ding ding :D I don't work for one of those two, but I do work for a very large CDN.
- cortesoft 10y agoI work for a large CDN.
- ceejayoz 10y ago> I work at MindGeek, depending on the time of the year it would be fair to say we rank within the top 25 bandwidth users in the world. "I'd have thought I've had heard of them..." One Wiki search later... yup. I've heard of them.
- Steeeve 10y agoHa! I went to their website... Hmm... never heard of them. Went to Wikipedia... Oh, now I know who they are! Yeah. Lot's of bandwidth.