6 ms·

Depends, usually you have to be able to re-build your prod infra within minutes or maximum hours, otherwise you are doing devops wrong. The whole point of autom

Depends, usually you have to be able to re-build your prod infra within minutes or maximum hours, otherwise you are doing devops wrong. The whole point of automation is reproducible infrastructure that you can stand up quickly. With stateless approach you can just do this. Why would you do that? Imagine an outage in one of the 3 datacenters you are running your infra in the same region. You need to move 1/3 of the capacity to the remaining 2 datacenters. This is not too much different to re-building it.

raverbashing 10y ago
"have to be able" is very different from "have to" Sure, you want to be able to deploy quickly. But if there's no reason to, then don't. And I would be very scared if Docker images had a 1 day uptime max
cortesoft 10y ago
I know you probably aren't trying to address all cases, but just because you can't re-build your prod infrastructure in minutes or hours doesn't mean you aren't doing devops right. Many larger companies can't do this; my company has 70+ datacenters with tens of thousands of servers. We can't re-build our prod infra in minutes or hours. We are still doing devops right :D Like I said, I know you aren't talking about my situation when you made your statement... I just get frustrated when people act like there are hard and fast rules for everyone.
- StreamBright 10y ago
  Well I am not talking about a 1M node outage. That you cannot fix with anything. I am talking about a maximum datacenter wide outage, that actually happens pretty often. Amazon has game days, Netflix has chaos monkeys for the same reason. Make sure that you can rebuild parts of your infra pretty quick.
  - cortesoft 10y ago
    Oh, of course. Our datacenters often go offline (both planned and unplanned), and we are always ready to handle that. We are pretty much constantly re-provisioning servers... with so many physical machines, hard drive and other hardware failures are a daily occurrence.
  - toomuchtodo 10y ago
    As long as your VMs are prebaked and your cloud provider supports ASG-esq primatives (i.e. "I need X instances running at a time, instantiate with such and such meta data") anyone can rebuild their prod infra quickly. You don't need Docker or containers to do that.
    - cortesoft 10y ago
      Some of us work for those cloud providers :)
      - toomuchtodo 10y ago
        I have no doubt ;) it's why I always strive for accuracy here!
- yo-code-sucks 10y ago
  Symantec does this with 10's of thousands of nodes. Your point is moot.
- ckdarby 10y ago
  I'm going to have to call bullshit, Where do you work or what company do you own? I work at MindGeek, depending on the time of the year it would be fair to say we rank within the top 25 bandwidth users in the world. We are not even close to that amount of servers and we deal with some of the largest traffic in the world. What company is running in 70+ datacenters!? World's largest VPN provider? Security company providing all the data to the NSA? Maybe it is just my broad assumptions but I would hope that the major big 10 that come to mind such as Google, Amazon, Microsoft, etc would be able to rebuild their production regions in hours.
  - Sanddancer 10y ago
    Not the OP, but first thing that comes to mind as to why you'd need a lot of datacenters are CDNs like Cloudflare or Akamai. Stuff like that, you need lots of servers, lots of storage, and low latency. You'd also need a good amount of configurations because things like video streaming would require different server settings than, say, protecting a site that's being DDOSed.
    - cortesoft 10y ago
      Ding ding ding :D I don't work for one of those two, but I do work for a very large CDN.
  - cortesoft 10y ago
    I work for a large CDN.
  - ceejayoz 10y ago
    > I work at MindGeek, depending on the time of the year it would be fair to say we rank within the top 25 bandwidth users in the world. "I'd have thought I've had heard of them..." One Wiki search later... yup. I've heard of them.
    - Steeeve 10y ago
      Ha! I went to their website... Hmm... never heard of them. Went to Wikipedia... Oh, now I know who they are! Yeah. Lot's of bandwidth.
cerebellum42 10y ago
I think the point of the comment you're replying to is not really what you are replying to. Having the ability to rebuild quickly and frequently is great, and something you should aim for, but actually being forced to do it regularly whether you need it or not is pretty bad. Edit: didn't notice someone else had already said this, opened this tab like 15 minutes ago
- StreamBright 10y ago
  I see your point. The source of force that makes you do it should not come from Docker I agree. :) We are just lucky that we can do it when it happens.
marcosdumay 10y ago
Can != should
falcolas 10y ago
Aaah, the classing "you're doing <it> wrong" argument. I can come up with dozens of different environments where it is simply not feasable to rebuild an environment within two hours. - Any infrastructure with lots of data. Data just takes time to move; backups take time to restore. - You're on bare metal because running node on VMs isn't fast enough. - You're in a secure environment, where the plain old bureaucracy will get in the way of a full rebuild. - Anytime you have to change DNS. That's going to take days to get everything failed over. - Clients (or vendors) whitelist IPs, and you have to work through with them to fix the IPs. - Amazon gives you the dreaded "we don't have capacity to start your requested instance; give us a few hours to spin up more capacity" > Imagine an outage in one of the 3 datacenters you are running your infra in the same region. You need to move 1/3 of the capacity to the remaining 2 datacenters. Oh, this is very different. If your provider loses a datacenter, and your existing infrastructure can't handle it, you're already SOL - the APIs for spinning up instances and networking is going to be DDOSed to death by all of the various users. Basic HA dictates that you provision enough spare capacity that a DC (AZ) can go down and you can still serve all of your customers.
- alrs 10y ago
  "Any infrastructure with lots of data. Data just takes time to move; backups take time to restore." Exactly. Don't put data in Docker. Files go in an object store, databases need to go somewhere else.
- vacri 10y ago
  > Any infrastructure with lots of data. OP's first point is 'don't put data in docker'. Docker is not for your data. But more to the point, if you're rebuilding your data store a couple of times every day, a couple of hours downtime isn't going to be feasible. > You're on bare metal because running node on VMs isn't fast enough In such a situation, you should be able to image bare metal faster than 2 hours. DD a base image, run a config manager over it, and you should be done. Small shops that rarely bring up new infra wouldn't need this, but anyone running 'bare metal to scale' should. > bureaucracy Isn't part of the infra rebuild per se. > Anytime you have to change DNS. That's going to take days Depends on your DNS timeouts, but this is config, not infra. Even if it is infra, 48-hour DNS entries aren't a best-practice anymore (and if you're on AWS, most things default to a 5 min timeout) > Clients (or vendors) whitelist IPs, and you have to work through with them to fix the IPs I'd file this under 'bureaucracy' - it's part of your config, not part of your prod infra (which the GP was talking about). > Amazon gives you the dreaded... Well, yes, but this is on the same order as "what if there's a power outage at the datacentre". Every single deploy plan out there has an unknown-length outage if the 'upstream' dependencies aren't working. "What if there's a hostage event at our NOC?" blah blah. The point is that with upstream working as normal, you should be able to cover the common SPOFs and get your prod components up in a relatively short time.
deleted 10y ago
[deleted]

raverbashing 10y ago

"have to be able" is very different from "have to" Sure, you want to be able to deploy quickly. But if there's no reason to, then don't. And I would be very scared if Docker images had a 1 day uptime max

cortesoft 10y ago

I know you probably aren't trying to address all cases, but just because you can't re-build your prod infrastructure in minutes or hours doesn't mean you aren't doing devops right. Many larger companies can't do this; my company has 70+ datacenters with tens of thousands of servers. We can't re-build our prod infra in minutes or hours. We are still doing devops right :D Like I said, I know you aren't talking about my situation when you made your statement... I just get frustrated when people act like there are hard and fast rules for everyone.

StreamBright 10y ago
Well I am not talking about a 1M node outage. That you cannot fix with anything. I am talking about a maximum datacenter wide outage, that actually happens pretty often. Amazon has game days, Netflix has chaos monkeys for the same reason. Make sure that you can rebuild parts of your infra pretty quick.
- cortesoft 10y ago
  Oh, of course. Our datacenters often go offline (both planned and unplanned), and we are always ready to handle that. We are pretty much constantly re-provisioning servers... with so many physical machines, hard drive and other hardware failures are a daily occurrence.
- toomuchtodo 10y ago
  As long as your VMs are prebaked and your cloud provider supports ASG-esq primatives (i.e. "I need X instances running at a time, instantiate with such and such meta data") anyone can rebuild their prod infra quickly. You don't need Docker or containers to do that.
  - cortesoft 10y ago
    Some of us work for those cloud providers :)
    - toomuchtodo 10y ago
      I have no doubt ;) it's why I always strive for accuracy here!
yo-code-sucks 10y ago
Symantec does this with 10's of thousands of nodes. Your point is moot.
ckdarby 10y ago
I'm going to have to call bullshit, Where do you work or what company do you own? I work at MindGeek, depending on the time of the year it would be fair to say we rank within the top 25 bandwidth users in the world. We are not even close to that amount of servers and we deal with some of the largest traffic in the world. What company is running in 70+ datacenters!? World's largest VPN provider? Security company providing all the data to the NSA? Maybe it is just my broad assumptions but I would hope that the major big 10 that come to mind such as Google, Amazon, Microsoft, etc would be able to rebuild their production regions in hours.
- Sanddancer 10y ago
  Not the OP, but first thing that comes to mind as to why you'd need a lot of datacenters are CDNs like Cloudflare or Akamai. Stuff like that, you need lots of servers, lots of storage, and low latency. You'd also need a good amount of configurations because things like video streaming would require different server settings than, say, protecting a site that's being DDOSed.
  - cortesoft 10y ago
    Ding ding ding :D I don't work for one of those two, but I do work for a very large CDN.
- cortesoft 10y ago
  I work for a large CDN.
- ceejayoz 10y ago
  > I work at MindGeek, depending on the time of the year it would be fair to say we rank within the top 25 bandwidth users in the world. "I'd have thought I've had heard of them..." One Wiki search later... yup. I've heard of them.
  - Steeeve 10y ago
    Ha! I went to their website... Hmm... never heard of them. Went to Wikipedia... Oh, now I know who they are! Yeah. Lot's of bandwidth.

cerebellum42 10y ago

I think the point of the comment you're replying to is not really what you are replying to. Having the ability to rebuild quickly and frequently is great, and something you should aim for, but actually being forced to do it regularly whether you need it or not is pretty bad. Edit: didn't notice someone else had already said this, opened this tab like 15 minutes ago

StreamBright 10y ago
I see your point. The source of force that makes you do it should not come from Docker I agree. :) We are just lucky that we can do it when it happens.

marcosdumay 10y ago

Can != should

falcolas 10y ago

Aaah, the classing "you're doing <it> wrong" argument. I can come up with dozens of different environments where it is simply not feasable to rebuild an environment within two hours. - Any infrastructure with lots of data. Data just takes time to move; backups take time to restore. - You're on bare metal because running node on VMs isn't fast enough. - You're in a secure environment, where the plain old bureaucracy will get in the way of a full rebuild. - Anytime you have to change DNS. That's going to take days to get everything failed over. - Clients (or vendors) whitelist IPs, and you have to work through with them to fix the IPs. - Amazon gives you the dreaded "we don't have capacity to start your requested instance; give us a few hours to spin up more capacity" > Imagine an outage in one of the 3 datacenters you are running your infra in the same region. You need to move 1/3 of the capacity to the remaining 2 datacenters. Oh, this is very different. If your provider loses a datacenter, and your existing infrastructure can't handle it, you're already SOL - the APIs for spinning up instances and networking is going to be DDOSed to death by all of the various users. Basic HA dictates that you provision enough spare capacity that a DC (AZ) can go down and you can still serve all of your customers.

alrs 10y ago
"Any infrastructure with lots of data. Data just takes time to move; backups take time to restore." Exactly. Don't put data in Docker. Files go in an object store, databases need to go somewhere else.
vacri 10y ago
> Any infrastructure with lots of data. OP's first point is 'don't put data in docker'. Docker is not for your data. But more to the point, if you're rebuilding your data store a couple of times every day, a couple of hours downtime isn't going to be feasible. > You're on bare metal because running node on VMs isn't fast enough In such a situation, you should be able to image bare metal faster than 2 hours. DD a base image, run a config manager over it, and you should be done. Small shops that rarely bring up new infra wouldn't need this, but anyone running 'bare metal to scale' should. > bureaucracy Isn't part of the infra rebuild per se. > Anytime you have to change DNS. That's going to take days Depends on your DNS timeouts, but this is config, not infra. Even if it is infra, 48-hour DNS entries aren't a best-practice anymore (and if you're on AWS, most things default to a 5 min timeout) > Clients (or vendors) whitelist IPs, and you have to work through with them to fix the IPs I'd file this under 'bureaucracy' - it's part of your config, not part of your prod infra (which the GP was talking about). > Amazon gives you the dreaded... Well, yes, but this is on the same order as "what if there's a power outage at the datacentre". Every single deploy plan out there has an unknown-length outage if the 'upstream' dependencies aren't working. "What if there's a hostage event at our NOC?" blah blah. The point is that with upstream working as normal, you should be able to cover the common SPOFs and get your prod components up in a relatively short time.

deleted 10y ago

[deleted]