3 ms·

I broadly agree. However there are a few issues that I have with this piece > When an error occurs, operations can just redeploy I mean no. If there is an err

I broadly agree. However there are a few issues that I have with this piece > When an error occurs, operations can just redeploy I mean no. If there is an error you should figure out what the problem is first. Also if operations just blindly redeploy when things go wrong you have some large culture issues that need to be solved first. > Operations can quickly return to a previous state I kinda agree. However the hardest thing to fix normally is the state that was fucked up by what ever outage that happened. Assuming that you have the correct mechanism for inter-service comms, its really simple to kill/restart services, so long as you have no data in flight. Whats not easy to fix is if the state that was stored is causing the crash. Those are normally the big problems. Bad messages/config causing rolling outages. > Your operations team can adopt a workflow > [..]You were dropped with a bunch of co-workers and all of them did their own thing[..] I mean thats is a massive problem right there. There is either the right way to deploy things, or there isn't. That's a culture problem and needs to be fixed, and fixing it is very very hard. On the hole I broadly agree that Prod should be >95% immutable, and nowadays its fairly simple to achieve. One thing I would recommend is running "prod" in two regions with some sort of health checked loadbalancer infront. This allows testing of IaC stuff in prod without bringing the entire service to a grinding halt when stuff fails. Yes managing state can be much harder, and its important to partition your state by region so you are not hampered by cross region syncing. But the advantages are that you are much better placed to survive an outage, or resource crunch. But most importantly it allows you to completely kill and rebuild a "prod" instance from almost scratch. This is key for DR preparedness.

PretzelPirate 5y ago
> I mean no. If there is an error you should figure out what the problem is first. Figuring out what the problem is may take time. If a problem is impacting a customer’s experience, I’d rather recover faster and then pull metadata later to try and understand the issue.
- packetlost 5y ago
  I think it ultimately comes down to the situation and how well you know your application. Sometimes restarting it will fuck up the data even more. It almost certainly indicates an engineering flaw, but that doesn't mean it won't happen.
- papito 5y ago
  Sure, but it's easy to fall into the trap of "the fix is to just restart it". That's dereliction of duty.
- KaiserPro 5y ago
  > Figuring out what the problem is may take time. If a problem is impacting a customer’s experience, I’d rather recover faster and then pull metadata later to try and understand the issue. How do you know it'll fix the problem, if you don't know the cause? In my experience, about 50% of the time to fix an incident is cleaning up the mess you made performing a fix based on an hypothesis that turns out to be wrong. That means longer downtime. A classic example of this is when a 60 disk raid unit crashed when performing a rebuild under load. This happened to my coworker when they were doing the daily replacing a disk task. They put in the disks, shut the draw, pressed the button that changes the activity lights to see that everything was working, and the thing froze. They rightly shat their pants. They panicked, rebooted the array and attached fileserver. A load of data was lost (the file servers had at the time the maximum amount of ram you could shove in a standard 2u two proc intel server). When it happened again, another coworker waited for the raid enclosure to reboot. Low and behold the SAS links came back and we were able to do an online fsck inside 3 minutes. no data lost. If your facing an outage, breathing space to gather thoughts is time well invested. Blind hacking, or indeed decision paralysis is damaging
lazyant 5y ago
> I mean no. If there is an error you should figure out what the problem is first. First thing you want to do (after confirming the issue and communicating) is to "fly the plane" and try to contain/mitigate the issue before debugging for a RC.

PretzelPirate 5y ago

> I mean no. If there is an error you should figure out what the problem is first. Figuring out what the problem is may take time. If a problem is impacting a customer’s experience, I’d rather recover faster and then pull metadata later to try and understand the issue.

packetlost 5y ago
I think it ultimately comes down to the situation and how well you know your application. Sometimes restarting it will fuck up the data even more. It almost certainly indicates an engineering flaw, but that doesn't mean it won't happen.
papito 5y ago
Sure, but it's easy to fall into the trap of "the fix is to just restart it". That's dereliction of duty.
KaiserPro 5y ago
> Figuring out what the problem is may take time. If a problem is impacting a customer’s experience, I’d rather recover faster and then pull metadata later to try and understand the issue. How do you know it'll fix the problem, if you don't know the cause? In my experience, about 50% of the time to fix an incident is cleaning up the mess you made performing a fix based on an hypothesis that turns out to be wrong. That means longer downtime. A classic example of this is when a 60 disk raid unit crashed when performing a rebuild under load. This happened to my coworker when they were doing the daily replacing a disk task. They put in the disks, shut the draw, pressed the button that changes the activity lights to see that everything was working, and the thing froze. They rightly shat their pants. They panicked, rebooted the array and attached fileserver. A load of data was lost (the file servers had at the time the maximum amount of ram you could shove in a standard 2u two proc intel server). When it happened again, another coworker waited for the raid enclosure to reboot. Low and behold the SAS links came back and we were able to do an online fsck inside 3 minutes. no data lost. If your facing an outage, breathing space to gather thoughts is time well invested. Blind hacking, or indeed decision paralysis is damaging

lazyant 5y ago

> I mean no. If there is an error you should figure out what the problem is first. First thing you want to do (after confirming the issue and communicating) is to "fly the plane" and try to contain/mitigate the issue before debugging for a RC.