5 ms·

How do you troubleshoot?

mdaniel 2y ago
In my world, if a developer needs access to the Node upon which their app is deployed to troubleshoot, that's 100% a bug in their application. I am cognizant that being whole-hog on 12 Factor apps is a journey, but for my money get on the train because "let me just ssh in and edit this one config file" is the road to ruin when no one knows who edited what to set it to what new value. Running $(kubectl edit) allows $(kubectl rollout undo) to put it back, and also shows what was changed from what to what
- yjftsjthsd-h 2y ago
  How do you debug the worker itself?
  - mdaniel 2y ago
    Separate from my sibling comment about AWS SSM, I also believe that if one cannot know that a Node is sick by the metrics or log egress from it, that's a deployment bug. I'm firmly in the "Cattle" camp, and am getting closer and closer to the "Reverse Uptime" camp - made easier by ASG's newfound "Instance Lifespan" setting to make it basically one-click to get onboard that train Even as I type all these answers out, I'm super cognizant that there's not one hammer for all nails, and I am for sure guilty of yanking Nodes out of the ASG in order to figure out what the hell has gone wrong with them, but I try very very hard not to place my Nodes in a precarious situation to begin with so that such extreme troubleshooting becomes a minor severity incident and not Situation Normal
    - __turbobrew__ 2y ago
      If accidentally nuking a single node while debugging causes issues you have bigger problems. Especially if you are running kubernetes any node should be able to fall off the earth at any time without issues. I agree that you should set a maximum lifetime for a node on the order of a few weeks. I also agree that you shouldn’t be giving randos access to production infra, but and the end of the day there needs to be some people at the company who have the keys to the kingdom because you don’t know what you don’t know and you need to be able to deal with unexpected faults or outages of the telemetry and logging systems. I once bootstrapped an entire datacenter with tens of thousands of nodes from an SSH terminal after an abrupt power failure. It turns out infrastructure has lots of circular dependencies and we had to manually break that dependency.
      - ramzyo 2y ago
        Exactly this. Have heard it referred to as "break glass access". Some form of remote access, be it SSH or otherwise, in case of serious emergency.
    - viraptor 2y ago
      Passive metrics/logs won't let you debug all the issues. At some point you either need a system for automatic memory dumps and submitting bpf scripts to live nodes... or you need SSH access to do that.
      - otabdeveloper4 2y ago
        This "system for automatic dumps" 100 percent uses ssh under the hood. Probably with some eternal sudo administrator key. Personal ssh access is always better (from a security standpoint) than bot tokens and keys.
        viraptor 2y ago
        There's a thousand ways to do it without SSH. It can be built into the app itself. It can be a special authenticated route to a suid script. It can be built into the current orchestration system. It can be pull-based using the a queue for system monitoring commands. It can be part of the existing monitoring agent. It can be run through AWS SSM. There's really no reason it has to be SSH. And even got SSH you can have special keys with access authorised to only specific commands, so a service account would be better than personal in that case.
    - acdha 2y ago
      > Separate from my sibling comment about AWS SSM, This seems like it’s conceding the point since SSM also allows you to run commands on nodes - I use it interchangeably with SSH to have Ansible manage legacy servers. Maybe what you’re trying to say is that it shouldn’t be routine and that there should be more of a review process so it’s not just a random unrestricted shell session? I think that’s less controversial, and especially when combined with some kind of “taint” mode where your access to a server triggers a rebuild after the dust has settled.
      - mdaniel 2y ago
        Yes, you nailed it with "it shouldn't be routine" and there for sure should be a review process. My primary concern with the audit logs actually isn't security it's lowering the cowboy of the software lifecycle > combined with some kind of “taint” mode where your access to a server triggers a rebuild after the dust has settled. Oh, I love that idea: thanks for bringing it to my attention. I'll for sure incorporate that into my process going forward
        acdha 2y ago
        The first time I heard it was a very simple idea: they had a wrapper for the command which installed SSH keys on an EC2 instance which also set a delete-after tag which CloudCustodian queried.
  - from-nibly 2y ago
    You don't. you shoot it in the head and get a new one. If you need logging / telemetry bake it into the image.
    - otabdeveloper4 2y ago
      Are you from techsupport? Actually not every problem is solved with the "have you tried turning it off and back on again" trick.
      - mdaniel 2y ago
        No, what we're talking about is (to extend your very condescending tech support analogy) shipping the customer a new PC from the factory, and telling them to throw the old one away because it doesn't matter. It will only start to matter if they have 3 bad PCs in a row, at which time it becomes (a) a demonstrable failure and not just stray neutron rays (b) an incident which will carry a postmortum of how the organization could have prevented that failure for next time I did start the whole thread by saying "and then I grew up," and not everyone is at the same place in their organizational maturity model. So, if you're happy with the process you have now, keep using it. I was unhappy, so I studied hard, incorporated supporting technology, and lobbied my heart out for change. Without maturity levels we'd all still be using telnet and tar based version control
        otabdeveloper4 2y ago
        Once you reach the next maturity level you realize that some problems and bugs are from bad design, and cannot be fixed by restarting the server. Fixing bad design is an art, not an organizational discipline. (Sadly.)
- megous 2y ago
  Your world is very narrow and limited. Some devs also have to deal with customer provisioned HW infrastructure, with buggy interactions between HW/virtualization solutions that every 5 minutes duplicate all packets for a few seconds; with applications that interact with customer only onsite HW you only have remote access to via production deployment; with quirky virtualization like vmware stopping the vCPU on you for hundreds of ms if you load it too much which you'll not replicate locally; with things you can't predict you'll need to observe ahead of time, etc. And it does not involve editing any configs. It's just troubleshooting.
bigiain 2y ago
I think ssh-ing into production is a sign of not fully mature devops practices. We are still stuck there, but we're striving to get to the place where we can turn off sshd on Prod and rely on the CI/CD pipeline to blow away and reprovision instances, and be 100% confident we can test and troubleshoot in dev and stage and by looking at off-instance logs from Prod. How important it is to get there is something I ponder about my motivations for - it's cleary not worthwhile if your project is one or 2 prod servers perhaps running something like HA WordPress, but it's obvious that at Netflix type scale that nobody is sshing into individual instances to troubleshoot. We are a long way (a long long long long way) from Netflix scale, and are unlikely to ever get there. But somewhere between dozens and hundreds of instances is about where I reckon the work required to get close to there stars paying off.
- imiric 2y ago
  Right. The answer is having systems that are resilient to failure, and if they do fail being able to quickly replace any node, hopefully automatically, along with solid observability to give you insight into what failed and how to fix it. The process of logging into a machine to troubleshoot it in real-time while the system is on fire is so antiquated, not to mention stressful. On-call shouldn't really be a major part of our industry. Systems should be self-healing, and troubleshooting done during working hours. Achieving this is difficult, but we have the tools to do it. The hurdles are often organizational rather than technical.
  - bigiain 2y ago
    > The hurdles are often organizational rather than technical. Yeah. And in my opinion "organizational" reasons can (and should) include "we are just not at the scale where achieving that makes sense". If you have single digit numbers of machines, the whole solid observability/ automated node replacement/self-healing setup overhead is unlikely to pay off. Especially if the SLAs don't require 2am weekend hair-on-fire platform recovery. For a _lot_ things, you can almost completely avoid on-call incidents with straightforward redundant (over provisioned) HA architectures, no single points of failure, and sensible office hours only deployment rules (and never _ever_ deploy to Prod on a Friday afternoon). Scrappy startups, and web/mobile platforms for anything where a few hours of downtime is not going to be an existential threat to the money flow or a big story in the tech press - probably have more important things to be doing than setting up log aggregation and request tracing. Work towards that, sure, but probably prioritise the dev productivity parts first. Get your CI/CD pipeline rock solid. Get some decent monitoring of the redundant components of your HA setup (as well as the Prod load balancer monitoring) so you know when you're degraded but not down (giving you some breathing space to troubleshoot). And aspire to fully resilient systems and have a plan for what they might look like in the future to avoid painting yourself into a corner that makes it harder then necessary to get there one day. But if you've got a guy spending 6 months setting up chaos monkey and chaos doctor for your WordPress site that's only getting a few thousand visits a day, you're definitely going it wrong. Five nines are expensive. If your users are gonna be "happy enough" with three nines or even two nines, you've probably got way better things to do with that budget.
LtWorf 2y ago
He asks the senior developer to do it.

In my world, if a developer needs access to the Node upon which their app is deployed to troubleshoot, that's 100% a bug in their application. I am cognizant that being whole-hog on 12 Factor apps is a journey, but for my money get on the train because "let me just ssh in and edit this one config file" is the road to ruin when no one knows who edited what to set it to what new value. Running $(kubectl edit) allows $(kubectl rollout undo) to put it back, and also shows what was changed from what to what

yjftsjthsd-h 2y ago
How do you debug the worker itself?
- mdaniel 2y ago
  Separate from my sibling comment about AWS SSM, I also believe that if one cannot know that a Node is sick by the metrics or log egress from it, that's a deployment bug. I'm firmly in the "Cattle" camp, and am getting closer and closer to the "Reverse Uptime" camp - made easier by ASG's newfound "Instance Lifespan" setting to make it basically one-click to get onboard that train Even as I type all these answers out, I'm super cognizant that there's not one hammer for all nails, and I am for sure guilty of yanking Nodes out of the ASG in order to figure out what the hell has gone wrong with them, but I try very very hard not to place my Nodes in a precarious situation to begin with so that such extreme troubleshooting becomes a minor severity incident and not Situation Normal
  - __turbobrew__ 2y ago
    If accidentally nuking a single node while debugging causes issues you have bigger problems. Especially if you are running kubernetes any node should be able to fall off the earth at any time without issues. I agree that you should set a maximum lifetime for a node on the order of a few weeks. I also agree that you shouldn’t be giving randos access to production infra, but and the end of the day there needs to be some people at the company who have the keys to the kingdom because you don’t know what you don’t know and you need to be able to deal with unexpected faults or outages of the telemetry and logging systems. I once bootstrapped an entire datacenter with tens of thousands of nodes from an SSH terminal after an abrupt power failure. It turns out infrastructure has lots of circular dependencies and we had to manually break that dependency.
    - ramzyo 2y ago
      Exactly this. Have heard it referred to as "break glass access". Some form of remote access, be it SSH or otherwise, in case of serious emergency.
  - viraptor 2y ago
    Passive metrics/logs won't let you debug all the issues. At some point you either need a system for automatic memory dumps and submitting bpf scripts to live nodes... or you need SSH access to do that.
    - otabdeveloper4 2y ago
      This "system for automatic dumps" 100 percent uses ssh under the hood. Probably with some eternal sudo administrator key. Personal ssh access is always better (from a security standpoint) than bot tokens and keys.
      - viraptor 2y ago
        There's a thousand ways to do it without SSH. It can be built into the app itself. It can be a special authenticated route to a suid script. It can be built into the current orchestration system. It can be pull-based using the a queue for system monitoring commands. It can be part of the existing monitoring agent. It can be run through AWS SSM. There's really no reason it has to be SSH. And even got SSH you can have special keys with access authorised to only specific commands, so a service account would be better than personal in that case.
  - acdha 2y ago
    > Separate from my sibling comment about AWS SSM, This seems like it’s conceding the point since SSM also allows you to run commands on nodes - I use it interchangeably with SSH to have Ansible manage legacy servers. Maybe what you’re trying to say is that it shouldn’t be routine and that there should be more of a review process so it’s not just a random unrestricted shell session? I think that’s less controversial, and especially when combined with some kind of “taint” mode where your access to a server triggers a rebuild after the dust has settled.
    - mdaniel 2y ago
      Yes, you nailed it with "it shouldn't be routine" and there for sure should be a review process. My primary concern with the audit logs actually isn't security it's lowering the cowboy of the software lifecycle > combined with some kind of “taint” mode where your access to a server triggers a rebuild after the dust has settled. Oh, I love that idea: thanks for bringing it to my attention. I'll for sure incorporate that into my process going forward
      - acdha 2y ago
        The first time I heard it was a very simple idea: they had a wrapper for the command which installed SSH keys on an EC2 instance which also set a delete-after tag which CloudCustodian queried.
- from-nibly 2y ago
  You don't. you shoot it in the head and get a new one. If you need logging / telemetry bake it into the image.
  - otabdeveloper4 2y ago
    Are you from techsupport? Actually not every problem is solved with the "have you tried turning it off and back on again" trick.
    - mdaniel 2y ago
      No, what we're talking about is (to extend your very condescending tech support analogy) shipping the customer a new PC from the factory, and telling them to throw the old one away because it doesn't matter. It will only start to matter if they have 3 bad PCs in a row, at which time it becomes (a) a demonstrable failure and not just stray neutron rays (b) an incident which will carry a postmortum of how the organization could have prevented that failure for next time I did start the whole thread by saying "and then I grew up," and not everyone is at the same place in their organizational maturity model. So, if you're happy with the process you have now, keep using it. I was unhappy, so I studied hard, incorporated supporting technology, and lobbied my heart out for change. Without maturity levels we'd all still be using telnet and tar based version control
      - otabdeveloper4 2y ago
        Once you reach the next maturity level you realize that some problems and bugs are from bad design, and cannot be fixed by restarting the server. Fixing bad design is an art, not an organizational discipline. (Sadly.)
megous 2y ago
Your world is very narrow and limited. Some devs also have to deal with customer provisioned HW infrastructure, with buggy interactions between HW/virtualization solutions that every 5 minutes duplicate all packets for a few seconds; with applications that interact with customer only onsite HW you only have remote access to via production deployment; with quirky virtualization like vmware stopping the vCPU on you for hundreds of ms if you load it too much which you'll not replicate locally; with things you can't predict you'll need to observe ahead of time, etc. And it does not involve editing any configs. It's just troubleshooting.

bigiain 2y ago

I think ssh-ing into production is a sign of not fully mature devops practices. We are still stuck there, but we're striving to get to the place where we can turn off sshd on Prod and rely on the CI/CD pipeline to blow away and reprovision instances, and be 100% confident we can test and troubleshoot in dev and stage and by looking at off-instance logs from Prod. How important it is to get there is something I ponder about my motivations for - it's cleary not worthwhile if your project is one or 2 prod servers perhaps running something like HA WordPress, but it's obvious that at Netflix type scale that nobody is sshing into individual instances to troubleshoot. We are a long way (a long long long long way) from Netflix scale, and are unlikely to ever get there. But somewhere between dozens and hundreds of instances is about where I reckon the work required to get close to there stars paying off.

imiric 2y ago
Right. The answer is having systems that are resilient to failure, and if they do fail being able to quickly replace any node, hopefully automatically, along with solid observability to give you insight into what failed and how to fix it. The process of logging into a machine to troubleshoot it in real-time while the system is on fire is so antiquated, not to mention stressful. On-call shouldn't really be a major part of our industry. Systems should be self-healing, and troubleshooting done during working hours. Achieving this is difficult, but we have the tools to do it. The hurdles are often organizational rather than technical.
- bigiain 2y ago
  > The hurdles are often organizational rather than technical. Yeah. And in my opinion "organizational" reasons can (and should) include "we are just not at the scale where achieving that makes sense". If you have single digit numbers of machines, the whole solid observability/ automated node replacement/self-healing setup overhead is unlikely to pay off. Especially if the SLAs don't require 2am weekend hair-on-fire platform recovery. For a _lot_ things, you can almost completely avoid on-call incidents with straightforward redundant (over provisioned) HA architectures, no single points of failure, and sensible office hours only deployment rules (and never _ever_ deploy to Prod on a Friday afternoon). Scrappy startups, and web/mobile platforms for anything where a few hours of downtime is not going to be an existential threat to the money flow or a big story in the tech press - probably have more important things to be doing than setting up log aggregation and request tracing. Work towards that, sure, but probably prioritise the dev productivity parts first. Get your CI/CD pipeline rock solid. Get some decent monitoring of the redundant components of your HA setup (as well as the Prod load balancer monitoring) so you know when you're degraded but not down (giving you some breathing space to troubleshoot). And aspire to fully resilient systems and have a plan for what they might look like in the future to avoid painting yourself into a corner that makes it harder then necessary to get there one day. But if you've got a guy spending 6 months setting up chaos monkey and chaos doctor for your WordPress site that's only getting a few thousand visits a day, you're definitely going it wrong. Five nines are expensive. If your users are gonna be "happy enough" with three nines or even two nines, you've probably got way better things to do with that budget.

LtWorf 2y ago

He asks the senior developer to do it.