4 ms·

that’s a pretty good and detailed explanation. 2 things: 1) i hope they have a runbook for situations like this (ie the support engineer does not have to figure

that’s a pretty good and detailed explanation. 2 things: 1) i hope they have a runbook for situations like this (ie the support engineer does not have to figure all this on the fly) 2) the customer should have provides more details and maybe should have thought of the tweaks they made (classic solution is to compare 2 instances - one works one does not)

donalhunt 6y ago
You can't runbook these sort of issues... You can ensure you have the knowledge and tools to root cause the problem.
- rantwasp 6y ago
  you most definitely can and you should. there are steps that you can do to gather the info and the linked example shows basic things to try. when you exhaust the run-book is when you start digging
  - donalhunt 6y ago
    I agree with the point around information gathering. Good technical teams have guides for endusers to collect relevant information. In this case the engineering team may have shared information with the TSCs to help narrow down the root cause. This also highlights why support agents shouldn't be 100 engaged with customers. They need time to review and amend runbooks, consult with engineering teams, etc.
amessina1 6y ago
1) you cannot have a runbook for everything, and even if you have a runbook you the best you could have found in this case is that something weird was happening in the VM. The setting had an insanely big value but it was accepted by the kernel, so you would assume it was a valid one. 2) the customer provided a huge amount of details, but it is usually very hard to explain what did you change from the base image. Most customers might not be willing to provide the full spec of their running system, as they might contain information they don't want to disclose. It is easier when the issue starts right after a change has been made, but this was not the case.
- rantwasp 6y ago
  i’m not saying a runbook will catch everything. but it will give you a chance to solve the problem quicker.
  - amessina1 6y ago
    I totally agree, but keep in mind that it is an iterative process: you have a case, you apply the runbook/playbook you have, if they are not enough you use your skill/knowledge to solve the case, then you update the playbooks.
    - rantwasp 6y ago
      iteration is my middle name :) the point I was trying to make is you should be prepared - if only for the 95% of the cases that the runbook can solve. you also don't start with a complete runbook - you build it as you operate the service.
fenesiistvan 6y ago
I don't think that this was a "support engineer". He was a god mode developer!

donalhunt 6y ago

You can't runbook these sort of issues... You can ensure you have the knowledge and tools to root cause the problem.

rantwasp 6y ago
you most definitely can and you should. there are steps that you can do to gather the info and the linked example shows basic things to try. when you exhaust the run-book is when you start digging
- donalhunt 6y ago
  I agree with the point around information gathering. Good technical teams have guides for endusers to collect relevant information. In this case the engineering team may have shared information with the TSCs to help narrow down the root cause. This also highlights why support agents shouldn't be 100 engaged with customers. They need time to review and amend runbooks, consult with engineering teams, etc.

amessina1 6y ago

1) you cannot have a runbook for everything, and even if you have a runbook you the best you could have found in this case is that something weird was happening in the VM. The setting had an insanely big value but it was accepted by the kernel, so you would assume it was a valid one. 2) the customer provided a huge amount of details, but it is usually very hard to explain what did you change from the base image. Most customers might not be willing to provide the full spec of their running system, as they might contain information they don't want to disclose. It is easier when the issue starts right after a change has been made, but this was not the case.

rantwasp 6y ago
i’m not saying a runbook will catch everything. but it will give you a chance to solve the problem quicker.
- amessina1 6y ago
  I totally agree, but keep in mind that it is an iterative process: you have a case, you apply the runbook/playbook you have, if they are not enough you use your skill/knowledge to solve the case, then you update the playbooks.
  - rantwasp 6y ago
    iteration is my middle name :) the point I was trying to make is you should be prepared - if only for the 95% of the cases that the runbook can solve. you also don't start with a complete runbook - you build it as you operate the service.

fenesiistvan 6y ago

I don't think that this was a "support engineer". He was a god mode developer!