2 ms·

Have you tried turning it off and then on again? > We're no longer pursuing restarts as a path to remediation. Oh you have

cube00 10d ago
Kind of surprised they admit they're going to try restarting and see what happens. I'm sure it happens everywhere but nobody admits it. > We've attempted a rolling restart on one of the impacted instances to see if that resolves the issue. At least it didn't fix the problem so they can actually start finding the real cause. > We're no longer pursuing restarts as a path to remediation. Why isn't the AI they sell telling them what's wrong? Why do they need to take shots in the dark to "see if that resolves the issue"?
- deleted 10d ago
  [deleted]
- andrewinardeer 10d ago
  "Yeah, I'm with Rob. Just let's reboot and see what happens"
  - chihuahua 10d ago
    "If that doesn't work, clear the cache and reboot again."
    - cube00 10d ago
      [dead]
- jakevoytko 10d ago
  In my experience it’s a safe way to do something useful while everyone is getting their bearings. It immediately partitions the situation space between being persisted or systemic vs local or caused by long-running processes. Plus everyone’s going to ask if you’ve tried that already, so you might as well get it out of the way if it makes any amount of sense
  - vrosas 10d ago
    Followed quickly by "Redeploying with more log lines", the next logical step.
- swatcoder 10d ago
  I don't know, that reads exactly like an AI troubleshooter working through a plan without the implicit contextual understanding an experienced human might bring to either the actions or the communications. "Oops, we forgot to tell it that this is the hyperscaled Salesforce production environment and that its choices need to project competence and consider brand embarrassment. WILLFIX"
- b112 10d ago
  Restart should be a very last emergency step, as if it works, a restart often might wipe out evidence of why. So hopefully it's not done often.
  - CoffeeOnWrite 10d ago
    I wouldn't say so, rather you need to balance recovery time and evidence preservation. A good incident manager will give the service owning team a chance or two to debug, but not let them fall into the trap of needing to understand the problem fully before attempt a clumsy potential fix. And of course will take into account the total business impact of the ongoing disruption and the known and unknown risks of the proposed clumsy fix (it could make things worse).
    - b112 10d ago
      Yes, that's why it's the last emergency step. We're not disagreeing.
  - ocdtrekkie 10d ago
    Rebooting is the first step: If it fixes it you don't have a problem. If it doesn't, you know more about the problem. It's a joke, but like, after nearly two decades of engineering I have something break on me, due to updates. I figure the updates broke it, call the vendor, and they go... did you try rebooting it again? Rebooting it a second time fixed it.
  - wookmaster 9d ago
    The requirement is to get customers out of impact as #1 priority. If there's suspicions around memory/thread states a restart makes a lot of sense. Digging through logs and flight records takes a lot of time, customers are losing business in that time. If you're afraid to restart your service you need to work on your telemetry.
- _joel 10d ago
  Been many an MIR that I've seen where after initial assessment, the next log entry was "service restart attempted"
- SoftTalker 10d ago
  In my experience this is very common on linux hosts. Not that you would do a full system reboot as a generic first attempt (this was much more common when I worked with Windows) but restarting a wedged or misbehaving service/daemon is a pretty common thing.
reaperducer 10d ago
Have you tried turning it off and then on again? Flashbacks to "Thank you for calling Three-Ten-DELL. Have you tried turning it of and turning it back on again?"

cube00 10d ago

Kind of surprised they admit they're going to try restarting and see what happens. I'm sure it happens everywhere but nobody admits it. > We've attempted a rolling restart on one of the impacted instances to see if that resolves the issue. At least it didn't fix the problem so they can actually start finding the real cause. > We're no longer pursuing restarts as a path to remediation. Why isn't the AI they sell telling them what's wrong? Why do they need to take shots in the dark to "see if that resolves the issue"?

deleted 10d ago
[deleted]
andrewinardeer 10d ago
"Yeah, I'm with Rob. Just let's reboot and see what happens"
- chihuahua 10d ago
  "If that doesn't work, clear the cache and reboot again."
  - cube00 10d ago
    [dead]
jakevoytko 10d ago
In my experience it’s a safe way to do something useful while everyone is getting their bearings. It immediately partitions the situation space between being persisted or systemic vs local or caused by long-running processes. Plus everyone’s going to ask if you’ve tried that already, so you might as well get it out of the way if it makes any amount of sense
- vrosas 10d ago
  Followed quickly by "Redeploying with more log lines", the next logical step.
swatcoder 10d ago
I don't know, that reads exactly like an AI troubleshooter working through a plan without the implicit contextual understanding an experienced human might bring to either the actions or the communications. "Oops, we forgot to tell it that this is the hyperscaled Salesforce production environment and that its choices need to project competence and consider brand embarrassment. WILLFIX"
b112 10d ago
Restart should be a very last emergency step, as if it works, a restart often might wipe out evidence of why. So hopefully it's not done often.
- CoffeeOnWrite 10d ago
  I wouldn't say so, rather you need to balance recovery time and evidence preservation. A good incident manager will give the service owning team a chance or two to debug, but not let them fall into the trap of needing to understand the problem fully before attempt a clumsy potential fix. And of course will take into account the total business impact of the ongoing disruption and the known and unknown risks of the proposed clumsy fix (it could make things worse).
  - b112 10d ago
    Yes, that's why it's the last emergency step. We're not disagreeing.
- ocdtrekkie 10d ago
  Rebooting is the first step: If it fixes it you don't have a problem. If it doesn't, you know more about the problem. It's a joke, but like, after nearly two decades of engineering I have something break on me, due to updates. I figure the updates broke it, call the vendor, and they go... did you try rebooting it again? Rebooting it a second time fixed it.
- wookmaster 9d ago
  The requirement is to get customers out of impact as #1 priority. If there's suspicions around memory/thread states a restart makes a lot of sense. Digging through logs and flight records takes a lot of time, customers are losing business in that time. If you're afraid to restart your service you need to work on your telemetry.
_joel 10d ago
Been many an MIR that I've seen where after initial assessment, the next log entry was "service restart attempted"
SoftTalker 10d ago
In my experience this is very common on linux hosts. Not that you would do a full system reboot as a generic first attempt (this was much more common when I worked with Windows) but restarting a wedged or misbehaving service/daemon is a pretty common thing.

reaperducer 10d ago

Have you tried turning it off and then on again? Flashbacks to "Thank you for calling Three-Ten-DELL. Have you tried turning it of and turning it back on again?"