5 ms·

This is a handy list. > 4:07pm The package install has failed as it can't resolve the repositories. Something is wrong with the /etc/apt configuration… Cloud

This is a handy list. > 4:07pm The package install has failed as it can't resolve the repositories. Something is wrong with the /etc/apt configuration… Cloud definitely has downsides, and isn’t a fit for all scenarios but in my experience it’s great for situations like this. Instead of messing around trying to repair it, simply kill the machine, or take it out of the pool. Get a new one. New machine and app likely comes up clean. Incident resolves. Dig into machine off the hot path.

throw5323446 3y ago
> Instead of messing around trying to repair it, simply kill the machine, or take it out of the pool. Get a new one. "4:10pm the new machine still has the same performance issue"
- FridgeSeal 3y ago
  Sure, but more often than not - esp in cloud scenarios, sometimes you just get a machine that is having a bad day and it’s quicker to just eject it, let the rest of the infra pick up the slack, and then debug from there. Additionally if you’ve axed a machine, and got the same issue, you know it’s not a machine issue, so either go look at your networking layer or whatever configs you’re using to boot your machines from…
  - tjoff 3y ago
    > esp in cloud scenarios ... so the nice thing about the about the cloud is that you can workaround cloud-specific issues?
  - deleted 3y ago
    [deleted]
- jandrese 3y ago
  4:20pm Turns out it was DNS
  - Propelloni 3y ago
    That made me laugh. Thank you. Of course, it is not DNS. DNS has become the new cabling. DNS is not especially complicated, but cabling is neither. Yet, during dot.com and subsequent years the cabling was causing a lot of the problems so that we get used to first check the cabling. But it only took a few more years to realize that it is not always cabling, actually failures are normally distributed. Is it wrong to check DNS first? No, but please realize that DNS misconfiguration is not more common than other SNAFUS.
    - ninkendo 3y ago
      It’s not DNS There’s no way it’s DNS It was DNS
      - smackeyacky 3y ago
        Certificates are the new DNS for service breakages
- SerCe 3y ago
  That's actually amazing, a reproducible problem is a 90% solved problem!
Jedd 3y ago
You're describing one of the benefits of virtualised cattle, not necessarily or exclusively 'cloud'.
userbinator 3y ago
Dig into machine off the hot path. Unfortunately, no one has the time to do that (or let someone do it) after the problem is "solved", so over time the "rebuild from scratch" approach just results in a loss of actual troubleshooting skills and acquired knowledge --- the software equivalent of a "parts swapper" in the physical world.
- patrick451 3y ago
  The end state of a culture that embraces restart/reboot/clear-cache instead of real diagnoses and troubleshooting is a cohort of junior devs who just delete their git repo and reclone instead of figuring out what a detached HEAD is. I don't really fault the junior dev who does that. They are just following the "I don't understand something, so just start over" paradigm set by seniors.
  - yosefk 3y ago
    To be fair, with git, specifically, it's a good idea to at least clone for backup before things like major merges. There are lots of horror stories from people losing work to git workflow issues and I'd rather be ridiculed as an idiot who is afraid of "his tools" (as if I have anything like a choice when using git) and won't learn them properly than lose work thanks to a belief that this thing behaves in a way which can actually be learned and followed safely. A special case of this is git rebase after which you "can" access the original history in some obscure way until it's garbage-collected; or you could clone the repo before the merge and then you can access the original history straightforwardly and you decide when to garbage-collect it by deleting that repo.
    - theptip 3y ago
      Git is a lot less scary when you understand the reflog; commit or stash your local changes and then you can rebase without fear of losing anything. (As a bonus tip, place “mybranch.bak” branches as pointers to your pre-rebase commit sha to avoid having to dig around in the reflog at all.) I would never ridicule anyone for your approach, just gently encourage them to spend a few mins to grok the ‘git reflog’ command.
KingOfCoders 3y ago
Kill the machine might destroy evidence. It might be the case you have everything logged outside, but most often there is something missing.
- monkpit 3y ago
  Take it out of the pool then.

throw5323446 3y ago

> Instead of messing around trying to repair it, simply kill the machine, or take it out of the pool. Get a new one. "4:10pm the new machine still has the same performance issue"

FridgeSeal 3y ago
Sure, but more often than not - esp in cloud scenarios, sometimes you just get a machine that is having a bad day and it’s quicker to just eject it, let the rest of the infra pick up the slack, and then debug from there. Additionally if you’ve axed a machine, and got the same issue, you know it’s not a machine issue, so either go look at your networking layer or whatever configs you’re using to boot your machines from…
- tjoff 3y ago
  > esp in cloud scenarios ... so the nice thing about the about the cloud is that you can workaround cloud-specific issues?
- deleted 3y ago
  [deleted]
jandrese 3y ago
4:20pm Turns out it was DNS
- Propelloni 3y ago
  That made me laugh. Thank you. Of course, it is not DNS. DNS has become the new cabling. DNS is not especially complicated, but cabling is neither. Yet, during dot.com and subsequent years the cabling was causing a lot of the problems so that we get used to first check the cabling. But it only took a few more years to realize that it is not always cabling, actually failures are normally distributed. Is it wrong to check DNS first? No, but please realize that DNS misconfiguration is not more common than other SNAFUS.
  - ninkendo 3y ago
    It’s not DNS There’s no way it’s DNS It was DNS
    - smackeyacky 3y ago
      Certificates are the new DNS for service breakages
SerCe 3y ago
That's actually amazing, a reproducible problem is a 90% solved problem!

Jedd 3y ago

You're describing one of the benefits of virtualised cattle, not necessarily or exclusively 'cloud'.

userbinator 3y ago

Dig into machine off the hot path. Unfortunately, no one has the time to do that (or let someone do it) after the problem is "solved", so over time the "rebuild from scratch" approach just results in a loss of actual troubleshooting skills and acquired knowledge --- the software equivalent of a "parts swapper" in the physical world.

patrick451 3y ago
The end state of a culture that embraces restart/reboot/clear-cache instead of real diagnoses and troubleshooting is a cohort of junior devs who just delete their git repo and reclone instead of figuring out what a detached HEAD is. I don't really fault the junior dev who does that. They are just following the "I don't understand something, so just start over" paradigm set by seniors.
- yosefk 3y ago
  To be fair, with git, specifically, it's a good idea to at least clone for backup before things like major merges. There are lots of horror stories from people losing work to git workflow issues and I'd rather be ridiculed as an idiot who is afraid of "his tools" (as if I have anything like a choice when using git) and won't learn them properly than lose work thanks to a belief that this thing behaves in a way which can actually be learned and followed safely. A special case of this is git rebase after which you "can" access the original history in some obscure way until it's garbage-collected; or you could clone the repo before the merge and then you can access the original history straightforwardly and you decide when to garbage-collect it by deleting that repo.
  - theptip 3y ago
    Git is a lot less scary when you understand the reflog; commit or stash your local changes and then you can rebase without fear of losing anything. (As a bonus tip, place “mybranch.bak” branches as pointers to your pre-rebase commit sha to avoid having to dig around in the reflog at all.) I would never ridicule anyone for your approach, just gently encourage them to spend a few mins to grok the ‘git reflog’ command.

KingOfCoders 3y ago

Kill the machine might destroy evidence. It might be the case you have everything logged outside, but most often there is something missing.

monkpit 3y ago
Take it out of the pool then.