4 ms·
> It's not even sure you could diagnose the problem in the first place, let alone deploy a workaround. It's easy to detect: ntpdate -q will tell you the drift,
by acdha 5y ago
> It's not even sure you could diagnose the problem in the first place, let alone deploy a workaround.
It's easy to detect: ntpdate -q will tell you the drift, your logging would tell you when ntpd gave up because skew was too large.
Correction would depend on why it was happening: you might be able to tell ntpd/chrony to adjust more frequently or to accept larger deltas, but at that point I'd also say that the best path would be pulling that instance from service and replacing it so it's not critical.
> You'd have to send dozens of email to tech support who of course would say there is no problem because their machines have 999999999999999999% uptime and nothing could be wrong on their side, but hey they can sell you advanced support/engineering if you've got way too much money to help you find the problem in your code.
Has that been your experience? Based on mine it would be more like “we confirmed the problem you showed and have contacted the service team” followed by an apologetic call from your TAM. I've opened plenty of issues over the years but have never had someone insist that something is not a problem because their systems are perfect — they'd basically have to be saying you faked the evidence in their report.
- southerntofu 5y ago> ntpdate -q will tell you the drift, your logging (...) Sure, that's if you got root on the machine. I was not talking about VPS hosting which we've been doing for a long time, but rather so-called "serverless" services. > Has that been your experience? That's been my experience with most managed hosting i've had over the years, though i've had some good experiences too. I've never dealt with Amazon but i'm assuming since you can't run diagnostics (since you have no shell) and like any other business it's likely their first level of customer is inexperienced and reads from a script, i'm guessing you're gonna have a bad time if you encounter weird/silent errors from your cloud services. Some hardware errors are hard-enough to diagnose with root and a helpful customer support, i can't imagine without those.
- acdha 5y agoWhile it's true that you can't just shell into a managed host, you can run diagnostics in many cases[1]. I would also say that it's _far_ less common for those services to have hardware issues — part of what they're doing is optimizing to things like health-checks and rebuilds easy since the places with durable state are very clearly identified and isolated. I've never seen a hardware problem which the platform didn't detect a fault first in those environments (e.g. you see some requests have latency spike for a minute prior to a new instance coming online — it doesn't say “underlying hardware failure” but clearly something changed on the host or perhaps a neighbor got very noisy) and that includes things like the various Intel exploits where you could see everything rotate a week before the public advisory went out. I will say that I've had a few poor run-arounds with first-level support but I've never had them refuse to escalate or switch to a different tech if you say that first response wasn't satisfactory. 1. e.g. on AWS “Fargate” refers to the managed Firecracker VMs, as opposed to the traditional ECS/EKS versions using servers which you can access, but you can use https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/... to run a command inside the container environment.