3 ms·

Ran into this the other day messing around with a Consul/Nomad cluster on FreeBSD on AWS. I kept having one of the nodes dropping out because it had missed sen

Ran into this the other day messing around with a Consul/Nomad cluster on FreeBSD on AWS. I kept having one of the nodes dropping out because it had missed sending its heartbeats, even though it was under next to no load and otherwise healthy. It was a head scratcher for a bit, but once I ended up figuring out what was going on it was a super obvious issue. Running `date; sleep 1;` in a loop, both were printing numbers increasing one second at a time, but one node was printing twice as fast as the other! Of course there were timing issues when two nodes couldn’t agree on something as fundamental as how long a second was. (As an aside, thank you for all your work on FreeBSD! If I ever run into you around town I owe you a beer!)

cperciva 5y ago
You might be the second person to run into this. Just to confirm, you were using T3 instances? (BTW I have a Patreon for anyone who wants to contribute to my FreeBSD/EC2 work or can convince their employers to: https://www.patreon.com/cperciva https://www.patreon.com/cperciva )
- nucleardog 5y ago
  Well don't I feel special now! Yes, t3 instances. I wasn't seeing this behaviour on _all_ of the instances. Deploying and redeploying a three node cluster a few times trying to track down what was going on never left me in a stable state though so it certainly wasn't rare. This was in us-west-2. I didn't notice if the issue correlated with the AZ or anything else, as until I figured out what was going on I naturally assumed it was an issue on my end and once I found the issue I just patched over it with `kern.timecounter.hardware=ACPI-fast` and moved on since this was just some play projects. Haven't used FreeBSD at work, but I've been using it at home and for personal hosting on AWS for the past 20 odd years. Just pledged a bit on Patreon. Thanks for all the work you put in!
  - cperciva 5y ago
    Yeah it seems random whether a t3 instance is affected or not -- rather annoyingly, when I was trying to debug this none of my instances ever ended up being affected. Using ACPI-fast is the correct workaround on 13.0, at least until the underlaying EC2 issue gets fixed. On 14.0 we have kvmclock which works fine even on instances with broken TSCs, and that's much faster than ACPI-fast -- but still slightly slower than TSC.

cperciva 5y ago

You might be the second person to run into this. Just to confirm, you were using T3 instances? (BTW I have a Patreon for anyone who wants to contribute to my FreeBSD/EC2 work or can convince their employers to: https://www.patreon.com/cperciva https://www.patreon.com/cperciva )

nucleardog 5y ago
Well don't I feel special now! Yes, t3 instances. I wasn't seeing this behaviour on _all_ of the instances. Deploying and redeploying a three node cluster a few times trying to track down what was going on never left me in a stable state though so it certainly wasn't rare. This was in us-west-2. I didn't notice if the issue correlated with the AZ or anything else, as until I figured out what was going on I naturally assumed it was an issue on my end and once I found the issue I just patched over it with `kern.timecounter.hardware=ACPI-fast` and moved on since this was just some play projects. Haven't used FreeBSD at work, but I've been using it at home and for personal hosting on AWS for the past 20 odd years. Just pledged a bit on Patreon. Thanks for all the work you put in!
- cperciva 5y ago
  Yeah it seems random whether a t3 instance is affected or not -- rather annoyingly, when I was trying to debug this none of my instances ever ended up being affected. Using ACPI-fast is the correct workaround on 13.0, at least until the underlaying EC2 issue gets fixed. On 14.0 we have kvmclock which works fine even on instances with broken TSCs, and that's much faster than ACPI-fast -- but still slightly slower than TSC.