4 ms·
> When you check the elapsed time, you subtract the old time from the new time. But if the new time is less than the old time, you get an overflow. Because we
by wahern 5y ago
> When you check the elapsed time, you subtract the old time from the new time. But if the new time is less than the old time, you get an overflow. Because we save time as an unsigned integer, you get a very large number that will for sure be larger than any timeout value you set, including our default 3 minutes.
If they were using unsigned arithmetic then they would've gotten the expected value. For example, 0 - 0xffffffffffffffff is 1 when using 64-bit types, just as one might expect. Unsigned (modulo) arithmetic works exactly how you'd want in a situation like this; it works until the interval exceeds the range of the type. So either they're describing things wrong, or their code wasn't doing what they thought.
Of course, in any event this wouldn't be the root of their problem--time stamp counter drift across cores. The Linux kernel has alot of code to decide whether to use the RDTSC-optimized clock_gettime vDSO implementation or fallback to a syscall. And when it does use the RDTSC method, IIRC it has code to train the time stamps to minimize any cross-core drift, and mask out any potential difference (reducing granularity). The moral of the story is to just use clock_gettime+CLOCK_MONOTONIC; at least on Linux it'll use RDTSC in most cases where that's practical.
- antonio2368 5y agoSorry if something was explained awkwardly or incorrectly, I'm trying to get a hang of writing posts like this, technical posts that are interesting and fun to read. > If they were using unsigned arithmetic then they would've gotten the expected value. And we did, but we did not expect to get the input values we got, which are explained by your last paragraph: "time stamp counter drift across cores." This was the real problem, and something new we learned. > use clock_gettime+CLOCK_MONOTONIC I forgot to include this because I was focused on the solutions we ended up using, but after some research this is 100% correct! We were afraid that too many syscalls during the query execution can produce huge performance hits so we decided to look for a different solution. All in all, thanks for the comment!
- namibj 5y agoJust to correct you: `clock_gettime` is handled by the vDSO, and does not involve an actual syscall for over a decade now.
- antonio2368 5y agoI didn't think about the vDSO, and that it's used for `clock_gettime`, so thank you for the information and correction! It's really hard for me to get a feel of what is okay to use, what is the standard way to do something without actually implementing everything and seeing how it behaves, so information like this help me a lot.
- namibj 5y agoI recommend looking at it, as all those calls are very cheap: man 7 vdso