5 ms·

If only there was a simple way to exponentially limit your retries and back off a bit when the other side is constantly failing.

wincy 3y ago
You mean the concept I learned week three as a junior developer (thanks to great mentorship)? Yeah, probably absolutely impossible.
- hirsin 3y ago
  You're lucky. I've seen entire engineering orgs convinced of the opposite, and highly resistant to improvement.
clnq 3y ago
Exponential back-off is fancy. There are a million repeating behaviors in a large automotive codebase. Maybe it's too arduous to code up exponential back-off for all. Even if you have a simple library to do it, it could introduce some complexity. But it's quite easy to set timers to attempt things at regular intervals, rather than in an infinite loop until they succeed. One attempt every 5 seconds (assuming a 5-second timeout for the request) is 17,280 attempts a day. One attempt every 5 minutes is 288 attempts a day. One attempt every 15 minutes is 96 attempts a day. A simple repeating timer is a fantastic solution. Does the car need to always send telemetry immediately? Can there be a 15-minute latency? If so, just put that on a timer and enjoy 99.4% power savings for offline cars, with basically the same quality of telemetry when they're online. But what if a car needs to send telemetry at a given moment? Just move up the next timer function call. But what if I want to get telemetry when the 3G signal is intermittent and briefly available? Just call the function once the 3G signal is available. Do this with a separate cool-down for this behavior. But what if I need telemetry every second? Just dump all telemetry events into a buffer between successful sends, slap a timestamp on those events, and you can now send data for the past X hours all at once. Even from the black spots. There are so many options (even less fancy than exp back-off), but this was probably built by a junior programmer without any support network. This feels like a very common lack-of-experience mistake that would be caught in code review if the reviewer cared.
- adhesive_wombat 3y ago
  > Exponential back-off is fancy It's literally just a left bit-shift? It's as easy as a linear back-off. The only thing less fancy is no back-off at all.
  - _trampeltier 3y ago
    Ethernet on coax cables did it already after a collision (late 70s, thats 50 years ago). Funny how basic things are today to much work, but animations everywhere are a must.
  - clnq 3y ago
    Perhaps you are right. Though it seems easier to reason about a system with a hundred timers running at a set cadence than a hundred exponential back-offs. Maybe I just have more experience with the former. I guess you could always cap off the exponential interval increases at some max interval, making it basically a repeating timer. Though there's still the question of why would you need this for telemetry. It doesn't feel like this type of "marketing" telemetry needs to be quite real time.
    - peheje 3y ago
      "I guess you could always cap off the exponential interval increases at some max interval, making it basically a repeating timer." No you MUST have a max interval cap. Exponentials and all, you'll soon be waiting days or months for a retry.
      - mmac_ 3y ago
        It's ok, if you wait long enough it'll go back down to a small number. Or maybe a negative number and then the car crashes? Please cap your numbers for both user experience reasons, and safety reasons. Yes, have dealt with this before. Nothing like doing a fault analysis and telling someone that it'll fix itself in a few months. A capped fibonacci sequence is a very common back off timer.
    - 3y ago
cyanydeez 3y ago
But I need to unit test each recursion else I won't have completed test coverage!

wincy 3y ago

You mean the concept I learned week three as a junior developer (thanks to great mentorship)? Yeah, probably absolutely impossible.

hirsin 3y ago
You're lucky. I've seen entire engineering orgs convinced of the opposite, and highly resistant to improvement.

clnq 3y ago

Exponential back-off is fancy. There are a million repeating behaviors in a large automotive codebase. Maybe it's too arduous to code up exponential back-off for all. Even if you have a simple library to do it, it could introduce some complexity. But it's quite easy to set timers to attempt things at regular intervals, rather than in an infinite loop until they succeed. One attempt every 5 seconds (assuming a 5-second timeout for the request) is 17,280 attempts a day. One attempt every 5 minutes is 288 attempts a day. One attempt every 15 minutes is 96 attempts a day. A simple repeating timer is a fantastic solution. Does the car need to always send telemetry immediately? Can there be a 15-minute latency? If so, just put that on a timer and enjoy 99.4% power savings for offline cars, with basically the same quality of telemetry when they're online. But what if a car needs to send telemetry at a given moment? Just move up the next timer function call. But what if I want to get telemetry when the 3G signal is intermittent and briefly available? Just call the function once the 3G signal is available. Do this with a separate cool-down for this behavior. But what if I need telemetry every second? Just dump all telemetry events into a buffer between successful sends, slap a timestamp on those events, and you can now send data for the past X hours all at once. Even from the black spots. There are so many options (even less fancy than exp back-off), but this was probably built by a junior programmer without any support network. This feels like a very common lack-of-experience mistake that would be caught in code review if the reviewer cared.

adhesive_wombat 3y ago
> Exponential back-off is fancy It's literally just a left bit-shift? It's as easy as a linear back-off. The only thing less fancy is no back-off at all.
- _trampeltier 3y ago
  Ethernet on coax cables did it already after a collision (late 70s, thats 50 years ago). Funny how basic things are today to much work, but animations everywhere are a must.
- clnq 3y ago
  Perhaps you are right. Though it seems easier to reason about a system with a hundred timers running at a set cadence than a hundred exponential back-offs. Maybe I just have more experience with the former. I guess you could always cap off the exponential interval increases at some max interval, making it basically a repeating timer. Though there's still the question of why would you need this for telemetry. It doesn't feel like this type of "marketing" telemetry needs to be quite real time.
  - peheje 3y ago
    "I guess you could always cap off the exponential interval increases at some max interval, making it basically a repeating timer." No you MUST have a max interval cap. Exponentials and all, you'll soon be waiting days or months for a retry.
    - mmac_ 3y ago
      It's ok, if you wait long enough it'll go back down to a small number. Or maybe a negative number and then the car crashes? Please cap your numbers for both user experience reasons, and safety reasons. Yes, have dealt with this before. Nothing like doing a fault analysis and telling someone that it'll fix itself in a few months. A capped fibonacci sequence is a very common back off timer.
  - 3y ago

cyanydeez 3y ago

But I need to unit test each recursion else I won't have completed test coverage!