5 ms·

Exponential back-off is fancy. There are a million repeating behaviors in a large automotive codebase. Maybe it's too arduous to code up exponential back-off fo

by clnq 3y ago

Exponential back-off is fancy. There are a million repeating behaviors in a large automotive codebase. Maybe it's too arduous to code up exponential back-off for all. Even if you have a simple library to do it, it could introduce some complexity. But it's quite easy to set timers to attempt things at regular intervals, rather than in an infinite loop until they succeed. One attempt every 5 seconds (assuming a 5-second timeout for the request) is 17,280 attempts a day. One attempt every 5 minutes is 288 attempts a day. One attempt every 15 minutes is 96 attempts a day. A simple repeating timer is a fantastic solution. Does the car need to always send telemetry immediately? Can there be a 15-minute latency? If so, just put that on a timer and enjoy 99.4% power savings for offline cars, with basically the same quality of telemetry when they're online. But what if a car needs to send telemetry at a given moment? Just move up the next timer function call. But what if I want to get telemetry when the 3G signal is intermittent and briefly available? Just call the function once the 3G signal is available. Do this with a separate cool-down for this behavior. But what if I need telemetry every second? Just dump all telemetry events into a buffer between successful sends, slap a timestamp on those events, and you can now send data for the past X hours all at once. Even from the black spots. There are so many options (even less fancy than exp back-off), but this was probably built by a junior programmer without any support network. This feels like a very common lack-of-experience mistake that would be caught in code review if the reviewer cared.

adhesive_wombat 3y ago
> Exponential back-off is fancy It's literally just a left bit-shift? It's as easy as a linear back-off. The only thing less fancy is no back-off at all.
- _trampeltier 3y ago
  Ethernet on coax cables did it already after a collision (late 70s, thats 50 years ago). Funny how basic things are today to much work, but animations everywhere are a must.
- clnq 3y ago
  Perhaps you are right. Though it seems easier to reason about a system with a hundred timers running at a set cadence than a hundred exponential back-offs. Maybe I just have more experience with the former. I guess you could always cap off the exponential interval increases at some max interval, making it basically a repeating timer. Though there's still the question of why would you need this for telemetry. It doesn't feel like this type of "marketing" telemetry needs to be quite real time.
  - peheje 3y ago
    "I guess you could always cap off the exponential interval increases at some max interval, making it basically a repeating timer." No you MUST have a max interval cap. Exponentials and all, you'll soon be waiting days or months for a retry.
    - mmac_ 3y ago
      It's ok, if you wait long enough it'll go back down to a small number. Or maybe a negative number and then the car crashes? Please cap your numbers for both user experience reasons, and safety reasons. Yes, have dealt with this before. Nothing like doing a fault analysis and telling someone that it'll fix itself in a few months. A capped fibonacci sequence is a very common back off timer.
  - dublin 3y ago
    Most (actually almost all) IoT systems are not well-thought-out in this regard. It requires a bit of paranoiac foresight. Designers far too often think it's really, really important that remote devices stay connected to the mothership, when in reality, most IoT devices would be far better off NEVER requiring a cloud connection, but leveraging one if/when available. *Rule One of IoT network design: NEVER assume that you even have a network * If at all possible, and to the maximum degree possible, an IoT device must work whether there's a network available or not.
- refulgentis 3y ago
  what's a left bit shift ;)
- Tempest1981 3y ago
  Also good to sprinkle in some randomness.
- sokoloff 3y ago
  It’s a left shift and a non-volatile* variable to store it. * at least across module sleep/wake
- crazygringo 3y ago
  No. It's setting an async timer in a module that might not support that at all. It's establishing a variable for that in a scope the programmer might not normally have access to. It's making decisions about how quickly to back off and is there an eventual minimum floor. It's making decisions about what events ought to reset the timer, such as starting the car. And it's about creating a series of tests to verify this behavior, which is awfully non-obvious when the behavior might take 10 minutes to verify and tests are supposed to complete in 5 min. It's fancy.
  - adhesive_wombat 3y ago
    If you cannot mock that controller state machine properly, you should not be writing software that goes in cars. At least the penalty is you just repeatedly cause every car of the model in the entire country to fail instead of crashing into a wall, but this kind of thing is why the T in IoT is for "shit".
    - crazygringo 3y ago
      I agree completely. And it just supports my point, which is that it's not "literally just a left bit-shift". It's quite a lot of work to do, actually. Which -- of course -- should be done. But there are commenters here acting like it's a single line of code.
      - adhesive_wombat 3y ago
        That's not work unique to exponential back-off. You'd have to do all that for any back-off, which is why I said it's more complex than no back-off. Exponential back off literally is about a line of code difference (change "x += a" to "x <<= 1" (ok, a different radix isn't so silly easy, but anything is better then no back-off). All the rest, maximums, wrap around protection, reset events, all needs thought in either case. Actually the pathological cases in a linear system are perhaps sneakier as they could creep up on you ("it won't overflow, right? That would take months!") But I think we all know what actually happened here was a rush to release and then a failure to follow up with "the update that actually does it right". The Jira ticket is probably still "open - should do, version: next" somewhere. And there's a "told you so" email chain linked in there somewhere.
nucleardog 3y ago
Not to argue the overall point (retrying on a tight loop is not a good solution), but many of the models support communication the other way—e.g., remotely starting the car, adjusting the HVAC, etc. So there is some justification for trying to keep a cell signal established or checking in more frequently than every 15 minutes or so. Someone attempting to remote start their car not having it actually respond for 15 minutes is likely to already be outside trying to leave and pissed off that they’re walking out to a cold car with iced up windows and thinking that the “remote start didn’t work”.
_xerces_ 3y ago
When I was involved in certifying new devices to get on a carrier network, the carrier insisted on exponential backoff as well as some randomness so you wouldn't get millions of voic..devices crying out at the same time when power was lost or restored.
- dublin 3y ago
  It's not like this is a new problem, and it's been solved for decades - it's just good network design. One example: This is exactly why randomness is required by Ethernet's ARP protocol: to avoid ARP storms when the power comes back on and all nodes try to get back on the network at the same time. (And ARP is still powering most IP connections (v4) today, since IPv6 is painfully fiddly by design, so no one uses v6 unless they have to...)

by clnq 3y ago

adhesive_wombat 3y ago

> Exponential back-off is fancy It's literally just a left bit-shift? It's as easy as a linear back-off. The only thing less fancy is no back-off at all.

_trampeltier 3y ago
Ethernet on coax cables did it already after a collision (late 70s, thats 50 years ago). Funny how basic things are today to much work, but animations everywhere are a must.
clnq 3y ago
Perhaps you are right. Though it seems easier to reason about a system with a hundred timers running at a set cadence than a hundred exponential back-offs. Maybe I just have more experience with the former. I guess you could always cap off the exponential interval increases at some max interval, making it basically a repeating timer. Though there's still the question of why would you need this for telemetry. It doesn't feel like this type of "marketing" telemetry needs to be quite real time.
- peheje 3y ago
  "I guess you could always cap off the exponential interval increases at some max interval, making it basically a repeating timer." No you MUST have a max interval cap. Exponentials and all, you'll soon be waiting days or months for a retry.
  - mmac_ 3y ago
    It's ok, if you wait long enough it'll go back down to a small number. Or maybe a negative number and then the car crashes? Please cap your numbers for both user experience reasons, and safety reasons. Yes, have dealt with this before. Nothing like doing a fault analysis and telling someone that it'll fix itself in a few months. A capped fibonacci sequence is a very common back off timer.
- dublin 3y ago
  Most (actually almost all) IoT systems are not well-thought-out in this regard. It requires a bit of paranoiac foresight. Designers far too often think it's really, really important that remote devices stay connected to the mothership, when in reality, most IoT devices would be far better off NEVER requiring a cloud connection, but leveraging one if/when available. *Rule One of IoT network design: NEVER assume that you even have a network * If at all possible, and to the maximum degree possible, an IoT device must work whether there's a network available or not.
refulgentis 3y ago
what's a left bit shift ;)
Tempest1981 3y ago
Also good to sprinkle in some randomness.
sokoloff 3y ago
It’s a left shift and a non-volatile* variable to store it. * at least across module sleep/wake
crazygringo 3y ago
No. It's setting an async timer in a module that might not support that at all. It's establishing a variable for that in a scope the programmer might not normally have access to. It's making decisions about how quickly to back off and is there an eventual minimum floor. It's making decisions about what events ought to reset the timer, such as starting the car. And it's about creating a series of tests to verify this behavior, which is awfully non-obvious when the behavior might take 10 minutes to verify and tests are supposed to complete in 5 min. It's fancy.
- adhesive_wombat 3y ago
  If you cannot mock that controller state machine properly, you should not be writing software that goes in cars. At least the penalty is you just repeatedly cause every car of the model in the entire country to fail instead of crashing into a wall, but this kind of thing is why the T in IoT is for "shit".
  - crazygringo 3y ago
    I agree completely. And it just supports my point, which is that it's not "literally just a left bit-shift". It's quite a lot of work to do, actually. Which -- of course -- should be done. But there are commenters here acting like it's a single line of code.
    - adhesive_wombat 3y ago
      That's not work unique to exponential back-off. You'd have to do all that for any back-off, which is why I said it's more complex than no back-off. Exponential back off literally is about a line of code difference (change "x += a" to "x <<= 1" (ok, a different radix isn't so silly easy, but anything is better then no back-off). All the rest, maximums, wrap around protection, reset events, all needs thought in either case. Actually the pathological cases in a linear system are perhaps sneakier as they could creep up on you ("it won't overflow, right? That would take months!") But I think we all know what actually happened here was a rush to release and then a failure to follow up with "the update that actually does it right". The Jira ticket is probably still "open - should do, version: next" somewhere. And there's a "told you so" email chain linked in there somewhere.

nucleardog 3y ago

Not to argue the overall point (retrying on a tight loop is not a good solution), but many of the models support communication the other way—e.g., remotely starting the car, adjusting the HVAC, etc. So there is some justification for trying to keep a cell signal established or checking in more frequently than every 15 minutes or so. Someone attempting to remote start their car not having it actually respond for 15 minutes is likely to already be outside trying to leave and pissed off that they’re walking out to a cold car with iced up windows and thinking that the “remote start didn’t work”.

_xerces_ 3y ago

When I was involved in certifying new devices to get on a carrier network, the carrier insisted on exponential backoff as well as some randomness so you wouldn't get millions of voic..devices crying out at the same time when power was lost or restored.

dublin 3y ago
It's not like this is a new problem, and it's been solved for decades - it's just good network design. One example: This is exactly why randomness is required by Ethernet's ARP protocol: to avoid ARP storms when the power comes back on and all nodes try to get back on the network at the same time. (And ARP is still powering most IP connections (v4) today, since IPv6 is painfully fiddly by design, so no one uses v6 unless they have to...)