7 ms·

This seems to be a very common type of software bug. On fail retry, without any limits of how frequent or how many time.

resolutebat 3y ago
If only there was a simple way to exponentially limit your retries and back off a bit when the other side is constantly failing.
- wincy 3y ago
  You mean the concept I learned week three as a junior developer (thanks to great mentorship)? Yeah, probably absolutely impossible.
  - hirsin 3y ago
    You're lucky. I've seen entire engineering orgs convinced of the opposite, and highly resistant to improvement.
- clnq 3y ago
  Exponential back-off is fancy. There are a million repeating behaviors in a large automotive codebase. Maybe it's too arduous to code up exponential back-off for all. Even if you have a simple library to do it, it could introduce some complexity. But it's quite easy to set timers to attempt things at regular intervals, rather than in an infinite loop until they succeed. One attempt every 5 seconds (assuming a 5-second timeout for the request) is 17,280 attempts a day. One attempt every 5 minutes is 288 attempts a day. One attempt every 15 minutes is 96 attempts a day. A simple repeating timer is a fantastic solution. Does the car need to always send telemetry immediately? Can there be a 15-minute latency? If so, just put that on a timer and enjoy 99.4% power savings for offline cars, with basically the same quality of telemetry when they're online. But what if a car needs to send telemetry at a given moment? Just move up the next timer function call. But what if I want to get telemetry when the 3G signal is intermittent and briefly available? Just call the function once the 3G signal is available. Do this with a separate cool-down for this behavior. But what if I need telemetry every second? Just dump all telemetry events into a buffer between successful sends, slap a timestamp on those events, and you can now send data for the past X hours all at once. Even from the black spots. There are so many options (even less fancy than exp back-off), but this was probably built by a junior programmer without any support network. This feels like a very common lack-of-experience mistake that would be caught in code review if the reviewer cared.
  - adhesive_wombat 3y ago
    > Exponential back-off is fancy It's literally just a left bit-shift? It's as easy as a linear back-off. The only thing less fancy is no back-off at all.
    - _trampeltier 3y ago
      Ethernet on coax cables did it already after a collision (late 70s, thats 50 years ago). Funny how basic things are today to much work, but animations everywhere are a must.
    - clnq 3y ago
      Perhaps you are right. Though it seems easier to reason about a system with a hundred timers running at a set cadence than a hundred exponential back-offs. Maybe I just have more experience with the former. I guess you could always cap off the exponential interval increases at some max interval, making it basically a repeating timer. Though there's still the question of why would you need this for telemetry. It doesn't feel like this type of "marketing" telemetry needs to be quite real time.
      - peheje 3y ago
        "I guess you could always cap off the exponential interval increases at some max interval, making it basically a repeating timer." No you MUST have a max interval cap. Exponentials and all, you'll soon be waiting days or months for a retry.
        mmac_ 3y ago
        It's ok, if you wait long enough it'll go back down to a small number. Or maybe a negative number and then the car crashes? Please cap your numbers for both user experience reasons, and safety reasons. Yes, have dealt with this before. Nothing like doing a fault analysis and telling someone that it'll fix itself in a few months. A capped fibonacci sequence is a very common back off timer.
      - 3y ago
- cyanydeez 3y ago
  But I need to unit test each recursion else I won't have completed test coverage!
petre 3y ago
It doesn't need to phone home when the ignition is off and could have been programmed to go into standby. You can hardly crash a car with the ignition off. Even if you do, it could be woken up using an accelerometer and try to phone home just a a few times. This is shitty engineering. I am quite amazed, because the Japanese normally don't do this. Probably another stupid marketing decision.
- avidphantasm 3y ago
  Most cars I’ve ever had don’t have auto-off headlights because the extra parts cost a few dollars and that eats into margins and growth must continue infinitely thanks to rent-seeking shareholders. Capitalism will corrupt engineering whenever it is in the interests of capitalists to do so.
  - petre 3y ago
    You don't need extra parts, these are no headlights, but an IoT modem that already has power saving fearures on chip. It just needs an ignition signal and programming to go to sleep when it's not there. You must have had very crappy cars, because I rented a 2022 Renault Clio this spring and it had very good ergonomics. I was quite amazed. Only an older Hyunday i3 that I know of had annoying electrical quirks. But hey, there are tiktok videos on how to steal current Hyundai and Kia cars. https://www.theverge.com/23742425/kia-boys-car-theft-steal-tiktok-hyundai-usb https://www.theverge.com/23742425/kia-boys-car-theft-steal-t...
  - slavboj 3y ago
    Communist automotive engineering departments by contrast are notorious for their near-decadent focus on customer satisfaction.
    - kjkjadksj 3y ago
      Is a lada really much different in that regard as a cold war era gm car?
      - 3y ago
Arrath 3y ago
It sure has been an issue with every cellphone I've ever owned, their radios flip out when I go somewhere without signal (tunnels, the subway, into nature) and will thrash the battery in a mad search for connectivity.

resolutebat 3y ago

If only there was a simple way to exponentially limit your retries and back off a bit when the other side is constantly failing.

wincy 3y ago
You mean the concept I learned week three as a junior developer (thanks to great mentorship)? Yeah, probably absolutely impossible.
- hirsin 3y ago
  You're lucky. I've seen entire engineering orgs convinced of the opposite, and highly resistant to improvement.
clnq 3y ago
Exponential back-off is fancy. There are a million repeating behaviors in a large automotive codebase. Maybe it's too arduous to code up exponential back-off for all. Even if you have a simple library to do it, it could introduce some complexity. But it's quite easy to set timers to attempt things at regular intervals, rather than in an infinite loop until they succeed. One attempt every 5 seconds (assuming a 5-second timeout for the request) is 17,280 attempts a day. One attempt every 5 minutes is 288 attempts a day. One attempt every 15 minutes is 96 attempts a day. A simple repeating timer is a fantastic solution. Does the car need to always send telemetry immediately? Can there be a 15-minute latency? If so, just put that on a timer and enjoy 99.4% power savings for offline cars, with basically the same quality of telemetry when they're online. But what if a car needs to send telemetry at a given moment? Just move up the next timer function call. But what if I want to get telemetry when the 3G signal is intermittent and briefly available? Just call the function once the 3G signal is available. Do this with a separate cool-down for this behavior. But what if I need telemetry every second? Just dump all telemetry events into a buffer between successful sends, slap a timestamp on those events, and you can now send data for the past X hours all at once. Even from the black spots. There are so many options (even less fancy than exp back-off), but this was probably built by a junior programmer without any support network. This feels like a very common lack-of-experience mistake that would be caught in code review if the reviewer cared.
- adhesive_wombat 3y ago
  > Exponential back-off is fancy It's literally just a left bit-shift? It's as easy as a linear back-off. The only thing less fancy is no back-off at all.
  - _trampeltier 3y ago
    Ethernet on coax cables did it already after a collision (late 70s, thats 50 years ago). Funny how basic things are today to much work, but animations everywhere are a must.
  - clnq 3y ago
    Perhaps you are right. Though it seems easier to reason about a system with a hundred timers running at a set cadence than a hundred exponential back-offs. Maybe I just have more experience with the former. I guess you could always cap off the exponential interval increases at some max interval, making it basically a repeating timer. Though there's still the question of why would you need this for telemetry. It doesn't feel like this type of "marketing" telemetry needs to be quite real time.
    - peheje 3y ago
      "I guess you could always cap off the exponential interval increases at some max interval, making it basically a repeating timer." No you MUST have a max interval cap. Exponentials and all, you'll soon be waiting days or months for a retry.
      - mmac_ 3y ago
        It's ok, if you wait long enough it'll go back down to a small number. Or maybe a negative number and then the car crashes? Please cap your numbers for both user experience reasons, and safety reasons. Yes, have dealt with this before. Nothing like doing a fault analysis and telling someone that it'll fix itself in a few months. A capped fibonacci sequence is a very common back off timer.
    - 3y ago
cyanydeez 3y ago
But I need to unit test each recursion else I won't have completed test coverage!

petre 3y ago

It doesn't need to phone home when the ignition is off and could have been programmed to go into standby. You can hardly crash a car with the ignition off. Even if you do, it could be woken up using an accelerometer and try to phone home just a a few times. This is shitty engineering. I am quite amazed, because the Japanese normally don't do this. Probably another stupid marketing decision.

avidphantasm 3y ago
Most cars I’ve ever had don’t have auto-off headlights because the extra parts cost a few dollars and that eats into margins and growth must continue infinitely thanks to rent-seeking shareholders. Capitalism will corrupt engineering whenever it is in the interests of capitalists to do so.
- petre 3y ago
  You don't need extra parts, these are no headlights, but an IoT modem that already has power saving fearures on chip. It just needs an ignition signal and programming to go to sleep when it's not there. You must have had very crappy cars, because I rented a 2022 Renault Clio this spring and it had very good ergonomics. I was quite amazed. Only an older Hyunday i3 that I know of had annoying electrical quirks. But hey, there are tiktok videos on how to steal current Hyundai and Kia cars. https://www.theverge.com/23742425/kia-boys-car-theft-steal-tiktok-hyundai-usb https://www.theverge.com/23742425/kia-boys-car-theft-steal-t...
- slavboj 3y ago
  Communist automotive engineering departments by contrast are notorious for their near-decadent focus on customer satisfaction.
  - kjkjadksj 3y ago
    Is a lada really much different in that regard as a cold war era gm car?
    - 3y ago

Arrath 3y ago

It sure has been an issue with every cellphone I've ever owned, their radios flip out when I go somewhere without signal (tunnels, the subway, into nature) and will thrash the battery in a mad search for connectivity.