5 ms·

Every automotive code base I've seen has been a dumpster fire. This (mostly) isn't the fault of the developers and Rust won't fix it. The business process creat

Every automotive code base I've seen has been a dumpster fire. This (mostly) isn't the fault of the developers and Rust won't fix it. The business process creates this outcome. No one likes it but little is done to address it. The software development timeline is slaved to the rigid manufacturing timeline. There is no allowance for slippage, so making things work By Any Means Necessary is the rule of the day. Concurrency problems often are not debugged; locks and similar are just sprinkled around until they disappear in testing. It is TDD taken to an extreme -- passing the enumerated tests, technically, is the definition of done even if the code is obviously poor or broken. There are a vast number of software and hardware components sourced from myriad vendors that are swapped out every year to save a few dollars or deal with a supply chain issue. There is no stable baseline but all of these components need to work together. The code has to constantly adapt to the reality that someone sourced parts for a new model year that work differently or are in a different programming language. This code is usually required to be supported for 10+ years. The developers don't just have this year's tech stack and code base to deliver on a rigid schedule, they are required to maintain several older implementations concurrently, often built in different languages and architectures. Basically, the automotive industry has failed badly at the transition from being hardware manufacturing companies to being hardware + software manufacturing companies, and try to run their software processes the same way they ran their traditional hardware processes. Choice of programming language is the least of their concerns, that isn't the source of their code quality issues.

iancmceachern 4y ago
This is the case for a lot of engineering, I've seen it time and again. The business processes force the engineering work product to be a dumpster fire. The great companies dont.
ParetoOptimal 4y ago
> Choice of programming language is the least of their concerns, that isn't the source of their code quality issues. I'm curious to see if the Rust programmers affect the organizations as well by introducing unions to combat this kind if thing. On another note I disagree that language will have no meanigful effect, even in a hostile environment such as this.
- jandrewrogers 4y ago
  A substantial amount of this development is already done in managed/safe languages when possible. In many cases, the software developers are represented by a works council or similar. These are generally capable and skilled developers that want to do a good job, not half-literate code monkeys slinging bad C in a sweatshop. Whatever ethos you imagine Rust developers uniquely have, it doesn't address any of the constraints and incentives that are creating the code quality problems in automotive. Even the best developers in the world would have a difficult time producing a quality product in that environment.
  - ParetoOptimal 4y ago
    > Whatever ethos you imagine Rust developers uniquely have They tend to seem more of the type to go on strike, form a union, or leave a job because of political views. > Even the best developers in the world would have a difficult time producing a quality product in that environment. Right, i'm saying I think at least more experienced rust developers would refuse to work in that environment. They seem like they would risk more of their own well-being for ethical reasons, for better or worse. I suppose if Volvo started depending on highly socially conscious domain experts and had a choice between changing working conditions and ditching rust though, they'd change the latter. I'm happy to be proven wrong, but experienced C programmers seem to be more conservative "don't mix politics and programming" types.
    - dmitriid 4y ago
      > I suppose if Volvo started depending on highly socially conscious domain experts It's amazing that you're saying all that and the same time completely oblivious to the fact that Volvo is in Sweden. Which has unions, labor protection, amazing working conditions etc. etc. The reason Volvo is doing what it's doing is due to Volvo's commitment to safety [1] and not because of some mythical exclusively social ethos of some programmers. [1] https://www.volvogroup.com/en/about-us/traffic-safety/safety-vision.html https://www.volvogroup.com/en/about-us/traffic-safety/safety...
    - pjmlp 4y ago
teaearlgraycold 4y ago
> Concurrency problems often are not debugged; locks and similar are just sprinkled around until they disappear in testing. It does sound like Rust could help here
- Snild 4y ago
  Not with liberal use of `unsafe` it can't.
  - 19h 4y ago
    IME It's often harder to do it the unsafe way rather than to just do it the safe way
    - klabb3 4y ago
      At the end of the day it's embedded code that's inherently unsafe, so the benefits are all in the code that belongs in the safe zone. Unsafe Rust is at best equivalent to C/C++ and at worst it's a false sense of security. Mixing safe and unsafe without a clear boundary is not something I can speak to with confidence but my gut-feeling is that it's worse than C/C++. Even OS kernels are borderline, and it's still being figured out how exactly Rust fits, but yes we are slowly moving in that direction. Kernels have a lot of logic that qualify for the safe zone, and software engineering culture that is... let's say ahead of automotive.
      - kennywinker 4y ago
        > embedded code that's inherently unsafe I'm not really an expert in embedded systems, so take this with a grain of salt, but that doesn't track with my experience or with my theoretical understanding of embedded systems. First off, embedded systems are just regular systems with tighter constraints and less high-level constructs. With that understanding, if embedded code is unsafe... so is any code on any system. Second, embedded systems have peripherals that are accessed via direct memory reads and writes. Which is only unsafe if access to those peripherals is shared in an unsafe way. Sure, the rust code that reads and writes to memory addresses will use `unsafe`, but everything above that can be written without using unsafe. Which I guess makes your statement technically correct, but imo not practically correct
dmitriid 4y ago
It's even worse than that. The standards they run on have contradicting errors in the dozens or hundreds. See this talk https://youtu.be/hXnS_Xjwk2Y?t=899 https://youtu.be/hXnS_Xjwk2Y?t=899 starting at 15 minutes in (that includes testing for Volvo). Relevant (for those who don't want to watch video): - 3000 pages of specs - 1 million LoC from suppliers (because you use different systems from different suppliers) - 200 problems - of which more than 100 problems were in the standard itself (standard being AUTOSAR that described how different parts from different suppliers communicate with each other)
drdaeman 4y ago
The language won't fix it (it's just a tool after all), though it may enforce certain rules that help the developers not shoot themselves in the leg. Rust has some fairly strong guards, as long as you don't circumvent them. At least it takes some conscious effort to make a Rust program segfault, and almost every IoT appliance (non-automotive, though, I haven't worked with those) I've had that I've reverse engineered had a command that could crash it - I've rebooted my cat feeder way too many times to my liking. What could be an important change is the culture. At the very least, developers who tend to pick Rust may be (hopefully) more inclined to do things the right way and don't just slap shit on the codebase until it sticks. I've seen way too much of this attitude in embedded developers. And another important change could be management policies. If they are conscious to pick something uncommon, they might have better tolerance toward spending more time but doing things properly, than pushing people to release whatever seem to work (somehow). I'm pretty sure a lot of bullshit in IoT happens because management want to release things as fast as it's possible, and developers either don't care or can't fight it.
- stubish 4y ago
  The language will fix a chunk of it. Much buggy code simply cannot be compiled, and if you can't compile it you can't ship it. You are mostly left with logic and design errors, which can become more obvious due to the language. Middle management will hate it because due to the language they lose much of their ability to compromise on quality, passing the buck. Upper management may love it or hate it, depending on how serious they are about improving quality, as the language forces the spending for that level of quality.
jrsj 4y ago
Generally speaking they also don't pay software development well so I would guess they aren't exactly getting the best talent either. I think cars are cool, but I wouldn't take a 50% pay cut to develop software for them.
- oppositelock 4y ago
  It's not a question of talent, just timing and process. The timing is rigid and there's a hard, non-negotiable deadline when metal starts being stamped. The process around certifying makes you work at a tiny fraction of your normal speed, because most time is spent filling out auditing requirements.
  - AlotOfReading 4y ago
    You still need to pay enough to attract the kind of people who understand what "undefined behavior" is and why you should avoid it. That number is somewhat higher than what most mid-level automotive programmers are paid, in my experience.
1-6 4y ago
Volvo should actually be contributing to a consortium of automotive computer developers that pool resources together to build shared software on cars. The ultimate outcome would be like an Android-like license that allows car manufacturers the freedom to build-on-top of a software stack that gets security updates. Cars get tricky, though, because bugs can lead to loss of life.
- pjmlp 4y ago
  I bet Android Auto devices get even less updates than most Android based phones and tablets.
  - int_19h 4y ago
    Isn't Android Auto just proprietary audio/video/touch protocol over USB-C, with no actual Android bits on the car side?
mechanical_bear 4y ago
How many automotive code bases have you seen and in what sort of capacity? Also, how recent?
sebazzz 4y ago
Cars have been hardware+software for some time now, aren't they? ECM have software and have been around since at least 1998.

iancmceachern 4y ago

This is the case for a lot of engineering, I've seen it time and again. The business processes force the engineering work product to be a dumpster fire. The great companies dont.

ParetoOptimal 4y ago

> Choice of programming language is the least of their concerns, that isn't the source of their code quality issues. I'm curious to see if the Rust programmers affect the organizations as well by introducing unions to combat this kind if thing. On another note I disagree that language will have no meanigful effect, even in a hostile environment such as this.

jandrewrogers 4y ago
A substantial amount of this development is already done in managed/safe languages when possible. In many cases, the software developers are represented by a works council or similar. These are generally capable and skilled developers that want to do a good job, not half-literate code monkeys slinging bad C in a sweatshop. Whatever ethos you imagine Rust developers uniquely have, it doesn't address any of the constraints and incentives that are creating the code quality problems in automotive. Even the best developers in the world would have a difficult time producing a quality product in that environment.
- ParetoOptimal 4y ago
  > Whatever ethos you imagine Rust developers uniquely have They tend to seem more of the type to go on strike, form a union, or leave a job because of political views. > Even the best developers in the world would have a difficult time producing a quality product in that environment. Right, i'm saying I think at least more experienced rust developers would refuse to work in that environment. They seem like they would risk more of their own well-being for ethical reasons, for better or worse. I suppose if Volvo started depending on highly socially conscious domain experts and had a choice between changing working conditions and ditching rust though, they'd change the latter. I'm happy to be proven wrong, but experienced C programmers seem to be more conservative "don't mix politics and programming" types.
  - dmitriid 4y ago
    > I suppose if Volvo started depending on highly socially conscious domain experts It's amazing that you're saying all that and the same time completely oblivious to the fact that Volvo is in Sweden. Which has unions, labor protection, amazing working conditions etc. etc. The reason Volvo is doing what it's doing is due to Volvo's commitment to safety [1] and not because of some mythical exclusively social ethos of some programmers. [1] https://www.volvogroup.com/en/about-us/traffic-safety/safety-vision.html https://www.volvogroup.com/en/about-us/traffic-safety/safety...
  - pjmlp 4y ago

teaearlgraycold 4y ago

> Concurrency problems often are not debugged; locks and similar are just sprinkled around until they disappear in testing. It does sound like Rust could help here

Snild 4y ago
Not with liberal use of `unsafe` it can't.
- 19h 4y ago
  IME It's often harder to do it the unsafe way rather than to just do it the safe way
  - klabb3 4y ago
    At the end of the day it's embedded code that's inherently unsafe, so the benefits are all in the code that belongs in the safe zone. Unsafe Rust is at best equivalent to C/C++ and at worst it's a false sense of security. Mixing safe and unsafe without a clear boundary is not something I can speak to with confidence but my gut-feeling is that it's worse than C/C++. Even OS kernels are borderline, and it's still being figured out how exactly Rust fits, but yes we are slowly moving in that direction. Kernels have a lot of logic that qualify for the safe zone, and software engineering culture that is... let's say ahead of automotive.
    - kennywinker 4y ago
      > embedded code that's inherently unsafe I'm not really an expert in embedded systems, so take this with a grain of salt, but that doesn't track with my experience or with my theoretical understanding of embedded systems. First off, embedded systems are just regular systems with tighter constraints and less high-level constructs. With that understanding, if embedded code is unsafe... so is any code on any system. Second, embedded systems have peripherals that are accessed via direct memory reads and writes. Which is only unsafe if access to those peripherals is shared in an unsafe way. Sure, the rust code that reads and writes to memory addresses will use `unsafe`, but everything above that can be written without using unsafe. Which I guess makes your statement technically correct, but imo not practically correct

dmitriid 4y ago

It's even worse than that. The standards they run on have contradicting errors in the dozens or hundreds. See this talk https://youtu.be/hXnS_Xjwk2Y?t=899 https://youtu.be/hXnS_Xjwk2Y?t=899 starting at 15 minutes in (that includes testing for Volvo). Relevant (for those who don't want to watch video): - 3000 pages of specs - 1 million LoC from suppliers (because you use different systems from different suppliers) - 200 problems - of which more than 100 problems were in the standard itself (standard being AUTOSAR that described how different parts from different suppliers communicate with each other)

drdaeman 4y ago

The language won't fix it (it's just a tool after all), though it may enforce certain rules that help the developers not shoot themselves in the leg. Rust has some fairly strong guards, as long as you don't circumvent them. At least it takes some conscious effort to make a Rust program segfault, and almost every IoT appliance (non-automotive, though, I haven't worked with those) I've had that I've reverse engineered had a command that could crash it - I've rebooted my cat feeder way too many times to my liking. What could be an important change is the culture. At the very least, developers who tend to pick Rust may be (hopefully) more inclined to do things the right way and don't just slap shit on the codebase until it sticks. I've seen way too much of this attitude in embedded developers. And another important change could be management policies. If they are conscious to pick something uncommon, they might have better tolerance toward spending more time but doing things properly, than pushing people to release whatever seem to work (somehow). I'm pretty sure a lot of bullshit in IoT happens because management want to release things as fast as it's possible, and developers either don't care or can't fight it.

stubish 4y ago
The language will fix a chunk of it. Much buggy code simply cannot be compiled, and if you can't compile it you can't ship it. You are mostly left with logic and design errors, which can become more obvious due to the language. Middle management will hate it because due to the language they lose much of their ability to compromise on quality, passing the buck. Upper management may love it or hate it, depending on how serious they are about improving quality, as the language forces the spending for that level of quality.

jrsj 4y ago

Generally speaking they also don't pay software development well so I would guess they aren't exactly getting the best talent either. I think cars are cool, but I wouldn't take a 50% pay cut to develop software for them.

oppositelock 4y ago
It's not a question of talent, just timing and process. The timing is rigid and there's a hard, non-negotiable deadline when metal starts being stamped. The process around certifying makes you work at a tiny fraction of your normal speed, because most time is spent filling out auditing requirements.
- AlotOfReading 4y ago
  You still need to pay enough to attract the kind of people who understand what "undefined behavior" is and why you should avoid it. That number is somewhat higher than what most mid-level automotive programmers are paid, in my experience.

1-6 4y ago

Volvo should actually be contributing to a consortium of automotive computer developers that pool resources together to build shared software on cars. The ultimate outcome would be like an Android-like license that allows car manufacturers the freedom to build-on-top of a software stack that gets security updates. Cars get tricky, though, because bugs can lead to loss of life.

pjmlp 4y ago
I bet Android Auto devices get even less updates than most Android based phones and tablets.
- int_19h 4y ago
  Isn't Android Auto just proprietary audio/video/touch protocol over USB-C, with no actual Android bits on the car side?

mechanical_bear 4y ago

How many automotive code bases have you seen and in what sort of capacity? Also, how recent?

sebazzz 4y ago

Cars have been hardware+software for some time now, aren't they? ECM have software and have been around since at least 1998.