4 ms·

With avionics code, it's not so much about trusting a programmer (be they human or artificial), but about thorough verification and sign-off with multiple parti

by tjr 4y ago

With avionics code, it's not so much about trusting a programmer (be they human or artificial), but about thorough verification and sign-off with multiple parties involved. (As an aerospace software person, I frequently like to try to persuade folks that this is actually "engineering" software!) I do not believe that current AI technology could write avionics software by itself, mainly because it's a very niche, very not-public field. I'm not sure what all would be involved with adequately training it, but I don't foresee this happening soon. (I could be wrong.) However, regardless of if, or when, AI tools are adequately trained to write avionics software, writing the code is only one piece of the puzzle. The work today includes writing requirements for the the code will do (which work is reviewed and signed off on), writing code based on the requirements (which work is reviewed and signed off on), writing verification tests based on the requirements (which work is reviewed and signed off on), running the tests (which test runs and test environment is reviewed and signed off on), preparing a document paper trail which shows everything has been done per FAA regulations, etc., etc. The first step to getting AI in the door here -- to do anything by itself -- would be to certify an AI tool for avionics software development. I suspect this by itself might be a non-starter, as the resulting outputs of neural-network-based AI tools are not sufficiently repeatable. But even if we were to imagine somehow certifying an AI tool to replace a human in the software development process, it seems not plausible that that same AI tool would replace humans in all parts of the software development process. I.e., if ChatGPT writes the code, ChatGPT can't also review the code and sign off on the code. Perhaps some other AI product could, but how different would it have to be? Is that a realistic plan, to have two completely different AI products? I don't have an answer for that, and I suspect it's a question the FAA will be trying to answer soon. The biggest challenge, I think, is lack of repeatability. If, given the same inputs, the AI tool always produced the same outputs, it would be much easier. Now, none of this precludes using AI tools to assist in any of these processes. It's completely removing a human from being involved which sets off the big alarms for FAA process.

p1esk 4y ago
The term "assist" is not clearly defined - every task you mentioned after "writing the code is only one piece of the puzzle": writing requirements, tests, running the tests, reviewing code, etc - is doable by an AI, in fact even GPT-4 can probably do a decent job today. The way you see AI being integrated into avionics SWE process - having to certify anything - I don't think that's how it's going to develop. AI is already in the door - the software engineers writing avionics code are probably already starting to use GPT-4 to assist them in some of the trivial coding tasks - with or without their managers knowledge or approval. This level of "assistance" is going to steadily increase in the next few years, as more and more capable models are released. Eventually AI will write everything, and humans will spend progressively less and less time reviewing the output, because they will increasingly trust the model to do a good job. One day, managers will adapt to the new way of doing things, and the trust in the models will be strong enough to get rid of human programmers - managers will interact with the models directly, for some time, until the managers themselves are no longer viewed necessary by the higher ups.
- tjr 4y ago
  The way you see AI being integrated into avionics SWE process - having to certify anything - I don't think that's how it's going to develop. What do you base this on? I find it hard to imagine the FAA relaxing their requirements for tool certification. EDIT: a recent paper on the subject: https://dl.gi.de/bitstream/handle/20.500.12116/40204/paper15.pdf?sequence=1&isAllowed=y https://dl.gi.de/bitstream/handle/20.500.12116/40204/paper15...
  - p1esk 4y ago
    Maybe I didn't express myself clearly. Let me try again. The way I see it develop in the next few years: 1. Avionics SWEs start using GPT models to help them do all the tasks you mentioned. This does not change any certification processes, because the resulting code will still be reviewed by humans (before being submitted for any certifications), and it will be certified the same as any human generated code. Managers can try to prohibit the use of AI models, but if models do a good job, people will find a way to bypass those restrictions. 2. Eventually, SWEs realize the models consistently do a better job than they possibly could, and start trusting the models. This will lead to less and less reviewing, and eventually SWEs will do little more than passing high level instructions from their managers to the models, and then passing the reports (also AI generated) about completed work back to the managers. 3. People who perform certifications of the code will follow the same path. AI models will become much better than humans in reviewing, testing, and whatever else is involved in the certification process. Eventually those people will too have very little to do other than write prompts. 4. Eventually management will convince themselves (and higherups) that things are working great, and as a cost saving measure, decide to get rid of the SWEs. 5. Same thing happens to the managers. If GPT-5/6/7 follow the same rate of improvement as what we saw with GPT-3/3.5/4 the above scenario will most likely happen within the next 5 years.
    - tjr 4y ago
      Maybe I am also not being clear. I agree with every single point you made, from the perspective of being, either now or in the future, technologically possible. However, unless the FAA accepts the output of AI tools in lieu of humans performing those same tasks, then it really doesn't matter. Without FAA approval, your code does not (legally) fly. [Speaking in the United States here; other regions have their own, likely similar, regulations.] Right now, there are software tools that replace humans in one function or another, in avionics software development. But the difference is these tools consistently produce expected output, and can be verified as such. The way the GPT-style tools currently behave, I think it would be very difficult to see the FAA approve them as a human-replacing code-generation tool. The path toward FAA-accepted usage of neural-network-based AI tools in avionics software is a point of research right now. Maybe we will get there. Maybe we will get partially there. I think that AI tools will make their way into avionics a lot more slowly than many other industries, not because of technical inadequacy, but because of procedural oversight.

by tjr 4y ago

p1esk 4y ago

The term "assist" is not clearly defined - every task you mentioned after "writing the code is only one piece of the puzzle": writing requirements, tests, running the tests, reviewing code, etc - is doable by an AI, in fact even GPT-4 can probably do a decent job today. The way you see AI being integrated into avionics SWE process - having to certify anything - I don't think that's how it's going to develop. AI is already in the door - the software engineers writing avionics code are probably already starting to use GPT-4 to assist them in some of the trivial coding tasks - with or without their managers knowledge or approval. This level of "assistance" is going to steadily increase in the next few years, as more and more capable models are released. Eventually AI will write everything, and humans will spend progressively less and less time reviewing the output, because they will increasingly trust the model to do a good job. One day, managers will adapt to the new way of doing things, and the trust in the models will be strong enough to get rid of human programmers - managers will interact with the models directly, for some time, until the managers themselves are no longer viewed necessary by the higher ups.

tjr 4y ago
The way you see AI being integrated into avionics SWE process - having to certify anything - I don't think that's how it's going to develop. What do you base this on? I find it hard to imagine the FAA relaxing their requirements for tool certification. EDIT: a recent paper on the subject: https://dl.gi.de/bitstream/handle/20.500.12116/40204/paper15.pdf?sequence=1&isAllowed=y https://dl.gi.de/bitstream/handle/20.500.12116/40204/paper15...
- p1esk 4y ago
  Maybe I didn't express myself clearly. Let me try again. The way I see it develop in the next few years: 1. Avionics SWEs start using GPT models to help them do all the tasks you mentioned. This does not change any certification processes, because the resulting code will still be reviewed by humans (before being submitted for any certifications), and it will be certified the same as any human generated code. Managers can try to prohibit the use of AI models, but if models do a good job, people will find a way to bypass those restrictions. 2. Eventually, SWEs realize the models consistently do a better job than they possibly could, and start trusting the models. This will lead to less and less reviewing, and eventually SWEs will do little more than passing high level instructions from their managers to the models, and then passing the reports (also AI generated) about completed work back to the managers. 3. People who perform certifications of the code will follow the same path. AI models will become much better than humans in reviewing, testing, and whatever else is involved in the certification process. Eventually those people will too have very little to do other than write prompts. 4. Eventually management will convince themselves (and higherups) that things are working great, and as a cost saving measure, decide to get rid of the SWEs. 5. Same thing happens to the managers. If GPT-5/6/7 follow the same rate of improvement as what we saw with GPT-3/3.5/4 the above scenario will most likely happen within the next 5 years.
  - tjr 4y ago
    Maybe I am also not being clear. I agree with every single point you made, from the perspective of being, either now or in the future, technologically possible. However, unless the FAA accepts the output of AI tools in lieu of humans performing those same tasks, then it really doesn't matter. Without FAA approval, your code does not (legally) fly. [Speaking in the United States here; other regions have their own, likely similar, regulations.] Right now, there are software tools that replace humans in one function or another, in avionics software development. But the difference is these tools consistently produce expected output, and can be verified as such. The way the GPT-style tools currently behave, I think it would be very difficult to see the FAA approve them as a human-replacing code-generation tool. The path toward FAA-accepted usage of neural-network-based AI tools in avionics software is a point of research right now. Maybe we will get there. Maybe we will get partially there. I think that AI tools will make their way into avionics a lot more slowly than many other industries, not because of technical inadequacy, but because of procedural oversight.