3 ms·

Maybe I am also not being clear. I agree with every single point you made, from the perspective of being, either now or in the future, technologically possible

by tjr 4y ago

Maybe I am also not being clear. I agree with every single point you made, from the perspective of being, either now or in the future, technologically possible. However, unless the FAA accepts the output of AI tools in lieu of humans performing those same tasks, then it really doesn't matter. Without FAA approval, your code does not (legally) fly. [Speaking in the United States here; other regions have their own, likely similar, regulations.] Right now, there are software tools that replace humans in one function or another, in avionics software development. But the difference is these tools consistently produce expected output, and can be verified as such. The way the GPT-style tools currently behave, I think it would be very difficult to see the FAA approve them as a human-replacing code-generation tool. The path toward FAA-accepted usage of neural-network-based AI tools in avionics software is a point of research right now. Maybe we will get there. Maybe we will get partially there. I think that AI tools will make their way into avionics a lot more slowly than many other industries, not because of technical inadequacy, but because of procedural oversight.

p1esk 4y ago
OK, I see your point. A couple of questions: 1. Do humans consistently produce expected output? Obviously not, so why would we trust them to write complicated software? 2. How does FAA know whether the code has been produced by a human, or by a human who asked GPT to write it? In both cases a human will have signed off on it.
- tjr 4y ago
  Couldn't humans make mistakes? Sure. And that's why there are so many checks and balances in avionics software, compared to most other software industries. The more safety critical it is, the more it is scrutinized. Why would we not trust a software tool to write code just as well as a human, when humans themselves are fallible? I have not discussed this with any FAA representative, and have not yet found any published reference material that specifically lays out this case, but just to offer my personal, unofficial opinion based on past experience: I believe that an AI tool (like ChatGPT or what have you) would be considered, per RTCA document DO-330, an AutoCode Generator. If the code generated by a tool (such as an AI tool) is subsequently treated just as if it was written by a human -- subject to the same code reviews, the same verification tests, the same structural code coverage analysis tests, etc. -- then, in my personal opinion, that ought to be no different than if a human wrote the code. ==> Thus, at least for the code-writing itself, your enumerated Step 1 should be reasonably attainable. It's only after that where things start getting tricky. No matter how good the code quality seems to be, no matter how flawless it seems to be, no matter how much you intuitively come to trust the AI tool ... it's work still has to be checked. If you could formally qualify the tool that is generating the code, then you can get away with a lot less checking. If you could formally qualify tools to do the code reviews and the verification testing and so forth, then yes, in theory you could automate the entire process. How would the FAA know what tool(s) you used to generate code? Unlike in most industries, part of formally releasing avionics software includes declaring the software development tools used to create it. You might not need to cite a tool that generated code which was subsequently human-reviewed and tested, but I would not be confident about that. If you qualified that tool to generate code which was not human-reviewed and tested, then you would certainly need to declare it.
  - p1esk 4y ago
    How a tool like a GPT model should be certified? Earlier you mentioned that given the same input, it should consistently produce the same output - that is technically achievable today (at the expense of creativity, etc). What else would convince you, or FAA, that the model is "good enough" to write avionics code? For example, we can subject it to the same tests we subject a human programmer. I spent two hours yesterday with GPT-4 discussing my ideas for a scientific paper. It was very productive - like talking to a friendly PhD adviser. The model discussed highly technical abstract concepts, and combined ideas in an interesting way. There was no misunderstanding of what I was describing, no logical inconsistencies, no signs indicating I'm talking to someone other than a smart and knowledgeable human being. It can also produce a code prototype of the algorithm it suggested, and successfully debug this prototype with me. The only drawback is its knowledge of relevant papers extended only up to September 2021 (limit of the training dataset). My point is I already feel the model we have today (GPT-4) is as useful as a smart colleague. Next year we will have GPT-5, which will be superior in every way, and it's quite possible that it will be better than me at what I do professionally (ML research/engineering). Keep in mind, GPT-4 was able to learn so much about the world while trained on text alone. Imagine how much faster and better these models will learn given access to video/audio data. GPT-4 can already imitate many distinct personalities quite well, it can already assume a persona of a complicated human being - people will soon become friends with such personas, some will even fall in love with them. I'm both awed and scared about what the next generation will be capable of. It might turn out that GPT-5 will be used to "certify" humans, not the other way around.
    - tjr 4y ago
      I have no reason to believe that a GPT model would not undergo the same criteria as any other software tool. The process is outlined in DO-330, but the essence of it is that, if a software tool is performing a function that (a) replaces a human who would otherwise be doing some particular task, and (b) either could introduce errors into a software system (for code generation), or could fail to detect them (for code verification), then the tool must be qualified. Qualifying a tool includes preparing a requirements document that describes the tool's behavior and limitations, and a set of tests that demonstrate that the tool performs as described within those limitations. (No tool is infinite, nor could any tool be tested infinitely. For an easy example, does it handle integers? What's the range of integers it handles?) Trivial in concept, but the more complex the tool, the more arduous the process of putting together materials that convince the FAA that the tool is acceptable. The DO-330 guidance for tool qualification is only about a decade old; prior to that, tool qualification guidelines were baked in to the more general avionics development guidelines document of DO-178B, which insisted that any software development tool be completely deterministic. DO-330 relaxes that somewhat, but in every case you must still demonstrate that, even if the tool output can vary, it varies within known expectations, and any variance does not change the effective meaning of the tool output. (It is my understanding that such variance is also only permitted for the least-critical category of software tools; the more safety-critical job the tool does, the less flexibility there is.) So preferably, for any given prompt, the AI tool will always produce the same output of code. In the most possible relaxed scenario, for any given prompt, the AI tool must at least always produce code that functions the same. For context, I've worked nearly 20 years in the aerospace industry, including a lot of work with tool qualification and product certification. I am not claiming that using AI tools is impossible. I am claiming that, compared to many other technological industries, there is a huge amount of red tape and bureaucracy here. Including AI/ML techniques in this work is slow-moving research effort, rather than the fast pace of many other places. Given that we are literally talking about software systems that are depended upon to keep people safe and alive, we must be more certain than usual.

by tjr 4y ago

p1esk 4y ago

OK, I see your point. A couple of questions: 1. Do humans consistently produce expected output? Obviously not, so why would we trust them to write complicated software? 2. How does FAA know whether the code has been produced by a human, or by a human who asked GPT to write it? In both cases a human will have signed off on it.

tjr 4y ago
Couldn't humans make mistakes? Sure. And that's why there are so many checks and balances in avionics software, compared to most other software industries. The more safety critical it is, the more it is scrutinized. Why would we not trust a software tool to write code just as well as a human, when humans themselves are fallible? I have not discussed this with any FAA representative, and have not yet found any published reference material that specifically lays out this case, but just to offer my personal, unofficial opinion based on past experience: I believe that an AI tool (like ChatGPT or what have you) would be considered, per RTCA document DO-330, an AutoCode Generator. If the code generated by a tool (such as an AI tool) is subsequently treated just as if it was written by a human -- subject to the same code reviews, the same verification tests, the same structural code coverage analysis tests, etc. -- then, in my personal opinion, that ought to be no different than if a human wrote the code. ==> Thus, at least for the code-writing itself, your enumerated Step 1 should be reasonably attainable. It's only after that where things start getting tricky. No matter how good the code quality seems to be, no matter how flawless it seems to be, no matter how much you intuitively come to trust the AI tool ... it's work still has to be checked. If you could formally qualify the tool that is generating the code, then you can get away with a lot less checking. If you could formally qualify tools to do the code reviews and the verification testing and so forth, then yes, in theory you could automate the entire process. How would the FAA know what tool(s) you used to generate code? Unlike in most industries, part of formally releasing avionics software includes declaring the software development tools used to create it. You might not need to cite a tool that generated code which was subsequently human-reviewed and tested, but I would not be confident about that. If you qualified that tool to generate code which was not human-reviewed and tested, then you would certainly need to declare it.
- p1esk 4y ago
  How a tool like a GPT model should be certified? Earlier you mentioned that given the same input, it should consistently produce the same output - that is technically achievable today (at the expense of creativity, etc). What else would convince you, or FAA, that the model is "good enough" to write avionics code? For example, we can subject it to the same tests we subject a human programmer. I spent two hours yesterday with GPT-4 discussing my ideas for a scientific paper. It was very productive - like talking to a friendly PhD adviser. The model discussed highly technical abstract concepts, and combined ideas in an interesting way. There was no misunderstanding of what I was describing, no logical inconsistencies, no signs indicating I'm talking to someone other than a smart and knowledgeable human being. It can also produce a code prototype of the algorithm it suggested, and successfully debug this prototype with me. The only drawback is its knowledge of relevant papers extended only up to September 2021 (limit of the training dataset). My point is I already feel the model we have today (GPT-4) is as useful as a smart colleague. Next year we will have GPT-5, which will be superior in every way, and it's quite possible that it will be better than me at what I do professionally (ML research/engineering). Keep in mind, GPT-4 was able to learn so much about the world while trained on text alone. Imagine how much faster and better these models will learn given access to video/audio data. GPT-4 can already imitate many distinct personalities quite well, it can already assume a persona of a complicated human being - people will soon become friends with such personas, some will even fall in love with them. I'm both awed and scared about what the next generation will be capable of. It might turn out that GPT-5 will be used to "certify" humans, not the other way around.
  - tjr 4y ago
    I have no reason to believe that a GPT model would not undergo the same criteria as any other software tool. The process is outlined in DO-330, but the essence of it is that, if a software tool is performing a function that (a) replaces a human who would otherwise be doing some particular task, and (b) either could introduce errors into a software system (for code generation), or could fail to detect them (for code verification), then the tool must be qualified. Qualifying a tool includes preparing a requirements document that describes the tool's behavior and limitations, and a set of tests that demonstrate that the tool performs as described within those limitations. (No tool is infinite, nor could any tool be tested infinitely. For an easy example, does it handle integers? What's the range of integers it handles?) Trivial in concept, but the more complex the tool, the more arduous the process of putting together materials that convince the FAA that the tool is acceptable. The DO-330 guidance for tool qualification is only about a decade old; prior to that, tool qualification guidelines were baked in to the more general avionics development guidelines document of DO-178B, which insisted that any software development tool be completely deterministic. DO-330 relaxes that somewhat, but in every case you must still demonstrate that, even if the tool output can vary, it varies within known expectations, and any variance does not change the effective meaning of the tool output. (It is my understanding that such variance is also only permitted for the least-critical category of software tools; the more safety-critical job the tool does, the less flexibility there is.) So preferably, for any given prompt, the AI tool will always produce the same output of code. In the most possible relaxed scenario, for any given prompt, the AI tool must at least always produce code that functions the same. For context, I've worked nearly 20 years in the aerospace industry, including a lot of work with tool qualification and product certification. I am not claiming that using AI tools is impossible. I am claiming that, compared to many other technological industries, there is a huge amount of red tape and bureaucracy here. Including AI/ML techniques in this work is slow-moving research effort, rather than the fast pace of many other places. Given that we are literally talking about software systems that are depended upon to keep people safe and alive, we must be more certain than usual.