3 ms·

I worked in control firmware and systems middleware/OSes for chemical processing equipment (and related control systems). None of the super fancy, huge plants;

I worked in control firmware and systems middleware/OSes for chemical processing equipment (and related control systems). None of the super fancy, huge plants; more single room sized processing pipelines for R&D uses. That said, the bigger chambers were like 0.2m^3 and operated at 2500PSI @ 100C, so you'd definitely know if one catastrophically failed. We didn't necessarily develop a lot of process in-house, because our senior engineers had backgrounds with Boeing and/or NASA, which both had extensive ideas about how to design reliable systems. If I were summarizing, there's only really two points that cover about 90% of what you need to do for high assurance software: 1. Realize almost all bugs are because of politics and economics, not technological or engineering faults per se. That is, we make choices about how we set up our culture and corporate system which incentivizes people to create and hide bugs, while also failing to incentivize others to help fix that bug. The first step in combating bugs must be to change the fundamental incentives which create them. In particular, a focus on the success or failure of a team as a whole. Development is a communal activity, and the entire team either succeeds or fails as a unit. Someone else committed a bug (and it got merged all the way to deployment!) that brought down the system? It's because you didn't provide the necessary support, teamwork, and help with engineering your peer needed to succeed. What can you do to help them succeed next time? After all, everyone is human and makes mistakes. What's important is that the people are interlocked in layers, where one person can catch another's mistake (without punishing the person who made a mistake, because that just incentivizes them to hide them!) and help fix it before the code reaches client machines. Successes may be individual, but failure is always the system's fault, never an individuals. 2. Almost all technical bugs, in any field, are because of leaky abstractions and implicit assumptions. From abstract ones like mathematics to physical ones like carpentry. Be explicit. About every possible detail. And if you think you're being too explicit, you probably forgot 80% of the cases. Ever see the average house blueprint? Puts software engineering design to shame, easily. If you want to build something that runs as reliably as a highly tuned, expertly designed engine, you can't start with anything less than as detailed of a specification as they use. Be explicit. Once you get to the point where you're working in genuine engineering teams rather than as individual engineers on a team and you have explicit, detailed specifications, the technologies to actually convert that reliably in to software that runs stably are pretty straight-forward. The reason we don't see this all the time is simply that it's expensive: the politics require a lot of redundancy of time spent (eg, code has to be read several times by several people); the explicitness requires a lot more upfront planning and effort in documentation, which requires more time invested per unit of actual coding; etc. Of course, much like we have building codes for houses and larger structures, I think it's perfectly fair to expect minimal standards from software engineers. (Especially now that, eg, IoT botnets are DoSing things.)

nickpsecurity 10y ago
Appreciate your write-up. It all sounds great. I'd say the tech to go from detailed specs to reliable or secure systems isn't necessarily straight-forward except in the easy cases. It can take some serious work by smart people, esp if it's formal verification or spotting unknowns. We can easily get 90% of the way there, though, without the hard stuff just by people giving a shit and doing stuff like you said. Of course, Cleanroom Software Engineering that does a lot of what you said without as much formal stuff as Praxis's method. Thing you should remember in these conversations is to point out to other party that what you recommend and what these methods did knocks out tons of debugging and maintenance costs. Since both phases cost labor, with maintenance fixes multiples of development cost, there were many cases where Cleanroom projects actually cost less since they knocked out huge issues early on. Can't bank on that being normal with it often being 20-30% or so more. It was around the same or less cost in many case studies, though, due to knocking out problems earlier in development. Btw, I found a nice write-up on Cleanroom without tons of math or formality with case studies on college students if you're interested in references like that: http://infohost.nmt.edu/~al/cseet-paper.html http://infohost.nmt.edu/~al/cseet-paper.html I like dropping them on people in regular development in these discussions when they say you can't engineer software, it takes ridiculous skill, or it would cost what NASA's team spends (rolls eyes). Eye opening experience for some.
- SomeStupidPoint 10y ago
  > I'd say the tech to go from detailed specs to reliable or secure systems isn't necessarily straight-forward except in the easy cases. It can take some serious work by smart people, esp if it's formal verification or spotting unknowns. I was being a little facetious. Of course the technical work is highly complex -- some of the brightest minds of our time work on DARPA related projects on foundational mathematics related to automated theorem proving. (Looking at you HoTT crew and related projects!) Why is DARPA funding automated theorem proving? Because they want to create high assurance software for government infrastructure to counter the threat of AI-based cyberwarfare, and our current mathematical techniques aren't up to muster. However, we've made tremendous progress on that problem in the span of around 100 years. By contrast, the problem of how to not incentivize your workers to do a shitty job, causing you problems later on... has been with us, I think it's safe to say, for thousands of years. (And it too, has attracted some of the brightest minds over the years.) So in a relative way, the technical aspects are "straight-forward" to address, compared to the underlying political problem. And the more technical methods only add some assurance that you've correctly implemented the spec as defined, not that you're doing the right thing. It's certainly good to, eg, check that you're using total functions or not misassigning memory, but it's not magic. So while they catch a lot of dangerous problems, they can also lead to false confidence about the existence of other classes of problems. > they say you can't engineer software, it takes ridiculous skill, or it would cost what NASA's team spends It mostly just requires that we consistently act with discipline and professionalism, which are both tiring compared to not doing them, so by and large, people just don't bother. I know I don't when I can get away with not doing it (even if I know that's a bad habit).

nickpsecurity 10y ago

Appreciate your write-up. It all sounds great. I'd say the tech to go from detailed specs to reliable or secure systems isn't necessarily straight-forward except in the easy cases. It can take some serious work by smart people, esp if it's formal verification or spotting unknowns. We can easily get 90% of the way there, though, without the hard stuff just by people giving a shit and doing stuff like you said. Of course, Cleanroom Software Engineering that does a lot of what you said without as much formal stuff as Praxis's method. Thing you should remember in these conversations is to point out to other party that what you recommend and what these methods did knocks out tons of debugging and maintenance costs. Since both phases cost labor, with maintenance fixes multiples of development cost, there were many cases where Cleanroom projects actually cost less since they knocked out huge issues early on. Can't bank on that being normal with it often being 20-30% or so more. It was around the same or less cost in many case studies, though, due to knocking out problems earlier in development. Btw, I found a nice write-up on Cleanroom without tons of math or formality with case studies on college students if you're interested in references like that: http://infohost.nmt.edu/~al/cseet-paper.html http://infohost.nmt.edu/~al/cseet-paper.html I like dropping them on people in regular development in these discussions when they say you can't engineer software, it takes ridiculous skill, or it would cost what NASA's team spends (rolls eyes). Eye opening experience for some.

SomeStupidPoint 10y ago
> I'd say the tech to go from detailed specs to reliable or secure systems isn't necessarily straight-forward except in the easy cases. It can take some serious work by smart people, esp if it's formal verification or spotting unknowns. I was being a little facetious. Of course the technical work is highly complex -- some of the brightest minds of our time work on DARPA related projects on foundational mathematics related to automated theorem proving. (Looking at you HoTT crew and related projects!) Why is DARPA funding automated theorem proving? Because they want to create high assurance software for government infrastructure to counter the threat of AI-based cyberwarfare, and our current mathematical techniques aren't up to muster. However, we've made tremendous progress on that problem in the span of around 100 years. By contrast, the problem of how to not incentivize your workers to do a shitty job, causing you problems later on... has been with us, I think it's safe to say, for thousands of years. (And it too, has attracted some of the brightest minds over the years.) So in a relative way, the technical aspects are "straight-forward" to address, compared to the underlying political problem. And the more technical methods only add some assurance that you've correctly implemented the spec as defined, not that you're doing the right thing. It's certainly good to, eg, check that you're using total functions or not misassigning memory, but it's not magic. So while they catch a lot of dangerous problems, they can also lead to false confidence about the existence of other classes of problems. > they say you can't engineer software, it takes ridiculous skill, or it would cost what NASA's team spends It mostly just requires that we consistently act with discipline and professionalism, which are both tiring compared to not doing them, so by and large, people just don't bother. I know I don't when I can get away with not doing it (even if I know that's a bad habit).