4 ms·

Interesting they seem to ad-hoc the lifecycle processes, or at least no mention of any industry standards mandatory (RTCA/DO-178C, ISO26262 or even sys eng like

Interesting they seem to ad-hoc the lifecycle processes, or at least no mention of any industry standards mandatory (RTCA/DO-178C, ISO26262 or even sys eng like SAE ARP4754, etc). Also, no talk of formal methods (formal modeling or formal verification, a la DO-333 with something like Simulink Design Verf or TLA+). Works for them obviously!

procarch2019 5y ago
I’m not sure that I took this away from the article. Looked like they just didn’t go into detail. They mentioned storing data for keeping track of equipment health. To me that sounds like a super broad stroke that would be equivalent to historized time series data as well as contextualized asset data. I’m not familiar with the standards you called out (not an aerospace engineer), but I would imagine it’s similar to what we do in the controls engineering and automation space. I’m actually really curious about what software/methods they use to contextualize assets and what those data points are. How do They digitize the equipment data that comes from visual inspections/manual testing. I wish it was a little more detailed in general.
- icegreentea2 5y ago
  I am quite interested to see how they deal with regulations and standards (I'm in medical device space). I scanned through their job postings and I can't find a single standard listed (this is up to the Sr Engineer/Developer positions), as well as a lack of dedicated software QA/test positions. Obviously they are doing some amazing, and quite high quality work, so I am quite interested in what their internal setup is like.
  - procarch2019 5y ago
    Yes, considering how fast they move I’m wondering if it’s closer to self regulated/peer quality control. Some QA/QC can make work come to a screeching halt in biopharma space (makes sense when a “batch” in the cell and gene therapy space is a single person and their wellbeing). It’s not like regulation and quality have not fallen by the wayside or been too relaxed for other aerospace companies, especially in recent history. Quality should be meaningful and not just a rubber stamp. It’s a fine line
    - icegreentea2 5y ago
      Even in a self-regulated/peer quality control context, I think it's quite amazing to not have people tasked for dedicated (or more focused towards) testing and QA, even within the software/engineering group. I wonder how developer/engineers are expected to split their focus between design and development of their own areas of responsibilities, and doing peer review and testing of other people's areas of responsibilities. One of the dangers of this setup is always that people (and their managers) prioritize output of their own areas, resulting in neglecting testing and review of other areas. It's certainly possible to make the setup, but it takes good people (and team cohesion/morale) up and down the chain to make this work. I definitely agree that it's a fine line.
      - procarch2019 5y ago
        I’ll preface this with “I think I’m good people.” Context switching is hard especially when understaffed and most certainly when doing different types of work. I’m still “executing” engineering projects, but I’m also performing inside sales functions, helping advance and maintain our dev systems, and now manage 2 functional groups. Unfortunately stuff does fall by the way side all the time. Prioritizing things is key, but at the end of the day there is a capacity to what one can do. It takes good management skills to realize the capacity of the employees and when things need to change. Luckily for me I’m working with groups in all three of the areas I was working in to offload those extra responsibilities to other people so I can focus on what I’m supposed to be doing (managing). It would be interesting to see how they operate.
        ethbr0 5y ago
        From seeing this work and not work at different companies, I've decided at root it's a company culture issue. Specifically, "How does my company deal with failure?" Which is another way of saying "Are people disincentivized from telling the truth at my company?" You can never have a process that requires truth-to-power as a regular occurrence that's successful in an environment where there are career penalties to speaking uncomfortable truths. That's why legacy providers & government have engineered a system that accepts +50% wasted time, in exchange for not requiring truth because almost everything is checked and verified. Good culture: We failed. I succeeded. Bad culture: You failed. I succeeded.
V_Terranova_Jr 5y ago
It's been ~a decade since I was knowledgeable in SpaceX software development details, but at least then, they were not the kind of shop that emphasized rigor and formality. Their culture is oriented toward rapid implementation & test, pragmatism, and empiricism. That has its pluses and minuses. My personal view is that more of the industry, including SpaceX, should be moving toward engineering things to be more correct-by-construction, including formal modeling & verification as you mention. As one example, I don't hear most aerospace firms talking about SeL4 for their flight computer RTOSes. For another, you might be surprised (frightened?) by how many vehicles implement their flight software as big single address space executables written in C (sometimes C++) despite running on modern superscalar processors with MMUs. Even static analysis in some of these codebases is relatively recent - they are heavily dependent on functional testing for safety.
- timClicks 5y ago
  SeL4 is now mostly defunct. The whole team was laid off during a pivot towards AI and blockchain.
  - V_Terranova_Jr 5y ago
    - I don't think SeL4 is "defunct" even though CSIRO's decision on the Trustworthy Systems group was profoundly incompetent; here's a relatively recent post from Gernot Heiser: https://microkerneldude.org/2021/09/28/where-is-sel4-heading/ https://microkerneldude.org/2021/09/28/where-is-sel4-heading... Also: https://sel4.systems/news/2021 https://sel4.systems/news/2021 Do you know more that would substantiate defunct-ness? - Your comment also reinforces my point: if aerospace seeks to be more rigorous, then more in aerospace should be figuring out how to shore up development of SeL4. Instead, they continue with no vision as to what should one day replace things like VxWorks.
- sam_bristow 5y ago
  Are the properties that have been proven for seL4 particularly relevant for avionics systems? I thought they mainly focussed on process isolation and security. A flight computer would be much more concerned about deterministic performance and real-time guarantees.

procarch2019 5y ago

I’m not sure that I took this away from the article. Looked like they just didn’t go into detail. They mentioned storing data for keeping track of equipment health. To me that sounds like a super broad stroke that would be equivalent to historized time series data as well as contextualized asset data. I’m not familiar with the standards you called out (not an aerospace engineer), but I would imagine it’s similar to what we do in the controls engineering and automation space. I’m actually really curious about what software/methods they use to contextualize assets and what those data points are. How do They digitize the equipment data that comes from visual inspections/manual testing. I wish it was a little more detailed in general.

icegreentea2 5y ago
I am quite interested to see how they deal with regulations and standards (I'm in medical device space). I scanned through their job postings and I can't find a single standard listed (this is up to the Sr Engineer/Developer positions), as well as a lack of dedicated software QA/test positions. Obviously they are doing some amazing, and quite high quality work, so I am quite interested in what their internal setup is like.
- procarch2019 5y ago
  Yes, considering how fast they move I’m wondering if it’s closer to self regulated/peer quality control. Some QA/QC can make work come to a screeching halt in biopharma space (makes sense when a “batch” in the cell and gene therapy space is a single person and their wellbeing). It’s not like regulation and quality have not fallen by the wayside or been too relaxed for other aerospace companies, especially in recent history. Quality should be meaningful and not just a rubber stamp. It’s a fine line
  - icegreentea2 5y ago
    Even in a self-regulated/peer quality control context, I think it's quite amazing to not have people tasked for dedicated (or more focused towards) testing and QA, even within the software/engineering group. I wonder how developer/engineers are expected to split their focus between design and development of their own areas of responsibilities, and doing peer review and testing of other people's areas of responsibilities. One of the dangers of this setup is always that people (and their managers) prioritize output of their own areas, resulting in neglecting testing and review of other areas. It's certainly possible to make the setup, but it takes good people (and team cohesion/morale) up and down the chain to make this work. I definitely agree that it's a fine line.
    - procarch2019 5y ago
      I’ll preface this with “I think I’m good people.” Context switching is hard especially when understaffed and most certainly when doing different types of work. I’m still “executing” engineering projects, but I’m also performing inside sales functions, helping advance and maintain our dev systems, and now manage 2 functional groups. Unfortunately stuff does fall by the way side all the time. Prioritizing things is key, but at the end of the day there is a capacity to what one can do. It takes good management skills to realize the capacity of the employees and when things need to change. Luckily for me I’m working with groups in all three of the areas I was working in to offload those extra responsibilities to other people so I can focus on what I’m supposed to be doing (managing). It would be interesting to see how they operate.
      - ethbr0 5y ago
        From seeing this work and not work at different companies, I've decided at root it's a company culture issue. Specifically, "How does my company deal with failure?" Which is another way of saying "Are people disincentivized from telling the truth at my company?" You can never have a process that requires truth-to-power as a regular occurrence that's successful in an environment where there are career penalties to speaking uncomfortable truths. That's why legacy providers & government have engineered a system that accepts +50% wasted time, in exchange for not requiring truth because almost everything is checked and verified. Good culture: We failed. I succeeded. Bad culture: You failed. I succeeded.

V_Terranova_Jr 5y ago

It's been ~a decade since I was knowledgeable in SpaceX software development details, but at least then, they were not the kind of shop that emphasized rigor and formality. Their culture is oriented toward rapid implementation & test, pragmatism, and empiricism. That has its pluses and minuses. My personal view is that more of the industry, including SpaceX, should be moving toward engineering things to be more correct-by-construction, including formal modeling & verification as you mention. As one example, I don't hear most aerospace firms talking about SeL4 for their flight computer RTOSes. For another, you might be surprised (frightened?) by how many vehicles implement their flight software as big single address space executables written in C (sometimes C++) despite running on modern superscalar processors with MMUs. Even static analysis in some of these codebases is relatively recent - they are heavily dependent on functional testing for safety.

timClicks 5y ago
SeL4 is now mostly defunct. The whole team was laid off during a pivot towards AI and blockchain.
- V_Terranova_Jr 5y ago
  - I don't think SeL4 is "defunct" even though CSIRO's decision on the Trustworthy Systems group was profoundly incompetent; here's a relatively recent post from Gernot Heiser: https://microkerneldude.org/2021/09/28/where-is-sel4-heading/ https://microkerneldude.org/2021/09/28/where-is-sel4-heading... Also: https://sel4.systems/news/2021 https://sel4.systems/news/2021 Do you know more that would substantiate defunct-ness? - Your comment also reinforces my point: if aerospace seeks to be more rigorous, then more in aerospace should be figuring out how to shore up development of SeL4. Instead, they continue with no vision as to what should one day replace things like VxWorks.
sam_bristow 5y ago
Are the properties that have been proven for seL4 particularly relevant for avionics systems? I thought they mainly focussed on process isolation and security. A flight computer would be much more concerned about deterministic performance and real-time guarantees.