8 ms·

How does one even begin to engineer tests for such a sequence of events? Yeah you can test each step individually, but then you have all the integration effects

by kortex 5y ago

How does one even begin to engineer tests for such a sequence of events? Yeah you can test each step individually, but then you have all the integration effects. How do you know you have enough coverage? $10B buys a lot of QA and I'm sure they try to engineer everything with the right margins, but it's still an unfathomable amount of state space. Are there techniques to stay sane and manage risk without just throwing money at it? I feel like that kind of knowledge could be useful for software test development.

wolverine876 5y ago
Simple: NASA Systems Engineering Handbook https://ntrs.nasa.gov/citations/20170001761 https://ntrs.nasa.gov/citations/20170001761 That's the 2017 version; maybe there's a later one. IIRC, it's an abridged from NASA Expanded Guidance for SE, but my link to that is broken.
- qwertyuiop_ 5y ago
  Wait no Agile ?
  - wolverine876 5y ago
    Did you read it all? ;)
double0jimb0 5y ago
Systems Engineering is the discipline that oversees this. They define what tests will be required to validate the thing will do what it is supposed to before any hardware is built. I don’t think there is a good analogy to typical software QA, which is usually a “make sure it doesn’t break anything that already works” type of discipline.
jdiez17 5y ago
There's no real silver bullet other than applying the systems engineering process diligently. You start by writing down your user requirements (what the system needs to deliver), and you follow the thread of figuring out that "to do X, this subsystem has to provide conditions A, B, C..." recursively, in a breadth-first search. The level of detail codified in these functional, performance and interface requirements depends on the level of assurance you need. Then, you need to validate that each requirement is met by your system. This can be done by test, analysis (mathematically proving some property), review of design, or inspection. It's true that you can't fully validate most space systems on Earth, because we can't simulate all environmental conditions simultaneously. That's why you ideally you want each requirement to be validated by two methods. When you find anomalies due to integration effects, it's usually because your interface requirements are not specified well enough ;)
- peterburkimsher 5y ago
  I agree, it's a recursive search! Translated into software testing: "level of detail codified in functional, performance and interface requirements" functions, usage frequency, APIs. "usually because your interface requirements are not specified well enough" It's probably a bug in the API. https://martinfowler.com/bliki/TwoHardThings.html https://martinfowler.com/bliki/TwoHardThings.html "There are only two hard things in Computer Science: cache invalidation, naming things, and off-by-one errors" https://en.wikipedia.org/wiki/Ctags https://en.wikipedia.org/wiki/Ctags Suggestion: use ctags to list all functions, variable names in your code. Look for ambiguity (e.g. variable name "i"). Look at neighbouring code. Zoom in and out. A small bug in the most-used code is actually more serious than a big bug in code that rarely gets used. "How long can you work on making a routine task before you're spending more time than you save?" https://xkcd.com/1205/ https://xkcd.com/1205/
- ausbah 5y ago
  this level of rigor always makes me snicker at the engineering in "software engineering"
  - markus_zhang 5y ago
    You can probably snicker at most engineering in "X engineering".
  - jl6 5y ago
    This is how you are supposed to build software too.
    - danielheath 5y ago
      Who supposed that? Broken software can be fixed cheaply after the fact. Yeah, it’s cheaper if you find the bugs earlier but it shouldn’t come as any surprise that pre-validation is more extensive in systems that are expensive to change.
      - s1artibartfast 5y ago
peterburkimsher 5y ago
You gave the answer! Integration tests. And they work recursively, with a Kalman filter to approximate even in noisy conditions. "USL was inspired by Hamilton's recognition of patterns or categories of errors occurring during Apollo software development. Errors at the interfaces between subsystem boundaries accounted for the majority of errors and were often the most subtle and most difficult to find. Each interface error was placed into a category identifying the means to prevent it by way of system definition. This process led to a set of six axioms, forming the basis for a mathematical constructive logical theory of control for designing systems that would eliminate entire classes of errors just by the way a system is defined.[3][4]" https://en.wikipedia.org/wiki/Universal_Systems_Language https://en.wikipedia.org/wiki/Universal_Systems_Language There's a diagram of rules on the USL Wikipedia page. The rules show triangle feedback loops with a parent, left, right child. Those are like generations of a Sierpiński triangle. Every part is trying to serve the Good Cause that it's working for, and love its neighbour. https://en.wikipedia.org/wiki/Minimal_realization https://en.wikipedia.org/wiki/Minimal_realization "any state-space model that is both controllable and observable and has the same input-output behaviour as the transfer function is said to be a minimal realization of the transfer function The realization is called "minimal" because it describes the system with the minimum number of states." https://en.wikipedia.org/wiki/Optimal_control https://en.wikipedia.org/wiki/Optimal_control "the problem of driving the output to a desired nonzero level can be solved after the zero output one is." An electronic analogy: find GND, then solve for 1. A common solution strategy in many optimal control problems is to solve for the costate (sometimes called the shadow price) A shadow price is a monetary value assigned to currently unknowable or difficult-to-calculate costs in the absence of correct market prices. It is based on the willingness to pay principle – the most accurate measure of the value of a good or service is what people are willing to give up in order to get it. The costate summarizes in one number the marginal value of expanding or contracting the state variable next turn. Each part looks ahead 1 generation, chooses left or right. https://en.wikipedia.org/wiki/Kalman_decomposition https://en.wikipedia.org/wiki/Kalman_decomposition convert a representation of any linear time-invariant (LTI) control system to a form in which the system can be decomposed into a standard form which makes clear the observable and controllable components of the system Take a big problem, break it down, look for I/O ports. Or in software test development: layers of abstraction. A suggestion: only add a layer of abstraction when it's too big to fit on the screen at once. Use tools like code folding, tree views. https://en.wikipedia.org/wiki/Optimal_control https://en.wikipedia.org/wiki/Optimal_control Optimise for time? When we're in a hurry we break things. Another suggestion: aim to minimise entropy, maximise connectedness. Thank you for asking a good question, and thank you for reading! Let's go and tidy up this world together, in software and hardware.
smarx007 5y ago
I would start with https://en.m.wikipedia.org/wiki/V-Model https://en.m.wikipedia.org/wiki/V-Model. System designs of everything in automotive, aerospace etc are based on a V model.
- Nicksil 5y ago
  https://en.wikipedia.org/wiki/V-Model https://en.wikipedia.org/wiki/V-Model
- hef19898 5y ago
  As with all good systematic approaches, Lean and Six Sigma fall in the same category, the V-model is great. It is also, basically, just codified common sense, and that codification is key and incredible important. And as with Lean and Six Sigma, people can apply the substance of it without rigidly following the form (read: checking boxes in process description) and be fine. Or they can can follow the form without the substance and be fine on paper and still produce crap systems engineering. Kind of like the old the saying: Inspection ready troops don't pass combat, combat ready troops don't pass inspection.
jacquesm 5y ago
That's the difference between engineering and 'move fast and break stuff'. Let's hope it all works out, if not, some expensive lessons are about to be learned.
virtue3 5y ago
The really scary thing is that because it's in L2 orbit (past the moon) it's not designed/intended to ever be serviced. So they can't go up and unfuck it if it's fucked.
- danny_codes 5y ago
  At least not until ~2023 or so, when Starship is ready.
  - virtue3 5y ago
    You are -severely- underestimating how big space is. roughly 30 days How long will it take Webb to get to L2? It will take roughly 30 days for Webb to reach the start of its orbit at L2, but it will take only 3 days to get as far away as the Moon's orbit, which is about a quarter of the way there.
  - randmeerkat 5y ago
    Or 2028, Musk has never set a deadline he hasn’t blown straight past.
    - danny_codes 5y ago
      That is definitely true. On the other hand, 2028 is better than the promises coming from the other option, SLS, which I believe will be ready never.
im_down_w_otp 5y ago
That kind of exhaustive integrated/system-level testing is precisely the kind of thing we're trying to enable at Auxon, FWIW. A really rough sketch, for sake of brevity, of the approach is cribbing from property-based testing, mutation testing, and model checking and applying them to system & software interactions instead of programs, functions, and source code. Our users are often Systems Engineers by title or more often by circumstance.

by kortex 5y ago

wolverine876 5y ago

Simple: NASA Systems Engineering Handbook https://ntrs.nasa.gov/citations/20170001761 https://ntrs.nasa.gov/citations/20170001761 That's the 2017 version; maybe there's a later one. IIRC, it's an abridged from NASA Expanded Guidance for SE, but my link to that is broken.

qwertyuiop_ 5y ago
Wait no Agile ?
- wolverine876 5y ago
  Did you read it all? ;)

double0jimb0 5y ago

Systems Engineering is the discipline that oversees this. They define what tests will be required to validate the thing will do what it is supposed to before any hardware is built. I don’t think there is a good analogy to typical software QA, which is usually a “make sure it doesn’t break anything that already works” type of discipline.

jdiez17 5y ago

There's no real silver bullet other than applying the systems engineering process diligently. You start by writing down your user requirements (what the system needs to deliver), and you follow the thread of figuring out that "to do X, this subsystem has to provide conditions A, B, C..." recursively, in a breadth-first search. The level of detail codified in these functional, performance and interface requirements depends on the level of assurance you need. Then, you need to validate that each requirement is met by your system. This can be done by test, analysis (mathematically proving some property), review of design, or inspection. It's true that you can't fully validate most space systems on Earth, because we can't simulate all environmental conditions simultaneously. That's why you ideally you want each requirement to be validated by two methods. When you find anomalies due to integration effects, it's usually because your interface requirements are not specified well enough ;)

peterburkimsher 5y ago
I agree, it's a recursive search! Translated into software testing: "level of detail codified in functional, performance and interface requirements" functions, usage frequency, APIs. "usually because your interface requirements are not specified well enough" It's probably a bug in the API. https://martinfowler.com/bliki/TwoHardThings.html https://martinfowler.com/bliki/TwoHardThings.html "There are only two hard things in Computer Science: cache invalidation, naming things, and off-by-one errors" https://en.wikipedia.org/wiki/Ctags https://en.wikipedia.org/wiki/Ctags Suggestion: use ctags to list all functions, variable names in your code. Look for ambiguity (e.g. variable name "i"). Look at neighbouring code. Zoom in and out. A small bug in the most-used code is actually more serious than a big bug in code that rarely gets used. "How long can you work on making a routine task before you're spending more time than you save?" https://xkcd.com/1205/ https://xkcd.com/1205/
ausbah 5y ago
this level of rigor always makes me snicker at the engineering in "software engineering"
- markus_zhang 5y ago
  You can probably snicker at most engineering in "X engineering".
- jl6 5y ago
  This is how you are supposed to build software too.
  - danielheath 5y ago
    Who supposed that? Broken software can be fixed cheaply after the fact. Yeah, it’s cheaper if you find the bugs earlier but it shouldn’t come as any surprise that pre-validation is more extensive in systems that are expensive to change.
    - s1artibartfast 5y ago

peterburkimsher 5y ago

You gave the answer! Integration tests. And they work recursively, with a Kalman filter to approximate even in noisy conditions. "USL was inspired by Hamilton's recognition of patterns or categories of errors occurring during Apollo software development. Errors at the interfaces between subsystem boundaries accounted for the majority of errors and were often the most subtle and most difficult to find. Each interface error was placed into a category identifying the means to prevent it by way of system definition. This process led to a set of six axioms, forming the basis for a mathematical constructive logical theory of control for designing systems that would eliminate entire classes of errors just by the way a system is defined.[3][4]" https://en.wikipedia.org/wiki/Universal_Systems_Language https://en.wikipedia.org/wiki/Universal_Systems_Language There's a diagram of rules on the USL Wikipedia page. The rules show triangle feedback loops with a parent, left, right child. Those are like generations of a Sierpiński triangle. Every part is trying to serve the Good Cause that it's working for, and love its neighbour. https://en.wikipedia.org/wiki/Minimal_realization https://en.wikipedia.org/wiki/Minimal_realization "any state-space model that is both controllable and observable and has the same input-output behaviour as the transfer function is said to be a minimal realization of the transfer function The realization is called "minimal" because it describes the system with the minimum number of states." https://en.wikipedia.org/wiki/Optimal_control https://en.wikipedia.org/wiki/Optimal_control "the problem of driving the output to a desired nonzero level can be solved after the zero output one is." An electronic analogy: find GND, then solve for 1. A common solution strategy in many optimal control problems is to solve for the costate (sometimes called the shadow price) A shadow price is a monetary value assigned to currently unknowable or difficult-to-calculate costs in the absence of correct market prices. It is based on the willingness to pay principle – the most accurate measure of the value of a good or service is what people are willing to give up in order to get it. The costate summarizes in one number the marginal value of expanding or contracting the state variable next turn. Each part looks ahead 1 generation, chooses left or right. https://en.wikipedia.org/wiki/Kalman_decomposition https://en.wikipedia.org/wiki/Kalman_decomposition convert a representation of any linear time-invariant (LTI) control system to a form in which the system can be decomposed into a standard form which makes clear the observable and controllable components of the system Take a big problem, break it down, look for I/O ports. Or in software test development: layers of abstraction. A suggestion: only add a layer of abstraction when it's too big to fit on the screen at once. Use tools like code folding, tree views. https://en.wikipedia.org/wiki/Optimal_control https://en.wikipedia.org/wiki/Optimal_control Optimise for time? When we're in a hurry we break things. Another suggestion: aim to minimise entropy, maximise connectedness. Thank you for asking a good question, and thank you for reading! Let's go and tidy up this world together, in software and hardware.

smarx007 5y ago

I would start with https://en.m.wikipedia.org/wiki/V-Model https://en.m.wikipedia.org/wiki/V-Model. System designs of everything in automotive, aerospace etc are based on a V model.

Nicksil 5y ago
https://en.wikipedia.org/wiki/V-Model https://en.wikipedia.org/wiki/V-Model
hef19898 5y ago
As with all good systematic approaches, Lean and Six Sigma fall in the same category, the V-model is great. It is also, basically, just codified common sense, and that codification is key and incredible important. And as with Lean and Six Sigma, people can apply the substance of it without rigidly following the form (read: checking boxes in process description) and be fine. Or they can can follow the form without the substance and be fine on paper and still produce crap systems engineering. Kind of like the old the saying: Inspection ready troops don't pass combat, combat ready troops don't pass inspection.

jacquesm 5y ago

That's the difference between engineering and 'move fast and break stuff'. Let's hope it all works out, if not, some expensive lessons are about to be learned.

virtue3 5y ago

The really scary thing is that because it's in L2 orbit (past the moon) it's not designed/intended to ever be serviced. So they can't go up and unfuck it if it's fucked.

danny_codes 5y ago
At least not until ~2023 or so, when Starship is ready.
- virtue3 5y ago
  You are -severely- underestimating how big space is. roughly 30 days How long will it take Webb to get to L2? It will take roughly 30 days for Webb to reach the start of its orbit at L2, but it will take only 3 days to get as far away as the Moon's orbit, which is about a quarter of the way there.
- randmeerkat 5y ago
  Or 2028, Musk has never set a deadline he hasn’t blown straight past.
  - danny_codes 5y ago
    That is definitely true. On the other hand, 2028 is better than the promises coming from the other option, SLS, which I believe will be ready never.

im_down_w_otp 5y ago

That kind of exhaustive integrated/system-level testing is precisely the kind of thing we're trying to enable at Auxon, FWIW. A really rough sketch, for sake of brevity, of the approach is cribbing from property-based testing, mutation testing, and model checking and applying them to system & software interactions instead of programs, functions, and source code. Our users are often Systems Engineers by title or more often by circumstance.