3 ms·
I figured with good annotations by programmers and Design-by-Contract the manual or automated testing would be easy enough that there could be a low number of t
by nickpsecurity 10y ago
I figured with good annotations by programmers and Design-by-Contract the manual or automated testing would be easy enough that there could be a low number of testers vs developers. 1:1 surprises me unless you're counting each developer writing their own basic tests or whatever. I'd be interested in hearing more about what tools or practices you use given I continuously survey safety-critical development practices.
- Jtsummers 10y ago1:1.5 tester:developer may be more accurate. We don't use a lot of good practices in my current office. I wish we did. We use C and fortran, one guy insists on C++ (which he calls C). We used to (before I came onto this project) do much better with using static analysis tools and good mapping of: (requirements <-> specification <-> design <-> code) <-> tests. The reason this stopped, people retired. The people that came in to replace them didn't understand the value of good practices and tools for those things because, well, they were awesome rockstar coders. And they are. But they assume everyone else is too. Which is a fatal mistake. They're terrible at management, but good engineers. A major problem, as well, is that we're on the maintenance end of the system. These aren't fresh projects or clean slate projects so we're stuck with the historical code (effectively). And too much of the initial (generally good) practices and processes didn't survive the transition. I've read a number of your posts, usually when I see your handle I make a point to read them regardless of how long they are. And sometimes I realize it's you just by the content and look for more in a thread. In principle I think we're on the same page. In practice, I made some expedient choices for myself that have kept me from being able to (on the job) make extensive use of these ideas for a while. One other reason, though, that we'd likely keep the same ratio is just the nature of what we test here. Radios with an esoteric messaging protocol. It's only used by us and a handful of others. A test case may involve dozens or hundreds of key messages to simulate a "conversation" between two or more radios to determine how our software behaves, with thousands more sent automatically. It's made more complex by the (IMO) relatively poorly designed protocol. It was satisfactory, maybe even good, originally but has become overly complex and possibly contradictory in its specification as new features were added over the years. Which is where better design (not just code development) would be nice. I started on, but never finished, going through all the messages that could be sent and what the responses ought to be and when to try to produce a better simulator than we presently use. One of my goals was to identify where the specification was contradictory, but I haven't been given time and I'm not willing to consider it after work hours. Another goal was to make it easier, with such a high level functional model/simulator, to develop some of our other systems that rely on this same protocol (or some subset of it). Something to verify/validate our implementation against. EDIT: In a past life I worked with LDRA as our primary static analysis tool for C. We had much better processes in place to track the relationships between spec, requirements, test procedures and code (we used Telelogic DOORS). On personal projects exploring these ideas I used Ada, and wanted to use SPARK Ada but found the documentation at the time to be sparse online, and my personal time waning so I dropped it.
- nickpsecurity 10y ago"The reason this stopped, people retired." Sorry to hear that. I'm short on suggestions for that one given I focus on getting tools adopted more than re-adopted. I'll have to look into that scenario a bit given I bet it plays out a lot. "I've read a number of your posts, usually when I see your handle I make a point to read them regardless of how long they are. And sometimes I realize it's you just by the content and look for more in a thread. In principle I think we're on the same page" Appreciate the kind words. I work hard to get the quality stuff out there to people like you. Hard to assess how it's being received. Comments like that and occasional emails are what I go on for impact assessment. "A test case may involve dozens or hundreds of key messages to simulate a "conversation" between two or more radios to determine how our software behaves, with thousands more sent automatically. " Ohhhh. Ok. I'm already seeing it based on prior experience testing both protocols and UI's. Yeah, that would take a lot of testing if specs were out of date or insufficient. With specs, you could use design-by-contract on it to support testing with interface checks and/or auto-generate tests depending on tooling. I mean, the Fortran static analysis tooling might be behind a bit... (read: nonexistent). "but never finished, going through all the messages that could be sent and what the responses ought to be and when to try to produce a better simulator than we presently use" Are you actually able to change the protocol in arbitrary ways so long as product functions properly in some other black box your company can control? Or do you have to maintain its existing, functional behavior for backward compatibility? If the former, you can gradually spec it out then maybe apply a protocol generator. If the latter, it might stay pretty manual with DbC, testing, and/or static analysis being best you're going to do. "In a past life I worked with LDRA " Oh, you got to work with one of the real ones. Good for you. I haven't been able to afford them so I keep trying to get comments on how effective or usable they are. Did it catch stuff well? Make it fairly easy to do requirements/spec/code consistency or lots of redundancy for user? On the latter, point is that high-assurance development requires a number of documents that all describe the same thing. Essentially views on the operation of the system. How easily a tool can keep them together and/or propagate changes in one to another is important to avoid people saying "screw it: i'll update that later [never]." So, I'm curious about its usability. Still no consensus on tools for that with Topcased being an open one some went with, some using traditional SCM w/ specs/docs in it, some HTML (eg LISP spec), and Karger et al on Caernarvon used Framemaker with lots of cross-links. "we used Telelogic DOORS" Could've been interesting but IBM acquired it for Rational suite. Nevermind. "wanted to use SPARK Ada but found the documentation at the time to be sparse online" Might be worth re-exploring given they've done a lot over the years. SPARK 2014 was quite an update. Almost nobody does anything with it but there is a DNS server written in it. Might contain insight on applying SPARK to protocol design/verification. Not to mention its other, potential benefits. :) http://ironsides.martincarlisle.com/ http://ironsides.martincarlisle.com/