7 ms·

Software Checklist

112 points by happyscrappy 13y ago

menzoic 13y ago
Isn't this what static analysis, penetration testing, and various other automated smoke tests made for? Instead of adding to a checklist these tests are updated to account for new vulnerabilities.
- jacques_chester 13y ago
  Checklists are just another tool. Where automated tools are available, they should be used. Where they aren't, we need other tools.
- krschultz 13y ago
  The checklist is about making sure you have used those tools. This is not how I would recommend running the average startup, but for mission critical software having someone other than the author review all the code using a checklist is probably the best way to ensure quality. The author of the code would have the same checklist and would be expected to hit all the points before handing the code over to the reviewer.
- epo 13y ago
  Automated tests are automated checklists.
nmrm 13y ago
This is essentially describing a "process". Except instead of emphasizing team-level details, it calls for an emphasis on code-level details. I really like this idea.
- jacques_chester 13y ago
  Watts Humphreys was probably the first to the idea of scaling development processes down, rather than up, arriving in the 90s at the Personal Software Process. The PSP is very data-intensive. You're meant to log pretty much everything you do. For example, you log your mistakes. Missed a semicolon? Log the mistake. Typo in a variable name? Log the mistake. Critical misunderstanding of how a library works? Log the mistake. After a while you use your data to correct your common errors with ... checklists! And other tools and practices, of course, but checklists are a big part of the PSP. It's worth reading A Discipline for Software Engineering, even if you never apply it.
humancontact 13y ago
I use this very frequently, helps a lot: http://www.boxuk.com/upload/2014/02/Relaunched-Ultimate-Website-Checklist-2.0.pdf http://www.boxuk.com/upload/2014/02/Relaunched-Ultimate-Webs...
Qworg 13y ago
Using a system like this really helps for complicated and/or important code. All of our safety critical and embedded code references a checklist before it is made into a pull request. It really helps catch all of the "rubber ducky" bugs.
nextos 13y ago
I think he's missing an important point. C should not be used for safety critical systems code if possible, period. A language that prevents those leaks should be employed in its place. Checklists are good, but you need to enforce them. If such a language is used instead, these errors are impossible by design.
- ColinWright 13y ago
  I think you're missing the point. Whether or not we're talking about C is a detail, what is more important is that the lessons we learn need to be encapsulated and encoded into procedures that everyone can apply and learn from. Equivalent mistakes can be made in any language powerful enough to do real work. Off-by-one, behaving inappropriately on parameters outside those expected, lots of things can go wrong in any language. Capturing your knowledge in a form that can be used for testing is the idea, and that seems to be missing entirely from computing.
  - jacques_chester 13y ago
    > lots of things can go wrong in any language Some languages, by design, exclude or drastically reduce certain classes of error. Edit: I think I'm missing the point too. Proposing the perfect world as an engineering solution is silly. So in the context of "it's written in C", checklists would be a step forward.
  - taejo 13y ago
    > what is more important is that the lessons we learn need to be encapsulated and encoded into procedures that everyone can apply and learn from. But in software, we can do better than that! We don't have to check a checklist ourselves if we can make a tool that makes it impossible to do the wrong thing (or rather, we don't have to have that thing on the checklist). Some places do have checklists for software engineering, if not software. For example, a simplified process for deploying code might be: 1. Check that code compiles 2. Run test suite 3. Get code reviewed 4. Push to production And having such a list does help to ensure that code doesn't get pushed to production without being tested and reviewed. But it's better to use a system which automatically runs the tests and checks that it's been reviewed before the code goes into production (unfortunately the review itself can't be automated -- though parts of it can be, and that's useful too).
cammil 13y ago
I think this is a great idea.
macobo 13y ago
Can anyone point to the studies done about using checklists in surgeries?
- deleted 13y ago
  [deleted]
- ColinWright 13y ago
  A quick Google search[0] returns these: http://news.bbc.co.uk/1/hi/health/7825780.stm http://news.bbc.co.uk/1/hi/health/7825780.stm http://www.nrls.npsa.nhs.uk/resources/?entryid45=59860 http://www.nrls.npsa.nhs.uk/resources/?entryid45=59860 http://www.who.int/patientsafety/safesurgery/ss_checklist/en/ http://www.who.int/patientsafety/safesurgery/ss_checklist/en... http://www.bmj.com/content/338/bmj.b220 http://www.bmj.com/content/338/bmj.b220 [0] https://www.google.com/search?q=surgery+checklists https://www.google.com/search?q=surgery+checklists
  - mcguire 13y ago
    You should probably also mention this: http://theincidentaleconomist.com/wordpress/when-checklists-work-and-when-they-dont/ http://theincidentaleconomist.com/wordpress/when-checklists-... Atul Gawande is probably the foremost proponent of surgical checklists and in that article is responding to an Ontario study reporting that checklists were not significant---it sounds like a pretty bad study. Surgeons don't particularly like checklists. "For starters, the most controversial idea for teams to accept is perhaps the simplest item in the checklist. Require all team members say their names prior to the launch of the procedure. "'This has been one of the most important things that help people feel comfortable speaking up' if they're unsure or unclear, for example, that this is the right patient, right site, right procedure. "'It acknowledges that you're part of a team and are allowed to speak.' "Gawande says that there has been resistance to accepting checklists at another level. 'The concept has forced us rethink what it means to be great at what we do. And I hadn't grasped this until I saw it recur over and over again. There's a set of values in the idea of a checklist, and they're in distinct conflict with some of the values we have in medicine.' "'We value physician autonomy, which works well when there are just two full time equivalents providing care, but when we have 19.5 FTEs trying to make things work, it becomes a problem.'"[1] Any of that sounding familiar? [1] http://www.healthleadersmedia.com/page-3/QUA-262159/Gawande-on-Checklists-Why-Dont-Hospitals-Use-Them http://www.healthleadersmedia.com/page-3/QUA-262159/Gawande-...
deleted 13y ago
[deleted]
leemcalilly 13y ago
With software this should be your test suite. No test is too simple. No feature or detail too small to test. Replace the line "relentless checklist" with "relentless testing" and you're good to go. Don't deploy before all tests are green and that's the same as not flying without running through your checklist. I grew up flying with my dad in his Cesna 172 and admiring how well the pre-flight checklist worked. When I first heard about TDD, I immediately recognized it as the same process, except automated.
- ColinWright 13y ago
  That's what I first thought. Then I realized that test suites as most people think of them would not have caught the Heartbleed bug, whereas a checklist like this would have done so. Testing is testing, but there are different ways of thinking about it, and different ways to have it embodied. Test suites are one embodiment, and higher-level checklists can complement them.
  - leemcalilly 13y ago
    I guess I've always thought of my integration tests as the higher-level checklist. I've only done web development, so that works. I could see how if you're writing software for something as important as OpenSSL then you might need an extra set of checks, but there's still room for human error. Seems like there should be someway to automate everything required to deploy software. But I realize this is an idealistic approach. The reality is that a good test suite with higher-level checklists is probably more practical.
  - judk 13y ago
    If your checklist says "memcpy is forbidden", and you encode that in a static analyer (test), heardbleed would be prevented.
rybosome 13y ago
My current team is the first one in my career as a software engineer in which our testing infrastructure is robust enough to think like this. I just resolved a bug this week by writing a test that failed due to its presence (and leaving a comment with the ID of the bug), changing the code, then re-running the test to ensure that it passed. This confidence in our code base is wonderful; I have never felt so little fear when deploying. Looking back, it amazes me that I ever didn't develop software this way.
- jnbiche 13y ago
  May I ask what language this code base is in?
  - vinceguidry 13y ago
    Probably Ruby. No other language makes testing that easy or fun.
    - bennyg 13y ago
      The new XCTest in Objective-C is fairly nice as well.
    - burntsushi 13y ago
      Go does. So does Rust. Both languages have some sort of unit testing facilities packaged with the standard distribution. And both are dead simple to use.
      - AYBABTME 13y ago
        You can't mock out your dependencies as easily in Go as you can monkey patch them in Ruby. However I've found the explicit error handling in Go to be more valuable in making reliable software than a torrent of test in Ruby, or Java.
- goblin89 13y ago
  Can you expound on the kind of infrastructure you have in mind? In my experience it's enough to have some tests and a documented way to run them. (Poorly written tests, though, is another story. Having to spend an hour figuring that out in order to write a new case isn't fun.)
hardwaresofton 13y ago
Tried to make something similar to this once: http://konvention.vadosware.com http://konvention.vadosware.com Suffered from the false premise that people would willingly contribute en masse though...
- icefox 13y ago
  On the main page the link for "Copy arrays with array#slice" is 404
  - hardwaresofton 13y ago
    Yeah, I haven't touched the site in what feels like forever...unfortunately, don't have the time to fill it up with information... I might return to it one day
dano 13y ago
Using wiki's in two different companies provided us with these checklist like capabilities. We used MoinMoin at a prior company and now use MediaWiki, but the technology isn't the point. Everyone contributes to the wiki and it accrues knowledge from experience. Some process documentation is carefully reviewed for accuracy, but it also left a bit free-form so as to accommodate situation specific variations from the norm. The verification team (I dislike the term QA) encapsulates as much as possible into the testing framework, but since the wiki based descriptions are so much more rich, they remain a go to resource. Recently I moved the IT/Networking staff to start making their project plans in the wiki. Their defect rate has plummeted to nearly zero - mostly because they are making plans! But what they've discovered is that they now have frameworks of plans (checklists) from which they can derive their next plan. Others have mentioned Dr. Gawande and I would recommend his book http://atulgawande.com/book/the-checklist-manifesto/ http://atulgawande.com/book/the-checklist-manifesto/ Good discussion. Don't have an internal wiki? Start one, you'll never go back.
- greenyoda 13y ago
  You can also read Gawande's original article in the New Yorker magazine that his book was based on: http://www.newyorker.com/reporting/2007/12/10/071210fa_fact_gawande?currentPage=all http://www.newyorker.com/reporting/2007/12/10/071210fa_fact_...
brudgers 13y ago
The optimist's response to "What could possibly go wrong?" is a swing for the fences. The pessimist's response is a checklist. It's seeking to avoid the worst outcomes and allowing success to come in due time. It's braking early and accelerating from the apex as opposed going full speed and breaking things fast.
- jacques_chester 13y ago
  This seems like a fallacy of false dilemmas. The optimist can swing harder because, thanks to professional pessimism, the bat won't crack halfway through the swing, the stands don't collapse before the ball reaches them, the ball doesn't explode on contact, and the rules for what constitutes a home run don't have to be constantly reinvented by each player individually.
  - brudgers 13y ago
    Yes, we agree that the optimist's imagination limits the list of things that could possibly go wrong to the absurd. And as you suggest perhaps an optimist's optimism derives from a concept of professionalism born the love child of dunning-kruger. Or perhaps the horns are grown on a strawman dividing the world into naught but optimists and pessimists according to the law of the excluded muddle.
mherrmann 13y ago
I find the comparison of software development and surgery/flying an air plane very lacking. It restricts the term "Software" in the post title to very critical systems (OpenSSL admittedly being one of them). The author proposes automated tests/Software "checklists" as the solution to all problems. The problem with automated tests is that they can only catch "expected" bugs. The author describes an idea for an automated test that _might_ have caught the Heartbleed bug. Big deal, now that we know all about the bug. A next unforeseen type of vulnerability will come, and no automated test will catch it. I think the article greatly exaggerates the applicability and capabilities of automated tests (or "check lists"). I'm a big fan of all things TDD and consider myself a very rigorous programmer (surely often to the detriment of speed/pragmaticism). But I don't need a check list of the complexity of a flight check to describe the rules I use in my work.
- deleted 13y ago
  [deleted]
- taejo 13y ago
  > A next unforeseen type of vulnerability will come, and no automated test will catch it. But Heartbleed wasn't a new type of vulnerability. It's the same type of vulnerability we've known about at least since the dawn of C. Almost all bugs are of known types. How often do we see new types of vulnerabilities? Maybe a couple a year. If we had only a couple vulnerabilities a year across all software, we'd be many orders of magnitude better off.
  - mherrmann 13y ago
    Sure, I'm not saying the proposed automated test is bad. I think it might be sensible to add it. As I said, I'm 100% for automated tests (I should be, as the co-founder of a startup specializing in test automation tools). But ascribing magical capabilities to catch all future bugs to automated tests / "check lists" is IMHO detrimental to the cause because it leads to too high expectations.
    - scott_w 13y ago
icefox 13y ago
Whenever I introduce a bug that could have been caught automatically I either add the missing unit test or I add a new git hook (there is of course a hook to run unit tests). From missing license files, build failures, to just spelling mistakes in the commit message there is no reason for these to exists in the repository. I created https://github.com/icefox/git-hooks https://github.com/icefox/git-hooks to manage hooks across repositories, users, and system wide. For example if in a project I add a new hook making sure I never commit an invalid xml file, all repos that have xml files will get the hook for free.
- lifeisstillgood 13y ago
  I am loving the look of the git-books - got to try it tomorrow
aegiso 13y ago
Thing is, software doesn't happen in a vacuum. Perhaps it's worth considering that maybe the reason that checklists aren't the norm in the FOSS "meritocracy" is that they hinder progress, for a certain value of progress. Maybe there was a stealth project that could have been OpenSSL, developed with scrict adherance to checklists, but OpenSSL won because it didn't have that burden? I suspect this applies more broadly to startups, too. Maybe checklists are a silent killer in the natural selection of the software ecosystem, and that's why so much of our software is tripping over peacock feathers? Just a thought.
goblin89 13y ago
> This can be automated! This could be mandated as a procedure to be gone through before the code is released. And doing this would prevent this kind of bug from ever happening again. I might be missing something, but this sounds to me very much like a usual test case. :) Having one in a suite of system or acceptance tests would indeed prevent further occurrences of the bug.
qwerta 13y ago
I tend to use checklists for everything. It saves a lot of thinking. I would recommend Zim or other personal wiki.
andyl 13y ago
Software already has checklists - using names like 'regression tests', 'automation scripts', etc.
JoachimSchipper 13y ago
Checklists have been proven to work, but the software I write (low-level high-security/crypto software) is less repetitive than (most) surgeries. I haven't managed to use anything checklist-like while writing code; however, quickly pattern-matching against a mental library has been exceedingly useful when _reviewing_ code. Here are some sources I would recommend to people building their own mental checklists: - The Art of Software Security Assessment (thanks to tptacek); I actually learnt most of this from bits 'n pieces scattered over the web, but it's a very convenient collection; - Common crypto flaws. Always look for authentication, _then_ encryption, _and_ replay protection. (The latter is often forgotten even in otherwise good protocols.) If a password is used anywhere, consider bruteforcing it (bcrypt may stop such attacks - or not.) Check that IVs are used as demanded by the mode (in particular, that CTR or GCM IVs are not reused.) Look for timing attacks; the easy-to-spot and high-impact ones are use of memcmp() with HMACs and use of CBC on unauthenticated data (padding oracle). - Buffer overflows may be bad, but integer under-/overflows are much more common (in decent code.) Basically, you need a check before any operation involving a user-supplied length, and this is very commonly forgotten. (Using functions with internal overflow checking, like a decent calloc, helps.) - Consider the wider context. Spotting a buffer overflow is easy. Noting that you forgot to <? include auth.php ?> is harder. You likely won't notice that the latest "cleanup" leaves the firmware updater open to the internet unless you actually spend five seconds thinking about the wider system. (I'm quite interested in other's tips; the above is clearly very slanted. Also, I feel vaguely guilty about doing all of this in memory instead of from a paper checklist - does anyone have positive or other experiences with those?)
matthewmacleod 13y ago
What I object to here is the core principle that there's something to fix in software development terms here. There's not. We've got robust, reliable tools for creating nearly bug-free software. We have the usual engineering principles; we have automated testing; we have virtual machines and documented environments and specifications and APIs and all the rest. The problem is, there is a cost associated with preventing software bugs, and it's a big one. There's a reason Facebook kept saying "move fast and break things" - their software was buggy as shit, but the cost of that was minimal; in situations where bugs don't matter as much, time invested in robust bug-free software is an undesirable expense. In the Heartbleed case, this is reversed. There's a highly-sensitive, critical bit of Internet infrastructure; there have been minimal resources invested into it, but it is relied on by a lot of key systems. So the problem is a "social" one, in a sense. OpenSSL as a project did not invest enough in reducing the occurrence of bugs, and this resulted in the vulnerability. It's not the maintainers' fault, really; it was irresponsible for so many of us to rely on the software without considering it's quality. I suspect we'll start seeing renewed focus on bits of important software like OpenSSL, and greater attention to the engineering tradeoffs. So there's an upside to the whole thing...
- gcv 13y ago
  > We have the usual engineering principles No. We don't. As Alan Kay said, programmers don't have a solid concept like an arch to build on. In real engineering, when you build a bridge, a single rusty rivet won't cause it to fall down. In software, a single pair of misplaced braces give you critical "goto fail" bugs. (And don't fool yourself into thinking that static analysis catches all these errors.) It has become fashionable to say that automated tests somehow turn programming into a rigorous discipline. This is bullshit. Tests are a decent band-aid which kinda help sometimes if you spend half your programming time writing them. And will still miss plenty of horrible bugs. In truth, programming remains a poorly-understood endeavor. Checklists as auto-written automatic tests might actually be a step in the right direction, especially if compilers start to analyze the kinds of mistakes individual programmers tend to make.
  - derefr 13y ago
halayli 13y ago
Rather than asking more from the developers, donate more money to the project and get more developers to test the software.

112 points by happyscrappy 13y ago

Isn't this what static analysis, penetration testing, and various other automated smoke tests made for? Instead of adding to a checklist these tests are updated to account for new vulnerabilities.

jacques_chester 13y ago
Checklists are just another tool. Where automated tools are available, they should be used. Where they aren't, we need other tools.
krschultz 13y ago
The checklist is about making sure you have used those tools. This is not how I would recommend running the average startup, but for mission critical software having someone other than the author review all the code using a checklist is probably the best way to ensure quality. The author of the code would have the same checklist and would be expected to hit all the points before handing the code over to the reviewer.
epo 13y ago
Automated tests are automated checklists.

nmrm 13y ago

This is essentially describing a "process". Except instead of emphasizing team-level details, it calls for an emphasis on code-level details. I really like this idea.

jacques_chester 13y ago
Watts Humphreys was probably the first to the idea of scaling development processes down, rather than up, arriving in the 90s at the Personal Software Process. The PSP is very data-intensive. You're meant to log pretty much everything you do. For example, you log your mistakes. Missed a semicolon? Log the mistake. Typo in a variable name? Log the mistake. Critical misunderstanding of how a library works? Log the mistake. After a while you use your data to correct your common errors with ... checklists! And other tools and practices, of course, but checklists are a big part of the PSP. It's worth reading A Discipline for Software Engineering, even if you never apply it.

humancontact 13y ago

I use this very frequently, helps a lot: http://www.boxuk.com/upload/2014/02/Relaunched-Ultimate-Website-Checklist-2.0.pdf http://www.boxuk.com/upload/2014/02/Relaunched-Ultimate-Webs...

Qworg 13y ago

Using a system like this really helps for complicated and/or important code. All of our safety critical and embedded code references a checklist before it is made into a pull request. It really helps catch all of the "rubber ducky" bugs.

nextos 13y ago

I think he's missing an important point. C should not be used for safety critical systems code if possible, period. A language that prevents those leaks should be employed in its place. Checklists are good, but you need to enforce them. If such a language is used instead, these errors are impossible by design.

ColinWright 13y ago
I think you're missing the point. Whether or not we're talking about C is a detail, what is more important is that the lessons we learn need to be encapsulated and encoded into procedures that everyone can apply and learn from. Equivalent mistakes can be made in any language powerful enough to do real work. Off-by-one, behaving inappropriately on parameters outside those expected, lots of things can go wrong in any language. Capturing your knowledge in a form that can be used for testing is the idea, and that seems to be missing entirely from computing.
- jacques_chester 13y ago
  > lots of things can go wrong in any language Some languages, by design, exclude or drastically reduce certain classes of error. Edit: I think I'm missing the point too. Proposing the perfect world as an engineering solution is silly. So in the context of "it's written in C", checklists would be a step forward.
- taejo 13y ago
  > what is more important is that the lessons we learn need to be encapsulated and encoded into procedures that everyone can apply and learn from. But in software, we can do better than that! We don't have to check a checklist ourselves if we can make a tool that makes it impossible to do the wrong thing (or rather, we don't have to have that thing on the checklist). Some places do have checklists for software engineering, if not software. For example, a simplified process for deploying code might be: 1. Check that code compiles 2. Run test suite 3. Get code reviewed 4. Push to production And having such a list does help to ensure that code doesn't get pushed to production without being tested and reviewed. But it's better to use a system which automatically runs the tests and checks that it's been reviewed before the code goes into production (unfortunately the review itself can't be automated -- though parts of it can be, and that's useful too).

cammil 13y ago

I think this is a great idea.

macobo 13y ago

Can anyone point to the studies done about using checklists in surgeries?

deleted 13y ago
[deleted]
ColinWright 13y ago
A quick Google search[0] returns these: http://news.bbc.co.uk/1/hi/health/7825780.stm http://news.bbc.co.uk/1/hi/health/7825780.stm http://www.nrls.npsa.nhs.uk/resources/?entryid45=59860 http://www.nrls.npsa.nhs.uk/resources/?entryid45=59860 http://www.who.int/patientsafety/safesurgery/ss_checklist/en/ http://www.who.int/patientsafety/safesurgery/ss_checklist/en... http://www.bmj.com/content/338/bmj.b220 http://www.bmj.com/content/338/bmj.b220 [0] https://www.google.com/search?q=surgery+checklists https://www.google.com/search?q=surgery+checklists
- mcguire 13y ago
  You should probably also mention this: http://theincidentaleconomist.com/wordpress/when-checklists-work-and-when-they-dont/ http://theincidentaleconomist.com/wordpress/when-checklists-... Atul Gawande is probably the foremost proponent of surgical checklists and in that article is responding to an Ontario study reporting that checklists were not significant---it sounds like a pretty bad study. Surgeons don't particularly like checklists. "For starters, the most controversial idea for teams to accept is perhaps the simplest item in the checklist. Require all team members say their names prior to the launch of the procedure. "'This has been one of the most important things that help people feel comfortable speaking up' if they're unsure or unclear, for example, that this is the right patient, right site, right procedure. "'It acknowledges that you're part of a team and are allowed to speak.' "Gawande says that there has been resistance to accepting checklists at another level. 'The concept has forced us rethink what it means to be great at what we do. And I hadn't grasped this until I saw it recur over and over again. There's a set of values in the idea of a checklist, and they're in distinct conflict with some of the values we have in medicine.' "'We value physician autonomy, which works well when there are just two full time equivalents providing care, but when we have 19.5 FTEs trying to make things work, it becomes a problem.'"[1] Any of that sounding familiar? [1] http://www.healthleadersmedia.com/page-3/QUA-262159/Gawande-on-Checklists-Why-Dont-Hospitals-Use-Them http://www.healthleadersmedia.com/page-3/QUA-262159/Gawande-...

deleted 13y ago

[deleted]

leemcalilly 13y ago

With software this should be your test suite. No test is too simple. No feature or detail too small to test. Replace the line "relentless checklist" with "relentless testing" and you're good to go. Don't deploy before all tests are green and that's the same as not flying without running through your checklist. I grew up flying with my dad in his Cesna 172 and admiring how well the pre-flight checklist worked. When I first heard about TDD, I immediately recognized it as the same process, except automated.

ColinWright 13y ago
That's what I first thought. Then I realized that test suites as most people think of them would not have caught the Heartbleed bug, whereas a checklist like this would have done so. Testing is testing, but there are different ways of thinking about it, and different ways to have it embodied. Test suites are one embodiment, and higher-level checklists can complement them.
- leemcalilly 13y ago
  I guess I've always thought of my integration tests as the higher-level checklist. I've only done web development, so that works. I could see how if you're writing software for something as important as OpenSSL then you might need an extra set of checks, but there's still room for human error. Seems like there should be someway to automate everything required to deploy software. But I realize this is an idealistic approach. The reality is that a good test suite with higher-level checklists is probably more practical.
- judk 13y ago
  If your checklist says "memcpy is forbidden", and you encode that in a static analyer (test), heardbleed would be prevented.

rybosome 13y ago

My current team is the first one in my career as a software engineer in which our testing infrastructure is robust enough to think like this. I just resolved a bug this week by writing a test that failed due to its presence (and leaving a comment with the ID of the bug), changing the code, then re-running the test to ensure that it passed. This confidence in our code base is wonderful; I have never felt so little fear when deploying. Looking back, it amazes me that I ever didn't develop software this way.

jnbiche 13y ago
May I ask what language this code base is in?
- vinceguidry 13y ago
  Probably Ruby. No other language makes testing that easy or fun.
  - bennyg 13y ago
    The new XCTest in Objective-C is fairly nice as well.
  - burntsushi 13y ago
    Go does. So does Rust. Both languages have some sort of unit testing facilities packaged with the standard distribution. And both are dead simple to use.
    - AYBABTME 13y ago
      You can't mock out your dependencies as easily in Go as you can monkey patch them in Ruby. However I've found the explicit error handling in Go to be more valuable in making reliable software than a torrent of test in Ruby, or Java.
goblin89 13y ago
Can you expound on the kind of infrastructure you have in mind? In my experience it's enough to have some tests and a documented way to run them. (Poorly written tests, though, is another story. Having to spend an hour figuring that out in order to write a new case isn't fun.)

hardwaresofton 13y ago

Tried to make something similar to this once: http://konvention.vadosware.com http://konvention.vadosware.com Suffered from the false premise that people would willingly contribute en masse though...

icefox 13y ago
On the main page the link for "Copy arrays with array#slice" is 404
- hardwaresofton 13y ago
  Yeah, I haven't touched the site in what feels like forever...unfortunately, don't have the time to fill it up with information... I might return to it one day

dano 13y ago

Using wiki's in two different companies provided us with these checklist like capabilities. We used MoinMoin at a prior company and now use MediaWiki, but the technology isn't the point. Everyone contributes to the wiki and it accrues knowledge from experience. Some process documentation is carefully reviewed for accuracy, but it also left a bit free-form so as to accommodate situation specific variations from the norm. The verification team (I dislike the term QA) encapsulates as much as possible into the testing framework, but since the wiki based descriptions are so much more rich, they remain a go to resource. Recently I moved the IT/Networking staff to start making their project plans in the wiki. Their defect rate has plummeted to nearly zero - mostly because they are making plans! But what they've discovered is that they now have frameworks of plans (checklists) from which they can derive their next plan. Others have mentioned Dr. Gawande and I would recommend his book http://atulgawande.com/book/the-checklist-manifesto/ http://atulgawande.com/book/the-checklist-manifesto/ Good discussion. Don't have an internal wiki? Start one, you'll never go back.

greenyoda 13y ago
You can also read Gawande's original article in the New Yorker magazine that his book was based on: http://www.newyorker.com/reporting/2007/12/10/071210fa_fact_gawande?currentPage=all http://www.newyorker.com/reporting/2007/12/10/071210fa_fact_...

brudgers 13y ago

The optimist's response to "What could possibly go wrong?" is a swing for the fences. The pessimist's response is a checklist. It's seeking to avoid the worst outcomes and allowing success to come in due time. It's braking early and accelerating from the apex as opposed going full speed and breaking things fast.

jacques_chester 13y ago
This seems like a fallacy of false dilemmas. The optimist can swing harder because, thanks to professional pessimism, the bat won't crack halfway through the swing, the stands don't collapse before the ball reaches them, the ball doesn't explode on contact, and the rules for what constitutes a home run don't have to be constantly reinvented by each player individually.
- brudgers 13y ago
  Yes, we agree that the optimist's imagination limits the list of things that could possibly go wrong to the absurd. And as you suggest perhaps an optimist's optimism derives from a concept of professionalism born the love child of dunning-kruger. Or perhaps the horns are grown on a strawman dividing the world into naught but optimists and pessimists according to the law of the excluded muddle.

mherrmann 13y ago

I find the comparison of software development and surgery/flying an air plane very lacking. It restricts the term "Software" in the post title to very critical systems (OpenSSL admittedly being one of them). The author proposes automated tests/Software "checklists" as the solution to all problems. The problem with automated tests is that they can only catch "expected" bugs. The author describes an idea for an automated test that _might_ have caught the Heartbleed bug. Big deal, now that we know all about the bug. A next unforeseen type of vulnerability will come, and no automated test will catch it. I think the article greatly exaggerates the applicability and capabilities of automated tests (or "check lists"). I'm a big fan of all things TDD and consider myself a very rigorous programmer (surely often to the detriment of speed/pragmaticism). But I don't need a check list of the complexity of a flight check to describe the rules I use in my work.

deleted 13y ago
[deleted]
taejo 13y ago
> A next unforeseen type of vulnerability will come, and no automated test will catch it. But Heartbleed wasn't a new type of vulnerability. It's the same type of vulnerability we've known about at least since the dawn of C. Almost all bugs are of known types. How often do we see new types of vulnerabilities? Maybe a couple a year. If we had only a couple vulnerabilities a year across all software, we'd be many orders of magnitude better off.
- mherrmann 13y ago
  Sure, I'm not saying the proposed automated test is bad. I think it might be sensible to add it. As I said, I'm 100% for automated tests (I should be, as the co-founder of a startup specializing in test automation tools). But ascribing magical capabilities to catch all future bugs to automated tests / "check lists" is IMHO detrimental to the cause because it leads to too high expectations.
  - scott_w 13y ago

icefox 13y ago

Whenever I introduce a bug that could have been caught automatically I either add the missing unit test or I add a new git hook (there is of course a hook to run unit tests). From missing license files, build failures, to just spelling mistakes in the commit message there is no reason for these to exists in the repository. I created https://github.com/icefox/git-hooks https://github.com/icefox/git-hooks to manage hooks across repositories, users, and system wide. For example if in a project I add a new hook making sure I never commit an invalid xml file, all repos that have xml files will get the hook for free.

lifeisstillgood 13y ago
I am loving the look of the git-books - got to try it tomorrow

aegiso 13y ago

Thing is, software doesn't happen in a vacuum. Perhaps it's worth considering that maybe the reason that checklists aren't the norm in the FOSS "meritocracy" is that they hinder progress, for a certain value of progress. Maybe there was a stealth project that could have been OpenSSL, developed with scrict adherance to checklists, but OpenSSL won because it didn't have that burden? I suspect this applies more broadly to startups, too. Maybe checklists are a silent killer in the natural selection of the software ecosystem, and that's why so much of our software is tripping over peacock feathers? Just a thought.

goblin89 13y ago

> This can be automated! This could be mandated as a procedure to be gone through before the code is released. And doing this would prevent this kind of bug from ever happening again. I might be missing something, but this sounds to me very much like a usual test case. :) Having one in a suite of system or acceptance tests would indeed prevent further occurrences of the bug.

qwerta 13y ago

I tend to use checklists for everything. It saves a lot of thinking. I would recommend Zim or other personal wiki.

andyl 13y ago

Software already has checklists - using names like 'regression tests', 'automation scripts', etc.

JoachimSchipper 13y ago

Checklists have been proven to work, but the software I write (low-level high-security/crypto software) is less repetitive than (most) surgeries. I haven't managed to use anything checklist-like while writing code; however, quickly pattern-matching against a mental library has been exceedingly useful when _reviewing_ code. Here are some sources I would recommend to people building their own mental checklists: - The Art of Software Security Assessment (thanks to tptacek); I actually learnt most of this from bits 'n pieces scattered over the web, but it's a very convenient collection; - Common crypto flaws. Always look for authentication, _then_ encryption, _and_ replay protection. (The latter is often forgotten even in otherwise good protocols.) If a password is used anywhere, consider bruteforcing it (bcrypt may stop such attacks - or not.) Check that IVs are used as demanded by the mode (in particular, that CTR or GCM IVs are not reused.) Look for timing attacks; the easy-to-spot and high-impact ones are use of memcmp() with HMACs and use of CBC on unauthenticated data (padding oracle). - Buffer overflows may be bad, but integer under-/overflows are much more common (in decent code.) Basically, you need a check before any operation involving a user-supplied length, and this is very commonly forgotten. (Using functions with internal overflow checking, like a decent calloc, helps.) - Consider the wider context. Spotting a buffer overflow is easy. Noting that you forgot to <? include auth.php ?> is harder. You likely won't notice that the latest "cleanup" leaves the firmware updater open to the internet unless you actually spend five seconds thinking about the wider system. (I'm quite interested in other's tips; the above is clearly very slanted. Also, I feel vaguely guilty about doing all of this in memory instead of from a paper checklist - does anyone have positive or other experiences with those?)

matthewmacleod 13y ago

What I object to here is the core principle that there's something to fix in software development terms here. There's not. We've got robust, reliable tools for creating nearly bug-free software. We have the usual engineering principles; we have automated testing; we have virtual machines and documented environments and specifications and APIs and all the rest. The problem is, there is a cost associated with preventing software bugs, and it's a big one. There's a reason Facebook kept saying "move fast and break things" - their software was buggy as shit, but the cost of that was minimal; in situations where bugs don't matter as much, time invested in robust bug-free software is an undesirable expense. In the Heartbleed case, this is reversed. There's a highly-sensitive, critical bit of Internet infrastructure; there have been minimal resources invested into it, but it is relied on by a lot of key systems. So the problem is a "social" one, in a sense. OpenSSL as a project did not invest enough in reducing the occurrence of bugs, and this resulted in the vulnerability. It's not the maintainers' fault, really; it was irresponsible for so many of us to rely on the software without considering it's quality. I suspect we'll start seeing renewed focus on bits of important software like OpenSSL, and greater attention to the engineering tradeoffs. So there's an upside to the whole thing...

gcv 13y ago
> We have the usual engineering principles No. We don't. As Alan Kay said, programmers don't have a solid concept like an arch to build on. In real engineering, when you build a bridge, a single rusty rivet won't cause it to fall down. In software, a single pair of misplaced braces give you critical "goto fail" bugs. (And don't fool yourself into thinking that static analysis catches all these errors.) It has become fashionable to say that automated tests somehow turn programming into a rigorous discipline. This is bullshit. Tests are a decent band-aid which kinda help sometimes if you spend half your programming time writing them. And will still miss plenty of horrible bugs. In truth, programming remains a poorly-understood endeavor. Checklists as auto-written automatic tests might actually be a step in the right direction, especially if compilers start to analyze the kinds of mistakes individual programmers tend to make.
- derefr 13y ago

halayli 13y ago

Rather than asking more from the developers, donate more money to the project and get more developers to test the software.