10 ms·
How programmers make sure that their software is correct
- philk10 5y agoInteresting that the 2 books referenced at the end are from 2008 and 2012 (I read them when I was learning more about testing) - nothing new since then?
- pelasaco 5y agoI really liked it, and I will probably use it as documentation for newcomers in my team. However there is one point that I don't really agree: "it can be difficult to add tests to a large application that has been developed entirely without testing. And that is why testing should be first on your mind when building serious software." I've being doing it all my life. Join a new project, without any test, and start to add test code, while getting used with the code base. Actually, I would say that "adding tests" for a code base is the easiest and safest way to get used with the code base itself.
- cassianoleal 5y agoAs far as I'm aware, unless your software is Hello World simple, the only way to make sure it's correct is to not write it in the first place. After that point, everything is a compromise.
- mellavora 5y agoWell, my software is always correct. Sometimes the spec is a bit wrong, like forgetting to be explicit in precisely how the system should crash on a certain unexpected input.
- ChrisMarshallNY 5y agoGreat answer!
- unionpivo 5y ago> ... Hello World simple End even then only if you stick to English strig hello world. I have had hello worlds fail, in other languages :)
- benttoothpaste 5y agoHello world definitely can fail even in English. It's just that nobody checks to make sure the print operation succeeded.
- junon 5y agoFormalized verification would say otherwise.
- pydry 5y agoAt its worst, formalized verification is an elaborate and overcomplicated process for converting coding bugs into specification bugs.
- junon 5y agoThis is a huge copout and isn't at all correct. It dismisses a bunch of insanely intelligent peoples' life works.
- odonnellryan 5y agoCan't you use this same argument when being critical about any technology, or really anything complex at all? It is some kind of logical fallacy - just because really smart people worked on something and dedicated their lives to it doesn't mean it is good or correct or that it works.
- junon 5y agoThe point was more the parent comment was dismissing the work clearly without understanding any of it. They employed a sound byte to disclaim something. You're welcome to go read the prose on the subject. Parent commenter clearly hasn't.
- pydry 5y agoIt's a wry observation that I made when I was forced to learn it by those very same researchers who hyped it up as the future of software development 20 years ago. It was sold as the "rolls royce of testing" which I think overstated its capabilities and effectiveness.
- junon 5y ago
- ChrisMarshallNY 5y agoI write UI and communication software. It is pretty much impossible to write meaningful tests that cover everything. This is especially true, when you mix complex, asynchronous server interactions, with random-access UI. Today, I am fixing a couple of bugs in an app that interacts with three different servers (all asynchronously, but some semi-synchronously. The timing diagram is a bit ... intense), and also allows the user to do things like switch contexts, while some part of the server interaction is still unresolved. I write a bit about my approach to testing, here: https://littlegreenviper.com/miscellany/testing-harness-vs-unit/ https://littlegreenviper.com/miscellany/testing-harness-vs-u...
- monkeyjoe 5y agoI started reading through your site a bit. Great content. Wanted to let you know that the "Here is the Doxygen-generated documentation for the BADGER layer" link is broken on this page: https://littlegreenviper.com/miscellany/forensic-design-documentation/ https://littlegreenviper.com/miscellany/forensic-design-docu...
- ChrisMarshallNY 5y agoHey, thanks! I'll look at that.
- commandlinefan 5y agoYou can try, though - and demonstrate that you did. I've seen bugs that would have been trivially caught if the software had been run even one time - so it was clear that the software had never been run even one time.
- ohwellhere 5y agoThis is my favorite. Doesn't even compile, or crashes immediately, and yet... it's somehow in a PR.
- klyrs 5y agoI've done that. Spend days testing a patch, getting everything just right. Then, as you're doing final touchup, you introduce a syntax error with a wrong keystroke and fail to notice before pushing to the PR branch. But, of course, this is why we use CI.
- gitgrump 5y agoI've had "senior engineers" and engineering directors submit PRs that just flat out don't work. Like, I ran a minimum working example and it just... didn't work. I swear to God, some people are actually professionally negligent at this point.
- codingdave 5y agoYes, tests are good. At the same time, my measuring stick is always, "Is the customer happy?" Software will never be perfect, so a zero bug rate is not a reasonable goal. Happy customers is quite a reasonable goal, and when they are not, their concerns direct where you need more quality.
- jeffreygoesto 5y agoTell that to NASA or your local nuclear powerstation provider.
- deleted 5y ago[deleted]
- buscoquadnary 5y agoIt's funny you say that because the next thing I pictured was Homer Simpson and realized that thousands of brilliant engineers from all walks of life may eventually have all their effort wasted because someone left their station for donut day.
- mhh__ 5y agoGood engineering means building systems that can survive Homer Simpson.
- codingdave 5y agoIt still holds true - their standards to reach the level of "happy" will clearly be more stringent than others. So will hospitals or any customer whose work can make a life or death difference. But they still care about the result of the product, and they set the standard you need to hit.
- jeffreygoesto 5y agoYou are right. Thank you for being kind and constructive even though I was snarky and did not fully get that you in fact did not exclude these cases.
- bondarchuk 5y ago.. they don't!
- zokier 5y agoI find odd that there was no discussion about figuring out edge cases and testing specifically for those instead of random values. Especially for functions with large domains, having few hand-picked test values could give much more confidence than thousands of random values. Additionally with generated test values figuring out the validation logic becomes complex, and in complexity lies potential for bugs. If your testing code more complex than the code being tested, do you really gain that much confidence about the correctness? In comparison with hard-coded test values you can also hard-code the expected results, which can be gotten from some reference or just manually checked for correctness. Of course ideally you'd have both random fuzzing and more manually defined tests, but that is setting quite high bar.
- Quekid5 5y agoThat's par for the course if you use an actual framework for Property-based Testing. All the ones I've tried certainly do mix in the usual suspects (-1, 0, 1, MAX_INT, empty list, etc.) into the randomly sampled data. The author of this piece probably chose to make the post framework-agnostic by rolling their own random generation, but left that nuance out.
- pydry 5y agoI find it interesting that while logic bugs (e.g. off by one errors, switched if conditions, etc) make up less than 1/3 of bugs I encounter day to day, it seems to comprise of about 80% of all discussion around testing. Bugs caused by misunderstood or unexpected interactions between subsystems and specification bugs seem to predominate these days IME but discussion about sophisticated ways of dealing with these things seems fairly scant.
- smugglerFlynn 5y agoI guess it depends on a system you build and its maturity - in my area (automation for finance and revenue management) 80% of bugs are related to business logic and misinterpretation of user requirements.
- marginalia_nu 5y agoErrors are highly domain specific. When I worked in finance, I would have very much agreed with your description. But then, the code I wrote was fairly trivial, the challenge was communicating with the users and sometimes getting the users to understand what they were asking for wasn't what they wanted. I, the same developer, spent a good hour recently trying to iron out the bugs in a piece of code that concatenated two integers bit-wise and calculated the offset point. That is, given two integers whose bits are 00000000,00000000,00000000,0PPPPPPP 00000000,000000QQ,QQQQQQQQ,QQQQQQQQ The code should produce the shortest an array of bytes that looked like ZZZZZZZZ,0000000P,PPPPPPQQ,QQQQQQQQ,QQQQQQQQ Where Z in this case would be 18, the offset of P's bits. This is finicky in a way I've never encountered while I worked in finance. It's not exorbitantly difficult, but there are just a lot of potential logic errors to make. The code, ultimately, looked like this (it's Java hence the extra awkward bit twiddling) int pw = bitWidth(p); int qw = bitWidth(q); int pwr = (pw%8); int totalSize = (pw + qw)/8 + (((pw+qw)%8)>0?1:0); byte[] data = new byte[totalSize + 1]; int b = data.length - 1; while (p > 0) { data[b]=(byte)(p&0xFF); p = p >>> 8; if (p != 0) { b--; } } if (pwr != 0) { while (q > 0) { data[b--] |= ((byte) ((q << (pwr)) & 0xFF)); q = q >>> (8-pwr); data[b] |= (byte) (q & 0xFF); q = q >>> pwr; } } else { while (q > 0) { data[--b] |= (byte)(q&0xFF); q = q >>> 8; } } data[0] = (byte) pw; return data; It's like 40 lines of code. It shouldn't be that hard. Yet it is. Bit-twiddling is error prone and hard to reason about and, dunno, maybe pwr isn't a great variable name, but the code really doesn't get particularly expressive even if you change it to pWidthRemainder or whatever. It's still basically a bunch of cthulhu fhtagn as far as I'm concerned.
- earedpiece 5y agoBut from a scientific standpoint, do you think a software can be 100% correct? Bugs will eventually be found. The best bet, when writing (100%) correct code, is through thorough testing, and hoping for the best, because when you are handling a million lines of code, errors, will definitely creep up. Personally, I feel achieving a 100% correct code is impossible, since developers are always updating, and adding new features to the code. Hence why a team, should be put in place to create patches quickly, when bugs are found.
- ebingdom 5y ago> But from a scientific standpoint, do you think a software can be 100% correct? Sure it can. I've written 100% correct code using Coq. For example, I wrote a relatively simple program (~1.7k LoC) to interpret a simple programming language, but it was definitely correct (certified by a machine-checked proof). Of course, for larger programs it's much harder to do that. But it's just a matter of how much time you're willing to invest.
- elihu 5y agoOn some level, that's sort of like asking whether we can be 100% sure that the solution to a math problem is correct. I mean, I'm pretty sure that 2+3=5, but what if I made a mistake somewhere? What if I'm falling prey to some cognitive bias that makes me blind to certain math errors? What if all of humanity is wrong about this math problem? If we can be convinced that we're 100% sure that 2+3=5, then what about a more complicated problem? What's the most complex math problem we can be sure about? What if we have to use a computer to check its correctness? (In general, I think it's usually good enough to say we're pretty sure something is correct if we have some solid basis for believing that it is. I'm okay with being 99.99% sure that 2+3=5, but in the world of software "I'm 50% sure this software is correct" is a standard that most software I use on a daily basis fails to meet.)
- mbrodersen 5y agoCompCert is an example of 100% proven correct software. There are other examples out there. Verification tools are getting better every day, making it less costly to prove code correct.
- smasher164 5y agoI'm not sure why the example used for formal verification is Z3, and not proof assistants like Coq or Agda. SMT solvers have made verification a lot more accessible, but Coq is strictly more expressive than Z3.
- ebingdom 5y agoCoq requires thinking like a functional programmer, and most programmers are resistant to that paradigm for some reason.
- continuational 5y agoSorely lacking from this article is the topic of static type systems and immutable data structures. One guarantees the absence of certain classes of bugs; the other reduce the number of moving parts in a system.
- nathan_compton 5y ago"Our most important goal in writing software is that it be correct." Because I'm a particular kind of idiot, it took me about 10 years of working in and around software engineering to understand that this is just not true. In many, many cases the most important goal for software is for it to take up some space in the market (either the external market or the internal corporate reputation market). In order for software to meet this goal, it must more or less function, but it doesn't need to be _correct_. It can crash, make mistakes, lose data, whatever. In many situations this goal can be understood as "gathering capital." You write software to impress investors or to attract users or to convince other people in the corporate hierarchy that you are a going concern. Thus the valuable part of the software is not its correctness but its _surface_. How shiny is the software? Does it impress and/or please users? Does it impress investors? If you've read Guy Debord you could have predicted this is how software would work. Software is spectacle.
- liketochill 5y agoI hope that my life or preservation of the environment does not depend on the software you are working on being correct. In fields where that is the case, methods for ensuring correctness are important.
- thewarrior 5y agoIt already does.
- hiptobecubic 5y agoYes but I think we would say that incorrect software in those circumstances "doesn't function." Most software is "usably wrong," which is good enough to build a career on.
- AnimalMuppet 5y agoGood enough because it's good enough (most of the time) for the user to do what they need to do. That's all that's needed. Not perfection. Even more: Perfection takes a lot of time. Perfection therefore means less software reaching the user. So, is the imperfect software more useful to the user than no software at all? In most cases, yes, even with all the problems the software causes. (Because if it were otherwise, wouldn't the users stop using the software?)
- wruza 5y agoAmong other, they demand/receive better requirements. Underspecified and underconstrained systems leave much space for variations of conditions “they may find themselves working in”. I put that in quotes because software is not sentient, and it’s really a programmer who encounters that enormous space at the implementation (or ignores it for some reason). When that happens, they often can’t ask anyone what to do, because nobody except them could understand the problem deep in the structure of a program/process. It leads to defensive approaches which may or may not trigger in the wild, producing cryptic messages, because it was hard to formulate it in the first place. Unrestricted underspecifity puts roots deep into the code, when they could be cut much earlier. Programmer’s clients and employers are rarely aware of that, and it’s a programmer’s responsibility to specify the details as deep as client is able to grasp. It’s another (imo most important) level of correctness, unrelated to rigid math-like correctness of a result of programming of completely coherent requirements, which almost none of us ever receive.
- peheje 5y agoVery well explained. Looking back I remember the projects where you had one of those unicorn colleagues/customers that truly could tie business and technical complexities together. It's often those projects that end up the best, because you actually CAN ask some about "uh I can see that this weird state could appear in the code, is this something we should work around in a special way or is the state not representable?"
- seumars 5y agoI'm surprised at the lack of stack overflow jokes in this thread.
- lmilcin 5y agoI am both accountant (actually educated) and developer. I will not rant (today) on how I disagree with unit testing and there are better ways to get the result. I just wanted to point out double bookkeeping analogy is bad. Double bookkeeping is there to make certain types of errors very visible. Double bookkeeping comes from idea that things do not just show up or disappear, but they change state -- value appearing on one account must have come from another. And so double bookkeeping is basically saying -- we want to have a system where it is immediately visible or even impossible to write something down that does not preserve this invariant. It has nothing to do with whether the operation is accounted for correctly. You can still mess up accounts very easily. Double bookkeeping does not mean double checking. You just select two accounts and put the same value on opposite sides of those accounts. This is a single operation. Everything about it can be wrong -- the value can be wrong, the accounts can be wrong and nothing is going to catch that error. Actually, using invariants is one technique I use to ensure my code works correctly. Has nothing to do with unit tests. I just write code, whenever it is possible, in a way that makes it impossible to put it in a wrong state.
- Supermancho 5y agoRegardless of the relative value to ascribe to unit testing, the confidence value is obvious from the intentional and functional double checking, which can be done piecemeal. There is still a problem with added side effects, which unit tests dont pick up on. This will probably be addressed with fn signatures...which doesnt do much but alert for changes. Eventually unit testing can start to approach formalism with monitored functions, wherein side effects must be asserted for a unit test to succeed.
- davesailer 5y agoRight now I'm reading "PSP, A Self-Improvement Process for Software Engineers" by Watts Humphrey (2005): https://resources.sei.cmu.edu/library/asset-view.cfm?assetID=30595 https://resources.sei.cmu.edu/library/asset-view.cfm?assetID... PSP = "Personal Software Process", the step before TSP, "Team Software Process". Personal software process (PSP): https://en.wikipedia.org/wiki/Personal_software_process https://en.wikipedia.org/wiki/Personal_software_process Team software process (TSP): https://en.wikipedia.org/wiki/Team_software_process https://en.wikipedia.org/wiki/Team_software_process It's there for anyone who is interested. Right now I'm skimming it to see what I can learn. Can be quite involved but can reduce errors and improve quality by orders of magnitude. (Proven.) Given that, Humphrey repeatedly emphasizes that communication and understanding about what needs to be built are the most important parts, and that trying to test quality into larger systems can literally take years (of full-time work, by teams of dozens of developers). Since I'm retired and reviewing all the ways that I wasted my life, then for me it's only an intellectual exercise, but this could help someone still in the thick of it. Lots of food for thought.
- awildfivreld 5y agoI have recently realized that I am not progressing my skills in the way I wanted, partly because I cannot seem to get off the ground. I read through the first chapter of PSP now, and this is exactly what I need! Thank you so much for sharing this.
- jeff-davis 5y agoThe article doesn't mention the additional problems with stateful software, which account for a lot of the worst problems in practice. Consider an application with persistent state. Version 1 has a bug that introduces some slight inconsistency in the state file. Version 7 reads the state file, assumes that the state file is consistent, and ends up producing crazy results. Version 7 is "correct" but still fails. One answer is to always validate the state file, but that may be impractical due to size or complexity. A better answer is to use a database that offers declarative constraints that help prevent inconsistencies. This is such a good solution that something written in PHP could be very robust in practice if it uses a good database, whereas something written in haskell that uses a database without good constraints might fail miserably.
- zozbot234 5y agoYou don't need a full database, you just need well-specified serialization/deserialization. Typically this includes explicit versioning mechanisms, so the app can be aware of what version it is dealing with and convert the format accordingly.
- jeff-davis 5y agoser/de only solves parsing problems and not much else. What about using duplicate user IDs? Or maybe many objects in the system need both the employee ID and the name (and the employee ID determines the name, obviously), and the employee changes their name, but it only updates some objects and not all? There are lots of ways data can be subtly inconsistent and a database is a big help preventing it.
- bgro 5y agoI think a lot of the common memes around programmers like these is due to the wrong-think gatekeeping during the interview process that everybody is copying from FAANG. I have 5-10 years programming experience and find an unlimited amount of bugs in other developer's work of any and all skill levels. On occasion, something like a billing calculation will have a function doing the opposite of what they intend to do but somehow nobody has caught it in 5 years. I've also seen ancient security reports that just have a chunk of code that doesn't work like it sounds it should, causing 25% of results to be missed. These are just a couple of actual examples. I focus on generally not screwing up my work like that as well as finding these problems in my team's code. It's mind blowing to me this has absolutely 0 value though. I and others like me are gatekept out of a lot of work since we process things differently. Developers can't seem to imagine a world where somebody thinks differently than them and memorizing leetcode is just as easy for everybody as it is them. "If you can't talk me through what you're doing while writing your memorized perfectly compilable code on a whiteboard or google doc, I guess you're either just lazy and not really invested, or a bad developer. Probably both." The only thing that technically matters to get into the dev roles is memorizing leetcode problems in order to pass the interview so that A) You have a job at all (even small companies are doing these now)and B) You can get your actual salary corrected every couple years. If it's not memorizing leetcode, it's how many (inefficient) lines of code did you contribute? (Thankfully this is finally starting to be phased out as a measurement AND bragging point.) The people who make it through all this then go on to store their critical world-infrastructure Exchange version number in a 32 bit integer because "It worked and compiled when I tested it, so it's probably good enough." Nobody dares test it due to the toxic hostility back, or because a senior dev wrote it, or "I ran the code and it compiled for me, so it's good." There's just so much talking down to others and gatekeeping about how leetcode solvers are superior and that it's the only possible starting block for a "real dev". I can't believe the frequency of simple yet major mistakes these people go on to actually do after they spend so much time bragging about it.
- rdtsc 5y ago> There are generally two main categories of tests. There are unit tests [and] Integration tests Later on there is a mention of fuzzing. However, there is something better called property tests. Where the on failure there is automatic "shrinking" of input complexity to find a minimally failing test scenario. For Erlang and Elixir I recommend https://propertesting.com/ https://propertesting.com/ book and the PropEr test framework https://github.com/proper-testing/proper https://github.com/proper-testing/proper. As an example, here is the use of an advanced feature called targeted property based testing: https://proper-testing.github.io/tutorials/PropEr_testing_with_search_strategies.html https://proper-testing.github.io/tutorials/PropEr_testing_wi... used to test a labyrinth. EDIT: as spockz mentioned below, the original idea was developed by QuviQ as QuickCheck http://www.quviq.com/products/erlang-quickcheck/ http://www.quviq.com/products/erlang-quickcheck/. It's a commercial offering but there is a QuickCheck mini offered as a free download as well.
- spockz 5y agoThe original is quickcheck for erlang which also has an Haskell flavour. It has inspired scalacheck for scala. I test pure logic with property based testing and simple integration testing to ensure all the steps are aligned properly.
- rdtsc 5y agoGood point. I edited my comment to mention QuviQ and their QuickCheck product.
- mananaysiempre 5y agoMy understanding was that property tests and fuzzing are orthogonal in roughly the same way unit and integration tests are: you do property test to ensure well-delimited pieces of your code satisfy well-defined constraints, and you fuzz to ensure the whole thing just doesn’t go completely nuts when faced with something unusual. But I could be wrong because I never could figure out how to set up fuzzing for some sort of moderately-involved realistic thing (a CSV parser? something like that).
- interactivecode 5y agoOn a side note I always find it interesting to see the static type and algorithm fans taut how correct code is. Great we now got typescript everywhere… it’s still amazingly difficult to test and validate user interfaces, styles and layouts.
- RoddaWallPro 5y agoI'm reminded of what I'm going to call a poem, but which isn't a poem, but more of a long saying. I can't remember the exact wording (if someone remembers and has the link, please link it!) but roughly: If you tested your program, congratulations. Your program is now tested, incorrectly. If you tested your program, and verified that the tests are correct, congratulations. Your program is now verified to do the wrong thing, correctly. It has at least two other lines which I can't even think of the paraphrasing for.
- tails4e 5y agoIn hardware (chip) design code verification is done by seperarate engineers. Design engineers have some test code to kick the tyres, but sign off is done independently and rigoirlsy, with functional coverage linked to specification documentation. This is a cautious approach, but hardware bugs can be multi million dollars, so it pays to have this level of testing.