8 ms·

QA = Time and Money. How much should you invest?

mistercheese 11y ago
thoughtful article
MichaelCrawford 11y ago
As a consultant, I used to ask my clients to QA my deliveries. Few of them were willing to, to the extent they did they could not or would not provide meaningful bug reports. This led to my adoption of test-driven development - mostly but not all automated testing - with the eventual result that I advertised that my final deliverables would be ready to ship to end-users, without any requirement for QA by my clients.
- 3am 11y ago
  Test driven development usually refers to unit testing or integration testing. It would be interesting to know if you went beyond that and the nature of contract work you were doing (testing for an internal service with a RESTful API being very different from a mobile app, or a website).
  - jacques_chester 11y ago
    TDD, in my view, starts from the outermost layer -- the end user -- and moves inwards progressively to the unit tests.
    - MichaelCrawford 11y ago
      John Lakos differs from you. In his book "Large Scale C++ Software Development" he advocates starting at the lowest level of your own code, that is, subroutines that only make system calls or call libraries that are provided by the system. Then you unit test the second level - subroutines that only call that first level, or that call the system or libraries. main() sits at the top level. Not every program is straightforward to levelize. His point is that one should do so. Also the unit tests for any one level should focus on what is new on that specific level, with the assumption that all the lower levels are flawless. In general they won't really be but when that's the case you write a unit test for the lower level. This keeps the LOC on the tests about the same as the LOC in the deliverable. It is unfortunate that his book focusses on C++; really he should have written a separate book on testing that was language-agnostic. There is very little of that section that's really specific to C++. In addition to unit tests I do integration tests. If there's a file format involved I create lots of input files that contain various edge cases.
      - 3am 11y ago
        Outstanding answer - thanks for the pointer to the Lakos book.
      - geoelectric 11y ago
        With the caveat I haven't read Lakos' book, there's a pretty big backlash going on in the industry against hyper-granular unit testing. Comes down to people realizing that when their implementation needs to change, they often have to change a bunch of unit tests that were written to the implementation. One problem is that modularity should let you change implementation without friction; that's the whole point of modularity to begin with. So there's not much profit in writing a bunch of client code (tests) that intentionally break modularity and put friction back on the process. It's not quite as bad as just random breakage, because at least you know where it all is, but it's still painful. Another problem is that if you're not careful, you end up with a set of tests that don't tell you when something works as expected or not, it just tells you when something changed. Having a test that tells you something isn't coded as expected anymore is pretty useless. You know you changed it. And still another problem is that the testing itself has become too invasive. We're architecting things for dependency injection that would never really need it if it weren't for invasive tests. It's fine and well to drop some testing hooks in, but if you're having to completely invert control in your code to do it and that makes things significantly more complex, that may not be great. I think there were some fairly recent Martin Fowler posts on this, but I believe his point was that we're very possibly doing it wrong: if the point is to guarantee an interface then tests should be to the interface. And maybe injecting test doubles is good in some cases but too invasive in others--especially when it's done to test an implementation--and so forth. So not sure where Lakos is on the scale there, but what you describe about putting in layers upon granular layers of tests strikes me as setting yourself up for these issues. I'm a pretty big believer in testing to the interface, and do try to draw a line between functions that are there to serve a particular implementation and functions that describe something more abstract. I test to the latter. Modularity is the key to maintenance; the biggest benefit of this type of testing to me is to validate your modularity, even more than validating the code within the modules.
- ThatMightBePaul 11y ago
  TDD is great. It should be a given on any project that isn't just a spike. I'm curious what kind of products you were building? Simple websites, or full fledged applications? 100% unit test coverage can still miss a host of gnarly bugs. A lot of the ugliness of QA comes from how A interacts with B. Or how A + B + C + D work together. Or at least, that's my 2 cents :D
  - geoelectric 11y ago
    You also have to qualify 100% coverage: SUT lines, branches, or paths? Those are all very different levels of rigor. The two most useful code coverage stats I've found are 0 and "less than yesterday." The former tells me something pretty important about the culture of the team who owns that code; and having rules against the latter is about the only way to ensure a test-on-checkin policy.
    - ThatMightBePaul 11y ago
      I love this metric, and am totally stealing it.
      - geoelectric 11y ago
        All yours!
  - MichaelCrawford 11y ago
    I've done a wide variety of projects - cross-platform GUI, a windows database kernel implemented as a dll - that is, not a server - embedded storage firmware, mac os x device drivers (kernel extensions). For some but not all of my projects, failure was not an option. Imagine I lost a disk sector in my storage firmware.
- Osiris 11y ago
  I've been taking this approach. I'm the sole developer on my company's API so it's important to me to not waste time fixing issues that get brought up by the UI team when they start working on with the API. To that end, I write integration tests in that all my tests call the REST endpoint then verify the result of the API call. My tests treat the API call as a black box. A single endpoint can have dozens or hundreds of tests. By the time the API gets to the UI guys, I know it's going to work correctly (except in those cases where I missed testing certain scenarios). Not only does it save me from bug reports, but it makes me look good because everything I ship works.
geoelectric 11y ago
This is probably the most honest article I've seen, ever, regarding how much QA/testing companies at various stages need. I'm impressed, because there's a real tendency within QA culture to say "the ideal is all the things all the time, we're the gatekeepers," which has forced quality into a bad all-or-nothing situation. We all know intuitively that it's not all needed a lot of the time and the success stories are clear, so the trust in any of it is undermined. So then you get a dev culture saying "QA is unnecessary, we can handle all of it," which is generally true at the beginning but can send your company off a cliff if you don't manage transition. The sad part is that even once quality processes get around that, the all-or-nothing attitude leads to a pyrrhic victory. Quality teams still end up only doing what can be afforded (because math) but everyone thinks they should be doing more. That leads to the pervasive opinion of ineffective quality teams. And, unfortunately, the scramble of trying to do everything at once (and therefore mastering none of it) often makes that a self-fulfilling prophecy. And I love that it stresses that QA is about hedging bets. Bugs get out, period. The gatekeeper attitude is what leads to the do all the things attitude in the first place. "At any cost" is generally a bad way to strategize. As for this chart, I'd have added exploratory/informal to seed B2C--exploratory is biggest bang for buck for finding new bugs, and incubating products are -all- new bugs--but I suspect they're lumping what I'd have recommended under dogfooding. So yeah, very nice. Of course, the best thing to do in any situation is consider the context of your company, product, and market. Think about how maintainable it really needs to be and what would cause the most damage: losing customer money, leaking their data, embarrassing you in the market, eroding their trust. Are there only a few customers to lose or are you mass market and can afford a round of sufficiently obscure failure? Those are the things to prioritize when you start picking your battles. But this is such a great set of guidelines for where to start that conversation. (Edits for typos only)
- ThatMightBePaul 11y ago
  Thanks for the good word (I'm the author). I 100% agree about setting the right culture / the difficulty of managing the transition. It takes a ton of work to establish the right infrastructure + culture. Specifically for the reason you mentioned: a lot of folks see QA in black and white. It's doubly hard when you're growing fast.
EliRivers 11y ago
This article, as is typical around here, addresses a very small part of the industry. Here's a view from a different part; the part where a project has a lifecycle of a decade or so. I used to work for a software consultancy that had been acquired by a US defence company (second tier). Over the previous 15 years, the software consultancy had elevated their QA to the point that bugs in the delivered product were literally unknown. Nobody ever saw any in the delivered product, over a decade. Every requirement could be traced to the design that would meet it, to the code that implemented it, to the test that verified it. The test documentation was beautiful, and anyone could (and did) execute a complete set of tests, writing their own personal signature on every confirmation of test success. The signed test specs were scanned before being sealed in an envelope against the customer ever choosing to inspect. QA had (and did) veto over requirements, design, and tests. We were working with two other major, globally known defence names, and the customer had learned through experience that when something went wrong, come to us last. Sometimes, the customer came to us to ask us to work out which of the other companies involved had screwed up (because sometimes, the other companies were themselves unable to work it out). Once, they sued one of the other names for incompetency and gave us their work because we knew how to fix it better than they did, because our interface tests highlighted everything that their software didn't do in accordance with the spec. We could replay any test, at any version, at any time, and compare the results now with the signed results then from the sealed envelope. The time savings we had because we got things right the first time, by doing it properly, was immense. Where this is going is that this was priceless. We had the customer's absolute trust, based on experience. Nobody ever found a bug in our delivered software (I'm not saying there weren't any; I'm saying that our QA was thorough enough that the customer was never going to do anything with it that we hadn't tested already). Code coverage, all warnings on, static analysis, valgrind, cross-platform compilation and testing, we did everything. Customers outright told us that we were worth paying extra because of the peace of mind and the reduced risk, so they paid us more and we spent less doing it. QA is time and money. The better your QA, the more time you will have and the more money you will make.
- geoelectric 11y ago
  So, this I agree with too, in contrast with my other comment. There are particular industries (engineering, banking, aerospace, defense, for some examples) where it -has- to be correct. These are also the industries where at least the final product should probably be waterfall or spiral, and generally you need formalism all the way down. The problem comes when people bring lessons learned from that to companies that don't need that level of assurance. Documentation that's never read again is a waste. Spending time to make tests replayable if they'll never be replayed again is a waste. And so forth. As for bugs being literally unknown, I'm jealous--it'd be nice to claim I ever guided a project to that state. But keep in mind that's different than no bugs period, just means no bugs in what was tested or used. If you're delivering a product with very specific use cases, flows and a tight scope with strong bumpers around anything unintended, it's pretty straightforward to guarantee all that. But if you're delivering a sandbox product on the internet to a bunch of randoms, much less so. After all, ten checkboxes on a dialog are 1024 testcases. That's not even accounting for free-entry fields. You'll never cover all combinations of all uses of the whole app interface, ever, period, and equivalences and fuzz testing only get you so far. That's a simple (if slightly fantastic) example for UI testing, but the same basic thing applies to most other kinds of testing. At the end of the day you often have to pick your battles. So I really disagree on the better is more. There is absolutely a diminishing return. It just so happens that for what -you- were doing, that point of diminishing return is pretty high.
codelion 11y ago
I am not sure you want to delay penetration testing until you are post IPO.
- ThatMightBePaul 11y ago
  Agreed. Caveat: do you need to do pen testing every release? I'm guessing, not. YMMV. If you're a defense contractor (as some other commenters mentioned), your priorities are probably quite different.
  - infosectosser 11y ago
    I head infosec for a “Series A - C” B2B company and a fairly standard request from a potential customer is to see not only our own penetration test reports but third party penetration test results, as well. As a result, we run automated pen tests on weekends and before major releases. We also work with an application security firm every 6-12 months. For what it’s worth, we don’t do anything nearly as intense as defense contracting or handling financial info. That said, I liked the article - thanks for sharing.
    - ThatMightBePaul 11y ago
      Np! Thanks for sharing. Glad to hear security is taking the front seat some places. Anecdotes like this help me expand my world view. <3
  - geoelectric 11y ago
    If I were handling private customer data or money, I'd absolutely pen test early. Comes back to that "what do you have to lose" consideration re: context. If the worst that happens is my own service goes down, I might delay it.
ffn 11y ago
Can someone please explain why one prefers unit testing to integration / acceptance testing when you're a seed stage company? I mean, your primary objective as an early stage company should be getting an operational product out to the market, so it makes sense that you test that your users can operate your product - aka integration / acceptance test with crapybara or phantomjs. On the other hand, if you're just unit testing you're bound to miss the bigger picture (usually this shows up in the form of misspellings in your html views, some CSRP bullshit, external apis misbehaving, etc.), and will wind up with all tests that pass but a product that still doesn't work.
- PointerReaper 11y ago
  In my line of accessibility work, I find QA fills the role of testing to educate. If the engineers don't have the experience, looping issues into the backlog, gives them the experience to train up the skill set. They might have had verified experience before hiring, but getting team collaboration helps mature new people or smooth out the rough skill sets of a dev who might be awesome in one particular area but meh in others. I suspect this could be the case for any subject matter expert where the skill set is viewed as fringe or not expecting the team will be required to have that exact knowledge before hire.
- geoelectric 11y ago
  Unit testing is easier to do with the developer resources on hand since they probably already know how to write them, probably necessary for quick pace in dev anyway, and gives you a large portion of the risk mitigation testing in general will give you. You also check in unit tests with the code if you do it right, whereas integration or acceptance tests require additional passes once all the code covered is checked in, so it's more painful to the schedule. Integration tests also require more development to allow integration while still being hermetic (isolating out any external factors, since you might have to architect to share injected dependencies between modules) -or- more maintenance if you don't make them hermetic and they have to track external dependencies. They require more time to triage failures as well if they're not hermetic, as external failure is a possibility; you can't just trust the test results. And final acceptance tests should never be hermetic--they should accept based on realistic environments--so they're almost always a little high-maintenance. So while the value is there, the cost is much higher and it'd be more of a distraction from primary development. Cost and benefit have to be considered. So in this model the integration/acceptance part is handled by dogfooding, which is the lowest cost way to get a decent chunk of it, even though it only walks happy-path for the most part. But that's why I said I'd add exploratory to B2C seed, is to get back some of the integration testing on non-happy paths and obscure paths. Edit: find docs on the test pyramid and consider that the shrinking number of tests as you go up also implies somewhat lower technical debt as you skip going up. The shrinkage is a function of the cost vs. benefit--you're just taking it to its logical conclusion of omission.
joshdotsmith 11y ago
Wow, crazy timing. We interviewed Fred at Rainforest on the Talking Code podcast – and that interview launches tonight at midnight (PDT). I'm going back through and picking out my favorite quotes right this moment in preparation for the episode launch. Some of my favorites: "How did we get to the place where the generally accepted best practice is not very good?" (referring to automated testing) "The visual representation is only somewhat semi linked to the actual underlying code that represents it." "A really good QA person can actually start to QA the requirements themselves." "Good QA people are the customer before the customer." "Shipping a bug at Facebook scale is pretty catastrophic." "For tiny company X, a YC company, shipping a bug – while painful – is not going to be the end of the world." "For 95% of people, QA is not a competitive advantage, so it doesn’t make sense to have it in house." This has honestly been one of my favorite interviews ever, and I'm just really lucky it happened on my own podcast. http://talkingcode.com/podcast/episode-6-fred-stevens-smith/ http://talkingcode.com/podcast/episode-6-fred-stevens-smith/
- MichaelCrawford 11y ago
  "QA the requirements". Safety-critical software does not commonly has bugs. The problem it faces, is incorrect requirements.
- ThatMightBePaul 11y ago
  Look forward to hearing this. Will Venmo $5 in exchange for the outtakes of Fred cursing ;)
calinet6 11y ago
QA is interesting, sure. I'm more interested in the Q part, and you don't get that just by having a role or department checking work before it hits the customer. This article does a great job of describing the minimum level of quality allowable to scrape past certain stages of growth. The question you should be asking is: what level of quality, built in from the beginning, intentionally and with full support, will create a product so valuable to the customer as to generate a wild feedback loop of success—not just scraping by at a minimum acceptable level. What level of quality will generate returns, not just reduce your debt to an acceptable level? And more importantly, how do you achieve it? Quality is your value to the customer. If you think of it systemically, you won't need a QA. As Dr. Deming said, "Inspection does not improve the quality, nor guarantee quality. Inspection is too late. The quality, good or bad, is already in the product." And Harold F. Dodge: "You can not inspect quality into a product." Deming advocated instead for a holistic understanding of the factors driving quality, including management and leadership, the processes used, and continuous improvement of systems. This is the type of discussion I always see lacking from discussions of software QA. On the manufacturing timeline, it's like we're in the 1920's. It's all very realistic, but we could be so much further along.
- geoelectric 11y ago
  What you describe is QA. Testing is technically QC: Quality Control. It's only one facet of QA, though the terms are usually conflated. Most companies only have a QC department.
  - umanwizard 11y ago
    You're proposing a definition that is out of line with how 99.99% of people use it. Even if your definition is more logical, this isn't how language works.
    - geoelectric 11y ago
      I'm not proposing anything. This is a well-known concept among quality professionals. I'm going to guess you have a different specialty. Google "quality assurance control" and you'll find a lot more, but here's one source: http://whatis.techtarget.com/definition/quality-control-QC http://whatis.techtarget.com/definition/quality-control-QC
ecopoesis 11y ago
Zero QA, or at least zero QA by dedicated QA people. Developers end up using QA as a crutch. The end up not testing their code completely, and sometimes they don't test it at all. Too many times I've seen code checked in, after going through code review, that just doesn't work. I'm not talking edge cases, it does not function at all. Get rid of the crutch, and force developers to be responsible for their code. Developers should do all the QA on their code. If you must have QA, don't waste them on checking if features are I plmented correctly, repeating a job developers should do. Instead, have them actually check quality: have them look at the whole app or site holistically. Have them try to find the bugs that developers won't be able to find.

thoughtful article

As a consultant, I used to ask my clients to QA my deliveries. Few of them were willing to, to the extent they did they could not or would not provide meaningful bug reports. This led to my adoption of test-driven development - mostly but not all automated testing - with the eventual result that I advertised that my final deliverables would be ready to ship to end-users, without any requirement for QA by my clients.

3am 11y ago
Test driven development usually refers to unit testing or integration testing. It would be interesting to know if you went beyond that and the nature of contract work you were doing (testing for an internal service with a RESTful API being very different from a mobile app, or a website).
- jacques_chester 11y ago
  TDD, in my view, starts from the outermost layer -- the end user -- and moves inwards progressively to the unit tests.
  - MichaelCrawford 11y ago
    John Lakos differs from you. In his book "Large Scale C++ Software Development" he advocates starting at the lowest level of your own code, that is, subroutines that only make system calls or call libraries that are provided by the system. Then you unit test the second level - subroutines that only call that first level, or that call the system or libraries. main() sits at the top level. Not every program is straightforward to levelize. His point is that one should do so. Also the unit tests for any one level should focus on what is new on that specific level, with the assumption that all the lower levels are flawless. In general they won't really be but when that's the case you write a unit test for the lower level. This keeps the LOC on the tests about the same as the LOC in the deliverable. It is unfortunate that his book focusses on C++; really he should have written a separate book on testing that was language-agnostic. There is very little of that section that's really specific to C++. In addition to unit tests I do integration tests. If there's a file format involved I create lots of input files that contain various edge cases.
    - 3am 11y ago
      Outstanding answer - thanks for the pointer to the Lakos book.
    - geoelectric 11y ago
      With the caveat I haven't read Lakos' book, there's a pretty big backlash going on in the industry against hyper-granular unit testing. Comes down to people realizing that when their implementation needs to change, they often have to change a bunch of unit tests that were written to the implementation. One problem is that modularity should let you change implementation without friction; that's the whole point of modularity to begin with. So there's not much profit in writing a bunch of client code (tests) that intentionally break modularity and put friction back on the process. It's not quite as bad as just random breakage, because at least you know where it all is, but it's still painful. Another problem is that if you're not careful, you end up with a set of tests that don't tell you when something works as expected or not, it just tells you when something changed. Having a test that tells you something isn't coded as expected anymore is pretty useless. You know you changed it. And still another problem is that the testing itself has become too invasive. We're architecting things for dependency injection that would never really need it if it weren't for invasive tests. It's fine and well to drop some testing hooks in, but if you're having to completely invert control in your code to do it and that makes things significantly more complex, that may not be great. I think there were some fairly recent Martin Fowler posts on this, but I believe his point was that we're very possibly doing it wrong: if the point is to guarantee an interface then tests should be to the interface. And maybe injecting test doubles is good in some cases but too invasive in others--especially when it's done to test an implementation--and so forth. So not sure where Lakos is on the scale there, but what you describe about putting in layers upon granular layers of tests strikes me as setting yourself up for these issues. I'm a pretty big believer in testing to the interface, and do try to draw a line between functions that are there to serve a particular implementation and functions that describe something more abstract. I test to the latter. Modularity is the key to maintenance; the biggest benefit of this type of testing to me is to validate your modularity, even more than validating the code within the modules.
ThatMightBePaul 11y ago
TDD is great. It should be a given on any project that isn't just a spike. I'm curious what kind of products you were building? Simple websites, or full fledged applications? 100% unit test coverage can still miss a host of gnarly bugs. A lot of the ugliness of QA comes from how A interacts with B. Or how A + B + C + D work together. Or at least, that's my 2 cents :D
- geoelectric 11y ago
  You also have to qualify 100% coverage: SUT lines, branches, or paths? Those are all very different levels of rigor. The two most useful code coverage stats I've found are 0 and "less than yesterday." The former tells me something pretty important about the culture of the team who owns that code; and having rules against the latter is about the only way to ensure a test-on-checkin policy.
  - ThatMightBePaul 11y ago
    I love this metric, and am totally stealing it.
    - geoelectric 11y ago
      All yours!
- MichaelCrawford 11y ago
  I've done a wide variety of projects - cross-platform GUI, a windows database kernel implemented as a dll - that is, not a server - embedded storage firmware, mac os x device drivers (kernel extensions). For some but not all of my projects, failure was not an option. Imagine I lost a disk sector in my storage firmware.
Osiris 11y ago
I've been taking this approach. I'm the sole developer on my company's API so it's important to me to not waste time fixing issues that get brought up by the UI team when they start working on with the API. To that end, I write integration tests in that all my tests call the REST endpoint then verify the result of the API call. My tests treat the API call as a black box. A single endpoint can have dozens or hundreds of tests. By the time the API gets to the UI guys, I know it's going to work correctly (except in those cases where I missed testing certain scenarios). Not only does it save me from bug reports, but it makes me look good because everything I ship works.

geoelectric 11y ago

This is probably the most honest article I've seen, ever, regarding how much QA/testing companies at various stages need. I'm impressed, because there's a real tendency within QA culture to say "the ideal is all the things all the time, we're the gatekeepers," which has forced quality into a bad all-or-nothing situation. We all know intuitively that it's not all needed a lot of the time and the success stories are clear, so the trust in any of it is undermined. So then you get a dev culture saying "QA is unnecessary, we can handle all of it," which is generally true at the beginning but can send your company off a cliff if you don't manage transition. The sad part is that even once quality processes get around that, the all-or-nothing attitude leads to a pyrrhic victory. Quality teams still end up only doing what can be afforded (because math) but everyone thinks they should be doing more. That leads to the pervasive opinion of ineffective quality teams. And, unfortunately, the scramble of trying to do everything at once (and therefore mastering none of it) often makes that a self-fulfilling prophecy. And I love that it stresses that QA is about hedging bets. Bugs get out, period. The gatekeeper attitude is what leads to the do all the things attitude in the first place. "At any cost" is generally a bad way to strategize. As for this chart, I'd have added exploratory/informal to seed B2C--exploratory is biggest bang for buck for finding new bugs, and incubating products are -all- new bugs--but I suspect they're lumping what I'd have recommended under dogfooding. So yeah, very nice. Of course, the best thing to do in any situation is consider the context of your company, product, and market. Think about how maintainable it really needs to be and what would cause the most damage: losing customer money, leaking their data, embarrassing you in the market, eroding their trust. Are there only a few customers to lose or are you mass market and can afford a round of sufficiently obscure failure? Those are the things to prioritize when you start picking your battles. But this is such a great set of guidelines for where to start that conversation. (Edits for typos only)

ThatMightBePaul 11y ago
Thanks for the good word (I'm the author). I 100% agree about setting the right culture / the difficulty of managing the transition. It takes a ton of work to establish the right infrastructure + culture. Specifically for the reason you mentioned: a lot of folks see QA in black and white. It's doubly hard when you're growing fast.

EliRivers 11y ago

This article, as is typical around here, addresses a very small part of the industry. Here's a view from a different part; the part where a project has a lifecycle of a decade or so. I used to work for a software consultancy that had been acquired by a US defence company (second tier). Over the previous 15 years, the software consultancy had elevated their QA to the point that bugs in the delivered product were literally unknown. Nobody ever saw any in the delivered product, over a decade. Every requirement could be traced to the design that would meet it, to the code that implemented it, to the test that verified it. The test documentation was beautiful, and anyone could (and did) execute a complete set of tests, writing their own personal signature on every confirmation of test success. The signed test specs were scanned before being sealed in an envelope against the customer ever choosing to inspect. QA had (and did) veto over requirements, design, and tests. We were working with two other major, globally known defence names, and the customer had learned through experience that when something went wrong, come to us last. Sometimes, the customer came to us to ask us to work out which of the other companies involved had screwed up (because sometimes, the other companies were themselves unable to work it out). Once, they sued one of the other names for incompetency and gave us their work because we knew how to fix it better than they did, because our interface tests highlighted everything that their software didn't do in accordance with the spec. We could replay any test, at any version, at any time, and compare the results now with the signed results then from the sealed envelope. The time savings we had because we got things right the first time, by doing it properly, was immense. Where this is going is that this was priceless. We had the customer's absolute trust, based on experience. Nobody ever found a bug in our delivered software (I'm not saying there weren't any; I'm saying that our QA was thorough enough that the customer was never going to do anything with it that we hadn't tested already). Code coverage, all warnings on, static analysis, valgrind, cross-platform compilation and testing, we did everything. Customers outright told us that we were worth paying extra because of the peace of mind and the reduced risk, so they paid us more and we spent less doing it. QA is time and money. The better your QA, the more time you will have and the more money you will make.

geoelectric 11y ago
So, this I agree with too, in contrast with my other comment. There are particular industries (engineering, banking, aerospace, defense, for some examples) where it -has- to be correct. These are also the industries where at least the final product should probably be waterfall or spiral, and generally you need formalism all the way down. The problem comes when people bring lessons learned from that to companies that don't need that level of assurance. Documentation that's never read again is a waste. Spending time to make tests replayable if they'll never be replayed again is a waste. And so forth. As for bugs being literally unknown, I'm jealous--it'd be nice to claim I ever guided a project to that state. But keep in mind that's different than no bugs period, just means no bugs in what was tested or used. If you're delivering a product with very specific use cases, flows and a tight scope with strong bumpers around anything unintended, it's pretty straightforward to guarantee all that. But if you're delivering a sandbox product on the internet to a bunch of randoms, much less so. After all, ten checkboxes on a dialog are 1024 testcases. That's not even accounting for free-entry fields. You'll never cover all combinations of all uses of the whole app interface, ever, period, and equivalences and fuzz testing only get you so far. That's a simple (if slightly fantastic) example for UI testing, but the same basic thing applies to most other kinds of testing. At the end of the day you often have to pick your battles. So I really disagree on the better is more. There is absolutely a diminishing return. It just so happens that for what -you- were doing, that point of diminishing return is pretty high.

codelion 11y ago

I am not sure you want to delay penetration testing until you are post IPO.

ThatMightBePaul 11y ago
Agreed. Caveat: do you need to do pen testing every release? I'm guessing, not. YMMV. If you're a defense contractor (as some other commenters mentioned), your priorities are probably quite different.
- infosectosser 11y ago
  I head infosec for a “Series A - C” B2B company and a fairly standard request from a potential customer is to see not only our own penetration test reports but third party penetration test results, as well. As a result, we run automated pen tests on weekends and before major releases. We also work with an application security firm every 6-12 months. For what it’s worth, we don’t do anything nearly as intense as defense contracting or handling financial info. That said, I liked the article - thanks for sharing.
  - ThatMightBePaul 11y ago
    Np! Thanks for sharing. Glad to hear security is taking the front seat some places. Anecdotes like this help me expand my world view. <3
- geoelectric 11y ago
  If I were handling private customer data or money, I'd absolutely pen test early. Comes back to that "what do you have to lose" consideration re: context. If the worst that happens is my own service goes down, I might delay it.

ffn 11y ago

Can someone please explain why one prefers unit testing to integration / acceptance testing when you're a seed stage company? I mean, your primary objective as an early stage company should be getting an operational product out to the market, so it makes sense that you test that your users can operate your product - aka integration / acceptance test with crapybara or phantomjs. On the other hand, if you're just unit testing you're bound to miss the bigger picture (usually this shows up in the form of misspellings in your html views, some CSRP bullshit, external apis misbehaving, etc.), and will wind up with all tests that pass but a product that still doesn't work.

PointerReaper 11y ago
In my line of accessibility work, I find QA fills the role of testing to educate. If the engineers don't have the experience, looping issues into the backlog, gives them the experience to train up the skill set. They might have had verified experience before hiring, but getting team collaboration helps mature new people or smooth out the rough skill sets of a dev who might be awesome in one particular area but meh in others. I suspect this could be the case for any subject matter expert where the skill set is viewed as fringe or not expecting the team will be required to have that exact knowledge before hire.
geoelectric 11y ago
Unit testing is easier to do with the developer resources on hand since they probably already know how to write them, probably necessary for quick pace in dev anyway, and gives you a large portion of the risk mitigation testing in general will give you. You also check in unit tests with the code if you do it right, whereas integration or acceptance tests require additional passes once all the code covered is checked in, so it's more painful to the schedule. Integration tests also require more development to allow integration while still being hermetic (isolating out any external factors, since you might have to architect to share injected dependencies between modules) -or- more maintenance if you don't make them hermetic and they have to track external dependencies. They require more time to triage failures as well if they're not hermetic, as external failure is a possibility; you can't just trust the test results. And final acceptance tests should never be hermetic--they should accept based on realistic environments--so they're almost always a little high-maintenance. So while the value is there, the cost is much higher and it'd be more of a distraction from primary development. Cost and benefit have to be considered. So in this model the integration/acceptance part is handled by dogfooding, which is the lowest cost way to get a decent chunk of it, even though it only walks happy-path for the most part. But that's why I said I'd add exploratory to B2C seed, is to get back some of the integration testing on non-happy paths and obscure paths. Edit: find docs on the test pyramid and consider that the shrinking number of tests as you go up also implies somewhat lower technical debt as you skip going up. The shrinkage is a function of the cost vs. benefit--you're just taking it to its logical conclusion of omission.

joshdotsmith 11y ago

Wow, crazy timing. We interviewed Fred at Rainforest on the Talking Code podcast – and that interview launches tonight at midnight (PDT). I'm going back through and picking out my favorite quotes right this moment in preparation for the episode launch. Some of my favorites: "How did we get to the place where the generally accepted best practice is not very good?" (referring to automated testing) "The visual representation is only somewhat semi linked to the actual underlying code that represents it." "A really good QA person can actually start to QA the requirements themselves." "Good QA people are the customer before the customer." "Shipping a bug at Facebook scale is pretty catastrophic." "For tiny company X, a YC company, shipping a bug – while painful – is not going to be the end of the world." "For 95% of people, QA is not a competitive advantage, so it doesn’t make sense to have it in house." This has honestly been one of my favorite interviews ever, and I'm just really lucky it happened on my own podcast. http://talkingcode.com/podcast/episode-6-fred-stevens-smith/ http://talkingcode.com/podcast/episode-6-fred-stevens-smith/

MichaelCrawford 11y ago
"QA the requirements". Safety-critical software does not commonly has bugs. The problem it faces, is incorrect requirements.
ThatMightBePaul 11y ago
Look forward to hearing this. Will Venmo $5 in exchange for the outtakes of Fred cursing ;)

calinet6 11y ago

QA is interesting, sure. I'm more interested in the Q part, and you don't get that just by having a role or department checking work before it hits the customer. This article does a great job of describing the minimum level of quality allowable to scrape past certain stages of growth. The question you should be asking is: what level of quality, built in from the beginning, intentionally and with full support, will create a product so valuable to the customer as to generate a wild feedback loop of success—not just scraping by at a minimum acceptable level. What level of quality will generate returns, not just reduce your debt to an acceptable level? And more importantly, how do you achieve it? Quality is your value to the customer. If you think of it systemically, you won't need a QA. As Dr. Deming said, "Inspection does not improve the quality, nor guarantee quality. Inspection is too late. The quality, good or bad, is already in the product." And Harold F. Dodge: "You can not inspect quality into a product." Deming advocated instead for a holistic understanding of the factors driving quality, including management and leadership, the processes used, and continuous improvement of systems. This is the type of discussion I always see lacking from discussions of software QA. On the manufacturing timeline, it's like we're in the 1920's. It's all very realistic, but we could be so much further along.

geoelectric 11y ago
What you describe is QA. Testing is technically QC: Quality Control. It's only one facet of QA, though the terms are usually conflated. Most companies only have a QC department.
- umanwizard 11y ago
  You're proposing a definition that is out of line with how 99.99% of people use it. Even if your definition is more logical, this isn't how language works.
  - geoelectric 11y ago
    I'm not proposing anything. This is a well-known concept among quality professionals. I'm going to guess you have a different specialty. Google "quality assurance control" and you'll find a lot more, but here's one source: http://whatis.techtarget.com/definition/quality-control-QC http://whatis.techtarget.com/definition/quality-control-QC

ecopoesis 11y ago

Zero QA, or at least zero QA by dedicated QA people. Developers end up using QA as a crutch. The end up not testing their code completely, and sometimes they don't test it at all. Too many times I've seen code checked in, after going through code review, that just doesn't work. I'm not talking edge cases, it does not function at all. Get rid of the crutch, and force developers to be responsible for their code. Developers should do all the QA on their code. If you must have QA, don't waste them on checking if features are I plmented correctly, repeating a job developers should do. Instead, have them actually check quality: have them look at the whole app or site holistically. Have them try to find the bugs that developers won't be able to find.