3 ms·

Note that Evan Priestley's post on Facebook and testing, linked in the article, ends with the statement "This process works for Facebook partly because Facebook

Note that Evan Priestley's post on Facebook and testing, linked in the article, ends with the statement "This process works for Facebook partly because Facebook does not, by and large, need to produce particularly high-quality software."

sriramk 15y ago
Though Evan's post made me write this, I'm actually talking about different aspects to testing than he is. But to respond to you - neither do a lot of others (need high quality software i.e). Agility->quality is a continuum. On one end, you can instantly deploy any piece of code to prod as soon as it is written. On another end, you test code for several years to make sure it is rock solid (stuff that goes into nuclear reactors). Large organizations move more to the right than they need to. Sometimes it makes more sense to optimize for bandwidth of code over quality (I believe Zuckerberg said this first).
- jrockway 15y ago
  This is not what I've seen. Large organizations are resistant to change, but it doesn't mean they test anything. I used to work for Bank of America. bankofamerica.com was down for over a week, despite a change control policy that basically prevents code from ever going into production. Big organizations like policies, but the people that make the policies do not have the knowledge required to implement or enforce them. And when they do, they don't work anyway; you can wish for perfect software, but when you hire dumb people to write it for you, it's going to break. Big companies don't want downtime, but they hire non-programmers to make this happen. The result is downtime.
  - absconditus 15y ago
    I work on medical software for a large bureaucratic company. We have a large QA department and more processes and sign-offs than I have ever seen before. The software is the lowest quality software that I have ever worked on.
    - bmajz 15y ago
      There are a bunch of different issues to be parsed out here: First and foremost, I think what this comes down to is understanding the priorities for your particular organization. There are some places where a high test:dev ratio may be critical e.g. the core pieces of a new OS or firmware for factory robots. For things like that, there is a really high cost to rolling out updates and that cost can be mitigated by paying for a second look at quality up front. For a consumer-facing web app, the cost of making a fix is often pretty low and can be mitigated by other means e.g. limited rollouts or beta previews. I think the real problem is treating various pieces of software in the same way, when in fact the fundamentals of the model are quite different. This also applies to using the same metrics (be that coverage, test pass time etc.) on each project. You have to find what works for you and not assume that the problem can be solved the same way. Second, it's not fair to go easy on the quality of test/QA folks. This is where a lot of organizations that actually invest in dedicated testers fail. Testers must be held to a similar bar as developers, just with a different functional role. As others have pointed out, the disbalance that widely varying talent levels cause in organizational health is quite disastrous. It's the same as any other piece of hiring. Make sure you have the right person for the job. That almost always means someone who knows computers/code/workflow/users at the same level as developers or product managers. Finally, as a former Windows engineer, I don't agree with the narrative that the Vista fiasco was a result of over-automation and not enough end-user scenario testing. It was more a problem of poor processes that couldn't handle the scale of the team and a lack of clear planning/leadership to account for that. Windows 7 had, if anything, more automation and more testers than Vista and was obviously a much better product. A lot of things have to go wrong to get something as ugly as the Longhorn Reset (http://en.wikipedia.org/wiki/Development_of_Windows_Vista#Mid-2004_to_Mid-2005:_Development_.22reset.22 http://en.wikipedia.org/wiki/Development_of_Windows_Vista#Mi...) and the one that OP pointed out was likely one of many.

sriramk 15y ago

Though Evan's post made me write this, I'm actually talking about different aspects to testing than he is. But to respond to you - neither do a lot of others (need high quality software i.e). Agility->quality is a continuum. On one end, you can instantly deploy any piece of code to prod as soon as it is written. On another end, you test code for several years to make sure it is rock solid (stuff that goes into nuclear reactors). Large organizations move more to the right than they need to. Sometimes it makes more sense to optimize for bandwidth of code over quality (I believe Zuckerberg said this first).

jrockway 15y ago
This is not what I've seen. Large organizations are resistant to change, but it doesn't mean they test anything. I used to work for Bank of America. bankofamerica.com was down for over a week, despite a change control policy that basically prevents code from ever going into production. Big organizations like policies, but the people that make the policies do not have the knowledge required to implement or enforce them. And when they do, they don't work anyway; you can wish for perfect software, but when you hire dumb people to write it for you, it's going to break. Big companies don't want downtime, but they hire non-programmers to make this happen. The result is downtime.
- absconditus 15y ago
  I work on medical software for a large bureaucratic company. We have a large QA department and more processes and sign-offs than I have ever seen before. The software is the lowest quality software that I have ever worked on.
  - bmajz 15y ago
    There are a bunch of different issues to be parsed out here: First and foremost, I think what this comes down to is understanding the priorities for your particular organization. There are some places where a high test:dev ratio may be critical e.g. the core pieces of a new OS or firmware for factory robots. For things like that, there is a really high cost to rolling out updates and that cost can be mitigated by paying for a second look at quality up front. For a consumer-facing web app, the cost of making a fix is often pretty low and can be mitigated by other means e.g. limited rollouts or beta previews. I think the real problem is treating various pieces of software in the same way, when in fact the fundamentals of the model are quite different. This also applies to using the same metrics (be that coverage, test pass time etc.) on each project. You have to find what works for you and not assume that the problem can be solved the same way. Second, it's not fair to go easy on the quality of test/QA folks. This is where a lot of organizations that actually invest in dedicated testers fail. Testers must be held to a similar bar as developers, just with a different functional role. As others have pointed out, the disbalance that widely varying talent levels cause in organizational health is quite disastrous. It's the same as any other piece of hiring. Make sure you have the right person for the job. That almost always means someone who knows computers/code/workflow/users at the same level as developers or product managers. Finally, as a former Windows engineer, I don't agree with the narrative that the Vista fiasco was a result of over-automation and not enough end-user scenario testing. It was more a problem of poor processes that couldn't handle the scale of the team and a lack of clear planning/leadership to account for that. Windows 7 had, if anything, more automation and more testers than Vista and was obviously a much better product. A lot of things have to go wrong to get something as ugly as the Longhorn Reset (http://en.wikipedia.org/wiki/Development_of_Windows_Vista#Mid-2004_to_Mid-2005:_Development_.22reset.22 http://en.wikipedia.org/wiki/Development_of_Windows_Vista#Mi...) and the one that OP pointed out was likely one of many.