7 ms·

A customer recently asked me to look into a program that used to run in 5 minutes but now took 1 to 4 hours. It's used by thousands of people all over the worl

by edw519 17y ago

A customer recently asked me to look into a program that used to run in 5 minutes but now took 1 to 4 hours. It's used by thousands of people all over the world all day long. It iterated through an array doing 3 SQL SELECTs against non-indexed files for each element. There used to be about 50 elements in the array; now there were more than 5000. I rewrote the whole thing in one day to do a total of 4 SELECTs and run in 12 seconds. But it took 6 days to get through QA (while the users continued to suffer). QA's biggest complaint? I indented 4 spaces instead of the (unpublished) standard of 5 spaces. Of all the things I have to deal with, nothing pisses me off more. Software QA is becoming more and more like TSA security at the airport: illogical, and obviously so. Last year, flagrantly unacceptable code was promoted without question while its replacement was held up on a meaningless detail. I got the feeling from the programmer's quotes in this essay that the same thing is happening at the app store. We programmers are a funny lot. Make us struggle for business or technical reasons and we adapt beautifully. Make us struggle for something stupid and we just get pissed off and do something else. What a pity.

cschep 17y ago
Five spaces?! Ludicrous.
lt 17y ago
Not that the indentation complaints are valid, but the fix should definitely go through QA while the users continue to suffer. While I'm sure it works on the developer machine, it is much better to have a program that takes a long time and works right than for it to give incorrect results to users, or make a different part of the application stop working. If the feature was really critical and important, QA should find a way to expedite the verification, but it should never be skipped in order to ship the fix faster.
- deleted 17y ago
  [deleted]
- jrockway 17y ago
  I've seen lots of software that's gone through QA but didn't work. Ever use Microsoft Windows? QA is important, but determining how many spaces a line of code is indented is not QA. That is a code review, which is important, but not critical. There should be automated acceptance tests, so that new releases can be shipped immediately. (It's also worth noting that stuff like indenting, naming conventions, etc. can be automatically checked at checkin time. See Perl::Critic, for example.) If there is some feature or corner case that the users test but isn't in the acceptance test suite... well... yeah, the software is going to be unreliable. So don't do that. Write a test. (I worked somewhere with a QA department once. They didn't like it when the programmers wrote tests, because they felt their job would be threatened. It was true, so we were asked to only write unit tests; no "integration" or "acceptance" tests. I quit a few weeks later.)
  - bestes 17y ago
    I did QA for years (8+) and would love to have developers write a test. Specifically NOT unit tests, but the harder more interesting one. I think it happened once. Or, was it twice?
    - erso 17y ago
      You mean like functional tests, end-to-end integration tests, and view tests? No developer should limit themselves to unit tests.
      - Quarrelsome 17y ago
        Bullshit, its attitudes like this that lead to crap software. Developers should work with QA to find the bugs, they're not "QA's bugs" they're OUR Bugs, we're a team, right? When you change a component you should also be able to write a simple manual test to prove the functionality works you should also perform this test yourself. As an optional if the QA people are good then describe what else might be effected by the code (if you edited a generic component). Good QA people will run your test, work out their own variations and also check other areas.
- rapind 17y ago
  That's only true some times. Unless the data is mission critical, or you introduce a really major bug, the company in question could be losing customer's and / or significant revenue due to the slow down of the application. I'm not saying you're wrong, I'm just saying there's no hard rule on this. I've had to push code many many times without any outside testing and while occasionally it's been detrimental (but fixed quick, because I didn't have to go through QA), the large majority of times has been beneficial. A large and formal QA process can kill a product / project just a quickly as cowboy code.
- Periodic 17y ago
  QA is basically your testing framework. It's there to do all the tests to make sure the software actually does what you say it does. No developer is good at testing their own code because deep down no developer wants to break the code they just lovingly crafted.
  - joe_the_user 17y ago
    I've always been treated well by QA. I'm sure there are problem QA departments out there but I doubt they are the rule. It's amazing to me that the parent got down moded for supporting the task of QA.
    - dasil003 17y ago
      QA is awesome. I think people are responding to the suggestion that in general developers will subconsciously protect their ego by willfully ignoring flaws in their own software. That is just tremendously offensive to any real hacker, and if it were true the world of software would be a pale shadow of what it is today.
      - xsmasher 17y ago
        I don't think it's a slam on developers - I've hit and played outfield, and I know what each side feels like. It's not that developers wants to ship bad software, but haven't you ever avoided running a test because you're afraid of opening a can of worms? QA wants to open that can. There's a perverse glee in seeing all the worms get out. When a new bug is found (a good one, not BS indentation) then QA has a justifiable joy in finding it before it got to the customer. The developer may also be devoted to quality, but doesn't your heart sink a little when you see a new bug? Overall you both want to ship good product, but your minute-to-minute motivations are just different.
  - rapind 17y ago
    As amusing as that statement is (and I think you meant it jokingly), I definitely have to disagree. There's just too many different developer styles to say No developer. Personally I like to ship quality code and am pretty anal about it's quality as well as it's interface, so you can bet I refactor my code a lot... and I mean a lot. (In fact I just refactored this comment to fix a spelling mistake... but I published it without writing any automated tests, or running it by QA... somehow I don't think many people noticed)
Xichekolas 17y ago
I find when this stuff happens with QA teams, it's almost always a case of protecting turf. When no one is looking (the first, buggy, submission is made), they will do the minimum necessary and approve it, often quickly. Then, when everything goes to hell, all eyes are on them. The developers submit a fix with the note "this needs to be approved ASAP to put out fires"... the QA person resents being told what to do, and suddenly feels compelled to justify his job by finding anything and everything that could possibly be considered 'wrong'. There is also the issue that things like indentation are more visibly 'wrong' than obscure bugs that only show up in production environments due to the way database replication is set up. Now, I'm not saying all QA people are like this. I've worked with some downright awesome QA people before. But awesome QA people don't reject things for spite, and usually don't stay in QA long before moving up.
- cakesy 17y ago
  Very cynical. It is more like there is a problem with an important application, everybody gets the blame. The developer, QA, PM. So everybody wants to make sure it goes well. The developer. The QA. The QA doesn't always understand all the contraints put on him, that he has to follow. For example, if this is a fortran program, the indenting will cause it to break. So rather than second guessing every step, he follow every step of the book. So if there is another problem, he doesn't take the blame.
brown9-2 17y ago
This might be a controversial opinion, but does anyone else think that if the QA team is reading your source code then you are doing QA wrong? Or do you really mean "held up by code review" when you say "get through QA"?
- emmett 17y ago
  No, at some companies people really do have the QA department ensure "code quality". After all, it's Quality Assurance, so they must know about code Quality!
  - rbanffy 17y ago
    The best QA teams have developers and security people in them exactly for this reason. Quality is not what the users see - it's a correct implementation of what the specs say, built upon a series of basic assumptions that apply to all software they ship. Treating QA as a step after development is a mistake that causes endless suffering to users.
    - shpxnvz 17y ago
      It seems to me like treating QA as developers who are not allowed to write code, but can criticize it, would tend to result in a situation exactly like the one described. Indeed QA is not a step after development, but I'd argue that encouraging developers to become better at QA themselves is a more effective approach than having non-coders inspecting sources after the fact.
petenixey 17y ago
The value of QA is a very delicate balance: Value - stuff should work after it's been through them Costs that I've seen - more stuff doesn't get done/fixed because of the delay time associated with QA - things don't get scheduled because of lack of QA bandwidth - additional specifications required in order to go through QA - programmers / PMs worry less about catching bugs because they feel it'll come back from QA When is it better to QA something rather than just measure when it's broken? Everything we do goes through QA but I'm sure most of it is a net loss. Anyone have a good way of knowing what to QA and what to simply monitor?
bumblebird 17y ago
Agreed. The last company I worked at we had some similar idiocy. We had a backend process that connected out to a network and did some things. However, the network it connected to changed their protocol, so our backend process was broken - didn't do anything - completely non functional. We created a fix, but had to wait for it to go through QA for ages. What's the worst that could happen? It doesn't do anything like the current version? It wasn't even a massive update. Maddening.
- lt 17y ago
  The worst? Data Loss, or corruption, which is worse than "not working". Likewise, more likely it could just not work well, be buggy and require several updates to get it right, burning out your team's (or your company's) image with the customer. Before you deployed your fix, it's whoever changed the protocol at fault. After that, it's you that can't fix it. Again, not justifing it taking ages, just saying there's never an excuse to skip QA.
  - bumblebird 17y ago
    There wasn't any data. It did X with the external network. The worst case was that it maxes out the connection and burns through bandwidth. But that's something that can be monitored. The customer would have rather had something working with the network, than something broken with the network. In that particular circumstance, there's always an excuse to skip QA. IMHO.
    - random42 17y ago
      there's always an excuse to skip QA. IMHO. No, there isn't!
  - roundsquare 17y ago
    The worst? Data Loss, or corruption, which is worse than "not working". Thats a good point (even though in this case it turns out not to be true). Seems like the right answer is to back out the change.
dylanz 17y ago
You indented with spaces instead of tabs!? Just kidding of course, but the QA team might have been on the other side of that holy war. I'm going to have to start using your TSA parallel though... it's so painfully true.
indiejade 17y ago
We programmers are a funny lot. Make us struggle for business or technical reasons and we adapt beautifully. Make us struggle for something stupid and we just get pissed off and do something else. This is the best YC quote of all time. Thx, edwin!
rbanffy 17y ago
What kind of insane person indents at 5 spaces. I get 2, 4, 8, but 5 is beyond me.

by edw519 17y ago

cschep 17y ago

Five spaces?! Ludicrous.

lt 17y ago

Not that the indentation complaints are valid, but the fix should definitely go through QA while the users continue to suffer. While I'm sure it works on the developer machine, it is much better to have a program that takes a long time and works right than for it to give incorrect results to users, or make a different part of the application stop working. If the feature was really critical and important, QA should find a way to expedite the verification, but it should never be skipped in order to ship the fix faster.

deleted 17y ago
[deleted]
jrockway 17y ago
I've seen lots of software that's gone through QA but didn't work. Ever use Microsoft Windows? QA is important, but determining how many spaces a line of code is indented is not QA. That is a code review, which is important, but not critical. There should be automated acceptance tests, so that new releases can be shipped immediately. (It's also worth noting that stuff like indenting, naming conventions, etc. can be automatically checked at checkin time. See Perl::Critic, for example.) If there is some feature or corner case that the users test but isn't in the acceptance test suite... well... yeah, the software is going to be unreliable. So don't do that. Write a test. (I worked somewhere with a QA department once. They didn't like it when the programmers wrote tests, because they felt their job would be threatened. It was true, so we were asked to only write unit tests; no "integration" or "acceptance" tests. I quit a few weeks later.)
- bestes 17y ago
  I did QA for years (8+) and would love to have developers write a test. Specifically NOT unit tests, but the harder more interesting one. I think it happened once. Or, was it twice?
  - erso 17y ago
    You mean like functional tests, end-to-end integration tests, and view tests? No developer should limit themselves to unit tests.
    - Quarrelsome 17y ago
      Bullshit, its attitudes like this that lead to crap software. Developers should work with QA to find the bugs, they're not "QA's bugs" they're OUR Bugs, we're a team, right? When you change a component you should also be able to write a simple manual test to prove the functionality works you should also perform this test yourself. As an optional if the QA people are good then describe what else might be effected by the code (if you edited a generic component). Good QA people will run your test, work out their own variations and also check other areas.
rapind 17y ago
That's only true some times. Unless the data is mission critical, or you introduce a really major bug, the company in question could be losing customer's and / or significant revenue due to the slow down of the application. I'm not saying you're wrong, I'm just saying there's no hard rule on this. I've had to push code many many times without any outside testing and while occasionally it's been detrimental (but fixed quick, because I didn't have to go through QA), the large majority of times has been beneficial. A large and formal QA process can kill a product / project just a quickly as cowboy code.
Periodic 17y ago
QA is basically your testing framework. It's there to do all the tests to make sure the software actually does what you say it does. No developer is good at testing their own code because deep down no developer wants to break the code they just lovingly crafted.
- joe_the_user 17y ago
  I've always been treated well by QA. I'm sure there are problem QA departments out there but I doubt they are the rule. It's amazing to me that the parent got down moded for supporting the task of QA.
  - dasil003 17y ago
    QA is awesome. I think people are responding to the suggestion that in general developers will subconsciously protect their ego by willfully ignoring flaws in their own software. That is just tremendously offensive to any real hacker, and if it were true the world of software would be a pale shadow of what it is today.
    - xsmasher 17y ago
      I don't think it's a slam on developers - I've hit and played outfield, and I know what each side feels like. It's not that developers wants to ship bad software, but haven't you ever avoided running a test because you're afraid of opening a can of worms? QA wants to open that can. There's a perverse glee in seeing all the worms get out. When a new bug is found (a good one, not BS indentation) then QA has a justifiable joy in finding it before it got to the customer. The developer may also be devoted to quality, but doesn't your heart sink a little when you see a new bug? Overall you both want to ship good product, but your minute-to-minute motivations are just different.
- rapind 17y ago
  As amusing as that statement is (and I think you meant it jokingly), I definitely have to disagree. There's just too many different developer styles to say No developer. Personally I like to ship quality code and am pretty anal about it's quality as well as it's interface, so you can bet I refactor my code a lot... and I mean a lot. (In fact I just refactored this comment to fix a spelling mistake... but I published it without writing any automated tests, or running it by QA... somehow I don't think many people noticed)

Xichekolas 17y ago

I find when this stuff happens with QA teams, it's almost always a case of protecting turf. When no one is looking (the first, buggy, submission is made), they will do the minimum necessary and approve it, often quickly. Then, when everything goes to hell, all eyes are on them. The developers submit a fix with the note "this needs to be approved ASAP to put out fires"... the QA person resents being told what to do, and suddenly feels compelled to justify his job by finding anything and everything that could possibly be considered 'wrong'. There is also the issue that things like indentation are more visibly 'wrong' than obscure bugs that only show up in production environments due to the way database replication is set up. Now, I'm not saying all QA people are like this. I've worked with some downright awesome QA people before. But awesome QA people don't reject things for spite, and usually don't stay in QA long before moving up.

cakesy 17y ago
Very cynical. It is more like there is a problem with an important application, everybody gets the blame. The developer, QA, PM. So everybody wants to make sure it goes well. The developer. The QA. The QA doesn't always understand all the contraints put on him, that he has to follow. For example, if this is a fortran program, the indenting will cause it to break. So rather than second guessing every step, he follow every step of the book. So if there is another problem, he doesn't take the blame.

brown9-2 17y ago

This might be a controversial opinion, but does anyone else think that if the QA team is reading your source code then you are doing QA wrong? Or do you really mean "held up by code review" when you say "get through QA"?

emmett 17y ago
No, at some companies people really do have the QA department ensure "code quality". After all, it's Quality Assurance, so they must know about code Quality!
- rbanffy 17y ago
  The best QA teams have developers and security people in them exactly for this reason. Quality is not what the users see - it's a correct implementation of what the specs say, built upon a series of basic assumptions that apply to all software they ship. Treating QA as a step after development is a mistake that causes endless suffering to users.
  - shpxnvz 17y ago
    It seems to me like treating QA as developers who are not allowed to write code, but can criticize it, would tend to result in a situation exactly like the one described. Indeed QA is not a step after development, but I'd argue that encouraging developers to become better at QA themselves is a more effective approach than having non-coders inspecting sources after the fact.

petenixey 17y ago

The value of QA is a very delicate balance: Value - stuff should work after it's been through them Costs that I've seen - more stuff doesn't get done/fixed because of the delay time associated with QA - things don't get scheduled because of lack of QA bandwidth - additional specifications required in order to go through QA - programmers / PMs worry less about catching bugs because they feel it'll come back from QA When is it better to QA something rather than just measure when it's broken? Everything we do goes through QA but I'm sure most of it is a net loss. Anyone have a good way of knowing what to QA and what to simply monitor?

bumblebird 17y ago

Agreed. The last company I worked at we had some similar idiocy. We had a backend process that connected out to a network and did some things. However, the network it connected to changed their protocol, so our backend process was broken - didn't do anything - completely non functional. We created a fix, but had to wait for it to go through QA for ages. What's the worst that could happen? It doesn't do anything like the current version? It wasn't even a massive update. Maddening.

lt 17y ago
The worst? Data Loss, or corruption, which is worse than "not working". Likewise, more likely it could just not work well, be buggy and require several updates to get it right, burning out your team's (or your company's) image with the customer. Before you deployed your fix, it's whoever changed the protocol at fault. After that, it's you that can't fix it. Again, not justifing it taking ages, just saying there's never an excuse to skip QA.
- bumblebird 17y ago
  There wasn't any data. It did X with the external network. The worst case was that it maxes out the connection and burns through bandwidth. But that's something that can be monitored. The customer would have rather had something working with the network, than something broken with the network. In that particular circumstance, there's always an excuse to skip QA. IMHO.
  - random42 17y ago
    there's always an excuse to skip QA. IMHO. No, there isn't!
- roundsquare 17y ago
  The worst? Data Loss, or corruption, which is worse than "not working". Thats a good point (even though in this case it turns out not to be true). Seems like the right answer is to back out the change.

dylanz 17y ago

You indented with spaces instead of tabs!? Just kidding of course, but the QA team might have been on the other side of that holy war. I'm going to have to start using your TSA parallel though... it's so painfully true.

indiejade 17y ago

We programmers are a funny lot. Make us struggle for business or technical reasons and we adapt beautifully. Make us struggle for something stupid and we just get pissed off and do something else. This is the best YC quote of all time. Thx, edwin!

rbanffy 17y ago

What kind of insane person indents at 5 spaces. I get 2, 4, 8, but 5 is beyond me.