5 ms·

I disagree with almost every single point of this article except that buying can be better than building (note this still isn't always true, especially if you a

I disagree with almost every single point of this article except that buying can be better than building (note this still isn't always true, especially if you are in a very large company that would get charged usurious rates based on usage). The other points really only hold true for things that are allowed to break. As someone who writes code that should never, ever break, and gets support tickets even for one-in-a-million breakages, testing is absolutely paramount. I can see how certain products, like a web-app, could follow these principles and benefit from not being over-tested, but it's definitely not an absolute. Of course, my code does still break. The author is right that this is still an absolute. But the blast radius of the breakage and the frequency are both quite low because of the measures we take to prevent them. Regarding the point about ditching QA and testing in production: both are possible. You can make QA a shard of your production environment (there are many different ways to implement this based on your tech). That is the ideal way to structure it so that it's actually pretty representative of a real environment without requiring lots of duct tape.

stefanmichael 6y ago
I think you're right. Context matters quite a bit. I have worked in places (quant / hft / algo trading firms) where "but we can fix it fast!" is not a defensible position and "fast" is not fast enough. Thinking about this article through the lens of an application where the cost of failure is quite low and time to fix is quite fast, the article starts making sense.
rualca 6y ago
> I disagree with almost every single point of this article except that buying can be better than building I agree, and I found the comment on how QAs somehow made quality worse to be specially absurd and detached from reality. I mean, the author went as far as presenting the brilliant solution to his problem by suggesting... Automating manual testing? Where did he came up with this stuff? I mean, for a couple of decades now, manual tests are only reserved to either tests that are not possible to automate, of exploratory testing. It's a suggestion in line of "let them eat cake", because it shows total detachment from the real world and specially how things are actually done in the industry. And how exactly can QAs degrade quality if all they do is check if the work done by developers does indeed work and complies with requirements ? It's mind boggling. QAs don't change code. QAs see the mess you do before it hits the customer. The absurdity of this mess reaches a point where the author complains that a failure by the developer to update tests to reflect his changes is somehow the fault of the guys tasked with running the tests? I mean, come on. All in all I was left with the impression that the author is totally oblivious to how he has been introducing and living with abhorrent software development practices and, to further compound the problems he has been creating for him and for others, he's shifting the blame away from himself to everyone around him.
- tuwtuwtuwtuw 6y ago
  > how exactly can QAs degrade quality if all they do is check if the work done by developers does indeed work and complies with requirements ? One thing I have seen far too many times is that a strong QA team may lead to developers not feeling the responsibility of making sure things works themselves. This often leads to worse quality. This is or course a social issue rather than a technical and I'm sure that many developers do not suffer from this mindsdt. But it's been there to some degree in almost all companies I have consulted for. Surely you have seen this mindset yourself?
  - garethrowlands 6y ago
    I've seen this and agree with you. Though, if the nature of your QA teams led to worse quality, you might want to find a term other than "strong" to describe them. Because "strong" implies you want more of that, which isn't what you meant at all. Perhaps "dominant"?
- nullsense 6y ago
  On the contrary it's all largely advice based off Accelerate / State of DevOps reports around how high performing organizations that ship code on demand are able to do so without increasing change failure rate. He mispoke when he said manual QA makes the quality worse. But the data do suggest that tossing your code over the wall to pass through a manual QA gate before being scheduled for a release doesn't actually improve quality when compared with shipping every merged PR to production provided it passes the automated pipeline and doing a staged rollout of the code to production users via a decent feature flag system. Continuous Delivery has been a thing for quite a while it seems but I think it's still relatively early in terms of adoption. There are a number of conference talks from recent years about testing in production. I don't live and work in Silicon Valley and most of the companies where I am aren't doing it, but I know of one that is and it sounds pretty incredible from what I've heard.
deleted 6y ago
[deleted]
dmitriid 6y ago
> As someone who writes code that should never, ever break, and gets support tickets even for one-in-a-million breakages, testing is absolutely paramount. Better put: you should test to acceptable limits. A nuclear reactor or medical software should be tested with any and all possible and impossible scenarios in mind. Some forum or chat software? Not so much. And obviously there are 50 million shades in between :)
- toast0 6y ago
  The trick is, one person's impossible scenario is another person's normal operating condition. I've made chat software work when DNS is broken. Other people say that's an impossible situation.
  - rmchugh 6y ago
    Definitely not an impossible situation. Could even be a regular situation given a flaky DNS provider and a sufficient volume of requests.
  - iib 6y ago
    That sounds like an interesting problem. Would you want to expand on the solution?
    - lstamour 6y ago
      First thing that occurred to me was WebRTC for P2P chat using an IP address in the STUN fallback list if DNS fails? As in https://github.com/michal-wrzosek/p2p-chat https://github.com/michal-wrzosek/p2p-chat
    - toast0 6y ago
      The solution is fairly simple in principle. You need to have fallback IPs in your clients, that they can connect to in case DNS fails. But that also means you need to have IPs that will last for as long as your client lasts. Which means you need to have a defined lifetime for your client. And you need to have some mechanism to validate the IPs are still your servers and not been taken over by someone else (mutual authentication).

stefanmichael 6y ago

I think you're right. Context matters quite a bit. I have worked in places (quant / hft / algo trading firms) where "but we can fix it fast!" is not a defensible position and "fast" is not fast enough. Thinking about this article through the lens of an application where the cost of failure is quite low and time to fix is quite fast, the article starts making sense.

rualca 6y ago

> I disagree with almost every single point of this article except that buying can be better than building I agree, and I found the comment on how QAs somehow made quality worse to be specially absurd and detached from reality. I mean, the author went as far as presenting the brilliant solution to his problem by suggesting... Automating manual testing? Where did he came up with this stuff? I mean, for a couple of decades now, manual tests are only reserved to either tests that are not possible to automate, of exploratory testing. It's a suggestion in line of "let them eat cake", because it shows total detachment from the real world and specially how things are actually done in the industry. And how exactly can QAs degrade quality if all they do is check if the work done by developers does indeed work and complies with requirements ? It's mind boggling. QAs don't change code. QAs see the mess you do before it hits the customer. The absurdity of this mess reaches a point where the author complains that a failure by the developer to update tests to reflect his changes is somehow the fault of the guys tasked with running the tests? I mean, come on. All in all I was left with the impression that the author is totally oblivious to how he has been introducing and living with abhorrent software development practices and, to further compound the problems he has been creating for him and for others, he's shifting the blame away from himself to everyone around him.

tuwtuwtuwtuw 6y ago
> how exactly can QAs degrade quality if all they do is check if the work done by developers does indeed work and complies with requirements ? One thing I have seen far too many times is that a strong QA team may lead to developers not feeling the responsibility of making sure things works themselves. This often leads to worse quality. This is or course a social issue rather than a technical and I'm sure that many developers do not suffer from this mindsdt. But it's been there to some degree in almost all companies I have consulted for. Surely you have seen this mindset yourself?
- garethrowlands 6y ago
  I've seen this and agree with you. Though, if the nature of your QA teams led to worse quality, you might want to find a term other than "strong" to describe them. Because "strong" implies you want more of that, which isn't what you meant at all. Perhaps "dominant"?
nullsense 6y ago
On the contrary it's all largely advice based off Accelerate / State of DevOps reports around how high performing organizations that ship code on demand are able to do so without increasing change failure rate. He mispoke when he said manual QA makes the quality worse. But the data do suggest that tossing your code over the wall to pass through a manual QA gate before being scheduled for a release doesn't actually improve quality when compared with shipping every merged PR to production provided it passes the automated pipeline and doing a staged rollout of the code to production users via a decent feature flag system. Continuous Delivery has been a thing for quite a while it seems but I think it's still relatively early in terms of adoption. There are a number of conference talks from recent years about testing in production. I don't live and work in Silicon Valley and most of the companies where I am aren't doing it, but I know of one that is and it sounds pretty incredible from what I've heard.

deleted 6y ago

[deleted]

dmitriid 6y ago

> As someone who writes code that should never, ever break, and gets support tickets even for one-in-a-million breakages, testing is absolutely paramount. Better put: you should test to acceptable limits. A nuclear reactor or medical software should be tested with any and all possible and impossible scenarios in mind. Some forum or chat software? Not so much. And obviously there are 50 million shades in between :)

toast0 6y ago
The trick is, one person's impossible scenario is another person's normal operating condition. I've made chat software work when DNS is broken. Other people say that's an impossible situation.
- rmchugh 6y ago
  Definitely not an impossible situation. Could even be a regular situation given a flaky DNS provider and a sufficient volume of requests.
- iib 6y ago
  That sounds like an interesting problem. Would you want to expand on the solution?
  - lstamour 6y ago
    First thing that occurred to me was WebRTC for P2P chat using an IP address in the STUN fallback list if DNS fails? As in https://github.com/michal-wrzosek/p2p-chat https://github.com/michal-wrzosek/p2p-chat
  - toast0 6y ago
    The solution is fairly simple in principle. You need to have fallback IPs in your clients, that they can connect to in case DNS fails. But that also means you need to have IPs that will last for as long as your client lasts. Which means you need to have a defined lifetime for your client. And you need to have some mechanism to validate the IPs are still your servers and not been taken over by someone else (mutual authentication).