5 ms·

> AWS is pretty reliable for the most part so I am pretty surprise that the code quality is that bad. I'm not totally surprised because of two factors: very st

> AWS is pretty reliable for the most part so I am pretty surprise that the code quality is that bad. I'm not totally surprised because of two factors: very stable product definitions and lots and lots of users. A number of years back, I was talking with people at a famous and popular site with a broad audience. I asked them how much unit testing they did. They said that particular isolated pieces sometimes had tests. But most of the user-facing stuff didn't because they had one-button rollout and one-button rollback. Instead of bothering with unit tests, they'd just frequently release changes, watch the metrics and the customer support queue, and quickly roll back if they'd introduced a bug.

zorked 5y ago
For very, very popular services, a second of being live will exercise more code paths and edge cases than even the most dedicated testing team could ever dream of. We hear a hell of a lot about testing but the most fundamental piece of software quality nowadays is the release strategy: running on tee'd live production traffic, canarying, metrics and alerting, quick roll backs, etc.
- bradleyjg 5y ago
  Depends on if you are serving ads over cat pictures or routing air traffic. Different solutions for different problems.
- sreque 5y ago
  It's a very short-sighted view on testing, although I'm not surprised SREs would say it. The biggest problem with software deployment is that it is owned and managed by people who have no vested interest in developer productivity, including devops engineers. A major goal of any org should be developer productivity; otherwise you are just hemorrhaging money and talent. When I say developer productivity, I mean: How confidently and quickly can I make a shippable, rollback-free change to a unit of software? If you are the dos equis man of testing, "I don't always test my code, but when I do, I do it in production", then you can't confidently make any change without risking a production outage, so you play lots of games, like you mentioned, around canarying, rolling out to a small percentage of users, etc., but at the end of the day your developer productivity has absolutely tanked. The goal of any system maintenance should be that a developer can quickly make and test a change locally and be highly confident that the change is correct. The canarying, phased rollouts, and other such systems should not be the primary means of testing code correctness.
  - wpietri 5y ago
    Yeah, I really appreciate excellent rollout strategies, although I suspect a lot of them are more developed out of self defense by SRE teams. I see it as a series of safety nets: I'm still going to write tests for my code so that I don't have far to fall if I make a mistake. But I also want a safe rollout so if I miss the first net I don't splatter on the pavement. And I totally agree with out about developer productivity. It's just not a consideration in most places. For example, in a factory or a restaurant, meetings are things that happen rarely and in constrained time slots, because everybody realizes that production is primary. But in most software companies, actually getting work done is second priority to meetings.
    - sreque 5y ago
      Agreed. I was an SRE for over a year and the philosophy is that anything that is shipped can be broken. SRE is all about detecting, limiting, and mitigating damage. I think this is the right philosophy for SREs but should not be the total picture in the org. I am also agreed in that I anecdotally often see a disregard for automated testing. I am still trying to understand how to eliminate this tendency. I know that in every software project I've had a major hand in building, I've helped ensure automated testing, with a heavy emphasis on unit testing, become a major part of team culture, and I've always felt the tests more than paid for themselves over time, even in the relative short-term.
      - wpietri 5y ago
        For sure. I'm still trying to understand it too. Two things that have helped me: To contain time pressure, I like a kanban board with small units of work. If the team has a history of steady delivery of small lumps of useful stuff, managers are more willing to trust that we know what we're doing. To mitigate the natural human lack of humility, I start with the rule that every bug requires a test before fixing. People may act as if they won't make mistakes, but it's much harder to claim that when we have a live bug. And then I like to introduce pair programming. Collaboratively adding failing tests and fixing them (as with ping-pong pairing) makes it fun. What approaches do you use to shift things over?
- notacoward 5y ago
  That's an overly general statement. Can you do that for front-end code that stores all of its state elsewhere? Sure. Can you do it for a storage system? Absolutely freaking not. If you introduce a bug that loses or corrupts data, there's no going back. You will have committed the worst sin that somebody in that specialty can commit. Better to test as much as you can, at every level. Other kinds of code are often somewhere in between. Also, even if it's true that being live will exercise more edge cases etc., it's a terrible way to test changes during early development. For one thing, there's no isolation. It becomes harder to determine which of several recent changes caused a problem, and that burden unfairly falls on the person who's on call instead of the person who introduced the error. And decent unit/functional tests allow "dumb" mistakes (we all make them) to be caught earlier than waiting in a deploy queue, allowing faster iteration. "Most recent change probably caused the problem" is a very useful heuristic, but the more low-assurance changes you allow in the less useful it becomes. To drive the point home even further: I have found data-loss bugs in focused testing that didn't show up in prod for months. I know because in many cases I was able to add logging for the preconditions when I fixed the bug. No logs for months, then some completely unrelated and completely valid change by another engineer tickles the preconditions and BAM. That would have been an absolute nightmare for other members of my team, possibly even after I was gone. Based on those experiences, I will never believe that foregoing systematic early tests can be valid. The systems most of us work on are too complex for that. "Test in prod" only works for trivial code and/or trivial teams. Not in the grown-up world.
  - deleted 5y ago
    [deleted]
  - ed_elliott_asc 5y ago
    Everyone should be testing in prod, in that you release code and see metrics and monitoring to show that everything is working. Testing in production is not going “let’s see if this will work” it is “we will release and validate that everything is working as expected” People need to get over the old school cowboys who jump on prod to see if something works.
    - 5y ago
- polotics 5y ago
  Tell that to the millisecond of testing in production that makes the MRI fry the patient's brain, to the one that trades one trillion instead of one thousand naked puts, to the nuclear armaggedon launch check that canaries humanity...
- treis 5y ago
  >For very, very popular services, a second of being live will exercise more code paths and edge cases than even the most dedicated testing team could ever dream of. Most of the code we care about is to handle anomalous situations. That AZ going down a week or two is a good example. It's when stuff like that happens that a bunch of code springs to life to keep things running. And indeed, things didn't exactly roll over just fine for us.

zorked 5y ago

For very, very popular services, a second of being live will exercise more code paths and edge cases than even the most dedicated testing team could ever dream of. We hear a hell of a lot about testing but the most fundamental piece of software quality nowadays is the release strategy: running on tee'd live production traffic, canarying, metrics and alerting, quick roll backs, etc.

bradleyjg 5y ago
Depends on if you are serving ads over cat pictures or routing air traffic. Different solutions for different problems.
sreque 5y ago
It's a very short-sighted view on testing, although I'm not surprised SREs would say it. The biggest problem with software deployment is that it is owned and managed by people who have no vested interest in developer productivity, including devops engineers. A major goal of any org should be developer productivity; otherwise you are just hemorrhaging money and talent. When I say developer productivity, I mean: How confidently and quickly can I make a shippable, rollback-free change to a unit of software? If you are the dos equis man of testing, "I don't always test my code, but when I do, I do it in production", then you can't confidently make any change without risking a production outage, so you play lots of games, like you mentioned, around canarying, rolling out to a small percentage of users, etc., but at the end of the day your developer productivity has absolutely tanked. The goal of any system maintenance should be that a developer can quickly make and test a change locally and be highly confident that the change is correct. The canarying, phased rollouts, and other such systems should not be the primary means of testing code correctness.
- wpietri 5y ago
  Yeah, I really appreciate excellent rollout strategies, although I suspect a lot of them are more developed out of self defense by SRE teams. I see it as a series of safety nets: I'm still going to write tests for my code so that I don't have far to fall if I make a mistake. But I also want a safe rollout so if I miss the first net I don't splatter on the pavement. And I totally agree with out about developer productivity. It's just not a consideration in most places. For example, in a factory or a restaurant, meetings are things that happen rarely and in constrained time slots, because everybody realizes that production is primary. But in most software companies, actually getting work done is second priority to meetings.
  - sreque 5y ago
    Agreed. I was an SRE for over a year and the philosophy is that anything that is shipped can be broken. SRE is all about detecting, limiting, and mitigating damage. I think this is the right philosophy for SREs but should not be the total picture in the org. I am also agreed in that I anecdotally often see a disregard for automated testing. I am still trying to understand how to eliminate this tendency. I know that in every software project I've had a major hand in building, I've helped ensure automated testing, with a heavy emphasis on unit testing, become a major part of team culture, and I've always felt the tests more than paid for themselves over time, even in the relative short-term.
    - wpietri 5y ago
      For sure. I'm still trying to understand it too. Two things that have helped me: To contain time pressure, I like a kanban board with small units of work. If the team has a history of steady delivery of small lumps of useful stuff, managers are more willing to trust that we know what we're doing. To mitigate the natural human lack of humility, I start with the rule that every bug requires a test before fixing. People may act as if they won't make mistakes, but it's much harder to claim that when we have a live bug. And then I like to introduce pair programming. Collaboratively adding failing tests and fixing them (as with ping-pong pairing) makes it fun. What approaches do you use to shift things over?
notacoward 5y ago
That's an overly general statement. Can you do that for front-end code that stores all of its state elsewhere? Sure. Can you do it for a storage system? Absolutely freaking not. If you introduce a bug that loses or corrupts data, there's no going back. You will have committed the worst sin that somebody in that specialty can commit. Better to test as much as you can, at every level. Other kinds of code are often somewhere in between. Also, even if it's true that being live will exercise more edge cases etc., it's a terrible way to test changes during early development. For one thing, there's no isolation. It becomes harder to determine which of several recent changes caused a problem, and that burden unfairly falls on the person who's on call instead of the person who introduced the error. And decent unit/functional tests allow "dumb" mistakes (we all make them) to be caught earlier than waiting in a deploy queue, allowing faster iteration. "Most recent change probably caused the problem" is a very useful heuristic, but the more low-assurance changes you allow in the less useful it becomes. To drive the point home even further: I have found data-loss bugs in focused testing that didn't show up in prod for months. I know because in many cases I was able to add logging for the preconditions when I fixed the bug. No logs for months, then some completely unrelated and completely valid change by another engineer tickles the preconditions and BAM. That would have been an absolute nightmare for other members of my team, possibly even after I was gone. Based on those experiences, I will never believe that foregoing systematic early tests can be valid. The systems most of us work on are too complex for that. "Test in prod" only works for trivial code and/or trivial teams. Not in the grown-up world.
- deleted 5y ago
  [deleted]
- ed_elliott_asc 5y ago
  Everyone should be testing in prod, in that you release code and see metrics and monitoring to show that everything is working. Testing in production is not going “let’s see if this will work” it is “we will release and validate that everything is working as expected” People need to get over the old school cowboys who jump on prod to see if something works.
  - 5y ago
polotics 5y ago
Tell that to the millisecond of testing in production that makes the MRI fry the patient's brain, to the one that trades one trillion instead of one thousand naked puts, to the nuclear armaggedon launch check that canaries humanity...
treis 5y ago
>For very, very popular services, a second of being live will exercise more code paths and edge cases than even the most dedicated testing team could ever dream of. Most of the code we care about is to handle anomalous situations. That AZ going down a week or two is a good example. It's when stuff like that happens that a bunch of code springs to life to keep things running. And indeed, things didn't exactly roll over just fine for us.