3 ms·

That's an overly general statement. Can you do that for front-end code that stores all of its state elsewhere? Sure. Can you do it for a storage system? Absolut

That's an overly general statement. Can you do that for front-end code that stores all of its state elsewhere? Sure. Can you do it for a storage system? Absolutely freaking not. If you introduce a bug that loses or corrupts data, there's no going back. You will have committed the worst sin that somebody in that specialty can commit. Better to test as much as you can, at every level. Other kinds of code are often somewhere in between. Also, even if it's true that being live will exercise more edge cases etc., it's a terrible way to test changes during early development. For one thing, there's no isolation. It becomes harder to determine which of several recent changes caused a problem, and that burden unfairly falls on the person who's on call instead of the person who introduced the error. And decent unit/functional tests allow "dumb" mistakes (we all make them) to be caught earlier than waiting in a deploy queue, allowing faster iteration. "Most recent change probably caused the problem" is a very useful heuristic, but the more low-assurance changes you allow in the less useful it becomes. To drive the point home even further: I have found data-loss bugs in focused testing that didn't show up in prod for months. I know because in many cases I was able to add logging for the preconditions when I fixed the bug. No logs for months, then some completely unrelated and completely valid change by another engineer tickles the preconditions and BAM. That would have been an absolute nightmare for other members of my team, possibly even after I was gone. Based on those experiences, I will never believe that foregoing systematic early tests can be valid. The systems most of us work on are too complex for that. "Test in prod" only works for trivial code and/or trivial teams. Not in the grown-up world.

deleted 5y ago
[deleted]
ed_elliott_asc 5y ago
Everyone should be testing in prod, in that you release code and see metrics and monitoring to show that everything is working. Testing in production is not going “let’s see if this will work” it is “we will release and validate that everything is working as expected” People need to get over the old school cowboys who jump on prod to see if something works.
- notacoward 5y ago
  Yes, everyone should release code and watch metrics etc. but I think that's at the very edge of what "testing" encompasses. Between model checking, traditional forms of testing, and shadow-traffic testing (which can test higher per-server load than prod), finding something after deploy should be like a parachute failure. Yes those happen, yes there should be a reserve, but if it happens more than once in a blue moon you have a process problem somewhere (quite likely between teams/services but still).
  - mavelikara 5y ago
    Tangential, where can I learn more about shadow-traffic testing? Books, blogs, tools etc.
    - notacoward 5y ago
      Cindy Sridharan often writes well about these topics. https://copyconstruct.medium.com/testing-in-production-the-safe-way-18ca102d0ef1 https://copyconstruct.medium.com/testing-in-production-the-s... For this one I'd start about half way down, under the heading "Shadowing (also known as Dark Traffic Testing or Mirroring)" Unfortunately the terminology is a bit fragmented - shadowing, mirroring, teeing, dark traffic (ick), ad nauseam. Whatever you call it, it's often pretty high overhead to add the infrastructure, unless you're already using some sort of "service mesh" (Envoy/Istio/Caddy/whatever) that supports it. Even then, if you're dealing specifically with a storage system then there can be some thorny issues - idempotent vs. non-idempotent vs. destructive requests, requests which require other objects (files/objects or directories/buckets) to exist or be in specific states, etc. I'm not going to pretend it's easy. If you can do it, though, it can be an incredibly valuable tool. There ain't nothing like the real traffic, baby. ;) My favorite feature, which I alluded to earlier, is that you can shadow traffic from a larger production cluster onto a smaller shadow cluster and give it a serious stress test. All sorts of bugs tend to fall out that way. The one thing you can't really catch, even with a good shadow, is interactions with other services - including things like permissions or quotas. But if those are the only things you have to shake out in true production, you're doing well.
joshuamorton 5y ago
Teeing/dark launch/dual write strategies solve most of the issue for databases. Sure you run into concerns when changing the framework that manages that, but that's usually a far smaller surface area than your entire storage layer. That said, you should have tests anyway.

deleted 5y ago

[deleted]

ed_elliott_asc 5y ago

Everyone should be testing in prod, in that you release code and see metrics and monitoring to show that everything is working. Testing in production is not going “let’s see if this will work” it is “we will release and validate that everything is working as expected” People need to get over the old school cowboys who jump on prod to see if something works.

notacoward 5y ago
Yes, everyone should release code and watch metrics etc. but I think that's at the very edge of what "testing" encompasses. Between model checking, traditional forms of testing, and shadow-traffic testing (which can test higher per-server load than prod), finding something after deploy should be like a parachute failure. Yes those happen, yes there should be a reserve, but if it happens more than once in a blue moon you have a process problem somewhere (quite likely between teams/services but still).
- mavelikara 5y ago
  Tangential, where can I learn more about shadow-traffic testing? Books, blogs, tools etc.
  - notacoward 5y ago
    Cindy Sridharan often writes well about these topics. https://copyconstruct.medium.com/testing-in-production-the-safe-way-18ca102d0ef1 https://copyconstruct.medium.com/testing-in-production-the-s... For this one I'd start about half way down, under the heading "Shadowing (also known as Dark Traffic Testing or Mirroring)" Unfortunately the terminology is a bit fragmented - shadowing, mirroring, teeing, dark traffic (ick), ad nauseam. Whatever you call it, it's often pretty high overhead to add the infrastructure, unless you're already using some sort of "service mesh" (Envoy/Istio/Caddy/whatever) that supports it. Even then, if you're dealing specifically with a storage system then there can be some thorny issues - idempotent vs. non-idempotent vs. destructive requests, requests which require other objects (files/objects or directories/buckets) to exist or be in specific states, etc. I'm not going to pretend it's easy. If you can do it, though, it can be an incredibly valuable tool. There ain't nothing like the real traffic, baby. ;) My favorite feature, which I alluded to earlier, is that you can shadow traffic from a larger production cluster onto a smaller shadow cluster and give it a serious stress test. All sorts of bugs tend to fall out that way. The one thing you can't really catch, even with a good shadow, is interactions with other services - including things like permissions or quotas. But if those are the only things you have to shake out in true production, you're doing well.

joshuamorton 5y ago

Teeing/dark launch/dual write strategies solve most of the issue for databases. Sure you run into concerns when changing the framework that manages that, but that's usually a far smaller surface area than your entire storage layer. That said, you should have tests anyway.