3 ms·

Everyone should be testing in prod, in that you release code and see metrics and monitoring to show that everything is working. Testing in production is not go

Everyone should be testing in prod, in that you release code and see metrics and monitoring to show that everything is working. Testing in production is not going “let’s see if this will work” it is “we will release and validate that everything is working as expected” People need to get over the old school cowboys who jump on prod to see if something works.

notacoward 5y ago
Yes, everyone should release code and watch metrics etc. but I think that's at the very edge of what "testing" encompasses. Between model checking, traditional forms of testing, and shadow-traffic testing (which can test higher per-server load than prod), finding something after deploy should be like a parachute failure. Yes those happen, yes there should be a reserve, but if it happens more than once in a blue moon you have a process problem somewhere (quite likely between teams/services but still).
- mavelikara 5y ago
  Tangential, where can I learn more about shadow-traffic testing? Books, blogs, tools etc.
  - notacoward 5y ago
    Cindy Sridharan often writes well about these topics. https://copyconstruct.medium.com/testing-in-production-the-safe-way-18ca102d0ef1 https://copyconstruct.medium.com/testing-in-production-the-s... For this one I'd start about half way down, under the heading "Shadowing (also known as Dark Traffic Testing or Mirroring)" Unfortunately the terminology is a bit fragmented - shadowing, mirroring, teeing, dark traffic (ick), ad nauseam. Whatever you call it, it's often pretty high overhead to add the infrastructure, unless you're already using some sort of "service mesh" (Envoy/Istio/Caddy/whatever) that supports it. Even then, if you're dealing specifically with a storage system then there can be some thorny issues - idempotent vs. non-idempotent vs. destructive requests, requests which require other objects (files/objects or directories/buckets) to exist or be in specific states, etc. I'm not going to pretend it's easy. If you can do it, though, it can be an incredibly valuable tool. There ain't nothing like the real traffic, baby. ;) My favorite feature, which I alluded to earlier, is that you can shadow traffic from a larger production cluster onto a smaller shadow cluster and give it a serious stress test. All sorts of bugs tend to fall out that way. The one thing you can't really catch, even with a good shadow, is interactions with other services - including things like permissions or quotas. But if those are the only things you have to shake out in true production, you're doing well.

notacoward 5y ago

Yes, everyone should release code and watch metrics etc. but I think that's at the very edge of what "testing" encompasses. Between model checking, traditional forms of testing, and shadow-traffic testing (which can test higher per-server load than prod), finding something after deploy should be like a parachute failure. Yes those happen, yes there should be a reserve, but if it happens more than once in a blue moon you have a process problem somewhere (quite likely between teams/services but still).

mavelikara 5y ago
Tangential, where can I learn more about shadow-traffic testing? Books, blogs, tools etc.
- notacoward 5y ago
  Cindy Sridharan often writes well about these topics. https://copyconstruct.medium.com/testing-in-production-the-safe-way-18ca102d0ef1 https://copyconstruct.medium.com/testing-in-production-the-s... For this one I'd start about half way down, under the heading "Shadowing (also known as Dark Traffic Testing or Mirroring)" Unfortunately the terminology is a bit fragmented - shadowing, mirroring, teeing, dark traffic (ick), ad nauseam. Whatever you call it, it's often pretty high overhead to add the infrastructure, unless you're already using some sort of "service mesh" (Envoy/Istio/Caddy/whatever) that supports it. Even then, if you're dealing specifically with a storage system then there can be some thorny issues - idempotent vs. non-idempotent vs. destructive requests, requests which require other objects (files/objects or directories/buckets) to exist or be in specific states, etc. I'm not going to pretend it's easy. If you can do it, though, it can be an incredibly valuable tool. There ain't nothing like the real traffic, baby. ;) My favorite feature, which I alluded to earlier, is that you can shadow traffic from a larger production cluster onto a smaller shadow cluster and give it a serious stress test. All sorts of bugs tend to fall out that way. The one thing you can't really catch, even with a good shadow, is interactions with other services - including things like permissions or quotas. But if those are the only things you have to shake out in true production, you're doing well.