6 ms·

After reading a few smug comments, I've concluded that some of the folks in this thread have never worked on an application where the production scale is many,

After reading a few smug comments, I've concluded that some of the folks in this thread have never worked on an application where the production scale is many, many orders of magnitude greater than preproduction environments. There's no substitution for "testing in prod." If you're not "testing in prod" with your really-big application, you're not testing enough.

detaro 5y ago
"some of the"? or the one you already replied to?
- jdauriemma 5y ago
  The one I replied to, plus others. Hence, "some."
tomrod 5y ago
Some places you simply can't. Medical devices, airplanes, etc. But non-critical services? Sure
- jdauriemma 5y ago
  You absolutely can and should test in prod in these cases.
- frozenport 5y ago
  We call that FDA testing
  - ekianjo 5y ago
    FDA does not test anything.
    - Supermancho 5y ago
      I'm not sure how this is a useful observation. Units don't test anything, but it's still called Unit Testing.
      - thewakalix 5y ago
        They test my patience!
- anaganisk 5y ago
  They will and do something called acceptance testing, which is nothing but testing in prod.
  - detaro 5y ago
    Not quite. Nobody acceptance tests an airliner on a passenger flight. If we acceptance test a medical diagnostics device, we don't run the test the doctor later relies on with the DUT.
    - jdauriemma 5y ago
      This is a common misconception about testing in prod. It’s not about the logic. To quote the article: “Once you deploy, you aren’t testing code anymore, you’re testing systems.”
      - detaro 5y ago
        I don't quite get the objection. For me, "testing in prod" is "we observe the actual running system, with production traffic and users directly interacting with it and its results being live". That's not quite what "acceptance testing" in the mentioned-above domains is. If you have an acceptance testing stage, prod follows it.
        joshuamorton 5y ago
        I think the argument breaks down when you have systems as opposed to objects. You can't acceptance test "Google's search", because at a minimum doing so would require some kind of reverse proxy that is itself part of the system and can't be acceptance tested without...and turtles etc. Another way of putting this might be that there isn't a good way to acceptance test "airplane manufacturing companies". There isn't a set of acceptance tests you can run to ensure that Boeing is performing to spec before having it build real airplanes.
        detaro 5y ago
dnautics 5y ago
Our production scale is just copied stage environments because our clients are trivially shardable and they only use our service from 9-5 m-f, but hoo boy are there a lot of them
- secondaryacct 5y ago
  Yeah it s nice to be trivially shardable ! Ours is not: stock market connections to 13 countries with clients having global trading controls across these countries - so we can shard some parts and not others and each country having specificities, guaranteeing a cross market feature works for a client, if it impacts control, is tested in prod, either after market close on mock exchanges provided by the various countries, or during trading hours on small orders. No amount of testing in the bank has ever been able to spot the weirdest issues, so we continue while really trying hard to make the prod "pilots" (we try not to call these tests) as routine as possible. But, we still find crazy issues, notably those only the client notices (so wrong specs they couldnt prevalidate by plugging their monster system on our monster system in QA)
  - dnautics 5y ago
    why can't you shard by clients (not regions)? Is there some reason why clients must know about other clients?
    - secondaryacct 5y ago
      Let s imagine we build all the trading robots to have one server per group of client. That would def work, we probably cant pay for one set of server (for HA) per client since we have hundreds, but then all these per client servers have to queue up for the exchange line access which has per country controls (cant move the market more than x %, cant trade x or y stock at z time whatever). So yes in a way, end of the line, the clients must "know abt each other" so we can respect the 13 different law codes. We split by exchange traditionally because it made the most sense when we started 30 years ago in Asia, but... We re more and more splitting clients in groups where we can and allocating them cpu power indeed. I envy the stock market people because while they must handle even higher volume, they can shard per stock itself and have just one jursidiction.
0x0nyandesu 5y ago
Same. I'm processing fitness data 24/7 and I can easily repull and overwrite the normalization. There's no actual down side to testing in prod with this style of data syncing. In fact the dev versions of our oauth2 app on fitbit have only one test account so there's no real data for me to access. 99% of the edge case issues happen with real data only and you will never catch them without it. I only use the dev version for setting up new features not yet in prod but once I've confirmed basic workability the rest of the dev is on my admin or test accounts but with production data.
- deleted 5y ago
  [deleted]
hinkley 5y ago
There is more than one way to define 'testing'. That's not responsible for all of the misunderstandings, but it's responsible for quite a few. For some people, feature toggles fall under testing in production. For other people, they don't. It's not whether production is orders of magnitude greater that preprod. It's whether production is orders of magnitude more diverse. If you've designed your product so that every user is a snowflake, that's on you, or at least the sales team. That functionality surface area is all undocumented features. Bragging about not being able to test all of the features you've promised users is like the opposite of a humble brag. What would you call that, a Dunning-Kruger brag? I've worked at a few places that didn't understand this. A couple eventually got it (begrudgingly, and with a hint of resentment. The other never did, and hemorrhaged money and good people.
quadrifoliate 5y ago
> After reading a few smug comments...I've concluded that some of the folks in this thread have never worked on an application where the production scale is many, many orders of magnitude greater than preproduction environments. I think your comment is smug as well. The author of the article has considerably overloaded the term "testing", reasonably having a lot of people have a knee-jerk reaction of "Don't do that". I get the impression that this is by design, and the article is intended to provoke discussion via flamewar. I think the best way to combat this is to simply avoid too much discussion on such articles until they are rewritten to be more clear and less liable to cause flamewars. I have some views on the thesis of the article, and practical experience backing those views – but I simply won't express them because I don't want to get embroiled in the numerous minor arguments caused by the confusing terminology, with little to no actionable information. An example of a pattern that you'll see repeating across all the comments in this thread: Person A: "We take our time and test the application thoroughly under all sorts of load in a preproduction environment. We run end-to-end tests on every build. We value work/life balance, and try to minimize testing-in-production." Person B: "When did the article say not to do that? It never says you shouldn't test outside prod, it says you should also test in prod and that's a superpower! You're totally misunderstanding the article!"
- jdauriemma 5y ago
  I think your reply to my smug comment is also smug! It’s smugness all the way down!
  - quadrifoliate 5y ago
    It's possible, I did feel a bit self-satisfied after making it! Maybe it applies upwards as well? The article is kind of all smug about "I test in prod" too :)
    - jdauriemma 5y ago
      Full-stack smugness
    - deleted 5y ago
BerislavLopac 5y ago
As the old saying goes: Every team has a testing environment; some teams also have a completely separate production environment.
k3liutZu 5y ago
That's why I usually develop in _prod_ (well _clients_)
Uberphallus 5y ago
Exactly. Here we crunch loads of data, and by a lot, I mean our production databases run on a three digit cluster of four digit terabytes of data. Good luck having that scale in preproduction environments. We have loads of redundancy, and dedicated "test in production" machines/datacenters to test actual production loads in actual production-scale sets machines. Now, tests in production usually involves a pair of hands and 2 other people looking behind (dev + ops + dba), and requires a well defined rollback procedure and post-mortem. We still have absurdly high SLA (99.999%).
tiku 5y ago
Well this is why it is a superpower. On such scaled applications a bug is simply lost in the system.
moksly 5y ago
Even with small applications that don’t have fatal consequences, you aren’t testing if you aren’t testing in production. Especially not in a world where it is so easy to control traffic to your application in a way where 95% get on the stable solution and 5% get on your staging solution. Hell, this is why you have LTS and non LTS solutions of the dev tools you use yourself.

detaro 5y ago

"some of the"? or the one you already replied to?

jdauriemma 5y ago
The one I replied to, plus others. Hence, "some."

tomrod 5y ago

Some places you simply can't. Medical devices, airplanes, etc. But non-critical services? Sure

jdauriemma 5y ago
You absolutely can and should test in prod in these cases.
frozenport 5y ago
We call that FDA testing
- ekianjo 5y ago
  FDA does not test anything.
  - Supermancho 5y ago
    I'm not sure how this is a useful observation. Units don't test anything, but it's still called Unit Testing.
    - thewakalix 5y ago
      They test my patience!
anaganisk 5y ago
They will and do something called acceptance testing, which is nothing but testing in prod.
- detaro 5y ago
  Not quite. Nobody acceptance tests an airliner on a passenger flight. If we acceptance test a medical diagnostics device, we don't run the test the doctor later relies on with the DUT.
  - jdauriemma 5y ago
    This is a common misconception about testing in prod. It’s not about the logic. To quote the article: “Once you deploy, you aren’t testing code anymore, you’re testing systems.”
    - detaro 5y ago
      I don't quite get the objection. For me, "testing in prod" is "we observe the actual running system, with production traffic and users directly interacting with it and its results being live". That's not quite what "acceptance testing" in the mentioned-above domains is. If you have an acceptance testing stage, prod follows it.
      - joshuamorton 5y ago
        I think the argument breaks down when you have systems as opposed to objects. You can't acceptance test "Google's search", because at a minimum doing so would require some kind of reverse proxy that is itself part of the system and can't be acceptance tested without...and turtles etc. Another way of putting this might be that there isn't a good way to acceptance test "airplane manufacturing companies". There isn't a set of acceptance tests you can run to ensure that Boeing is performing to spec before having it build real airplanes.
        detaro 5y ago

dnautics 5y ago

Our production scale is just copied stage environments because our clients are trivially shardable and they only use our service from 9-5 m-f, but hoo boy are there a lot of them

secondaryacct 5y ago
Yeah it s nice to be trivially shardable ! Ours is not: stock market connections to 13 countries with clients having global trading controls across these countries - so we can shard some parts and not others and each country having specificities, guaranteeing a cross market feature works for a client, if it impacts control, is tested in prod, either after market close on mock exchanges provided by the various countries, or during trading hours on small orders. No amount of testing in the bank has ever been able to spot the weirdest issues, so we continue while really trying hard to make the prod "pilots" (we try not to call these tests) as routine as possible. But, we still find crazy issues, notably those only the client notices (so wrong specs they couldnt prevalidate by plugging their monster system on our monster system in QA)
- dnautics 5y ago
  why can't you shard by clients (not regions)? Is there some reason why clients must know about other clients?
  - secondaryacct 5y ago
    Let s imagine we build all the trading robots to have one server per group of client. That would def work, we probably cant pay for one set of server (for HA) per client since we have hundreds, but then all these per client servers have to queue up for the exchange line access which has per country controls (cant move the market more than x %, cant trade x or y stock at z time whatever). So yes in a way, end of the line, the clients must "know abt each other" so we can respect the 13 different law codes. We split by exchange traditionally because it made the most sense when we started 30 years ago in Asia, but... We re more and more splitting clients in groups where we can and allocating them cpu power indeed. I envy the stock market people because while they must handle even higher volume, they can shard per stock itself and have just one jursidiction.

0x0nyandesu 5y ago

Same. I'm processing fitness data 24/7 and I can easily repull and overwrite the normalization. There's no actual down side to testing in prod with this style of data syncing. In fact the dev versions of our oauth2 app on fitbit have only one test account so there's no real data for me to access. 99% of the edge case issues happen with real data only and you will never catch them without it. I only use the dev version for setting up new features not yet in prod but once I've confirmed basic workability the rest of the dev is on my admin or test accounts but with production data.

deleted 5y ago
[deleted]

hinkley 5y ago

There is more than one way to define 'testing'. That's not responsible for all of the misunderstandings, but it's responsible for quite a few. For some people, feature toggles fall under testing in production. For other people, they don't. It's not whether production is orders of magnitude greater that preprod. It's whether production is orders of magnitude more diverse. If you've designed your product so that every user is a snowflake, that's on you, or at least the sales team. That functionality surface area is all undocumented features. Bragging about not being able to test all of the features you've promised users is like the opposite of a humble brag. What would you call that, a Dunning-Kruger brag? I've worked at a few places that didn't understand this. A couple eventually got it (begrudgingly, and with a hint of resentment. The other never did, and hemorrhaged money and good people.

quadrifoliate 5y ago

> After reading a few smug comments...I've concluded that some of the folks in this thread have never worked on an application where the production scale is many, many orders of magnitude greater than preproduction environments. I think your comment is smug as well. The author of the article has considerably overloaded the term "testing", reasonably having a lot of people have a knee-jerk reaction of "Don't do that". I get the impression that this is by design, and the article is intended to provoke discussion via flamewar. I think the best way to combat this is to simply avoid too much discussion on such articles until they are rewritten to be more clear and less liable to cause flamewars. I have some views on the thesis of the article, and practical experience backing those views – but I simply won't express them because I don't want to get embroiled in the numerous minor arguments caused by the confusing terminology, with little to no actionable information. An example of a pattern that you'll see repeating across all the comments in this thread: Person A: "We take our time and test the application thoroughly under all sorts of load in a preproduction environment. We run end-to-end tests on every build. We value work/life balance, and try to minimize testing-in-production." Person B: "When did the article say not to do that? It never says you shouldn't test outside prod, it says you should also test in prod and that's a superpower! You're totally misunderstanding the article!"

jdauriemma 5y ago
I think your reply to my smug comment is also smug! It’s smugness all the way down!
- quadrifoliate 5y ago
  It's possible, I did feel a bit self-satisfied after making it! Maybe it applies upwards as well? The article is kind of all smug about "I test in prod" too :)
  - jdauriemma 5y ago
    Full-stack smugness
  - deleted 5y ago

BerislavLopac 5y ago

As the old saying goes: Every team has a testing environment; some teams also have a completely separate production environment.

k3liutZu 5y ago

That's why I usually develop in _prod_ (well _clients_)

Uberphallus 5y ago

Exactly. Here we crunch loads of data, and by a lot, I mean our production databases run on a three digit cluster of four digit terabytes of data. Good luck having that scale in preproduction environments. We have loads of redundancy, and dedicated "test in production" machines/datacenters to test actual production loads in actual production-scale sets machines. Now, tests in production usually involves a pair of hands and 2 other people looking behind (dev + ops + dba), and requires a well defined rollback procedure and post-mortem. We still have absurdly high SLA (99.999%).

tiku 5y ago

Well this is why it is a superpower. On such scaled applications a bug is simply lost in the system.

moksly 5y ago

Even with small applications that don’t have fatal consequences, you aren’t testing if you aren’t testing in production. Especially not in a world where it is so easy to control traffic to your application in a way where 95% get on the stable solution and 5% get on your staging solution. Hell, this is why you have LTS and non LTS solutions of the dev tools you use yourself.