4 ms·

If you have so many flaky tests, and their flakiness is so intractable, that you actually need to come up with SLOs for handling the flakiness to negotiate bein

If you have so many flaky tests, and their flakiness is so intractable, that you actually need to come up with SLOs for handling the flakiness to negotiate being allowed to address the flakiness in the test base, then quite frankly, you should be looking for another job, one where you can actually go ahead and just fix these things and get back to shipping value to customers, instead of one where you play bureaucratic games with management that cares neither about craftsmanship nor about getting value out the door as quickly as possible.

Swizec 3y ago
Every flakey test is a production failure that really happens to some users sometimes. How often is just a matter of scale. A 1 in 10,000 failure can be a daily annoyance for your users even with just 100 daily active users who each make 10 actions on your app. At “internet scale” a 1 in 10k error frustrates a user every few seconds. If your tests are so flaky that you need SLOs … your poor users …
- drewcoo 3y ago
  > Every flakey test is a production failure that really happens to some users sometimes No. Tests can be flakey when the code they're testing is not. Every flakey test is not a customer problem. Training developers to ignore flakey tests also trains them to ignore real failures. Potentially many of them. Any of those might be real customer problems.
  - m3ll0 3y ago
    This! Flaky tests become a customer problem when developers learn to ignore them! You gotta weed them out.
- zarroboogs 3y ago
  > Every flakey test is a production failure that really happens to some users sometimes. How often is just a matter of scale. Not really, the one I’ve seen most often is a shared resource failing - for example a Gitlab Runner not handling a new VM for a DB so tests fail etc
  - sixstringtheory 3y ago
    Also, dirty test harness state not being cleaned up between tests/suites, and threading issues in the test code
- vlovich123 3y ago
  Depending on the problem domain, a lot of times the flakiness is in the way the test is written and not the code under test. You have to judge the relative cost of tracking down and solving that.
deleted 3y ago
[deleted]
crabbone 3y ago
This is a bunch of wishful thinking... Some software projects, let's call them "integration projects" use third-party software they can do nothing about. And it just doesn't work well. But you have to use it in testing because... well, you are the integrator. The users have already accepted the fact that the software you are integrating doesn't work well, so, it's "all good", except it makes it very hard to distinguish between failures that need to be addressed and those that don't. Just to give you one example of this situation: JupyterLab is an absolute pile of garbage in terms of how it's programmed. For example, the side-bar of the interface doesn't load properly quite often, and you need to click on "reload" button few times to get it to show up. Suppose now you are the integrator which provides some features that are supposed to be exposed to the user through JupyterLab interface. Well, what can you do? -- Yes. Nothing. Just suck it up. You can manipulate the threshold for how many times you will retry reloading the interface, but you absolutely have to have a threshold because sometimes the interface will never load (because of some other reason), and you will be stalling the test pipeline if you don't let this test fail. In general, the larger the SUT, the more "foreign" components it has, the harder it is to predict the test behavior, and the more flaky the tests are. But this isn't the only source of test flakiness. Hardware is another source. Especially in embedded software that has to be tested on hardware that the software company has limited access to (think something like Smart TV, where the TV vendor provides some means of accessing the OS running on the TV set, but they deliberately limit the access in such a way as to prevent the SW company from getting access to the proprietary bits installed by the vendor). So, sometimes things will fail. And you wouldn't know why and wouldn't be able to discover (as in, if you tried to break into vendor's part of the software, they'd sue you).
- couchand 3y ago
  It sounds like your integration architecture is a mash-up. It is indeed quite difficult to handle external systems that are so tightly coupled to your own. For standard server-server integrations, where the coupling is limited to an (hopefully eventually) well-documented API surface, it's much more straightforward to replace an external service with an internal mock. For CI purposes, we eventually develop a more-or-less complete internal version of every external service, quirks and all. It's fun to see their bugfixes show up in our git history in behavioral mocks. We don't put much emphasis into automating integration testing. We've found that with all the vagarities of flaky external services, it's usually necessary to have a human in the loop somewhere for integration tests.
kohsuke 3y ago
I don't think the point of SLO = flakiness out of control. The point of framing it as SLO is the realization that neither extremes are good. Flakiness cannot be allowed to get out of control that some efforts must be spent to contain it, but it's also unnecessarily perfectionist and thus the waste of precious engineering bandwidth to eliminate them completely. The whole point is to avoid "bureaucratic games", as you call it. My theory is that the lack of easy mechanism to measure the flakiness is stalling the progress. If the overall flakiness can be measured, and the top offending tests identified, then I think it becomes no brainer to spend efforts curtailing them back when the flakiness gets too high, but otherwise exclude flaky tests from, say, PR merge gate.

Swizec 3y ago

Every flakey test is a production failure that really happens to some users sometimes. How often is just a matter of scale. A 1 in 10,000 failure can be a daily annoyance for your users even with just 100 daily active users who each make 10 actions on your app. At “internet scale” a 1 in 10k error frustrates a user every few seconds. If your tests are so flaky that you need SLOs … your poor users …

drewcoo 3y ago
> Every flakey test is a production failure that really happens to some users sometimes No. Tests can be flakey when the code they're testing is not. Every flakey test is not a customer problem. Training developers to ignore flakey tests also trains them to ignore real failures. Potentially many of them. Any of those might be real customer problems.
- m3ll0 3y ago
  This! Flaky tests become a customer problem when developers learn to ignore them! You gotta weed them out.
zarroboogs 3y ago
> Every flakey test is a production failure that really happens to some users sometimes. How often is just a matter of scale. Not really, the one I’ve seen most often is a shared resource failing - for example a Gitlab Runner not handling a new VM for a DB so tests fail etc
- sixstringtheory 3y ago
  Also, dirty test harness state not being cleaned up between tests/suites, and threading issues in the test code
vlovich123 3y ago
Depending on the problem domain, a lot of times the flakiness is in the way the test is written and not the code under test. You have to judge the relative cost of tracking down and solving that.

deleted 3y ago

[deleted]

crabbone 3y ago

This is a bunch of wishful thinking... Some software projects, let's call them "integration projects" use third-party software they can do nothing about. And it just doesn't work well. But you have to use it in testing because... well, you are the integrator. The users have already accepted the fact that the software you are integrating doesn't work well, so, it's "all good", except it makes it very hard to distinguish between failures that need to be addressed and those that don't. Just to give you one example of this situation: JupyterLab is an absolute pile of garbage in terms of how it's programmed. For example, the side-bar of the interface doesn't load properly quite often, and you need to click on "reload" button few times to get it to show up. Suppose now you are the integrator which provides some features that are supposed to be exposed to the user through JupyterLab interface. Well, what can you do? -- Yes. Nothing. Just suck it up. You can manipulate the threshold for how many times you will retry reloading the interface, but you absolutely have to have a threshold because sometimes the interface will never load (because of some other reason), and you will be stalling the test pipeline if you don't let this test fail. In general, the larger the SUT, the more "foreign" components it has, the harder it is to predict the test behavior, and the more flaky the tests are. But this isn't the only source of test flakiness. Hardware is another source. Especially in embedded software that has to be tested on hardware that the software company has limited access to (think something like Smart TV, where the TV vendor provides some means of accessing the OS running on the TV set, but they deliberately limit the access in such a way as to prevent the SW company from getting access to the proprietary bits installed by the vendor). So, sometimes things will fail. And you wouldn't know why and wouldn't be able to discover (as in, if you tried to break into vendor's part of the software, they'd sue you).

couchand 3y ago
It sounds like your integration architecture is a mash-up. It is indeed quite difficult to handle external systems that are so tightly coupled to your own. For standard server-server integrations, where the coupling is limited to an (hopefully eventually) well-documented API surface, it's much more straightforward to replace an external service with an internal mock. For CI purposes, we eventually develop a more-or-less complete internal version of every external service, quirks and all. It's fun to see their bugfixes show up in our git history in behavioral mocks. We don't put much emphasis into automating integration testing. We've found that with all the vagarities of flaky external services, it's usually necessary to have a human in the loop somewhere for integration tests.

kohsuke 3y ago

I don't think the point of SLO = flakiness out of control. The point of framing it as SLO is the realization that neither extremes are good. Flakiness cannot be allowed to get out of control that some efforts must be spent to contain it, but it's also unnecessarily perfectionist and thus the waste of precious engineering bandwidth to eliminate them completely. The whole point is to avoid "bureaucratic games", as you call it. My theory is that the lack of easy mechanism to measure the flakiness is stalling the progress. If the overall flakiness can be measured, and the top offending tests identified, then I think it becomes no brainer to spend efforts curtailing them back when the flakiness gets too high, but otherwise exclude flaky tests from, say, PR merge gate.