3 ms·

Is monitoring separate from testing? Should it be?

I realized I thought of e2e tests as always being pretty intimately connected with both monitoring and testing. Especially if you take the time to write your tests in something like Playwright, and they're doing complex user behaviors and measuring response times, you want those running both as part of QA/Test and as part of your general monitoring. I've recently had more than one discussion where it seemed like the distinction was very clear, and that stuff like Canary deploys were across the line from QA and now part of monitoring for the SRE/Operations. I have two questions: * is there a reason that things like Synthetic user testing shouldn't be used at the QA/Test/Staging, hell even Dev deployment stages? * Should developers be writing the tests that will later be used to do uptime and synthetics checks? Or is that monitoring better left to the SREs and Ops specialists? On the other hand is there a reason on the operations side that you wouldn't want to run the detailed tests that were written for QA? Full disclosure I work for Synthetics tool Checkly, not trying to promote Checkly just hoping to understand this a bit better.

dexwiz 3y ago
Integration tests are for finding issues and regressions in code. Monitoring and synthetic checks are for finding issues in infrastructure. At a basic level, they could be the same tests, but their differing goals means the ultimately diverge. E2E tests can be destructive as they usually run in a isolated sandbox. Synthetic tests shouldn't be effecting real user data. E2E tests can be expensive or long running. An E2E test suite for a mature application may take hours (or even days) to run, which means time and money. In larger orgs, the cost of running E2E tests can represent a significant dollar amount. Synthetic tests should be quick so they can give timely feedback and be reran often. E2E tests are often run from the same location as the code. Monitoring benefits when it sits in multiple locations, even in an external networks. That way connectivity issues can be discovered. It also avoids "it works on my machine" scenarios. Synthetic tests tend to be "happy path" looking for basic functionality. Can a user login? Can they perform basic common actions? Can they access the app from multiple regions? E2E tests often cover multiple edge cases that step outside the happy path. They are testing for correctness which is more in depth. Tests are always written inside a framework. Sure Devs could probably write the tests, but if devs are not familiar with the framework, don't have access to it, or are not the ones to address the failures, then I am not sure of the value. SRE/DevOps should have enough development chops to write monitoring tests. Finally, noise. Playwright and Selenium are known for flappy tests. Developers are usually more tolerant of noise. Since they don't drop everything for each test failure, a flappy test might just resolve itself before the dev gets to it. SREs are much less tolerant of noise, as any issue may represent a real dollar loss for the company. I don't want to get a page from SR just because a selenium test failed to complete.
deleted 3y ago
[deleted]

dexwiz 3y ago

Integration tests are for finding issues and regressions in code. Monitoring and synthetic checks are for finding issues in infrastructure. At a basic level, they could be the same tests, but their differing goals means the ultimately diverge. E2E tests can be destructive as they usually run in a isolated sandbox. Synthetic tests shouldn't be effecting real user data. E2E tests can be expensive or long running. An E2E test suite for a mature application may take hours (or even days) to run, which means time and money. In larger orgs, the cost of running E2E tests can represent a significant dollar amount. Synthetic tests should be quick so they can give timely feedback and be reran often. E2E tests are often run from the same location as the code. Monitoring benefits when it sits in multiple locations, even in an external networks. That way connectivity issues can be discovered. It also avoids "it works on my machine" scenarios. Synthetic tests tend to be "happy path" looking for basic functionality. Can a user login? Can they perform basic common actions? Can they access the app from multiple regions? E2E tests often cover multiple edge cases that step outside the happy path. They are testing for correctness which is more in depth. Tests are always written inside a framework. Sure Devs could probably write the tests, but if devs are not familiar with the framework, don't have access to it, or are not the ones to address the failures, then I am not sure of the value. SRE/DevOps should have enough development chops to write monitoring tests. Finally, noise. Playwright and Selenium are known for flappy tests. Developers are usually more tolerant of noise. Since they don't drop everything for each test failure, a flappy test might just resolve itself before the dev gets to it. SREs are much less tolerant of noise, as any issue may represent a real dollar loss for the company. I don't want to get a page from SR just because a selenium test failed to complete.

deleted 3y ago

[deleted]