3 ms·

Ask HN: How often are you frustrated with flaky tests?

• ComputerGuru 7y ago
  Write tests to be deterministic from the very start (mainly: learn to use synchronization primitives to synchronize events rather than trying on timing coincidences) and then you'll realize there's no such thing as a flaky test.
  • quickthrower2 7y ago
    Have you written Selenium tests before, pointing to a staging or production build?
    • ComputerGuru 7y ago
      Yes, I have and it's IMHO a very poorly designed API (or rather, abstraction) for reproducible tests, but yes, the premise still stands. Instead of crossing your fingers and hoping a navigation has occurred/completed, test deterministically for the current state (inspecting browser navigation state, page url, and window events) to decide whether a certain action can be taken or is valid at the time. Basically, you should build your own framework upon which real tests can be run.
• quickthrower2 7y ago
  I've worked somewhere before where there have been flakey tests and they had a system to detect them and list them, and they could be assigned to people. Pretty nice. I'd like it to exist as a service yes, if it can plug into any testing output, and accurately know that yes something is a flakey test. The telltale sign is a retry on the same branch fails then passes with no source code changes. I am not sure however if any company I worked for would PAY for such a system. That's a different question. But as a developer I'd like to use one if it is well honed.
• photonios 7y ago
  Just fix them? Where I work we explicitly do not automatically retry failing tests. Otherwise the incentive goes away to fix them. At some point somebody will be frustrated enough to fix the issue. Note: we have run hundreds of little browser-based tests, visual snapshots and thousands of unit and integration tests on each PR. Flaky tests get annoying real quick and get fixed quickly as a result.
• cpufry 7y ago
  not to be too dramatic, but they are bane of my existence