5 ms·

> Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inhere

> Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. At least for web applications, all end to end test suites are slow and flaky. This is not an exaggeration - all of them. There are no magical optimizations. This is something that every project runs into, over and over again. I will never willingly write an end to end test ever again. Unit / module tests + targeted integration tests are the only hope that we have.

jrochkind1 5y ago
> I will never willingly write an end to end test ever again. Unit / module tests + targeted integration tests are the only hope that we have. What are your "integration" tests that are not "end to end" tests like, how do they differ from end to end tests?
- Jtsummers 5y ago
  Integration tests may integrate smaller-than-the-whole groups of subsystems. It definitely gets fuzzy. A lot of people treat end-to-end and integration tests as equivalent, but piecing together everything-but-the-frontend and testing it is also an integration test, but not an end-to-end test. If we consider tests as existing at and covering different scales, unit tests are at the smallest scale and integration tests run the gamut from 2 units to the entire system.
- BurningFrog 5y ago
  Not OP, but testing groups of the "units" you unit tests can have a lot of value.
- amw-zero 5y ago
  Picture a test that doesn’t involve any clicking of UI elements. That’s a start. So right there you’re avoiding the complexities of UI rendering, you just call the commands that are invoked by clicking directly. Also, you can test frontend components together, and backend components together, but not cross the client-server boundary. Faster, more reliable. That leaves a very small amount of e2e tests that you even want to write, and by that point I’m totally fine with manual smoke testing or automating them. But they’re the vast minority of tests.
dcow 5y ago
Sounds like you’ve been subject to some pretty poor test setups. I’ve experienced good ones. My cynical take is that well maintained e2e tests aren’t a product priority in environments where they’re flaky and slow so they come as an afterthought. Not that they can’t be good. Usually product wants to ship code yesterday and doesn't care if there are bugs… so good test hygiene is nowhere to be seen.
- amw-zero 5y ago
  This argument does not account for the fact that all e2e tests are non-deterministic, so the quality of your “setup” is not relevant.
  - dcow 5y ago
    You must have worked on some incredibly bad software to have non-determinism dominate your life. If a request fails then retry it just like a user would. If it keeps failing there's a problem. If it works then move on. I doubt your bank just throws in the towel and says "whelp software systems are inherently non-deterministic so we'll just forget some transactions here, allow the wrong amount over there, forget tests they're hard we can handle a little chance in our payment flows". The closest thing I've heard to that is amazon very occasionally shipping multiples of the same item because it was allegedly more expensive to implement immediate consistency than to ship a few duplicate items.
mbrodersen 5y ago
I never had any problems writing reliable end-to-end tests. They are super useful for catching serious subtle bugs before the system goes into production. Not having solid end-to-end tests is a massive red flag for me.
- amw-zero 5y ago
  I flat out don’t believe you.
  - mbrodersen 5y ago
    Well it is true. The fact that you don’t believe it tells me you have a lot to learn. Writing good end-to-end tests is a skill you need to learn. Don’t assume that software developers can do it without proper training/learning. It is hard to do well.
    - amw-zero 5y ago
      My disbelief of you is from my own experiences writing them, and talking to dozens of colleagues across different companies. Every single company has to heavily parallelize their e2e tests, and pays a huge CI bill on top of effort maintaining an overly complex CI config. Even after this, every company has to have a retry mechanism for their e2e tests because at least one fails at least every test run. It’s also the first thing I ask on interviews. I have many dat points on this one, across many teams and companies. It’s an abomination. Maybe you’re not talking about interactive web applications. Or maybe you’re not talking about the scale where you have thousands of e2e tests written over multi-year projects. If you’re talking about a toy project, sure you might not run into nondeterminism. These problems present themselves in aggregate.
      - mbrodersen 5y ago
        I am talking about very large scale systems with web + mobile + desktop clients in an Enterprise environments. Think airlines and airports (operations and pairing/rostering). However it sounds as if your experience has been pretty bad. So I understand where you are coming from. My experience is that yes it is hard to do right but also that it is worth doing. YMMV of course.
sumedh 5y ago
> At least for web applications, all end to end test suites are slow and flaky. Then you should be asking why are they flaky in the test environment? Its probably because your services are running on very slow servers.
- amw-zero 5y ago
  No, it’s because end to end tests are non-deterministic with respect to execution time. They are deterministic only in their discrete states. This is why all end to end testing involves polling to wait for asynchronous operations to complete, which is by definition non-deterministic.
  - sumedh 5y ago
    No not really, what you are saying is that you dont how long a test will take to complete, it can take 1 min or it can take 1 hour. If it sometimes takes 1 hour then you have to put on your detective hat and go look in the logs to see which service is slowing the e2e flow.

jrochkind1 5y ago

> I will never willingly write an end to end test ever again. Unit / module tests + targeted integration tests are the only hope that we have. What are your "integration" tests that are not "end to end" tests like, how do they differ from end to end tests?

Jtsummers 5y ago
Integration tests may integrate smaller-than-the-whole groups of subsystems. It definitely gets fuzzy. A lot of people treat end-to-end and integration tests as equivalent, but piecing together everything-but-the-frontend and testing it is also an integration test, but not an end-to-end test. If we consider tests as existing at and covering different scales, unit tests are at the smallest scale and integration tests run the gamut from 2 units to the entire system.
BurningFrog 5y ago
Not OP, but testing groups of the "units" you unit tests can have a lot of value.
amw-zero 5y ago
Picture a test that doesn’t involve any clicking of UI elements. That’s a start. So right there you’re avoiding the complexities of UI rendering, you just call the commands that are invoked by clicking directly. Also, you can test frontend components together, and backend components together, but not cross the client-server boundary. Faster, more reliable. That leaves a very small amount of e2e tests that you even want to write, and by that point I’m totally fine with manual smoke testing or automating them. But they’re the vast minority of tests.

dcow 5y ago

Sounds like you’ve been subject to some pretty poor test setups. I’ve experienced good ones. My cynical take is that well maintained e2e tests aren’t a product priority in environments where they’re flaky and slow so they come as an afterthought. Not that they can’t be good. Usually product wants to ship code yesterday and doesn't care if there are bugs… so good test hygiene is nowhere to be seen.

amw-zero 5y ago
This argument does not account for the fact that all e2e tests are non-deterministic, so the quality of your “setup” is not relevant.
- dcow 5y ago
  You must have worked on some incredibly bad software to have non-determinism dominate your life. If a request fails then retry it just like a user would. If it keeps failing there's a problem. If it works then move on. I doubt your bank just throws in the towel and says "whelp software systems are inherently non-deterministic so we'll just forget some transactions here, allow the wrong amount over there, forget tests they're hard we can handle a little chance in our payment flows". The closest thing I've heard to that is amazon very occasionally shipping multiples of the same item because it was allegedly more expensive to implement immediate consistency than to ship a few duplicate items.

mbrodersen 5y ago

I never had any problems writing reliable end-to-end tests. They are super useful for catching serious subtle bugs before the system goes into production. Not having solid end-to-end tests is a massive red flag for me.

amw-zero 5y ago
I flat out don’t believe you.
- mbrodersen 5y ago
  Well it is true. The fact that you don’t believe it tells me you have a lot to learn. Writing good end-to-end tests is a skill you need to learn. Don’t assume that software developers can do it without proper training/learning. It is hard to do well.
  - amw-zero 5y ago
    My disbelief of you is from my own experiences writing them, and talking to dozens of colleagues across different companies. Every single company has to heavily parallelize their e2e tests, and pays a huge CI bill on top of effort maintaining an overly complex CI config. Even after this, every company has to have a retry mechanism for their e2e tests because at least one fails at least every test run. It’s also the first thing I ask on interviews. I have many dat points on this one, across many teams and companies. It’s an abomination. Maybe you’re not talking about interactive web applications. Or maybe you’re not talking about the scale where you have thousands of e2e tests written over multi-year projects. If you’re talking about a toy project, sure you might not run into nondeterminism. These problems present themselves in aggregate.
    - mbrodersen 5y ago
      I am talking about very large scale systems with web + mobile + desktop clients in an Enterprise environments. Think airlines and airports (operations and pairing/rostering). However it sounds as if your experience has been pretty bad. So I understand where you are coming from. My experience is that yes it is hard to do right but also that it is worth doing. YMMV of course.

sumedh 5y ago

> At least for web applications, all end to end test suites are slow and flaky. Then you should be asking why are they flaky in the test environment? Its probably because your services are running on very slow servers.

amw-zero 5y ago
No, it’s because end to end tests are non-deterministic with respect to execution time. They are deterministic only in their discrete states. This is why all end to end testing involves polling to wait for asynchronous operations to complete, which is by definition non-deterministic.
- sumedh 5y ago
  No not really, what you are saying is that you dont how long a test will take to complete, it can take 1 min or it can take 1 hour. If it sometimes takes 1 hour then you have to put on your detective hat and go look in the logs to see which service is slowing the e2e flow.