4 ms·

Signs you are doing continuous testing wrong

34 points by jhibbets 8y ago

nobody271 8y ago
I like test recorders (Selenium IDE). They aren't brittle if you manually enter your selectors. Plus you can hook a test recorder up to your own code to fill in any gaps. I do wish they showed mouse movement, though. One I'd like to add is to keep it simple. When I hear the word pipeline I immediately think "oh no". I'm telling you man, some shops add a mind blowing amount of unneeded complexity with their pipelines.
- weliketocode 8y ago
  'not brittle' if you 'manually enter' isn't exactly a glowing review. There's definitely some good with selenium, but I and others find it is often brittle and doesn't catch errors as reliably as would be ideal.
  - alex__c 8y ago
    The objects under test are brittle and bound to change dramatically. Selenium can't offer you insurance against this. I now remember a test suite we had on my project (automotive). We tested against the code structure... that is, we would follow a variable's lifetime and check that its value is correct throughout execution. Two releases later, guess which function got slightly refactored and threw off some 20 testcases?
macca321 8y ago
'Test Pyramid' considered harmful IMO. It discourages people from writing tests against the public interface of their system , and instead encourages testing against internals. This leads to a test pack that prevents refactoring, and can't be trusted to catch regression errors. This is partly down to the widely confused definition of a 'unit test' (which a lot of people interpret as 'one test per class'). If you haven't seen it, please watch Ian Cooper's "TDD, Where Did It All Go Wrong" https://www.youtube.com/watch?v=EZ05e7EMOLM https://www.youtube.com/watch?v=EZ05e7EMOLM Also, I thought continuous testing was having your test pack run all the time while coding, but hey ho.
- TarpitCarnivore 8y ago
  TBH I feel like integration and UI tests can be combined at this point. Writing a bunch of tests to only check UI elements feels like a waste to me. I don't mean to go full on BDD where every step is spec'd out, but I've done it by interacting with elements and validating data with the API.

34 points by jhibbets 8y ago

I like test recorders (Selenium IDE). They aren't brittle if you manually enter your selectors. Plus you can hook a test recorder up to your own code to fill in any gaps. I do wish they showed mouse movement, though. One I'd like to add is to keep it simple. When I hear the word pipeline I immediately think "oh no". I'm telling you man, some shops add a mind blowing amount of unneeded complexity with their pipelines.

weliketocode 8y ago
'not brittle' if you 'manually enter' isn't exactly a glowing review. There's definitely some good with selenium, but I and others find it is often brittle and doesn't catch errors as reliably as would be ideal.
- alex__c 8y ago
  The objects under test are brittle and bound to change dramatically. Selenium can't offer you insurance against this. I now remember a test suite we had on my project (automotive). We tested against the code structure... that is, we would follow a variable's lifetime and check that its value is correct throughout execution. Two releases later, guess which function got slightly refactored and threw off some 20 testcases?

macca321 8y ago

'Test Pyramid' considered harmful IMO. It discourages people from writing tests against the public interface of their system , and instead encourages testing against internals. This leads to a test pack that prevents refactoring, and can't be trusted to catch regression errors. This is partly down to the widely confused definition of a 'unit test' (which a lot of people interpret as 'one test per class'). If you haven't seen it, please watch Ian Cooper's "TDD, Where Did It All Go Wrong" https://www.youtube.com/watch?v=EZ05e7EMOLM https://www.youtube.com/watch?v=EZ05e7EMOLM Also, I thought continuous testing was having your test pack run all the time while coding, but hey ho.

TarpitCarnivore 8y ago
TBH I feel like integration and UI tests can be combined at this point. Writing a bunch of tests to only check UI elements feels like a waste to me. I don't mean to go full on BDD where every step is spec'd out, but I've done it by interacting with elements and validating data with the API.