4 ms·

> Contract tests allow us to describe the interactions between our services through expectations to be met by their inputs and outputs. If you say so, and I wi

> Contract tests allow us to describe the interactions between our services through expectations to be met by their inputs and outputs. If you say so, and I wish you luck but... I've seen that tried many, many times and never seen it actually work out in practice. It seems like it ought to be workable - there are only a finite number of ways that each service can be invoked after all - but if the goal of automated testing is to find problems before they become production problems, I've never seen "defined contracts" fulfill that goal.

coryrc 5y ago
It says the majority of the problems caught in E2E were due to changed contacts, so it makes sense to have a dedicated test type for those errors. If the remaining caught bugs are few enough and their customers are willing to suffer occasional unavailability, just let them be caught in production and save that expense.
makeitdouble 5y ago
A complicating factor is that teams usually start with E2E when it’s simple, and only move to other approaches when the systems have become way too complicated. At that point I don’t know if there is any specific strategy that effectively catches a lot of bugs, short of sending to production and monitor the effect. At a company we just called those tests “sanity checks”, and the goal was just to make sure the most basic use cases would still work, and nothing more.
anamexis 5y ago
> but if the goal of automated testing is to find problems before they become production problems, I've never seen "defined contracts" fulfill that goal But, have you seen contract testing fail to catch problems that E2E tests did catch? I think both of them end up tending to be regression tests a lot of the time.
mbrodersen 5y ago
Agree. They are not testing the end-to-end contracts of how concurrent services interact. I predict it will end badly.

coryrc 5y ago

It says the majority of the problems caught in E2E were due to changed contacts, so it makes sense to have a dedicated test type for those errors. If the remaining caught bugs are few enough and their customers are willing to suffer occasional unavailability, just let them be caught in production and save that expense.

makeitdouble 5y ago

A complicating factor is that teams usually start with E2E when it’s simple, and only move to other approaches when the systems have become way too complicated. At that point I don’t know if there is any specific strategy that effectively catches a lot of bugs, short of sending to production and monitor the effect. At a company we just called those tests “sanity checks”, and the goal was just to make sure the most basic use cases would still work, and nothing more.

anamexis 5y ago

> but if the goal of automated testing is to find problems before they become production problems, I've never seen "defined contracts" fulfill that goal But, have you seen contract testing fail to catch problems that E2E tests did catch? I think both of them end up tending to be regression tests a lot of the time.

mbrodersen 5y ago

Agree. They are not testing the end-to-end contracts of how concurrent services interact. I predict it will end badly.