4 ms·
Reading this post-mortem was very useful, and I appreciate the segment engineering team sharing it. It seems like the primary problem causes were flaky/unrelia
by dbt00 8y ago
Reading this post-mortem was very useful, and I appreciate the segment engineering team sharing it.
It seems like the primary problem causes were flaky/unreliable tests, and difficulty making coordinated changes across many small repositories.
Having worked on similar projects before (and currently), with a small team driving microservices oriented projects, I would probably recommend:
1) single repository to allow easy coordinated changes.
2) a build system that only runs tests that are downstream of the change you made (Bazel is my favorite here, but others exist). This means all services use the HEAD version of libraries, and you find out if a library change broke one of the services you didn't think about. This also allows for faster performance.
3) Emphasis on making tests reliable. Mock out external services, or if you must reach out to dependencies use conditional test execution, like golang's Skip or junit's Assume if you can't verify a working connection.
If you still can't build a reliable service with those choices, then it's time to think about changing the architecture.