4 ms·

Another pain with monoliths is that they can only be deployed if the entire monolith is passing all tests. When you cannot deploy your changes because someone e

Another pain with monoliths is that they can only be deployed if the entire monolith is passing all tests. When you cannot deploy your changes because someone else on an entirely orthogonal team broke something in the monolith which is not related to you it gets old really quick. Large monolithic repos with many independent targets for testing and deployment work the best at huge scales. If you are only a few hundred engineers, monorepo with monolithic deployments and tests work fine.

bottlepalm 4y ago
Tests passing should be a gate to merge into main. Tests can also be run in parallel.
- kubanczyk 4y ago
  Two green changes merged to a green main can produce a red main. This is not a commonly known fact. Just to take an example of GitHub, this check is disabled by default: > Require branches to be up to date before merging > Whether pull requests targeting a matching branch must be tested with the latest code.
  - arp242 4y ago
    > Two green changes merged to a green main can produce a red main. I'm sure this happens occasionally, but I've never experienced it, and it seems to be rare enough that it's not that big of a concern. Especially since it'll be easily remedied by either just fixing the error or just reverting one or both of the changes.
    - bottlepalm 4y ago
      It's definitely an issue if you're treating main as 'good to deploy'. Also with more devs the chance of this happening is pretty good. Definitely want to enforce a branch is up to date before allowing merge (though with a lot of devs this can become difficult) not sure if there is some sort of merge/test pipeline solution out there.
    - lxgr 4y ago
      It completely depends on your organization's merge volume and your codebase's complexity. I've seen it happen many times working on a large monolith. But requiring a passing integration branch build before merging to master, or a merge queue, solves that.
  - byroot 4y ago
    > Require branches to be up to date before merging Because this really doesn't scale well with the number of developers. For larger teams the solution is to use a merge queue, e.g. https://shopify.engineering/successfully-merging-work-1000-developers https://shopify.engineering/successfully-merging-work-1000-d... I haven't tried it, but GitHub is now offering such feature in public-beta: https://github.blog/changelog/2023-02-08-pull-request-merge-queue-public-beta/ https://github.blog/changelog/2023-02-08-pull-request-merge-...
withinboredom 4y ago
I'd ask why people are merging things that break the tests? I worked on a monolith with hundreds of devs and I can count the time the tests failed because someone force-merged something in an emergency on one hand. It was generally unacceptable to merge something when tests failed; you had to have a really good reason.
- anonzzzies 4y ago
  Yes, this seems weird; merging breaking code is not an option. The 'breaking team' will have to wait/fix on their side, not us waiting on them for deployment of our working and tested features.
  - withinboredom 4y ago
    There are some extreme circumstances where pushing broken tests to production make sense. For example, if you push a simple change to simply `return false` and disable a feature in code. In this case, the tests using it will probably fail but the desired behavior happens in production. At this point, you have a bit more time to set the tests to 'skip' while the load is shed in production. Even if you break tests on purpose, you should fix them asap as you are blocking literally every other team in the company. Thus, you need a really good reason to do so (like if you didn't do it asap, global downtime would ensue).
  - Akronymus 4y ago
    Currently we have the problem of merges of two working branches occasionally resulting in a broken one. How does one solve that?
    - withinboredom 4y ago
      Don’t merge. Rebase only. Keep a linear history. When committing, do a ff-only of ‘main’ to your branch. Yes, this forces everyone to rebase before “merging” but in practice, this results in the least amount of failures, tests being run after you resolved any conflicts, etc. If you can use GitHub merge queues, that solves a ton of this, and you can run tests on the final merge before actually merging instead of relying on rebasing.
nikita2206 4y ago
IMO for these rare occasions it should be okay to unmerge PRs. Or what could also be done is you only deploy cherry-picked releases, granted that you have a way of tracking commits in the upstream such that no commit ever gets lost.
- __turbobrew__ 4y ago
  That is possible, but at that point you need a integrations/release team to babysit the builds and make sure everything integrates back together cleanly. The last thing I want to do is have my build broken by someone on another team and then have to track them down and babysitting the revert. That is easily an hour of my time wasted.
exitheone 4y ago
It's not only tests, a rollback because your pdf generation has a bug will also mean rolling back for example a customer facing new API, slowing down the API team until the monolith is fixed or the change reverted and rebuilt.
citrin_ru 4y ago
1. Monolithic app and VCS mono-repo are orthogonal and can be mixed in different combinations (with different tradeoffs). 2. How about an old way of blocking deployments less: parts of this monolithic apps developed as libraries with a stable API, a new version of a library released only after its own tests has passed, next you can increment dependency version in the monolith and run integration tests, if they failing you can revert to the old library version and still go ahead with the deployment (if you don't depend on something added in the latest library release). If you depend on this new feature and a component providing it is broken micro-services would not help you. > If you are only a few hundred engineers, monorepo with monolithic deployments and tests work fine. And here lies a very important problem IMHO - many (if not majority) of organizations (which do at least some software development) have less than 100 software developers but the industry best practices (which include micro-service architecture) are defined by FAANG-sized organizations and at least some of these practices are sub-optimal for small shops.
adql 4y ago
>Another pain with monoliths is that they can only be deployed if the entire monolith is passing all tests. When you cannot deploy your changes because someone else on an entirely orthogonal team broke something in the monolith which is not related to you it gets old really quick. You can just... not allow code not passing test into master branch. They can fuck around in their own one, that's what branches are for
- __turbobrew__ 4y ago
  Tests can pass before they are merged but fail after merge. Tests can also regress over time and become flaky.
drw85 4y ago
Aren't you just hiding the flaws in the overall system by confiding the tests to your microservice? That's my experience at least. Things still break, you just notice it later.

bottlepalm 4y ago

Tests passing should be a gate to merge into main. Tests can also be run in parallel.

kubanczyk 4y ago
Two green changes merged to a green main can produce a red main. This is not a commonly known fact. Just to take an example of GitHub, this check is disabled by default: > Require branches to be up to date before merging > Whether pull requests targeting a matching branch must be tested with the latest code.
- arp242 4y ago
  > Two green changes merged to a green main can produce a red main. I'm sure this happens occasionally, but I've never experienced it, and it seems to be rare enough that it's not that big of a concern. Especially since it'll be easily remedied by either just fixing the error or just reverting one or both of the changes.
  - bottlepalm 4y ago
    It's definitely an issue if you're treating main as 'good to deploy'. Also with more devs the chance of this happening is pretty good. Definitely want to enforce a branch is up to date before allowing merge (though with a lot of devs this can become difficult) not sure if there is some sort of merge/test pipeline solution out there.
  - lxgr 4y ago
    It completely depends on your organization's merge volume and your codebase's complexity. I've seen it happen many times working on a large monolith. But requiring a passing integration branch build before merging to master, or a merge queue, solves that.
- byroot 4y ago
  > Require branches to be up to date before merging Because this really doesn't scale well with the number of developers. For larger teams the solution is to use a merge queue, e.g. https://shopify.engineering/successfully-merging-work-1000-developers https://shopify.engineering/successfully-merging-work-1000-d... I haven't tried it, but GitHub is now offering such feature in public-beta: https://github.blog/changelog/2023-02-08-pull-request-merge-queue-public-beta/ https://github.blog/changelog/2023-02-08-pull-request-merge-...

withinboredom 4y ago

I'd ask why people are merging things that break the tests? I worked on a monolith with hundreds of devs and I can count the time the tests failed because someone force-merged something in an emergency on one hand. It was generally unacceptable to merge something when tests failed; you had to have a really good reason.

anonzzzies 4y ago
Yes, this seems weird; merging breaking code is not an option. The 'breaking team' will have to wait/fix on their side, not us waiting on them for deployment of our working and tested features.
- withinboredom 4y ago
  There are some extreme circumstances where pushing broken tests to production make sense. For example, if you push a simple change to simply `return false` and disable a feature in code. In this case, the tests using it will probably fail but the desired behavior happens in production. At this point, you have a bit more time to set the tests to 'skip' while the load is shed in production. Even if you break tests on purpose, you should fix them asap as you are blocking literally every other team in the company. Thus, you need a really good reason to do so (like if you didn't do it asap, global downtime would ensue).
- Akronymus 4y ago
  Currently we have the problem of merges of two working branches occasionally resulting in a broken one. How does one solve that?
  - withinboredom 4y ago
    Don’t merge. Rebase only. Keep a linear history. When committing, do a ff-only of ‘main’ to your branch. Yes, this forces everyone to rebase before “merging” but in practice, this results in the least amount of failures, tests being run after you resolved any conflicts, etc. If you can use GitHub merge queues, that solves a ton of this, and you can run tests on the final merge before actually merging instead of relying on rebasing.

nikita2206 4y ago

IMO for these rare occasions it should be okay to unmerge PRs. Or what could also be done is you only deploy cherry-picked releases, granted that you have a way of tracking commits in the upstream such that no commit ever gets lost.

__turbobrew__ 4y ago
That is possible, but at that point you need a integrations/release team to babysit the builds and make sure everything integrates back together cleanly. The last thing I want to do is have my build broken by someone on another team and then have to track them down and babysitting the revert. That is easily an hour of my time wasted.

exitheone 4y ago

It's not only tests, a rollback because your pdf generation has a bug will also mean rolling back for example a customer facing new API, slowing down the API team until the monolith is fixed or the change reverted and rebuilt.

citrin_ru 4y ago

1. Monolithic app and VCS mono-repo are orthogonal and can be mixed in different combinations (with different tradeoffs). 2. How about an old way of blocking deployments less: parts of this monolithic apps developed as libraries with a stable API, a new version of a library released only after its own tests has passed, next you can increment dependency version in the monolith and run integration tests, if they failing you can revert to the old library version and still go ahead with the deployment (if you don't depend on something added in the latest library release). If you depend on this new feature and a component providing it is broken micro-services would not help you. > If you are only a few hundred engineers, monorepo with monolithic deployments and tests work fine. And here lies a very important problem IMHO - many (if not majority) of organizations (which do at least some software development) have less than 100 software developers but the industry best practices (which include micro-service architecture) are defined by FAANG-sized organizations and at least some of these practices are sub-optimal for small shops.

adql 4y ago

>Another pain with monoliths is that they can only be deployed if the entire monolith is passing all tests. When you cannot deploy your changes because someone else on an entirely orthogonal team broke something in the monolith which is not related to you it gets old really quick. You can just... not allow code not passing test into master branch. They can fuck around in their own one, that's what branches are for

__turbobrew__ 4y ago
Tests can pass before they are merged but fail after merge. Tests can also regress over time and become flaky.

drw85 4y ago

Aren't you just hiding the flaws in the overall system by confiding the tests to your microservice? That's my experience at least. Things still break, you just notice it later.