4 ms·

My team is responsible for about 200 microservices being deployed, some of them have 10 or more pods. We don't do continuous delivery. Instead it's done by 5 di

My team is responsible for about 200 microservices being deployed, some of them have 10 or more pods. We don't do continuous delivery. Instead it's done by 5 different groups deploying once every week or two. Our production deployment jobs are in Jenkins and isolated. It's easy to check what was deployed when. We also have a script written that can run an environment report to see what versions and which microservices have been deployed. Along with their CPU/memory allocations, number of pods etc. Release management tracks which JIRA stories are in which release, they do it mainly by looking at master merges between prod deployments.

JamesSwift 6y ago
That seems overwhelming to me. Do you feel this is a tenable strategy? Or is the number of deployments becoming an issue?
- twh270 6y ago
  This is similar to what $myclient is doing. Parent comment doesn't mention whether identification of versions is done manually or whether they just grab master. If the latter, it's probably reasonable. At $myclient, every release to stage and prod requires teams to manually identify each version of each microservice as well as the stories (JIRA tickets) that are being deployed. This is extremely painful, time-consuming, and error-prone. Avoid at all cost; as the number of services grows, the pain/time/error cost appears to increase geometrically.
  - 100011_100001 6y ago
    Our master versions automatically create new images with unique ids and JIRA stories are associated with them. You can see that from JIRA to Gitlab and vice versa. Having said that there are people that do some manual correlation. Mainly to be able to determine that the fixes actually got in.
- 100011_100001 6y ago
  If you automate enough you almost don't need to know the specifics. We no longer monitor actively during production deployments. They just work, once every 5 months there is a deployment that needs to be escalated to us.
Crazyontap 6y ago
Any plans for showing user's birthday on the settings page? Sorry, I'm just kidding but that's the only thing I could think off when I heard the number 200!

JamesSwift 6y ago

That seems overwhelming to me. Do you feel this is a tenable strategy? Or is the number of deployments becoming an issue?

twh270 6y ago
This is similar to what $myclient is doing. Parent comment doesn't mention whether identification of versions is done manually or whether they just grab master. If the latter, it's probably reasonable. At $myclient, every release to stage and prod requires teams to manually identify each version of each microservice as well as the stories (JIRA tickets) that are being deployed. This is extremely painful, time-consuming, and error-prone. Avoid at all cost; as the number of services grows, the pain/time/error cost appears to increase geometrically.
- 100011_100001 6y ago
  Our master versions automatically create new images with unique ids and JIRA stories are associated with them. You can see that from JIRA to Gitlab and vice versa. Having said that there are people that do some manual correlation. Mainly to be able to determine that the fixes actually got in.
100011_100001 6y ago
If you automate enough you almost don't need to know the specifics. We no longer monitor actively during production deployments. They just work, once every 5 months there is a deployment that needs to be escalated to us.

Crazyontap 6y ago

Any plans for showing user's birthday on the settings page? Sorry, I'm just kidding but that's the only thing I could think off when I heard the number 200!