3 ms·
Great post, thanks for sharing. We should link that in the Pipeline Efficiency docs: https://docs.gitlab.com/ee/ci/pipelines/pipeline_efficiency.html https://do
by dnsmichi 5y ago
Great post, thanks for sharing. We should link that in the Pipeline Efficiency docs: https://docs.gitlab.com/ee/ci/pipelines/pipeline_efficiency.html https://docs.gitlab.com/ee/ci/pipelines/pipeline_efficiency....
I've given a talk about similar ideas for efficient pipelines at Continuous Lifecycle, the slides have many URLs inside to learn async: https://docs.google.com/presentation/d/1nq7Q4WMv6rQc6WFJCRqjtgtgQfgtKTtwSFw4IXXuRsA/edit https://docs.google.com/presentation/d/1nq7Q4WMv6rQc6WFJCRqj...
And if you want dive deeper, a free full day workshop with exercises to practice config, resource, caches, container images and more. I've created it for the Open Source Automation Days in early October.
Slides with exercises: https://docs.google.com/presentation/d/12ifd_w7G492FHRaS9CXAXOGky20pEQuV-Qox8V4Rq8s/edit https://docs.google.com/presentation/d/12ifd_w7G492FHRaS9CXA...
Exercises+solutions: https://gitlab.com/gitlab-de/workshops/ci-cd-pipeline-efficiency-workshop/pipeline-efficiency-workshop-exercises https://gitlab.com/gitlab-de/workshops/ci-cd-pipeline-effici...
I did not have time yet to write a blog post sharing more insights on the exercises, but they should be self-explaining on the slides, with solutions in the repository. Let me know how it goes, feel free to repurpose for your own blog posts, and send documentation updates please :)
- dnsmichi 5y ago> We should link that in the Pipeline Efficiency docs I've shared the resources and this HN topic with GitLab's technical writing team to make tutorials more visible on docs.gitlab.com https://gitlab.com/gitlab-org/technical-writing/-/issues/511#note_763331836 https://gitlab.com/gitlab-org/technical-writing/-/issues/511...
- iduoad 5y agoSo much to learn here, thank you for sharing!
- dnsmichi 5y agoI went ahead and blogged about the workshop, thanks y'all for the inspiration :) https://dev.to/dnsmichi/efficient-devsecops-pipelines-in-a-cloud-native-world-free-workshop-hmk https://dev.to/dnsmichi/efficient-devsecops-pipelines-in-a-c... If you learn a new trick or gem, please blog and share with our community :-) Overview of the topics inside the workshop: - Introduction: CI/CD meets Dev, Sec and Ops - CI/CD: Terminology and first steps - Analyse & Identify - Learn using the GitLab CI Pipeline Exporter to monitor the exercise project throughout the workshop. - Efficiency actions - Config Efficiency: CI/CD Variables in variables, job templates (YAML anchors, extends), includes (local, remote), rules and conditions (if, dynamic variables, conditional includes), !reference tags (script, rules), maintain own CI/CD templates (include templates, override config values), parent-child pipelines, multi project pipelines, better error messages to fix failures fast - Resource Use Efficiency: Identification, max pipeline duration analysis, fail fast with stages grouping, fail fast with async needs, analyse blocking stages pipeline (solution with needs), matrix builds for parallel execution (pratice: combine matrix and extends, combine matrix and !reference), extends merge strategies (with and without !reference) - CI/CD Infrastructure Efficiency: Optimization ideas, custom build images, optimize builds with C++ as example, GitLab runner resource analysis (sharing, tags, external dependencies, Kubernetes), local runner exercise, resource groups, storage usage analysis, caching (Python dependency exercise, including when:always on failed jobs) - Auto-scaling: Overview, AWS auto-scaling with GitLab Runner with Terraform, insights into Spot Runners on AWS Graviton - Group discussion - Deployment Strategies: IaC, GitOps, Terraform, Kubernetes, registries - Security: Secrets in CI/CD variables, Hashicorp Vault, secrets scanning, vulnerability scanning - Observability: CI/CD Runner monitoring, SLOs, quality gates, CI/CD Tracing - More efficiency ideas: Auto DevOps, Fast vs Resources, Conclusion and tips