5 ms·

Multirepo pipelines behind a paywall is a good reason to say "No, thanks".

by chupasaurus 7y ago

azdacha 7y ago
what is your alternative please ?
- chupasaurus 7y ago
  In one of the previous jobs I've used Jenkins Pipelines (and oh boy it was PITA to set it up in 2016) to handle jobs of different projects. On current one we are using Teamcity (still cheaper than Gitlab Silver license for us) which has the support of this feature as long as branch names have a common variable in their naming convention. I.e. Project1 has branches in form feature/TICKET_NAME or fix/TICKET_NAME, Project2 uses only TICKET_NAME, TeamCity VCS settings are based upon masks of branch names so ticket 12345 would be branch 12345 in P2 but fix/12345 or feature/12345 of P1 if one exists (priorities could be tweaked too).
majkinetor 7y ago
TBH, its easily done in shell via REST API. We use custom pipeline runner from simple Powershell scripts that works even better for us then default style. We set all jobs to manual and then our script triggers them depending on commit message, person, moon phase etc. But really, this should be in core. Its very hard to do multirepository stuff. Its not that easy to do mono repo stuff too - I really need a pipeline withint sub-project, a mother pipeline, option to run whatever one I want etc... Gitlb pipelines could be a lot better.
- hardwaresofton 7y ago
  I find it very easy to make multi-repo stuff, I've also wrote about it: https://vadosware.io/post/fun-with-gitlab-ci https://vadosware.io/post/fun-with-gitlab-ci Also, running pipelines in different projects and whenever you want is accessible from the web interface... It could be cleaner, but IMO adding even more syntax to the YAML file can also be a rabbit hole -- CURL from the script section seems like not a bad middle ground. Am I misunderstanding what is being referenced?
  - Robin_f 7y ago
    This post seems very focused on projects with just a single developer. The "recognizing and tagging" versions for example completely breaks down when you have multiple people commit and merging things to master and multiple pipelines running that are trying to recognize and tag versions.
    - majkinetor 7y ago
      This is what interests me a lot. I have found that there are moments that start to give chaotic behavior in CI that are influenced by the number of people working on mono repository and some other stuff like code maturity: 1 dev, 2-5 devs, 5-10 devs and 10+ devs ..... Something that works great for any number might start to break for other number. Something effective for huge team may be very bad for smaller team. Something that works great in one moment (first several months of the project when thing still get shape 0.X.Y versions) sux when we move to 1.x version and vice versa. I am not aware of anybody giving more detailed thought about this nor examine what could be most optimal solution for any type of team (if such thing exist) or on what precise moment you start doing certain practices (for example protecting master branch).
    - hardwaresofton 7y ago
      The post is very much not complete -- there is much more a team would have to standardize to fully take a set up like this into production, bu I'm not sure that it "completely breaks down" with more people contributing. As long as there is some release coordination, a system like this can work. In particular, I've found that the lightest way to get a system like this working for x > 1 developers is to have a release-vX.X.X branch for releases that are going out and vX.X.X tags for releases once they have actually landed. You mention the recognizing and tagging versions being an issue -- are you imagining a world where two people are releasing something at the same time? I'm not exactly sure If we assume that what that posts suggests isn't feasible, regardless of how you deploy and do your CI, if your release process can be done by a human, it can likely be automated. Gitlab CI is most robust and yet easiest to understand automated CI system I've seen for making that happen (at least that was my intended point).
- chupasaurus 7y ago
  I know about this way, but it's basically building CI upon CI, doesn't sound good from any perspective.
hardwaresofton 7y ago
Multi repo pipelines are not paywalled? I can have one repo trigger jobs in another today, I've written about it a little bit: https://vadosware.io/post/fun-with-gitlab-ci/ https://vadosware.io/post/fun-with-gitlab-ci/ Am I misunderstanding what you meant?
- chupasaurus 7y ago
  Answered to other sibling comment, could elaborate further for you though. In order to pull other pipelines through Gitlab's API, you have to build another CI on top of an existing one. What I mean is that you're making the a pipeline to handle projects, but there's a catch - you have to run another pipeline to set the variables in the One pipeline to rule them all because you can't override variables from the jobs themselves! And the main pipeline would run a bunch of scripts to pass variables and handling the state. Now let's compare that to other systems from my experience. Jenkins way: one Groovy script (might be written as a special job type from UI or through VCS) to do the job with some fancy features like pause to confirm the next step from the CI's Web UI, full control of everything. Teamcity way: either could be made from UI (with reverse dependency chain because you need to start connecting pipeline from the last jobs to first) or write yours from scratch with Kotlin DSL (there's still no direct support for pipelines[1]), you can handle the process through dependency settings and by setting variables from the jobs themselves. [2] An example of Teamcity approach. (black boxes are there because of my NDA) [1]https://blog.jetbrains.com/teamcity/2017/02/kotlin-configuration-scripts-extending-the-teamcity-dsl/ https://blog.jetbrains.com/teamcity/2017/02/kotlin-configura... [2]https://imgur.com/a/jkid47J https://imgur.com/a/jkid47J edit: added picture. edit2: added link to TC blog and rewrite the link to a screenshot.
  - hardwaresofton 7y ago
    Thanks for clarifying, I see exactly what you mean now. It looks like some Gitlabbers are in this thread now so hopefully they see this as well
  - boleary-gl 7y ago
    GitLab Product Manager for Verify (CI) here. This is really valuable feedback, thank you for it. It's something we are thinking about and working on fleshing out a vision for. Two epics we have around these ideas include [1] Making CI lovable for monorepos and [2] Making CI lovable for microservices. Both of these challenge our current assumptions around the project::pipeline relationship and will help make room for improvements around that model and provide flexibility to build more complex models that work for "real world" problems like the ones you've stated. However, I don't believe we have all the answers yet so I would love more feedback on these and the issues attached. [1] https://gitlab.com/groups/gitlab-org/-/epics/812 https://gitlab.com/groups/gitlab-org/-/epics/812 [2] https://gitlab.com/groups/gitlab-org/-/epics/813 https://gitlab.com/groups/gitlab-org/-/epics/813

Multirepo pipelines behind a paywall is a good reason to say "No, thanks".

azdacha 7y ago

what is your alternative please ?

chupasaurus 7y ago
In one of the previous jobs I've used Jenkins Pipelines (and oh boy it was PITA to set it up in 2016) to handle jobs of different projects. On current one we are using Teamcity (still cheaper than Gitlab Silver license for us) which has the support of this feature as long as branch names have a common variable in their naming convention. I.e. Project1 has branches in form feature/TICKET_NAME or fix/TICKET_NAME, Project2 uses only TICKET_NAME, TeamCity VCS settings are based upon masks of branch names so ticket 12345 would be branch 12345 in P2 but fix/12345 or feature/12345 of P1 if one exists (priorities could be tweaked too).

majkinetor 7y ago

TBH, its easily done in shell via REST API. We use custom pipeline runner from simple Powershell scripts that works even better for us then default style. We set all jobs to manual and then our script triggers them depending on commit message, person, moon phase etc. But really, this should be in core. Its very hard to do multirepository stuff. Its not that easy to do mono repo stuff too - I really need a pipeline withint sub-project, a mother pipeline, option to run whatever one I want etc... Gitlb pipelines could be a lot better.

hardwaresofton 7y ago
I find it very easy to make multi-repo stuff, I've also wrote about it: https://vadosware.io/post/fun-with-gitlab-ci https://vadosware.io/post/fun-with-gitlab-ci Also, running pipelines in different projects and whenever you want is accessible from the web interface... It could be cleaner, but IMO adding even more syntax to the YAML file can also be a rabbit hole -- CURL from the script section seems like not a bad middle ground. Am I misunderstanding what is being referenced?
- Robin_f 7y ago
  This post seems very focused on projects with just a single developer. The "recognizing and tagging" versions for example completely breaks down when you have multiple people commit and merging things to master and multiple pipelines running that are trying to recognize and tag versions.
  - majkinetor 7y ago
    This is what interests me a lot. I have found that there are moments that start to give chaotic behavior in CI that are influenced by the number of people working on mono repository and some other stuff like code maturity: 1 dev, 2-5 devs, 5-10 devs and 10+ devs ..... Something that works great for any number might start to break for other number. Something effective for huge team may be very bad for smaller team. Something that works great in one moment (first several months of the project when thing still get shape 0.X.Y versions) sux when we move to 1.x version and vice versa. I am not aware of anybody giving more detailed thought about this nor examine what could be most optimal solution for any type of team (if such thing exist) or on what precise moment you start doing certain practices (for example protecting master branch).
  - hardwaresofton 7y ago
    The post is very much not complete -- there is much more a team would have to standardize to fully take a set up like this into production, bu I'm not sure that it "completely breaks down" with more people contributing. As long as there is some release coordination, a system like this can work. In particular, I've found that the lightest way to get a system like this working for x > 1 developers is to have a release-vX.X.X branch for releases that are going out and vX.X.X tags for releases once they have actually landed. You mention the recognizing and tagging versions being an issue -- are you imagining a world where two people are releasing something at the same time? I'm not exactly sure If we assume that what that posts suggests isn't feasible, regardless of how you deploy and do your CI, if your release process can be done by a human, it can likely be automated. Gitlab CI is most robust and yet easiest to understand automated CI system I've seen for making that happen (at least that was my intended point).
chupasaurus 7y ago
I know about this way, but it's basically building CI upon CI, doesn't sound good from any perspective.

hardwaresofton 7y ago

Multi repo pipelines are not paywalled? I can have one repo trigger jobs in another today, I've written about it a little bit: https://vadosware.io/post/fun-with-gitlab-ci/ https://vadosware.io/post/fun-with-gitlab-ci/ Am I misunderstanding what you meant?

chupasaurus 7y ago
Answered to other sibling comment, could elaborate further for you though. In order to pull other pipelines through Gitlab's API, you have to build another CI on top of an existing one. What I mean is that you're making the a pipeline to handle projects, but there's a catch - you have to run another pipeline to set the variables in the One pipeline to rule them all because you can't override variables from the jobs themselves! And the main pipeline would run a bunch of scripts to pass variables and handling the state. Now let's compare that to other systems from my experience. Jenkins way: one Groovy script (might be written as a special job type from UI or through VCS) to do the job with some fancy features like pause to confirm the next step from the CI's Web UI, full control of everything. Teamcity way: either could be made from UI (with reverse dependency chain because you need to start connecting pipeline from the last jobs to first) or write yours from scratch with Kotlin DSL (there's still no direct support for pipelines[1]), you can handle the process through dependency settings and by setting variables from the jobs themselves. [2] An example of Teamcity approach. (black boxes are there because of my NDA) [1]https://blog.jetbrains.com/teamcity/2017/02/kotlin-configuration-scripts-extending-the-teamcity-dsl/ https://blog.jetbrains.com/teamcity/2017/02/kotlin-configura... [2]https://imgur.com/a/jkid47J https://imgur.com/a/jkid47J edit: added picture. edit2: added link to TC blog and rewrite the link to a screenshot.
- hardwaresofton 7y ago
  Thanks for clarifying, I see exactly what you mean now. It looks like some Gitlabbers are in this thread now so hopefully they see this as well
- boleary-gl 7y ago
  GitLab Product Manager for Verify (CI) here. This is really valuable feedback, thank you for it. It's something we are thinking about and working on fleshing out a vision for. Two epics we have around these ideas include [1] Making CI lovable for monorepos and [2] Making CI lovable for microservices. Both of these challenge our current assumptions around the project::pipeline relationship and will help make room for improvements around that model and provide flexibility to build more complex models that work for "real world" problems like the ones you've stated. However, I don't believe we have all the answers yet so I would love more feedback on these and the issues attached. [1] https://gitlab.com/groups/gitlab-org/-/epics/812 https://gitlab.com/groups/gitlab-org/-/epics/812 [2] https://gitlab.com/groups/gitlab-org/-/epics/813 https://gitlab.com/groups/gitlab-org/-/epics/813