4 ms·

Unfortunatelly there is still no way to run CI file locally which is major turn down. Deprecated `exec` command is hell to use. I should be able to install run

Unfortunatelly there is still no way to run CI file locally which is major turn down. Deprecated `exec` command is hell to use. I should be able to install runner on developers machine and just run and debug pipeline there, with or without Gitlab server present.

DogLover_ 7y ago
Deprecated `exec` command is hell to use Curious what makes it hell to use? Unfortunatelly there is still no way to run CI file locally which is major turn down I don't think thats correct. As you mention there is `exec`. Do you know of better alternatives?
- 0xEFF 7y ago
  My use case is to develop jobs quickly, locally. exec is a PITA because it doesn't replicate the same environment jobs actually run in and it doesn't have the same functionality as a "normal" runner so it ends up causing more problems than the alternative of simply not using it at all. As I mentioned above, exec is no longer maintained and has missing features.
- majkinetor 7y ago
  One example that comes to mind is codefresh.io. It allows local run of the pipeline (and even debug via breakpoints) and only requires docker to be installed locally. I currently use task system with Gitlab to make any job in 1-3 lines. Then I can easily replicate build locally. The task engine is either installed on runner's host itself or run inside docker container.
  - jl-gitlab 7y ago
    We are looking to add breakpoints in an upcoming release, building on our interactive web terminals feature. If you're interested in providing feedback the public issue is https://gitlab.com/gitlab-org/gitlab/issues/39527 https://gitlab.com/gitlab-org/gitlab/issues/39527.
ipodopt 7y ago
I am particular about how I code my builds so they are portable. I like debugging locally, its faster. Gitlab-Ci does not encourage this and I usually end up refactoring my colleagues code all the time to this end. Not sure if having a runner is the proper solution to this problem. Off the cuff I feel like there are better ways to go about it if I where making a CI server from scratch. Like only allowing single Docker commands per action? Then you would just those and have a nice portable and consistent environment. But Docker isn't very friendly as development tooling as you have to go through length to persist file artifacts and such. Would seem like there is some room in the space...
- orf 7y ago
  Gitlab executes shell commands, you can and should run those locally to debug. If your putting complex build logic inside gitlab definitions then perhaps you’re doing it wrong - “make test” or “make release” should be all that Gitlab runs.
  - ipodopt 7y ago
    Right, my point was that I shouldn't need to enforce this. Gitlab yaml allows and often goes out of its way to encourage non-portable practices.
    - orf 7y ago
      But... it just runs shell commands. It doesn’t encourage you to do more complex things, especially compared to other CI systems like Jenkins. My point is your complaint seems more general than Gitlab itself.
      - ipodopt 7y ago
        Well, let's see. I am going to exaggerate a touch here but here is what I see at my workplace: a. They will write an inline script. Solution: Rip it out into a file. b. Each Job is using a different container. Solution: use Docker with mapped volumes or we all have the same machines (this might not be possible for some workloads) and only use a single container. Why not run the shell directly you may ask? b.1 Each container has a different environment and my machine has one. b.2 Some things do not run natively on my machine that do on theirs. c. Variables are defined in the yaml instead of the scripts. Solution: Make a local .env file see b. So sure, a local gitlab runner us probably best patch here. But I am imagine if someone made a container runtime specifically for this then wrapped that in a CI config it could be very ergonomic. Which is what I would explore if I where to write a IC today. I am sure some build system today already does this... Things outside (kindof) of Gitlab's control that I might as well complain about while I am here: d. And regardless, there are transient dependencies everywhere (which I blame the all language creators for really, mfs :P ). So everything gets fucked eventually anyhow. Wouldn't it be nice if you could whip out a three year old project, run the build, and its exactly as you left it? e. People thinks all right to allow for mutable versions so I have to asdjnasdaskdjasdlasjdl;askjd;asldkj = asdasdljkdasl;kdjlaskdjaslkjd some non-human readable hash if I am feeling paranoid. d. and e. are firmly in the language creators hand I believe. But Gitlab does offer artifact repositories (which are non-portable and could also easily could be). Gitlab could also save dependency trees in different languages. But really I would gripe to the tool/language creators.
rumanator 7y ago
> Unfortunatelly there is still no way to run CI file locally which is major turn down. You can run CI files locally. Just install a GitLab runner and run it through the command line.
- 0xEFF 7y ago
  The functionality is no longer being maintained [1]. Notice: Not all features of .gitlab-ci.yml are supported by exec. [2] [1]: https://gitlab.com/gitlab-org/gitlab-runner/issues/2710 https://gitlab.com/gitlab-org/gitlab-runner/issues/2710 [2]: https://docs.gitlab.com/runner/commands/#gitlab-runner-exec https://docs.gitlab.com/runner/commands/#gitlab-runner-exec
  - deastman 7y ago
    Hi, Darren here - PDM for Runner (deastman@gitlab.com). Yes - as we mention on this page, https://docs.gitlab.com/runner/commands/#limitations-of-gitlab-runner-exec https://docs.gitlab.com/runner/commands/#limitations-of-gitl..., there are limitations with the current exec implementation. We also have this issue https://gitlab.com/gitlab-org/gitlab-runner/issues/2797#proposal https://gitlab.com/gitlab-org/gitlab-runner/issues/2797#prop... open on this topic. In reading the threads here, I will update that issue on GitLab with our latest thinking regarding implementation challenges, ideas, etc. At a high-level today, the Runner is really just a job execution engine for want of a better description. The Runner is focused on taking a job and executing it. So as we look at this problem of executing a pipeline locally, we have to think about all of the logic that's happening in the GitLab CI Rails app - stuff like artifacts passing for example. Then we have to figure out how to parse the .gitlab-ci.yml file. Ideas like, do we do this in the context of a specific project, or a connection to any GitLab API. So tons of pros and cons to consider for sure. Given the in-depth discussion on this topic, its clear that I and the team need to refocus on this topic sooner rather than later. In chatting with one of our lead backend engineers, his point of view is that " I totally agree. I use exec quite often and I'd love to have this command a first class citizen again." So I am going to spend some time in the next couple of days capturing some of the thoughts here and updating the issue in GitLab with our latest ideas and proposals. We definitely need to get this done, so any help from the community is welcome, as we think solving this is going be a bit of work and make take a few release cycles.
jl-gitlab 7y ago
We have a new PM for the runner and this is on his agenda to research and improve. The reality is that the job execution environment is complex - each job does just have a simple script section which is just shell commands, but there is a lot of context that impacts behavior (to which environment are you deploying, what environment variables are set specific to that environment, is it a protected environment with variables that are shareable to any random user, are artifacts being passed, and so on.) Making iterating on pipelines easier is definitely on our agenda, and adding your thoughts to issues like https://gitlab.com/gitlab-org/gitlab-runner https://gitlab.com/gitlab-org/gitlab-runner /issues/2226 or making new ones for your specific use case will help. We are also working on other oblique solutions to make pipelines easier to iterate on like https://gitlab.com/gitlab-org/gitlab/issues/39527 https://gitlab.com/gitlab-org/gitlab/issues/39527, which help solve the problem in a different way where you don't need to try to replicate the state of the world on your local environment but can still find and fix problems quickly. EDIT: I should also mention, https://gitlab.com/groups/gitlab-org/-/epics/2072 https://gitlab.com/groups/gitlab-org/-/epics/2072 is a big priority for us where we want to reduce the time to first green pipeline (not just for new users, but for new projects too). Feedback is welcome there as well.

DogLover_ 7y ago

Deprecated `exec` command is hell to use Curious what makes it hell to use? Unfortunatelly there is still no way to run CI file locally which is major turn down I don't think thats correct. As you mention there is `exec`. Do you know of better alternatives?

0xEFF 7y ago
My use case is to develop jobs quickly, locally. exec is a PITA because it doesn't replicate the same environment jobs actually run in and it doesn't have the same functionality as a "normal" runner so it ends up causing more problems than the alternative of simply not using it at all. As I mentioned above, exec is no longer maintained and has missing features.
majkinetor 7y ago
One example that comes to mind is codefresh.io. It allows local run of the pipeline (and even debug via breakpoints) and only requires docker to be installed locally. I currently use task system with Gitlab to make any job in 1-3 lines. Then I can easily replicate build locally. The task engine is either installed on runner's host itself or run inside docker container.
- jl-gitlab 7y ago
  We are looking to add breakpoints in an upcoming release, building on our interactive web terminals feature. If you're interested in providing feedback the public issue is https://gitlab.com/gitlab-org/gitlab/issues/39527 https://gitlab.com/gitlab-org/gitlab/issues/39527.

ipodopt 7y ago

I am particular about how I code my builds so they are portable. I like debugging locally, its faster. Gitlab-Ci does not encourage this and I usually end up refactoring my colleagues code all the time to this end. Not sure if having a runner is the proper solution to this problem. Off the cuff I feel like there are better ways to go about it if I where making a CI server from scratch. Like only allowing single Docker commands per action? Then you would just those and have a nice portable and consistent environment. But Docker isn't very friendly as development tooling as you have to go through length to persist file artifacts and such. Would seem like there is some room in the space...

orf 7y ago
Gitlab executes shell commands, you can and should run those locally to debug. If your putting complex build logic inside gitlab definitions then perhaps you’re doing it wrong - “make test” or “make release” should be all that Gitlab runs.
- ipodopt 7y ago
  Right, my point was that I shouldn't need to enforce this. Gitlab yaml allows and often goes out of its way to encourage non-portable practices.
  - orf 7y ago
    But... it just runs shell commands. It doesn’t encourage you to do more complex things, especially compared to other CI systems like Jenkins. My point is your complaint seems more general than Gitlab itself.
    - ipodopt 7y ago
      Well, let's see. I am going to exaggerate a touch here but here is what I see at my workplace: a. They will write an inline script. Solution: Rip it out into a file. b. Each Job is using a different container. Solution: use Docker with mapped volumes or we all have the same machines (this might not be possible for some workloads) and only use a single container. Why not run the shell directly you may ask? b.1 Each container has a different environment and my machine has one. b.2 Some things do not run natively on my machine that do on theirs. c. Variables are defined in the yaml instead of the scripts. Solution: Make a local .env file see b. So sure, a local gitlab runner us probably best patch here. But I am imagine if someone made a container runtime specifically for this then wrapped that in a CI config it could be very ergonomic. Which is what I would explore if I where to write a IC today. I am sure some build system today already does this... Things outside (kindof) of Gitlab's control that I might as well complain about while I am here: d. And regardless, there are transient dependencies everywhere (which I blame the all language creators for really, mfs :P ). So everything gets fucked eventually anyhow. Wouldn't it be nice if you could whip out a three year old project, run the build, and its exactly as you left it? e. People thinks all right to allow for mutable versions so I have to asdjnasdaskdjasdlasjdl;askjd;asldkj = asdasdljkdasl;kdjlaskdjaslkjd some non-human readable hash if I am feeling paranoid. d. and e. are firmly in the language creators hand I believe. But Gitlab does offer artifact repositories (which are non-portable and could also easily could be). Gitlab could also save dependency trees in different languages. But really I would gripe to the tool/language creators.

rumanator 7y ago

> Unfortunatelly there is still no way to run CI file locally which is major turn down. You can run CI files locally. Just install a GitLab runner and run it through the command line.

0xEFF 7y ago
The functionality is no longer being maintained [1]. Notice: Not all features of .gitlab-ci.yml are supported by exec. [2] [1]: https://gitlab.com/gitlab-org/gitlab-runner/issues/2710 https://gitlab.com/gitlab-org/gitlab-runner/issues/2710 [2]: https://docs.gitlab.com/runner/commands/#gitlab-runner-exec https://docs.gitlab.com/runner/commands/#gitlab-runner-exec
- deastman 7y ago
  Hi, Darren here - PDM for Runner (deastman@gitlab.com). Yes - as we mention on this page, https://docs.gitlab.com/runner/commands/#limitations-of-gitlab-runner-exec https://docs.gitlab.com/runner/commands/#limitations-of-gitl..., there are limitations with the current exec implementation. We also have this issue https://gitlab.com/gitlab-org/gitlab-runner/issues/2797#proposal https://gitlab.com/gitlab-org/gitlab-runner/issues/2797#prop... open on this topic. In reading the threads here, I will update that issue on GitLab with our latest thinking regarding implementation challenges, ideas, etc. At a high-level today, the Runner is really just a job execution engine for want of a better description. The Runner is focused on taking a job and executing it. So as we look at this problem of executing a pipeline locally, we have to think about all of the logic that's happening in the GitLab CI Rails app - stuff like artifacts passing for example. Then we have to figure out how to parse the .gitlab-ci.yml file. Ideas like, do we do this in the context of a specific project, or a connection to any GitLab API. So tons of pros and cons to consider for sure. Given the in-depth discussion on this topic, its clear that I and the team need to refocus on this topic sooner rather than later. In chatting with one of our lead backend engineers, his point of view is that " I totally agree. I use exec quite often and I'd love to have this command a first class citizen again." So I am going to spend some time in the next couple of days capturing some of the thoughts here and updating the issue in GitLab with our latest ideas and proposals. We definitely need to get this done, so any help from the community is welcome, as we think solving this is going be a bit of work and make take a few release cycles.

jl-gitlab 7y ago

We have a new PM for the runner and this is on his agenda to research and improve. The reality is that the job execution environment is complex - each job does just have a simple script section which is just shell commands, but there is a lot of context that impacts behavior (to which environment are you deploying, what environment variables are set specific to that environment, is it a protected environment with variables that are shareable to any random user, are artifacts being passed, and so on.) Making iterating on pipelines easier is definitely on our agenda, and adding your thoughts to issues like https://gitlab.com/gitlab-org/gitlab-runner https://gitlab.com/gitlab-org/gitlab-runner /issues/2226 or making new ones for your specific use case will help. We are also working on other oblique solutions to make pipelines easier to iterate on like https://gitlab.com/gitlab-org/gitlab/issues/39527 https://gitlab.com/gitlab-org/gitlab/issues/39527, which help solve the problem in a different way where you don't need to try to replicate the state of the world on your local environment but can still find and fix problems quickly. EDIT: I should also mention, https://gitlab.com/groups/gitlab-org/-/epics/2072 https://gitlab.com/groups/gitlab-org/-/epics/2072 is a big priority for us where we want to reduce the time to first green pipeline (not just for new users, but for new projects too). Feedback is welcome there as well.