4 ms·

All of the test runners have their issues. I've had as much problems with Jenkins as I've had with Github Runners and similar solutions. The best way out of t

by zdw 3y ago

All of the test runners have their issues. I've had as much problems with Jenkins as I've had with Github Runners and similar solutions. The best way out of this is to subdivide responsibilities: - Put all build/publish/test logic in Makefiles or scripts in the repo. This means that devs can run it locally as well. The only interaction between a test runner and the codebase should be running `make <target>`. - Put all permissions and code checkout and artifact publishing credentials in the test runner, but no logic. At most you would put processing of test output like making junit/coverage/tap more web-readable. That's it. Split the efforts cleanly, and things fall into place. Also, you can switch runners easily - less needs to be reimplemented in whatever runner config DSL is picked.

nerdponx 3y ago
This is basically what we do, using TeamCity. DevOps sets up all the permissions and VCS integration, and helps figure out all the AWS IAM stuff. Then we get a nice clean little sandbox to run our build and deploy scripts in, which we commit to version control, including CloudFormation templates. It works really well: it's a comfortable separation of responsibilities that allows our team to move quickly with relatively little risk, and it involves almost no hands-on attention from DevOps once it's up and running, so they can direct their efforts elsewhere.
sevagh 3y ago
>- Put all build/publish/test logic in Makefiles or scripts in the repo. This means that devs can run it locally as well. The only interaction between a test runner and the codebase should be running `make <target>`. This always breaks because then each platform has a specific way of defining env vars or secrets for those Makefiles and bash scripts. End result is devs still can't really run CI "proper" the way it's configured in a runner.
- zdw 3y ago
  This is fundamentally an interface problem. As you called out Env vars are a pretty common and well supported interface, across multiple platforms. Do you have a better option to suggest?
  - sevagh 3y ago
    No, and I don't think any CI platform deals with it in a great way. I mean that once you account for the fact that your dev team may not have access to all of the secrets, the single bash script/Makefile stops being able to shield you from the specific CI platform, and now you have to start using Jenkins secret storage, or GitHub Actions secrets, or whatever.
  - lmm 3y ago
    Set up whatever maintains your secrets (vault?) so that it works the same way in dev, CI, and real. Have whatever manages your dev versions of services you depend on (vagrant? kubernetes-docker-whateveritisthesedays?) integrate with that so that you find your service endpoints and credentials the same way in every environment.
- maratc 3y ago
  You're correct. I sometimes forward my devs the script they need to run locally in order to replicate what Jenkins does. It has the looks of make <target> THIS=THAT AND_THIS=THAT AND_THAT_THING=SOME_OTHER_THING ETC=ETC Sometimes it won't fit into a Slack message, so I attach it as a text snippet instead.
- MereInterest 3y ago
  Maybe this is just me, but needing to define environment variables or secrets for testing seems like a massive problem in the test design. Does that mean the tests are dependent on the use of an external service, and that service can't be run locally? The only part of a CI pipeline I can imagine requiring secrets would be a release/publish step. However, those would only occur after the tests run successfully, so nothing up until that point would require secrets.
  - Tainnor 3y ago
    Just to give some examples: - your code is using private dependencies, so you can't build the code without authentication - you have some integration tests that use test containers using private docker images, so you need to do a "docker login" before running tests Locally, that would work because you're locally authenticated (and likely have all dependencies already installed etc.)
    - MereInterest 3y ago
      Ah, I think I see. There is a dependency in the code that requires copying some external resource in order to build/run locally. The developer's local environment may have a different version of that resource than the CI, so a bug ends up not being reproducible. The developer may not run the CI directly, because the developer's access tokens should be separate from the CI's access tokens. I was picturing a case where the CI was allowed privileged access to a resource to which developers were not allowed access at all.
      - Tainnor 3y ago
        > I was picturing a case where the CI was allowed privileged access to a resource to which developers were not allowed access at all. That's also possible. Imagine a pipeline which carries out a deployment in an environment where normal devs are not allowed direct access to production machines / cluster (might happen in larger companies).
john-tells-all 3y ago
> - Put all build/publish/test logic in Makefiles or scripts in the repo. This means that devs can run it locally as well. This is critical! In many teams Devs are "lazy", if any errors or weirdness happens in CI they just wave their hands and say "not a feature, not my problem". Even though they know exactly what's in the test code, the deploy code, and can see the issue. Letting Devs see and run the CI scripts locally gives them _ownership_. They don't have to fix the issues, but they have control and understanding which radically reduces the arguments and pipeline drama :)
- roland35 3y ago
  It also takes a lot less time to fix the tests locally too - rather than tweak it... push the change... oops it failed again... repeat...
  - dharmab 3y ago
    This still happens, but the surface area is greatly reduced to figuring out that the directory, variables, credentials, dependencies between stages, etc. are set up correctly
Jnr 3y ago
I totally agree. Make sure developers can run the same tasks locally as well.
eysi 3y ago
This is such a great point! Your pipelines should mostly be CI provider agnostic and runnable from anywhere, including your laptop during development. If you ever need to change a CI provider, you just move your pipelines. I'm definitely biased, I'm working at Garden[0] which allows you to do just that but it's mostly applicable for teams using Kubernetes. I paused at your comment because Garden has sometimes been called "a makefile for the cloud". Whatever tooling you use, having portable pipelines that you can iterate on and debug from your laptop is the only sane approach imo. [0] https://docs.garden.io/overview/use-cases#faster-simpler-and-portable-ci https://docs.garden.io/overview/use-cases#faster-simpler-and...

by zdw 3y ago

nerdponx 3y ago

This is basically what we do, using TeamCity. DevOps sets up all the permissions and VCS integration, and helps figure out all the AWS IAM stuff. Then we get a nice clean little sandbox to run our build and deploy scripts in, which we commit to version control, including CloudFormation templates. It works really well: it's a comfortable separation of responsibilities that allows our team to move quickly with relatively little risk, and it involves almost no hands-on attention from DevOps once it's up and running, so they can direct their efforts elsewhere.

sevagh 3y ago

>- Put all build/publish/test logic in Makefiles or scripts in the repo. This means that devs can run it locally as well. The only interaction between a test runner and the codebase should be running `make <target>`. This always breaks because then each platform has a specific way of defining env vars or secrets for those Makefiles and bash scripts. End result is devs still can't really run CI "proper" the way it's configured in a runner.

zdw 3y ago
This is fundamentally an interface problem. As you called out Env vars are a pretty common and well supported interface, across multiple platforms. Do you have a better option to suggest?
- sevagh 3y ago
  No, and I don't think any CI platform deals with it in a great way. I mean that once you account for the fact that your dev team may not have access to all of the secrets, the single bash script/Makefile stops being able to shield you from the specific CI platform, and now you have to start using Jenkins secret storage, or GitHub Actions secrets, or whatever.
- lmm 3y ago
  Set up whatever maintains your secrets (vault?) so that it works the same way in dev, CI, and real. Have whatever manages your dev versions of services you depend on (vagrant? kubernetes-docker-whateveritisthesedays?) integrate with that so that you find your service endpoints and credentials the same way in every environment.
maratc 3y ago
You're correct. I sometimes forward my devs the script they need to run locally in order to replicate what Jenkins does. It has the looks of make <target> THIS=THAT AND_THIS=THAT AND_THAT_THING=SOME_OTHER_THING ETC=ETC Sometimes it won't fit into a Slack message, so I attach it as a text snippet instead.
MereInterest 3y ago
Maybe this is just me, but needing to define environment variables or secrets for testing seems like a massive problem in the test design. Does that mean the tests are dependent on the use of an external service, and that service can't be run locally? The only part of a CI pipeline I can imagine requiring secrets would be a release/publish step. However, those would only occur after the tests run successfully, so nothing up until that point would require secrets.
- Tainnor 3y ago
  Just to give some examples: - your code is using private dependencies, so you can't build the code without authentication - you have some integration tests that use test containers using private docker images, so you need to do a "docker login" before running tests Locally, that would work because you're locally authenticated (and likely have all dependencies already installed etc.)
  - MereInterest 3y ago
    Ah, I think I see. There is a dependency in the code that requires copying some external resource in order to build/run locally. The developer's local environment may have a different version of that resource than the CI, so a bug ends up not being reproducible. The developer may not run the CI directly, because the developer's access tokens should be separate from the CI's access tokens. I was picturing a case where the CI was allowed privileged access to a resource to which developers were not allowed access at all.
    - Tainnor 3y ago
      > I was picturing a case where the CI was allowed privileged access to a resource to which developers were not allowed access at all. That's also possible. Imagine a pipeline which carries out a deployment in an environment where normal devs are not allowed direct access to production machines / cluster (might happen in larger companies).

john-tells-all 3y ago

> - Put all build/publish/test logic in Makefiles or scripts in the repo. This means that devs can run it locally as well. This is critical! In many teams Devs are "lazy", if any errors or weirdness happens in CI they just wave their hands and say "not a feature, not my problem". Even though they know exactly what's in the test code, the deploy code, and can see the issue. Letting Devs see and run the CI scripts locally gives them _ownership_. They don't have to fix the issues, but they have control and understanding which radically reduces the arguments and pipeline drama :)

roland35 3y ago
It also takes a lot less time to fix the tests locally too - rather than tweak it... push the change... oops it failed again... repeat...
- dharmab 3y ago
  This still happens, but the surface area is greatly reduced to figuring out that the directory, variables, credentials, dependencies between stages, etc. are set up correctly

Jnr 3y ago

I totally agree. Make sure developers can run the same tasks locally as well.

eysi 3y ago

This is such a great point! Your pipelines should mostly be CI provider agnostic and runnable from anywhere, including your laptop during development. If you ever need to change a CI provider, you just move your pipelines. I'm definitely biased, I'm working at Garden[0] which allows you to do just that but it's mostly applicable for teams using Kubernetes. I paused at your comment because Garden has sometimes been called "a makefile for the cloud". Whatever tooling you use, having portable pipelines that you can iterate on and debug from your laptop is the only sane approach imo. [0] https://docs.garden.io/overview/use-cases#faster-simpler-and-portable-ci https://docs.garden.io/overview/use-cases#faster-simpler-and...