4 ms·

>- Put all build/publish/test logic in Makefiles or scripts in the repo. This means that devs can run it locally as well. The only interaction between a test ru

by sevagh 3y ago

>- Put all build/publish/test logic in Makefiles or scripts in the repo. This means that devs can run it locally as well. The only interaction between a test runner and the codebase should be running `make <target>`. This always breaks because then each platform has a specific way of defining env vars or secrets for those Makefiles and bash scripts. End result is devs still can't really run CI "proper" the way it's configured in a runner.

zdw 3y ago
This is fundamentally an interface problem. As you called out Env vars are a pretty common and well supported interface, across multiple platforms. Do you have a better option to suggest?
- sevagh 3y ago
  No, and I don't think any CI platform deals with it in a great way. I mean that once you account for the fact that your dev team may not have access to all of the secrets, the single bash script/Makefile stops being able to shield you from the specific CI platform, and now you have to start using Jenkins secret storage, or GitHub Actions secrets, or whatever.
- lmm 3y ago
  Set up whatever maintains your secrets (vault?) so that it works the same way in dev, CI, and real. Have whatever manages your dev versions of services you depend on (vagrant? kubernetes-docker-whateveritisthesedays?) integrate with that so that you find your service endpoints and credentials the same way in every environment.
maratc 3y ago
You're correct. I sometimes forward my devs the script they need to run locally in order to replicate what Jenkins does. It has the looks of make <target> THIS=THAT AND_THIS=THAT AND_THAT_THING=SOME_OTHER_THING ETC=ETC Sometimes it won't fit into a Slack message, so I attach it as a text snippet instead.
MereInterest 3y ago
Maybe this is just me, but needing to define environment variables or secrets for testing seems like a massive problem in the test design. Does that mean the tests are dependent on the use of an external service, and that service can't be run locally? The only part of a CI pipeline I can imagine requiring secrets would be a release/publish step. However, those would only occur after the tests run successfully, so nothing up until that point would require secrets.
- Tainnor 3y ago
  Just to give some examples: - your code is using private dependencies, so you can't build the code without authentication - you have some integration tests that use test containers using private docker images, so you need to do a "docker login" before running tests Locally, that would work because you're locally authenticated (and likely have all dependencies already installed etc.)
  - MereInterest 3y ago
    Ah, I think I see. There is a dependency in the code that requires copying some external resource in order to build/run locally. The developer's local environment may have a different version of that resource than the CI, so a bug ends up not being reproducible. The developer may not run the CI directly, because the developer's access tokens should be separate from the CI's access tokens. I was picturing a case where the CI was allowed privileged access to a resource to which developers were not allowed access at all.
    - Tainnor 3y ago
      > I was picturing a case where the CI was allowed privileged access to a resource to which developers were not allowed access at all. That's also possible. Imagine a pipeline which carries out a deployment in an environment where normal devs are not allowed direct access to production machines / cluster (might happen in larger companies).

by sevagh 3y ago

zdw 3y ago

This is fundamentally an interface problem. As you called out Env vars are a pretty common and well supported interface, across multiple platforms. Do you have a better option to suggest?

sevagh 3y ago
No, and I don't think any CI platform deals with it in a great way. I mean that once you account for the fact that your dev team may not have access to all of the secrets, the single bash script/Makefile stops being able to shield you from the specific CI platform, and now you have to start using Jenkins secret storage, or GitHub Actions secrets, or whatever.
lmm 3y ago
Set up whatever maintains your secrets (vault?) so that it works the same way in dev, CI, and real. Have whatever manages your dev versions of services you depend on (vagrant? kubernetes-docker-whateveritisthesedays?) integrate with that so that you find your service endpoints and credentials the same way in every environment.

maratc 3y ago

You're correct. I sometimes forward my devs the script they need to run locally in order to replicate what Jenkins does. It has the looks of make <target> THIS=THAT AND_THIS=THAT AND_THAT_THING=SOME_OTHER_THING ETC=ETC Sometimes it won't fit into a Slack message, so I attach it as a text snippet instead.

MereInterest 3y ago

Maybe this is just me, but needing to define environment variables or secrets for testing seems like a massive problem in the test design. Does that mean the tests are dependent on the use of an external service, and that service can't be run locally? The only part of a CI pipeline I can imagine requiring secrets would be a release/publish step. However, those would only occur after the tests run successfully, so nothing up until that point would require secrets.

Tainnor 3y ago
Just to give some examples: - your code is using private dependencies, so you can't build the code without authentication - you have some integration tests that use test containers using private docker images, so you need to do a "docker login" before running tests Locally, that would work because you're locally authenticated (and likely have all dependencies already installed etc.)
- MereInterest 3y ago
  Ah, I think I see. There is a dependency in the code that requires copying some external resource in order to build/run locally. The developer's local environment may have a different version of that resource than the CI, so a bug ends up not being reproducible. The developer may not run the CI directly, because the developer's access tokens should be separate from the CI's access tokens. I was picturing a case where the CI was allowed privileged access to a resource to which developers were not allowed access at all.
  - Tainnor 3y ago
    > I was picturing a case where the CI was allowed privileged access to a resource to which developers were not allowed access at all. That's also possible. Imagine a pipeline which carries out a deployment in an environment where normal devs are not allowed direct access to production machines / cluster (might happen in larger companies).