4 ms·
Ask HN: Is there a good way to run integration tests on Kubernetes?
Background: We currently use Argo Workflows to orchestrate end-to-end integration tests.
Basically, a set of workflow definitions define containers for all the apps that the tests need (typically 7-8 microservice apps, plus services like databases, caches, etc.). Then the last step of the workflow runs test.
The downside is that the workflow needs to duplicate the Kubernetes config that we already deploy with ArgoCD. We keep apps, tests, and ArgoCD definitions in separate repos, and this stuff gets out of sync.
Plus, Argo Workflows is not so great. Poor UI, very flaky, relatively slow, and of course everything is YAML, which is horrible to work with.
We've used CircleCI before, but it's really not good at defining big pipelines with lots of dependencies.
Is there anything better?
- verdverm 5y agoMaybe define with CUE and generate the Yaml. https://cuelang.org https://cuelang.org You'd then get reusable sections and imports.
- atombender 5y agoCue is nice, but I'm not sure that solves anything. For one, it would require that we first migrate all our current Kustomize-based Kubernetes manifests to Cue, and it would require that all apps be designed from the ground up as "repurposable" templates. I'm more interested in the orchestration aspect here. We already have apps expressed as deployable resources, but we need something that can do that.
- verdverm 5y agoah, maybe https://spinnaker.io/ https://spinnaker.io/ is what you are looking for?
- atombender 5y agoHm, that seems roughly analogous to ArgoCD?
- olafmol 5y agoHi OP/TS! We now have a CircleCI Config SDK in early preview. You can use this SDK to generate a Dynamic Config script to checkout the files you want and generate a very specific list of the jobs and workflows you want to run. More info on this: https://discuss.circleci.com/t/early-preview-circleci-config-sdk-visual-config-editor-write-config-with-code-or-drag-and-drop/40984#circleci-config-sdk-4 https://discuss.circleci.com/t/early-preview-circleci-config... and https://circleci.com/docs/2.0/dynamic-config/ https://circleci.com/docs/2.0/dynamic-config/ Disclaimer: i work for CircleCI. Happy to help if there are any additional questions. We are working on some very cool improvements and optimisations to boost CircleCI into 2022, so any feedback and suggestions are very welcome!
- atombender 5y agoHi, thanks. I guess this won't let us define steps based on Kubernetes manifests? Part of our challenge (though not all of it) is that apps need to be defined twice: A single app needs configs, secrets, usually multiple "daemon" processes and background jobs, and then you of course have things like databases. We have the Kubernetes manifests for production/staging, and then the end-to-end tests have to define everything all over again. A single end-to-end test may end up using a dozen apps, so this is a lot of duplication. Last I checked, CircleCI did not support running a single job across multiple machines? Some of our end-to-end tests run some very big things that require multiple machines.
- olafmol 5y agoHi @atombender, with the new SDK you would be able to parse your K8s manifests, and create (dynamic) configs with the parsed info. This is actually an interesting usecase, tnx! Could you explain what you mean with "running a single job across multiple machines"? How would that exactly work clustering- and result-agreggation wise? Of course it's perfectly possible to run parallel jobs over multiple machines (https://circleci.com/docs/2.0/parallelism-faster-jobs/ https://circleci.com/docs/2.0/parallelism-faster-jobs/) but i guess you mean something different?
- 5y ago