3 ms·

Hi OP/TS! We now have a CircleCI Config SDK in early preview. You can use this SDK to generate a Dynamic Config script to checkout the files you want and genera

Hi OP/TS! We now have a CircleCI Config SDK in early preview. You can use this SDK to generate a Dynamic Config script to checkout the files you want and generate a very specific list of the jobs and workflows you want to run. More info on this: https://discuss.circleci.com/t/early-preview-circleci-config-sdk-visual-config-editor-write-config-with-code-or-drag-and-drop/40984#circleci-config-sdk-4 https://discuss.circleci.com/t/early-preview-circleci-config... and https://circleci.com/docs/2.0/dynamic-config/ https://circleci.com/docs/2.0/dynamic-config/ Disclaimer: i work for CircleCI. Happy to help if there are any additional questions. We are working on some very cool improvements and optimisations to boost CircleCI into 2022, so any feedback and suggestions are very welcome!

atombender 5y ago
Hi, thanks. I guess this won't let us define steps based on Kubernetes manifests? Part of our challenge (though not all of it) is that apps need to be defined twice: A single app needs configs, secrets, usually multiple "daemon" processes and background jobs, and then you of course have things like databases. We have the Kubernetes manifests for production/staging, and then the end-to-end tests have to define everything all over again. A single end-to-end test may end up using a dozen apps, so this is a lot of duplication. Last I checked, CircleCI did not support running a single job across multiple machines? Some of our end-to-end tests run some very big things that require multiple machines.
- olafmol 5y ago
  Hi @atombender, with the new SDK you would be able to parse your K8s manifests, and create (dynamic) configs with the parsed info. This is actually an interesting usecase, tnx! Could you explain what you mean with "running a single job across multiple machines"? How would that exactly work clustering- and result-agreggation wise? Of course it's perfectly possible to run parallel jobs over multiple machines (https://circleci.com/docs/2.0/parallelism-faster-jobs/ https://circleci.com/docs/2.0/parallelism-faster-jobs/) but i guess you mean something different?
  - atombender 5y ago
    Hm, parsing the Kubernetes manifests sounds like a lot of work, and wouldn't guarantee 1:1 correctness. Some Kubernetes things (secrets, mounted configmaps, and ingresses, for example) don't even have CircleCI equivalents. By multiple machines I mean running apps (both our own, and also third-party software like databases) on multiple machines as part of the test. A single test may have to stand up too much for a single machine to handle. The beauty of Kubernetes is that this just works: Kubernetes will "pack" the nodes however it likes, and you just need to specify your resource requirements.

atombender 5y ago

Hi, thanks. I guess this won't let us define steps based on Kubernetes manifests? Part of our challenge (though not all of it) is that apps need to be defined twice: A single app needs configs, secrets, usually multiple "daemon" processes and background jobs, and then you of course have things like databases. We have the Kubernetes manifests for production/staging, and then the end-to-end tests have to define everything all over again. A single end-to-end test may end up using a dozen apps, so this is a lot of duplication. Last I checked, CircleCI did not support running a single job across multiple machines? Some of our end-to-end tests run some very big things that require multiple machines.

olafmol 5y ago
Hi @atombender, with the new SDK you would be able to parse your K8s manifests, and create (dynamic) configs with the parsed info. This is actually an interesting usecase, tnx! Could you explain what you mean with "running a single job across multiple machines"? How would that exactly work clustering- and result-agreggation wise? Of course it's perfectly possible to run parallel jobs over multiple machines (https://circleci.com/docs/2.0/parallelism-faster-jobs/ https://circleci.com/docs/2.0/parallelism-faster-jobs/) but i guess you mean something different?
- atombender 5y ago
  Hm, parsing the Kubernetes manifests sounds like a lot of work, and wouldn't guarantee 1:1 correctness. Some Kubernetes things (secrets, mounted configmaps, and ingresses, for example) don't even have CircleCI equivalents. By multiple machines I mean running apps (both our own, and also third-party software like databases) on multiple machines as part of the test. A single test may have to stand up too much for a single machine to handle. The beauty of Kubernetes is that this just works: Kubernetes will "pack" the nodes however it likes, and you just need to specify your resource requirements.