3 ms·

Kinda off-topic, but: > During the Q&A an attendee asked about Kubernetes CI. The audience member said that CI for the project cost "between $100,000 and $200,

by arp242 3y ago

Kinda off-topic, but: > During the Q&A an attendee asked about Kubernetes CI. The audience member said that CI for the project cost "between $100,000 and $200,000 a month" wtf?! That's a crazy amount of money. At 10,000 PRs a month, that's $10 to $20 per PR(!!)

zufallsheld 3y ago
Looking at one random PR there are about 15 jobs running. One of these ran for 40 minutes. Assuming every job starts a new virtual machine that would be 10 hours of CI time. If they use very beefy machines, we can probably somehow justify this cost.
fragmede 3y ago
how much should it cost per PR, in your head, and why?
- xboxnolifes 2y ago
  Assuming a PR is using 1 hour of compute time for CI related tasks, i expect it to take around 1 hour of compute cost. Is there any service that would realistically cost $10-20 per hour of used compute for CI related tasks?
debarshri 3y ago
I think gitops is the culprit. Around 2017-2018, everything was git centric and that was the best practice. With some sane process, you can definitely bring down that price ridiculously. I have seen in lot of organisations, every push triggers a build. That is wasted compute.
- pjc50 3y ago
  Every push should trigger a build. This should not cost a significant amount of real money, especially not compared to the salary of the developer waiting for the build.
  - mindwok 3y ago
    Why should every push trigger a build?
    - ffsm8 2y ago
      Because why not? There is no reason to filter/abstain from builds given the second statement. Especially if it has the potential to save time for the developer, whose time is going to cost significant more then the build ever will
      - arp242 2y ago
        Because they're spending up to $200,000 every month on it. That's why not. I don't think I've ever seen the "developer time is worth more!" true-ism being uncritically repeated than here. They can easily hire over 12 full-time devs for this money.
        ffsm8 2y ago
        given the second statement. You're still ignoring it. They've got an issue in their CI, it's not that they're triggering it to often. It's because each build costs way too much. Though we technically don't know that either, as the attendee didn't specify how many builds they did, technically. (At least I didn't see it in TFA) It was also explicitly stated that the goal was to have the Dev run the build locally and sign the commit with the build status so they could skip CI on the server and save cost there. Considering that context, I believe that the Dev time = money argument is justified.
      - deleted 2y ago
        [deleted]
eddiezane 2y ago
Said audience member here. "The Kubernetes project" includes a bit more than just https://github.com/kubernetes/kubernetes https://github.com/kubernetes/kubernetes, but yes that's accurate. Here's our GCP spend for the past month: https://imgur.com/a/VVJTSKx https://imgur.com/a/VVJTSKx. Note that does not include a separate AWS cluster that we are migrating jobs too. A large chunk of this comes from the nature of distributed tests. We need to reproduce the environment, spin up compute, etc. We do have a large problem with flaky tests on the project as well. Whether that's timeouts, memory/cpu consumption creep over time, loads of other things. We talk about how one day we'd like to get to the granulairty of being able to go to a SIG and say, "this flaky test of yours is costing the project $x in retries. Please dedicate some resources to fix it". How we distribute the artifacts is a whole different conversation. The container world is unique in that voluntary mirrors are not as possible as with linux packages and other binaries. If this space interests you please join us at either [SIG K8s Infra](https://github.com/kubernetes/community/tree/master/sig-k8s-infra https://github.com/kubernetes/community/tree/master/sig-k8s-...) or [SIG Testing](https://github.com/kubernetes/community/blob/master/sig-testing/README.md https://github.com/kubernetes/community/blob/master/sig-test...)!

by arp242 3y ago

zufallsheld 3y ago

Looking at one random PR there are about 15 jobs running. One of these ran for 40 minutes. Assuming every job starts a new virtual machine that would be 10 hours of CI time. If they use very beefy machines, we can probably somehow justify this cost.

fragmede 3y ago

how much should it cost per PR, in your head, and why?

xboxnolifes 2y ago
Assuming a PR is using 1 hour of compute time for CI related tasks, i expect it to take around 1 hour of compute cost. Is there any service that would realistically cost $10-20 per hour of used compute for CI related tasks?

debarshri 3y ago

I think gitops is the culprit. Around 2017-2018, everything was git centric and that was the best practice. With some sane process, you can definitely bring down that price ridiculously. I have seen in lot of organisations, every push triggers a build. That is wasted compute.

pjc50 3y ago
Every push should trigger a build. This should not cost a significant amount of real money, especially not compared to the salary of the developer waiting for the build.
- mindwok 3y ago
  Why should every push trigger a build?
  - ffsm8 2y ago
    Because why not? There is no reason to filter/abstain from builds given the second statement. Especially if it has the potential to save time for the developer, whose time is going to cost significant more then the build ever will
    - arp242 2y ago
      Because they're spending up to $200,000 every month on it. That's why not. I don't think I've ever seen the "developer time is worth more!" true-ism being uncritically repeated than here. They can easily hire over 12 full-time devs for this money.
      - ffsm8 2y ago
        given the second statement. You're still ignoring it. They've got an issue in their CI, it's not that they're triggering it to often. It's because each build costs way too much. Though we technically don't know that either, as the attendee didn't specify how many builds they did, technically. (At least I didn't see it in TFA) It was also explicitly stated that the goal was to have the Dev run the build locally and sign the commit with the build status so they could skip CI on the server and save cost there. Considering that context, I believe that the Dev time = money argument is justified.
    - deleted 2y ago
      [deleted]

eddiezane 2y ago

Said audience member here. "The Kubernetes project" includes a bit more than just https://github.com/kubernetes/kubernetes https://github.com/kubernetes/kubernetes, but yes that's accurate. Here's our GCP spend for the past month: https://imgur.com/a/VVJTSKx https://imgur.com/a/VVJTSKx. Note that does not include a separate AWS cluster that we are migrating jobs too. A large chunk of this comes from the nature of distributed tests. We need to reproduce the environment, spin up compute, etc. We do have a large problem with flaky tests on the project as well. Whether that's timeouts, memory/cpu consumption creep over time, loads of other things. We talk about how one day we'd like to get to the granulairty of being able to go to a SIG and say, "this flaky test of yours is costing the project $x in retries. Please dedicate some resources to fix it". How we distribute the artifacts is a whole different conversation. The container world is unique in that voluntary mirrors are not as possible as with linux packages and other binaries. If this space interests you please join us at either [SIG K8s Infra](https://github.com/kubernetes/community/tree/master/sig-k8s-infra https://github.com/kubernetes/community/tree/master/sig-k8s-...) or [SIG Testing](https://github.com/kubernetes/community/blob/master/sig-testing/README.md https://github.com/kubernetes/community/blob/master/sig-test...)!