9 ms·

Not sure if this story is true: Okta used to have a monolith architecture on AWS. All their 300 engineers used to work on the same service, so they had to run a

Not sure if this story is true: Okta used to have a monolith architecture on AWS. All their 300 engineers used to work on the same service, so they had to run all their tests for every commit. The cost? $60 per commit.

cellis 4y ago
Sounds pretty cheap, relative to 300 engineers.
- samwillis 4y ago
  Assuming a dev is committing (ball park) 4 times per day, that's in the region of $55k/year per developer, or $16.5M/year all in. Thats not cheap...
  - djur 4y ago
    I would not assume that 300 developers are all committing 4 times a day. That seems very high.
    - jimbokun 4y ago
      And there you have it...$60 per commit means slowing down your commit velocity! Which means its longer before changes get tested or reviewed or deployed. The cost (money and more importantly time) of running tests fundamentally changes the development process.
      - xorcist 4y ago
        More commits are not better.
        JMS2021 4y ago
        Unless you are a PHP developer
  - cellis 4y ago
    You only need to run CI on Production, and you can squash commits. Generally a release is once per week, and at Oktas level you have a release engineer.
    - samwillis 4y ago
      Yes, I suppose I could be taking the anecdote too literally. It's much more likely they are only running all test on push to main / against pull requests / on code review.
    - TheCoelacanth 4y ago
      300 developers' work being crammed together into a weekly release without having their contributions pass CI first sounds absolutely hellish for whatever poor bastard has to track down who broke the build.
      - nicoburns 4y ago
        There's tooling like bors that will rollup commits together and run CI on them before merging to master. If it fails then the commits will be retried individually.
        TheCoelacanth 4y ago
        I suppose so, but that cuts down on the number of CI runs when merging to master because the vast majority of commits being merged to master don't fail the build. In a typical bors workflow, that holds, because you do a CI run before attempting to merge, so commits don't break the build except in the rare case that two commits that are individually okay combine in a way that breaks. If you get rid of the individual run before merging, your combined runs will hardly ever pass, so it won't cut down on the number of runs.
  - ar_lan 4y ago
    4 commits per day per developer is exceptionally high, unless these are tiny, tiny changes. I'm a very productive engineer on my team and I average less than 1 commit per day.
theamk 4y ago
That's what, 15 minutes of FTE salary+overhead? Sounds pretty reasonable.
- samwillis 4y ago
  60x4x8x5x48 = $460,800/year full time salary+overhead? I'm not sure that's right.
  - theamk 4y ago
    That's $230K year and 100% overhead Sure, $230K a year is on the high side, but Okta is headquartered in SF, so it is plausible. I cannot find a documented source for overhead, but in private conversations multiple people told me that 100% is pretty reasonable amount, and often is even higher.
hibikir 4y ago
When your system is critical enough, those kind of prices are cheap for what you get. I can't tell you about Okta, but there used to be a (now apparently gone) article by Nelson Elhague about their test parallelization infrastructure. Many thousands of pretty slow test ran for every Jenkins build, massively parallelized across machines to get an acceptable response time. At the time he wrote this, there was no modularization to speak of, so every test really had to run for every build. Very expensive, but there's a lot of value on getting the equivalent of integration-level tests for almost every critical system automatically. As a monorepo grows, the technologies used have to change to make its disadvantages spiral out of control: Serious modularization of codebases, something like Bazel to make sure code isn't overbuilt, and a testing strategy that finds the right balance of finding problems and keeping expenses in control. Every company with a monorepo ends up walking that road, but the good part is that by the time you have a problem, you definitely have proof that your organization is in a situation where the problem is worth solving. This is very different from, say, what I saw at a company that had Fred George as CTO, and was going all in on microservices with one customer and about a dozen engineers, all of which wrote in the same JVM language. At that point, all the work that avoids running too many tests is probably going to be a net loss, and even more so when one considers opportunity costs.
- sleepybrett 4y ago
  ... when your pipeline architecture becomes more of a cost sink that development into your actual product.
mbo 4y ago
> so they had to run all their tests for every commit Did they _have_ to? They could have run tests for affected transitive dependencies, cached build artefacts, randomly sampled integration tests on master. There's plenty they could have done to reduce the bill.
mbrodersen 4y ago
Running the same tests using micro-services would be more expensive.

cellis 4y ago

Sounds pretty cheap, relative to 300 engineers.

samwillis 4y ago
Assuming a dev is committing (ball park) 4 times per day, that's in the region of $55k/year per developer, or $16.5M/year all in. Thats not cheap...
- djur 4y ago
  I would not assume that 300 developers are all committing 4 times a day. That seems very high.
  - jimbokun 4y ago
    And there you have it...$60 per commit means slowing down your commit velocity! Which means its longer before changes get tested or reviewed or deployed. The cost (money and more importantly time) of running tests fundamentally changes the development process.
    - xorcist 4y ago
      More commits are not better.
      - JMS2021 4y ago
        Unless you are a PHP developer
- cellis 4y ago
  You only need to run CI on Production, and you can squash commits. Generally a release is once per week, and at Oktas level you have a release engineer.
  - samwillis 4y ago
    Yes, I suppose I could be taking the anecdote too literally. It's much more likely they are only running all test on push to main / against pull requests / on code review.
  - TheCoelacanth 4y ago
    300 developers' work being crammed together into a weekly release without having their contributions pass CI first sounds absolutely hellish for whatever poor bastard has to track down who broke the build.
    - nicoburns 4y ago
      There's tooling like bors that will rollup commits together and run CI on them before merging to master. If it fails then the commits will be retried individually.
      - TheCoelacanth 4y ago
        I suppose so, but that cuts down on the number of CI runs when merging to master because the vast majority of commits being merged to master don't fail the build. In a typical bors workflow, that holds, because you do a CI run before attempting to merge, so commits don't break the build except in the rare case that two commits that are individually okay combine in a way that breaks. If you get rid of the individual run before merging, your combined runs will hardly ever pass, so it won't cut down on the number of runs.
- ar_lan 4y ago
  4 commits per day per developer is exceptionally high, unless these are tiny, tiny changes. I'm a very productive engineer on my team and I average less than 1 commit per day.

theamk 4y ago

That's what, 15 minutes of FTE salary+overhead? Sounds pretty reasonable.

samwillis 4y ago
60x4x8x5x48 = $460,800/year full time salary+overhead? I'm not sure that's right.
- theamk 4y ago
  That's $230K year and 100% overhead Sure, $230K a year is on the high side, but Okta is headquartered in SF, so it is plausible. I cannot find a documented source for overhead, but in private conversations multiple people told me that 100% is pretty reasonable amount, and often is even higher.

hibikir 4y ago

When your system is critical enough, those kind of prices are cheap for what you get. I can't tell you about Okta, but there used to be a (now apparently gone) article by Nelson Elhague about their test parallelization infrastructure. Many thousands of pretty slow test ran for every Jenkins build, massively parallelized across machines to get an acceptable response time. At the time he wrote this, there was no modularization to speak of, so every test really had to run for every build. Very expensive, but there's a lot of value on getting the equivalent of integration-level tests for almost every critical system automatically. As a monorepo grows, the technologies used have to change to make its disadvantages spiral out of control: Serious modularization of codebases, something like Bazel to make sure code isn't overbuilt, and a testing strategy that finds the right balance of finding problems and keeping expenses in control. Every company with a monorepo ends up walking that road, but the good part is that by the time you have a problem, you definitely have proof that your organization is in a situation where the problem is worth solving. This is very different from, say, what I saw at a company that had Fred George as CTO, and was going all in on microservices with one customer and about a dozen engineers, all of which wrote in the same JVM language. At that point, all the work that avoids running too many tests is probably going to be a net loss, and even more so when one considers opportunity costs.

sleepybrett 4y ago
... when your pipeline architecture becomes more of a cost sink that development into your actual product.

mbo 4y ago

> so they had to run all their tests for every commit Did they _have_ to? They could have run tests for affected transitive dependencies, cached build artefacts, randomly sampled integration tests on master. There's plenty they could have done to reduce the bill.

mbrodersen 4y ago

Running the same tests using micro-services would be more expensive.