7 ms·

How long should your CI take?

63 points by fosterfriends 3y ago

stephenturban 3y ago
Is CI even worth it anymore? To be honest, I prefer testing on my local machine. This seem unecessary.
- filterfiber 3y ago
  The CI shows the entire team that the tests are passing, and doesn't require other devs to pull it and run the tests themselves. It prevents devs from accidentally forgetting to run tests, or thinking "This is just a typo fix, it shouldnt break any tests". Tests might also run on multiple platforms/os's/architectures/versions, etc. The CD ensures that the code is only deployed after all tests passed, it was approved, merged, and ensures a deploy isn't skipped accidentally. Depending on what your doing it could take a while to fully build new binaries, build for multiple platforms/versions/architectures, etc.
- madduci 3y ago
  Yes it is very important, since not everyone has the discipline to run tests on their own machine, leading then to errors upstream.
- tetromino_ 3y ago
  Testing on your local machine won't catch portability issues (Linux vs macOS vs Windows, x64 vs Arm64). Unless you take great care in writing hermetic tests, testing on your local machine won't catch problems hidden by non-standard versions of libraries and tools in your local development environment (e.g. you installed modern versions of bash and python on your Mac, but your users are using what the OS comes with). And for large enough projects, running tests on a single machine is simply impractical: a full test suite run on a single machine would take hours; the only solution is to shard.
  - kjkjadksj 3y ago
    You can set up arbitrary environments locally too.
    - misnome 3y ago
      Yeah! And set up scripts so that you don't need to remember how to do all the setup/building. And you could set up a trigger to build it when you make a commit so that you don't need to manually trigger every machine separately. And then you don't need CI!
    - yoav 3y ago
      You can forget to commit a file and it will work on your machine but break everything for your coworkers locally and in production. I see this class of issue often in early startups that don’t have robust CI implemented yet. Like literally once a month or more per company.
- Jenk 3y ago
  The whole point is to continuously integrate your work with that of others. If you are a solo operation then sure, no diff. On a team? Different story.
- __alexs 3y ago
  Then CI cluster has 100x the core count of my laptop, so yes.
- jdblair 3y ago
  Ah yes, the "works on my machine" testing technique
- dataflow 3y ago
  Maybe not for a small 1-person project. If you have a team project you definitely need automated tests to run regularly, and everyone needs to be able to see when/where they fail.
- recursive 3y ago
  Seems weird to put these in conflict. I do both. There's some overlap, but they seem to serve different purposes.
- zimpenfish 3y ago
  Some places it isn't possible to run the test suite on your local machine[1] (access to resources, sheer CPU power, etc.) [1] Unit tests, maybe, but not integration.
- bluGill 3y ago
  I'm waiting for a CI build as I write this - If I ran it locally it would take 3 hours, but the CI system can split across multiple AWS nodes and so be done in 25 minutes (the time of the longest build). I tried getting a more powerful machine, turns out my build becomes memory bandwidth bound at about 48 cores (building on a RAM disk makes the build take longer which is how I know it is memory bandwidth).
  - ssdspoimdsjvv 3y ago
    In my case, it would take about 45 hours to run on a single machine. Our distributed testing infrastructure gets it done in about 2. But there's a limited amount of machines, so you usually have to add 2 more hours queuing to that.
- mmmpetrichor 3y ago
  Do you always remember to run every test? Does every other dev? Are there 100 other devs? If you check in bad code will it break something for 100 other devs? It all depends on the size of the org.
- nextlevelwizard 3y ago
  Depends what you are making, how many people depend on your code working and how many team members you have
joshka 3y ago
"Developer Productivity for Humans, Part 4: Build Latency, Predictability, and Developer Productivity" [1] (discussion[2]) in the IEEE Software magazine earlier this year is a better look at this. TL;DR: less time than it currently takes. [1]: https://www.computer.org/csdl/magazine/so/2023/04/10176199/1OAJyfknInm https://www.computer.org/csdl/magazine/so/2023/04/10176199/1... [2]: https://news.ycombinator.com/item?id=37377690 https://news.ycombinator.com/item?id=37377690
capableweb 3y ago
Seems to assume that people are actively/idly waiting for CI to pass and then start the review? Or they've done the review, wait for CI to pass then hit merge? Most projects I've worked on with multiple contributors on have had a merge queue where you simply tell it to merge if/when the CI passes, then you can move on with your day.
- zimpenfish 3y ago
  Anecdata of one, obviously, but I've never worked anywhere using Github that has had auto-merge turned on. You just have to wait for the "CI complete" email (or keep refreshing the page) and hit the merge button (or, in some places, ask the appropriate person to do that.)
  - veidr 3y ago
    me either, so now anecdata of 2; but also, sounds insane of all the things a human should be in the loop for, merging to main seems like one of them
    - veidr 3y ago
      (also nuking from orbit)
    - capableweb 3y ago
      > of all the things a human should be in the loop for, merging to main seems like one of them That's what the review is for. And I'd argue that the review should be decoupled from CI, because you're not reviewing if you think the tests is passing before/after merge, you're reviewing the code and docs themselves. And once the review is done, it should be fine to merge at any point, today or tomorrow, barring any merge conflicts of course. Merging to main should be the most mundane task ever, and even a robot should be able to do it with confidence.
      - 000ooo000 3y ago
        >once the review is done, it should be fine to merge at any point, today or tomorrow Sometimes there are considerations like QA/infrastructure resourcing that dictate when something can go into master. Ideally that's rare but I have worked in a place where it's a consideration for every single merge. To expand on this a little: we had a simple branching model. To/from master for work, branch from master to cut a release. Being highly regulated, as a matter of (unchangeable) policy, every change needed testing by QA team. QA team == QA guy. Among feature work without hard release dates, there was also bug fixes with some urgency, and regulatory work with hard release dates. Given we traded flexibility in our branching model for simplicity, we instead had to consider if something could be merged to master without impacting QA load of anything already merged but not released. It worked fine for us but were were a team of <5. My remark about trading flexibility is really the crux of it all though. There's compromises that need to made and this one made sense for us at the time.
eschneider 3y ago
The missing data in that research is org size. If you've got 5-10 developers in a group, I don't see any of this making much difference in anyone's productivity. OTOH, if you're in an org with 100+ developers and every merge that goes in before you means some sort of conflict resolution, then yes, this might actually be important.
- bluGill 3y ago
  If you have 100+ developers you should segregate your system so that merge conflicts outside your group of 10 are rare. Sure someone needs to upgrade a build tool and that may sometimes mean changing everything, but those are not common operation and only a few people out of your 100 ever feel that pain. (having been there, it is often best to announce it in advance and then merge Saturday when everyone else is at home - my company will let me take a different day off in the week if I work a saturday)
graypegg 3y ago
My favourite CI/CD build discovery was `find .` (recursive file listing) being called after each step. (maybe like 12 total steps, 6 being the usual dependancy fetch, unit tests, smoke tests etc, and the other 6 being just this file listing task.) CircleCI dutifully polling the ~750k-line stdout for the find command was taking up about 3 minutes of our 5 minute process. Was done so someone could debug a failing build... supposedly.
- capableweb 3y ago
  > CircleCI dutifully polling the ~750k-line stdout for the find command was taking up about 3 minutes of our 5 minute process. Was done so someone could debug a failing build... supposedly. Doing it for a one-time CI run is fine. Forgetting to remove it is less fine. Doing it on CircleCI who have had builds you can SSH into since forever is a lot less fine, it should never have hit the config at all.
Jackevansevo 3y ago
At every single company I've ever worked the company issued laptop has been significantly faster than the machines provisioned for CI (i.e. m1 mac vs Github action free tier runners). Consequently I don't usually push code without running the tests locally, it's such a faster feedback loop. I've always wondered if it would be possible to design some proof of work concept where you could hash your filesystem + test artifacts to verify tests passed for a specific change. FWIW in yeas of development I've never had an issue where "it works on my machine" doesn't equate to the exact same result in CI.
- mixmastamyk 3y ago
  Maybe but one important reason to have it done externally is to confirm it works elsewhere.
  - TylerE 3y ago
    One bug that's bitten me a time or two is the rather annoying case-preserving but not case-sensitive MacOS filesystem. This can mean a link works locally, but not when e.g. deployed to a linux server.
  - Jackevansevo 3y ago
    It's a valid reason to have stuff run in CI (i.e. consistent environment). But for my line of work I can't think of a single scenario where the architecture / platform has ever caused issues in tests, typically it's caught at build time / when resolving dependencies/packages.
    - mixmastamyk 3y ago
      Forgetting to commit a file is pretty common. Non-existent folder, not updating out-of-tree config, etc.
  - nextlevelwizard 3y ago
    Have easy way to run your tests in container. Then you don’t need to worry, unless you are deploying on strange hardware
mmmpetrichor 3y ago
I've worked in k8s hybrid cloud automation for the past few years, and it requires deploying a k8s cluster, installing our services, and then running tests against the cluster. Our pre-merge pipeline takes 3 hours minimum and often much longer. It's definitely a drag on productivity but in a lot of complex multi-cloud environments with tons of moving pieces, I haven't yet seen a CI system achieve much better.
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]

63 points by fosterfriends 3y ago

stephenturban 3y ago

Is CI even worth it anymore? To be honest, I prefer testing on my local machine. This seem unecessary.

filterfiber 3y ago
The CI shows the entire team that the tests are passing, and doesn't require other devs to pull it and run the tests themselves. It prevents devs from accidentally forgetting to run tests, or thinking "This is just a typo fix, it shouldnt break any tests". Tests might also run on multiple platforms/os's/architectures/versions, etc. The CD ensures that the code is only deployed after all tests passed, it was approved, merged, and ensures a deploy isn't skipped accidentally. Depending on what your doing it could take a while to fully build new binaries, build for multiple platforms/versions/architectures, etc.
madduci 3y ago
Yes it is very important, since not everyone has the discipline to run tests on their own machine, leading then to errors upstream.
tetromino_ 3y ago
Testing on your local machine won't catch portability issues (Linux vs macOS vs Windows, x64 vs Arm64). Unless you take great care in writing hermetic tests, testing on your local machine won't catch problems hidden by non-standard versions of libraries and tools in your local development environment (e.g. you installed modern versions of bash and python on your Mac, but your users are using what the OS comes with). And for large enough projects, running tests on a single machine is simply impractical: a full test suite run on a single machine would take hours; the only solution is to shard.
- kjkjadksj 3y ago
  You can set up arbitrary environments locally too.
  - misnome 3y ago
    Yeah! And set up scripts so that you don't need to remember how to do all the setup/building. And you could set up a trigger to build it when you make a commit so that you don't need to manually trigger every machine separately. And then you don't need CI!
  - yoav 3y ago
    You can forget to commit a file and it will work on your machine but break everything for your coworkers locally and in production. I see this class of issue often in early startups that don’t have robust CI implemented yet. Like literally once a month or more per company.
Jenk 3y ago
The whole point is to continuously integrate your work with that of others. If you are a solo operation then sure, no diff. On a team? Different story.
__alexs 3y ago
Then CI cluster has 100x the core count of my laptop, so yes.
jdblair 3y ago
Ah yes, the "works on my machine" testing technique
dataflow 3y ago
Maybe not for a small 1-person project. If you have a team project you definitely need automated tests to run regularly, and everyone needs to be able to see when/where they fail.
recursive 3y ago
Seems weird to put these in conflict. I do both. There's some overlap, but they seem to serve different purposes.
zimpenfish 3y ago
Some places it isn't possible to run the test suite on your local machine[1] (access to resources, sheer CPU power, etc.) [1] Unit tests, maybe, but not integration.
bluGill 3y ago
I'm waiting for a CI build as I write this - If I ran it locally it would take 3 hours, but the CI system can split across multiple AWS nodes and so be done in 25 minutes (the time of the longest build). I tried getting a more powerful machine, turns out my build becomes memory bandwidth bound at about 48 cores (building on a RAM disk makes the build take longer which is how I know it is memory bandwidth).
- ssdspoimdsjvv 3y ago
  In my case, it would take about 45 hours to run on a single machine. Our distributed testing infrastructure gets it done in about 2. But there's a limited amount of machines, so you usually have to add 2 more hours queuing to that.
mmmpetrichor 3y ago
Do you always remember to run every test? Does every other dev? Are there 100 other devs? If you check in bad code will it break something for 100 other devs? It all depends on the size of the org.
nextlevelwizard 3y ago
Depends what you are making, how many people depend on your code working and how many team members you have

joshka 3y ago

"Developer Productivity for Humans, Part 4: Build Latency, Predictability, and Developer Productivity" [1] (discussion[2]) in the IEEE Software magazine earlier this year is a better look at this. TL;DR: less time than it currently takes. [1]: https://www.computer.org/csdl/magazine/so/2023/04/10176199/1OAJyfknInm https://www.computer.org/csdl/magazine/so/2023/04/10176199/1... [2]: https://news.ycombinator.com/item?id=37377690 https://news.ycombinator.com/item?id=37377690

capableweb 3y ago

Seems to assume that people are actively/idly waiting for CI to pass and then start the review? Or they've done the review, wait for CI to pass then hit merge? Most projects I've worked on with multiple contributors on have had a merge queue where you simply tell it to merge if/when the CI passes, then you can move on with your day.

zimpenfish 3y ago
Anecdata of one, obviously, but I've never worked anywhere using Github that has had auto-merge turned on. You just have to wait for the "CI complete" email (or keep refreshing the page) and hit the merge button (or, in some places, ask the appropriate person to do that.)
- veidr 3y ago
  me either, so now anecdata of 2; but also, sounds insane of all the things a human should be in the loop for, merging to main seems like one of them
  - veidr 3y ago
    (also nuking from orbit)
  - capableweb 3y ago
    > of all the things a human should be in the loop for, merging to main seems like one of them That's what the review is for. And I'd argue that the review should be decoupled from CI, because you're not reviewing if you think the tests is passing before/after merge, you're reviewing the code and docs themselves. And once the review is done, it should be fine to merge at any point, today or tomorrow, barring any merge conflicts of course. Merging to main should be the most mundane task ever, and even a robot should be able to do it with confidence.
    - 000ooo000 3y ago
      >once the review is done, it should be fine to merge at any point, today or tomorrow Sometimes there are considerations like QA/infrastructure resourcing that dictate when something can go into master. Ideally that's rare but I have worked in a place where it's a consideration for every single merge. To expand on this a little: we had a simple branching model. To/from master for work, branch from master to cut a release. Being highly regulated, as a matter of (unchangeable) policy, every change needed testing by QA team. QA team == QA guy. Among feature work without hard release dates, there was also bug fixes with some urgency, and regulatory work with hard release dates. Given we traded flexibility in our branching model for simplicity, we instead had to consider if something could be merged to master without impacting QA load of anything already merged but not released. It worked fine for us but were were a team of <5. My remark about trading flexibility is really the crux of it all though. There's compromises that need to made and this one made sense for us at the time.

eschneider 3y ago

The missing data in that research is org size. If you've got 5-10 developers in a group, I don't see any of this making much difference in anyone's productivity. OTOH, if you're in an org with 100+ developers and every merge that goes in before you means some sort of conflict resolution, then yes, this might actually be important.

bluGill 3y ago
If you have 100+ developers you should segregate your system so that merge conflicts outside your group of 10 are rare. Sure someone needs to upgrade a build tool and that may sometimes mean changing everything, but those are not common operation and only a few people out of your 100 ever feel that pain. (having been there, it is often best to announce it in advance and then merge Saturday when everyone else is at home - my company will let me take a different day off in the week if I work a saturday)

graypegg 3y ago

My favourite CI/CD build discovery was `find .` (recursive file listing) being called after each step. (maybe like 12 total steps, 6 being the usual dependancy fetch, unit tests, smoke tests etc, and the other 6 being just this file listing task.) CircleCI dutifully polling the ~750k-line stdout for the find command was taking up about 3 minutes of our 5 minute process. Was done so someone could debug a failing build... supposedly.

capableweb 3y ago
> CircleCI dutifully polling the ~750k-line stdout for the find command was taking up about 3 minutes of our 5 minute process. Was done so someone could debug a failing build... supposedly. Doing it for a one-time CI run is fine. Forgetting to remove it is less fine. Doing it on CircleCI who have had builds you can SSH into since forever is a lot less fine, it should never have hit the config at all.

Jackevansevo 3y ago

At every single company I've ever worked the company issued laptop has been significantly faster than the machines provisioned for CI (i.e. m1 mac vs Github action free tier runners). Consequently I don't usually push code without running the tests locally, it's such a faster feedback loop. I've always wondered if it would be possible to design some proof of work concept where you could hash your filesystem + test artifacts to verify tests passed for a specific change. FWIW in yeas of development I've never had an issue where "it works on my machine" doesn't equate to the exact same result in CI.

mixmastamyk 3y ago
Maybe but one important reason to have it done externally is to confirm it works elsewhere.
- TylerE 3y ago
  One bug that's bitten me a time or two is the rather annoying case-preserving but not case-sensitive MacOS filesystem. This can mean a link works locally, but not when e.g. deployed to a linux server.
- Jackevansevo 3y ago
  It's a valid reason to have stuff run in CI (i.e. consistent environment). But for my line of work I can't think of a single scenario where the architecture / platform has ever caused issues in tests, typically it's caught at build time / when resolving dependencies/packages.
  - mixmastamyk 3y ago
    Forgetting to commit a file is pretty common. Non-existent folder, not updating out-of-tree config, etc.
- nextlevelwizard 3y ago
  Have easy way to run your tests in container. Then you don’t need to worry, unless you are deploying on strange hardware

mmmpetrichor 3y ago

I've worked in k8s hybrid cloud automation for the past few years, and it requires deploying a k8s cluster, installing our services, and then running tests against the cluster. Our pre-merge pipeline takes 3 hours minimum and often much longer. It's definitely a drag on productivity but in a lot of complex multi-cloud environments with tons of moving pieces, I haven't yet seen a CI system achieve much better.

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]