6 ms·

Please, step me through this. In a start-up: a) Developer builds locally b) Developer tests locally c) Developer pushes to a repository d) CI starts

Please, step me through this. In a start-up: a) Developer builds locally b) Developer tests locally c) Developer pushes to a repository d) CI starts e) CI ends f) Wait for human code review and approval g) Merge and deploy h) Observe that nothing broke / no need to revert. Is it even possible for (e) - (d) to be short enough, let alone (f) - (d), to keep the developer's attention instead of context switching? Most devs I know just context switch immediately after (c). If you care about developer productivity, you're much more likely to get results from focusing on (a) and (b), by hooking development laptops into caching and restricting system/test scope, than you are by reducing CI times, unless your CI takes some ungodly amount of time to run (several hours). For point of comparison: article examines how using monster machines can reduce the build time of Fedora to 27 minutes (not really a comparable example to most companies, but OK). My devs complain about an (admittedly unoptimized) CI time of 20 minutes (on a much simpler project than Fedora) that introduces context switching. Is the article really trying to get me to believe that Fedora developers wouldn't context switch on a 27 minute build, twiddling their thumbs for 27 minutes, but that a 35 or 55 minute build would? Something about getting under the 30 minute bar gets developers to keep their focus? I call bullshit.

isoprophlex 4y ago
Here's a single bit of anecdata: I'm currently working on a computer vision pipeline. Some pipeline components test and build in 3-5 minutes, which is the absolute upper limit on my attention. Other components take 10-20 minutes, and I always switch away to something else.
ParetoOptimal 4y ago
What about e.1) rerun failing CI build?
epage 4y ago
Agreed that there are categorical tiers of CI times (instant, check something, grab a coffee, grab a lunch, see you next day) and optimizations within a tier don't matter. I will say blocking review on CI is unneeded and, depending on your system, dramatically slows down down your whole process. PRs being posted is the main notification people get. If people instead poll for CI being done, then you'll get significant lag. This shouldn't negatively affect things. The biggest risk is CI pointing out a change big enough to invalidate the review but that has been rare in my experience. CI should just be treated as another reviewer.
BillyTheKing 4y ago
you could also skip f) by pair-program in a) and b), and then have the CI decide whether or not the code is deployable and then automatically merge into master or another pre-production branch
ninkendo 4y ago
To me, it's not about the context switching, but the gradual "learned helplessness" that comes from CI tests taking so long that it's no longer worth making small changes or fixes. Say you find a typo in a comment, or a small linting error/extra whitespace/etc, and you want to fix it. You know there's no behavioral change. Should be easy right? If your build and test suite takes hours, and the infrastructure is flaky (sometimes there are not enough job runners, sometimes the test setup fails, etc), you will often decide it's not worth it and move on. Or you'll decide to roll that change into the next PR, at which point PR's start to have a lot of unrelated work because "well, I gotta run CI for <new feature> anyway, may as well fix some linting errors while I'm at it", which is often not great practice. Or, somebody leaves a comment in your PR that is a nitpick about something which isn't that important, but you agree you should probably change it, but tests finally finished/passed. If you make a change to address the nit, tests will take hours to run again. Do you bother, or do you "leave it for a followup"? Or worse, do you force-merge the amended PR because there's no way that little nitpick will break anything? Leading to a culture of merging over incomplete tests? Tests taking a long time can gradually degrade the quality of a project in this manner. Little things never get done, or the culture starts being around skipping tests for changes that "shouldn't break anything", etc. I've seen it happen in $dayjob and it really kills the culture of quality for a project.
- solatic 4y ago
  > If your build and test suite takes hours... you will often decide it's not worth it and move on. I agree that there's a hazard here that should be defended against. My question is, if it's a small non-functional issue like that, why not mark the PR for auto-merge after you open it? Then it doesn't really matter how long the (wasteful) tests take to run, right? > tests will take hours to run again. Do you bother...? I think it's easy to blame CI in that case, but really, that sounds to me like a classic case of management prioritizing deadlines over quality. If management gives you the breathing room to and imposes the expectation not to merge unless the quality bar is reached, then you sit back and relax while you let this thing that's outside of your control take its time. Look, clearly there's a number for which the CI test suite is "too long". IME that number is several hours. What I fail to see is how the higher opex of larger machines justifies small decreases in build time, particularly when there are much better improvements to be realized by caching etc. that don't necessarily carry the additional opex.
bluGill 4y ago
the problem is how much can you context switch. My next story really needs the previous to be through at least (g) before I can start it. I keep a collection of unrelated stories, but right now I'm focused on one feature and so it is really hard to find something that is unrelated to the previous feature. I can make a snapshot at (c), and start the next feature from there, but now if code review finds a critical problem requiring some re-work I have to merge that into the previous code which may now not fit in. I'm currently working on a low priority story in a completely different area from the important feature my boss needs done because there is too much WIP from other stories. (I need to poke some people to look at my reviews right now - but I know they are working on critical must be done yesterday fixes)
baobabKoodaa 4y ago
In my current project I'm unable to do step b) because running the tests on my workstation would take longer than 8 hours. So I can only run the tests on CI, where it takes about 1 hour. So i have a 1-hour feedback loop when I'm fixing broken tests.

isoprophlex 4y ago

Here's a single bit of anecdata: I'm currently working on a computer vision pipeline. Some pipeline components test and build in 3-5 minutes, which is the absolute upper limit on my attention. Other components take 10-20 minutes, and I always switch away to something else.

ParetoOptimal 4y ago

What about e.1) rerun failing CI build?

epage 4y ago

Agreed that there are categorical tiers of CI times (instant, check something, grab a coffee, grab a lunch, see you next day) and optimizations within a tier don't matter. I will say blocking review on CI is unneeded and, depending on your system, dramatically slows down down your whole process. PRs being posted is the main notification people get. If people instead poll for CI being done, then you'll get significant lag. This shouldn't negatively affect things. The biggest risk is CI pointing out a change big enough to invalidate the review but that has been rare in my experience. CI should just be treated as another reviewer.

BillyTheKing 4y ago

you could also skip f) by pair-program in a) and b), and then have the CI decide whether or not the code is deployable and then automatically merge into master or another pre-production branch

ninkendo 4y ago

To me, it's not about the context switching, but the gradual "learned helplessness" that comes from CI tests taking so long that it's no longer worth making small changes or fixes. Say you find a typo in a comment, or a small linting error/extra whitespace/etc, and you want to fix it. You know there's no behavioral change. Should be easy right? If your build and test suite takes hours, and the infrastructure is flaky (sometimes there are not enough job runners, sometimes the test setup fails, etc), you will often decide it's not worth it and move on. Or you'll decide to roll that change into the next PR, at which point PR's start to have a lot of unrelated work because "well, I gotta run CI for <new feature> anyway, may as well fix some linting errors while I'm at it", which is often not great practice. Or, somebody leaves a comment in your PR that is a nitpick about something which isn't that important, but you agree you should probably change it, but tests finally finished/passed. If you make a change to address the nit, tests will take hours to run again. Do you bother, or do you "leave it for a followup"? Or worse, do you force-merge the amended PR because there's no way that little nitpick will break anything? Leading to a culture of merging over incomplete tests? Tests taking a long time can gradually degrade the quality of a project in this manner. Little things never get done, or the culture starts being around skipping tests for changes that "shouldn't break anything", etc. I've seen it happen in $dayjob and it really kills the culture of quality for a project.

solatic 4y ago
> If your build and test suite takes hours... you will often decide it's not worth it and move on. I agree that there's a hazard here that should be defended against. My question is, if it's a small non-functional issue like that, why not mark the PR for auto-merge after you open it? Then it doesn't really matter how long the (wasteful) tests take to run, right? > tests will take hours to run again. Do you bother...? I think it's easy to blame CI in that case, but really, that sounds to me like a classic case of management prioritizing deadlines over quality. If management gives you the breathing room to and imposes the expectation not to merge unless the quality bar is reached, then you sit back and relax while you let this thing that's outside of your control take its time. Look, clearly there's a number for which the CI test suite is "too long". IME that number is several hours. What I fail to see is how the higher opex of larger machines justifies small decreases in build time, particularly when there are much better improvements to be realized by caching etc. that don't necessarily carry the additional opex.

bluGill 4y ago

the problem is how much can you context switch. My next story really needs the previous to be through at least (g) before I can start it. I keep a collection of unrelated stories, but right now I'm focused on one feature and so it is really hard to find something that is unrelated to the previous feature. I can make a snapshot at (c), and start the next feature from there, but now if code review finds a critical problem requiring some re-work I have to merge that into the previous code which may now not fit in. I'm currently working on a low priority story in a completely different area from the important feature my boss needs done because there is too much WIP from other stories. (I need to poke some people to look at my reviews right now - but I know they are working on critical must be done yesterday fixes)

baobabKoodaa 4y ago

In my current project I'm unable to do step b) because running the tests on my workstation would take longer than 8 hours. So I can only run the tests on CI, where it takes about 1 hour. So i have a 1-hour feedback loop when I'm fixing broken tests.