61 ms·

To me, it's not about the context switching, but the gradual "learned helplessness" that comes from CI tests taking so long that it's no longer worth making sma

To me, it's not about the context switching, but the gradual "learned helplessness" that comes from CI tests taking so long that it's no longer worth making small changes or fixes. Say you find a typo in a comment, or a small linting error/extra whitespace/etc, and you want to fix it. You know there's no behavioral change. Should be easy right? If your build and test suite takes hours, and the infrastructure is flaky (sometimes there are not enough job runners, sometimes the test setup fails, etc), you will often decide it's not worth it and move on. Or you'll decide to roll that change into the next PR, at which point PR's start to have a lot of unrelated work because "well, I gotta run CI for <new feature> anyway, may as well fix some linting errors while I'm at it", which is often not great practice. Or, somebody leaves a comment in your PR that is a nitpick about something which isn't that important, but you agree you should probably change it, but tests finally finished/passed. If you make a change to address the nit, tests will take hours to run again. Do you bother, or do you "leave it for a followup"? Or worse, do you force-merge the amended PR because there's no way that little nitpick will break anything? Leading to a culture of merging over incomplete tests? Tests taking a long time can gradually degrade the quality of a project in this manner. Little things never get done, or the culture starts being around skipping tests for changes that "shouldn't break anything", etc. I've seen it happen in $dayjob and it really kills the culture of quality for a project.

solatic 4y ago
> If your build and test suite takes hours... you will often decide it's not worth it and move on. I agree that there's a hazard here that should be defended against. My question is, if it's a small non-functional issue like that, why not mark the PR for auto-merge after you open it? Then it doesn't really matter how long the (wasteful) tests take to run, right? > tests will take hours to run again. Do you bother...? I think it's easy to blame CI in that case, but really, that sounds to me like a classic case of management prioritizing deadlines over quality. If management gives you the breathing room to and imposes the expectation not to merge unless the quality bar is reached, then you sit back and relax while you let this thing that's outside of your control take its time. Look, clearly there's a number for which the CI test suite is "too long". IME that number is several hours. What I fail to see is how the higher opex of larger machines justifies small decreases in build time, particularly when there are much better improvements to be realized by caching etc. that don't necessarily carry the additional opex.
gampleman 4y ago
I think this is really important and under-examined viewpoint. Anecdotally, at work I usually contribute to 2 codebases. The first, we've invested into optimizing CI heavily, CI is generally not flakey and almost always takes less than 10 mins to run. It's also engineered to generally fail early most of the time, so if it's going to break, you often find out in < 2mins. The other codebase takes about 20mins, often takes >15mins to fail and is generally quite flakey. Subjectively speaking, the code quality on the first codebase is much higher, since making those small changes is generally quite painless.
- TYMorningCoffee 4y ago
  The causality could be reversed: the high quality causes faster less flakey builds. There could be a hidden variable like developers on the better repo are better developers or the better repository solves a less complex problem. However, I do agree with you. For me it's so much more pleasant working on the faster repo and it feels like it's because of the faster build. But the above criticism again applies to my feelings.
  - lowbloodsugar 4y ago
    I'm going to just put it out there that developers who tolerate a flakey CI are, by definition, worse developers than those who don't. Possibly because lower competence, but more likely less experience of much damage a flakey CI causes to productivity and morale.
    - ninkendo 4y ago
      I’m in an organization with tens of thousands of people, and the CI/build team is very far from me in the org chart. The CI system they’ve devised is extremely flaky, not because of anything in my code, but because the CI infrastructure itself is extremely unreliable. Am I by-definition a worse developer because I find myself in this scenario? If I were a better developer, what would I do? Change teams to the CI team so I can fix their shit? Just quit and join a different company? Not everyone is in the position to fix the things that are causing them pain.
amtamt 4y ago
> Say you find a typo in a comment, or a small linting error/extra whitespace/etc, and you want to fix it. You know there's no behavioral change. Should be easy right? There are things like ci.skip option in git for this.
- ninkendo 4y ago
  Right, but that leads to a culture of skipping CI for things, which can easily turn into skipping CI for things that are a bit less obviously-fine, which I’ve seen time and time again. Once force-merging over incomplete CI becomes a thing, people start using it more often and suddenly you get actual bugs slipping in.

solatic 4y ago

> If your build and test suite takes hours... you will often decide it's not worth it and move on. I agree that there's a hazard here that should be defended against. My question is, if it's a small non-functional issue like that, why not mark the PR for auto-merge after you open it? Then it doesn't really matter how long the (wasteful) tests take to run, right? > tests will take hours to run again. Do you bother...? I think it's easy to blame CI in that case, but really, that sounds to me like a classic case of management prioritizing deadlines over quality. If management gives you the breathing room to and imposes the expectation not to merge unless the quality bar is reached, then you sit back and relax while you let this thing that's outside of your control take its time. Look, clearly there's a number for which the CI test suite is "too long". IME that number is several hours. What I fail to see is how the higher opex of larger machines justifies small decreases in build time, particularly when there are much better improvements to be realized by caching etc. that don't necessarily carry the additional opex.

gampleman 4y ago

I think this is really important and under-examined viewpoint. Anecdotally, at work I usually contribute to 2 codebases. The first, we've invested into optimizing CI heavily, CI is generally not flakey and almost always takes less than 10 mins to run. It's also engineered to generally fail early most of the time, so if it's going to break, you often find out in < 2mins. The other codebase takes about 20mins, often takes >15mins to fail and is generally quite flakey. Subjectively speaking, the code quality on the first codebase is much higher, since making those small changes is generally quite painless.

TYMorningCoffee 4y ago
The causality could be reversed: the high quality causes faster less flakey builds. There could be a hidden variable like developers on the better repo are better developers or the better repository solves a less complex problem. However, I do agree with you. For me it's so much more pleasant working on the faster repo and it feels like it's because of the faster build. But the above criticism again applies to my feelings.
- lowbloodsugar 4y ago
  I'm going to just put it out there that developers who tolerate a flakey CI are, by definition, worse developers than those who don't. Possibly because lower competence, but more likely less experience of much damage a flakey CI causes to productivity and morale.
  - ninkendo 4y ago
    I’m in an organization with tens of thousands of people, and the CI/build team is very far from me in the org chart. The CI system they’ve devised is extremely flaky, not because of anything in my code, but because the CI infrastructure itself is extremely unreliable. Am I by-definition a worse developer because I find myself in this scenario? If I were a better developer, what would I do? Change teams to the CI team so I can fix their shit? Just quit and join a different company? Not everyone is in the position to fix the things that are causing them pain.

amtamt 4y ago

> Say you find a typo in a comment, or a small linting error/extra whitespace/etc, and you want to fix it. You know there's no behavioral change. Should be easy right? There are things like ci.skip option in git for this.

ninkendo 4y ago
Right, but that leads to a culture of skipping CI for things, which can easily turn into skipping CI for things that are a bit less obviously-fine, which I’ve seen time and time again. Once force-merging over incomplete CI becomes a thing, people start using it more often and suddenly you get actual bugs slipping in.