11 ms·

What Is Proper Continuous Integration?

73 points by markoa 10y ago

verytrivial 10y ago
> "Of course our build takes long, we have over 10,000 lines of code!" Well, my project misses the 10 minute target by a factor of four, but does include tests and have 670kloc of C++. I claim continuous in this context, thanks very much.
- marcv81 10y ago
  Not bashing at all (you are somewhat in a better place than my own organization) but a point could be made that you could break the code down into subprojects and manage dependencies.
  - verytrivial 10y ago
    Indeed it could, and you're looking at the result, down from 2.5h-ish when I arrived. Our pipeline runneth deep ... (Though of course we'll never stop whittling!)
    - marcv81 10y ago
      Aw man!
    - deleted 10y ago
      [deleted]
  - MaulingMonkey 10y ago
    I've been on projects that did this - just linking everything back together if you touch a widely used subproject can blow that 10 minute budget too, for a single build flavor. When porting things for game development, I routinely have 30+ build flavors (say, 5 platforms, 6 configs per), and want to test at least a few of them to have even a remote chance of the checkin going green when touching core platform abstractions or config-specific stuff. A lot can be done to reduce your typical or average build times (make everything data driven so you don't even have to rebuild the code!), but C++ makes it very hard to keep your worst-case build times in check. And I tend to be one of the guys tweaking and fixing things that invoke those worst-case build times. A simple example: Annotating logging macros to catch printf-style format string errors at build time instead of crashing at runtime if you're lucky. Properly testing involves rebuilding most source files, on at least 3 build flavors to test against MSVC, Clang, and GCC...
    - solarengineer 10y ago
      I quite agree with you! I'd enjoy working on this :) It looks like a nice application of true build pipelines using GoCD. If you're interested to discuss this offline, do let me know.
- pjc50 10y ago
  500kloc here! Produces a 200MB binary in debug, 8MB in release. We managed to get it down under about 15 minutes for the most common platform by using concatenation builds. It's much much faster to build half a dozen files than a thousand. At a previous employer we were up to 1.25mloc by the time I left and we had a different solution: ccache+distcc distributing the build across a room full of computers. Few files took longer than 5m to build all by themselves, and the link phase also took about 5m. (Splitting up large C++ projects that have grown organically is really hard. It requires re-architecting and commitment to spend time on this technical debt.)
- tyingq 10y ago
  Do you experience any real consequence from this, as the author suggests? I don't deny that can happen, but I have seen teams with build times that exceed 10 minutes that don't have issues. People aren't generally dumb, so they find a way to work that doesn't involve constantly sitting around waiting on the build.
  - Macha 10y ago
    Moved from a team that had build times of 15 minutes to a team that had build times of hours, it felt crippling :(
    - tyingq 10y ago
      Hours, sure. But your original 15 minute build time exceeds the author's 10 minute cutoff. I'm questioning whether that seemed to be a huge issue.
  - verytrivial 10y ago
    We do, yes. The main one being developers will often have to make a judgement call regarding only doing partial builds and/or coverage so that the commit is not always certain to work during CI (despite what they pwomise to do regarding pre-commit testing). It also means that they have a chunk of mental load that lingers after the commit, the bot might DING them and they'll need to switch back. I don't think the distinction between 10 and 40 minutes is as much as the 40 to 2.5 hours difference -- any problem worth solving is going to take you half and hour IMHO (but let's not bike-shed that), but 2.5 hours you can start a whole investigation from scratch etc...
- LegNeato 10y ago
  If you switch to a modern build system like Bazel or Buck with remote caching, you will likely get that down to < 5 mins per change...
  - verytrivial 10y ago
    They look interesting! I keep an eye on these re: Windows support (we already use IncrediBuild there, and distcc and ccache on Linux ...) We have some monster links that can take upwards of 8 minutes on our fastest machine with SSDs etc, so the 5 mins seems keen.
marcv81 10y ago
This is an okay and mostly correct introduction for anyone who has never heard of CI, but lacks any sort of depth. The author sounds like he just discovered the concept and wants to share it, but lacks practical experience.
- 2T1Qka0rEiPr 10y ago
  And yet is a blog post from a CI platform!
  - marcv81 10y ago
    Yes, I realized after commenting. It's not a bad post, but I'm not sure why they tried their luck here.
    - Traubenfuchs 10y ago
      Hackernews provides a relevantly sized audience. "Roughly 2.6M views a day, 300K daily uniques, 3 to 3.5M monthly uniques. It depends on how you count, of course." And that was two years ago... https://news.ycombinator.com/item?id=9219581 https://news.ycombinator.com/item?id=9219581
      - marcv81 10y ago
        Yes, I suppose I am aware of their product now...
- Traubenfuchs 10y ago
  It's an advertisement for their CI software in the form of an informative blog post to hide that it is an ad so they can post it on websites like hacker news. The original poster is a cofounder of Semaphore.
  - pc86 10y ago
    The domain is "semaphoreci.com" and the service is linked very early in the article, I don't think they're trying to hide anything. It's content marketing, just like 80% of the blog posts that end up here, except for Medium which seems to be exclusively marketing.
xyzzy123 10y ago
Well a major problem in many organisations is perceived sensitivity of source code, and therefore a reluctance to outsource many tasks which rely on access to such. I like to think "we'll get there".
SideburnsOfDoom 10y ago
> As with all ideas, everybody does their own version of it in practice. Yes. > Continuous integration (CI) is confusing. At core, no. Not really. See the 1st paragraph of https://en.wikipedia.org/wiki/Continuous_integration https://en.wikipedia.org/wiki/Continuous_integration : > In software engineering, continuous integration (CI) is the practice of merging all developer working copies to a shared mainline several times a day. So the 2 words are defined as follows: Continuous: at least several times a day. Integration: merge changes to master. It's not a tool: it's a practice, a workflow that builds, tests and tools can help you do safely. Some of these "CI Server" tools also support "build and test the branches" which is useful but is not CI. This is a common mistake. See also: https://trunkbaseddevelopment.com/ https://trunkbaseddevelopment.com/
- leblancfg 10y ago
  Really does sound like "Let me shame your practices by redefining what this means in order to sell this service."
- UK-AL 10y ago
  As long as those branches are merged in daily, it's still CI. Your working copy is just an informal branch. Sticking it in a feature branch while your working on it is just convince.
  - SideburnsOfDoom 10y ago
    Yes. I meant that "build and test the branches" is not itself CI. But it can be a useful practice before merging in a CI workflow. Or it can be a painful set of long-lived branches. But some people have the idea that the build and test of any branch, i.e the automated production of a binary that passes checks, is the "integration" itself. Look out for statements like "We do CI of branches on our CI server".
TurboHaskal 10y ago
Someone introduces $buzzword. As software is a popularity contest industry driven by hype and career building blogposts, now everyone needs to do $buzzword so that they can play SEO with their resumes. This is a highly competitive market after all and you gotta put food on the table. smug-faced guru enters the scene: "Oh it looks like you're not actually doing proper $buzzword. Shame on you for raising the hand!" Now you're trapped. You either a) do what the guru says or b) remove $buzzword from your resume before you kiss your children goodbye and prepare for your new homeless life.
deleted 10y ago
[deleted]
penetrarthur 10y ago
Well, what if the CI is first building xcode project and then deploying it on device and then running tests? 10 minutes?
__mp 10y ago
I think this really depends on the code base. If I had a website where a change would trigger more than 5 minutes of integrating and testing I would go mad. I'm working with a weather model and it takes around 1 hour to build and test everything: We build against 2 super computers, 3 different compilers, 2 different architectures (CPU, GPU) and single and double precision. Building alone takes 30 minutes (more in some cases). For testing we reserve 1x node with 8 GPUs and 9 CPU cores on one machine and 2x nodes with 1 GPUs in a 30 minutes debug slot. With a pull-request based workflow we are able to push to the master multiple times a day. 40 minutes might be achievable by massively revamping our build mechanism and getting jenkins to store gigabytes in build artifacts for each run. However, I do not think it is worth it, because changes can take multiple days, or in some rare cases months to implement. If people need to wait for an hour for their tests to validate that's not so bad.
- maxxxxx 10y ago
  Our tests can take hours to days and need multiple hardware configurations. I am already happy if all tests run once a week or even once a month. In the past a lot of tests would be run maybe once or twice during a year long release cycle so once a month is already huge progress.
  - __mp 10y ago
    I think this would be too expensive for us. Why do they take so long? What area are you working in? Would it make sense to scale down the tests to a sensible size and do the longer tests over the weekends?
pokemongoaway 10y ago
Quite a lot of words in the comments and article, but not the word "automation" for some reason :P
lucaspiller 10y ago
For those using Selenium or something similar, do you have any tips for making test suites fast? Usually I just test the main flows with Selenium, then go more indepth with unit-tests where parts of the application are mocked out - but it's a trade off between accuracy and speed.
- vojant 10y ago
  Run tests in parallel, only way too make it fast.
- cstejerean 10y ago
  That's a good trade off though, and not just for speed. Selenium tests are more difficult to write and take more effort to maintain. You should definitely test as much of your application as you can with unit tests and integration tests. The end to end tests with something like Selenium should then focus on the type of bugs that you couldn't catch with the lower level tests. I've seen situations where people had Selenium tests for every possible validation error on a form. A lower level test would have been sufficient to test all validation permutations and making sure they return the right error message, and then a single high level test can ensure that when an error message is returned it is properly rendered on the screen.
- markoa 10y ago
  I'd advise definitely covering all usage scenarios with Selenium. And then parallelize. Overall unit tests should greatly outnumber UI/integration/acceptance tests. And when you catch a bug with Selenium test, replicate it at the right place with a unit test. Look up "test pyramid" if you're perhaps not familiar with it.
- dhpe 10y ago
  - Test cases should be designed to allow parallel execution (they should not interfere with each other) - Do not use sleep() for synchronisation - use explicit waits (e.g. wait until element is visible, then proceed immediately)
- blktiger 10y ago
  If you haven't encountered the concept before, I recommend looking at the testing pyramid[1]. Basically the idea is that UI/Functional tests take a long time to run so you want just enough to test the main, important flow in the application while you want a ton of unit tests because they run quickly and give you very targeted feedback about what might be broken. [1] https://martinfowler.com/bliki/TestPyramid.html https://martinfowler.com/bliki/TestPyramid.html
draw_down 10y ago
What exactly is the significance of 10 minutes? Who came up with that part? These kind of purity tests can get a little silly, I think.
- jdlshore 10y ago
  The "ten-minute build" is an idea from Extreme Programming, and you can find it in Martin Fowler's article on continuous integration[1]. (That's probably where OP got it from.) The reason, as explained in Fowler's article, is to provide rapid feedback. So the person who came up with it is Kent Beck. [1] https://www.martinfowler.com/articles/continuousIntegration.html https://www.martinfowler.com/articles/continuousIntegration....
jpalomaki 10y ago
More important than build speed is the confidence that code which passed to build really works. I think one big reason why organizations don't integrate and deploy often is that people are afraid they will break something when they do this. Then you end up pushing this big and scary thing forward and forward. This of course does not take away any of the risks, but at least you need to close your eyes and wish for the best only once a month.
mazlix 10y ago
I really don't think build time is very important. With a separate CI server it doesn't much matter if takes an hour or <10 minutes for a build to complete, the gain for me is that it's no longer "your" time. It's some other server that's doing the work with a saved state (master). I'll typically just merge into master on GH, let the CI server take it from there and not bother checking the site manually when the build is complete, it's not necessary. Our QA and CI process handles that. I consider it a major advantage (benefits far outweigh risk) that I can typically merge my ticket into master, close my laptop and head home.
skiplecariboo 10y ago
Is that now a thing to put your css at the end of the page?

73 points by markoa 10y ago

> "Of course our build takes long, we have over 10,000 lines of code!" Well, my project misses the 10 minute target by a factor of four, but does include tests and have 670kloc of C++. I claim continuous in this context, thanks very much.

marcv81 10y ago
Not bashing at all (you are somewhat in a better place than my own organization) but a point could be made that you could break the code down into subprojects and manage dependencies.
- verytrivial 10y ago
  Indeed it could, and you're looking at the result, down from 2.5h-ish when I arrived. Our pipeline runneth deep ... (Though of course we'll never stop whittling!)
  - marcv81 10y ago
    Aw man!
  - deleted 10y ago
    [deleted]
- MaulingMonkey 10y ago
  I've been on projects that did this - just linking everything back together if you touch a widely used subproject can blow that 10 minute budget too, for a single build flavor. When porting things for game development, I routinely have 30+ build flavors (say, 5 platforms, 6 configs per), and want to test at least a few of them to have even a remote chance of the checkin going green when touching core platform abstractions or config-specific stuff. A lot can be done to reduce your typical or average build times (make everything data driven so you don't even have to rebuild the code!), but C++ makes it very hard to keep your worst-case build times in check. And I tend to be one of the guys tweaking and fixing things that invoke those worst-case build times. A simple example: Annotating logging macros to catch printf-style format string errors at build time instead of crashing at runtime if you're lucky. Properly testing involves rebuilding most source files, on at least 3 build flavors to test against MSVC, Clang, and GCC...
  - solarengineer 10y ago
    I quite agree with you! I'd enjoy working on this :) It looks like a nice application of true build pipelines using GoCD. If you're interested to discuss this offline, do let me know.
pjc50 10y ago
500kloc here! Produces a 200MB binary in debug, 8MB in release. We managed to get it down under about 15 minutes for the most common platform by using concatenation builds. It's much much faster to build half a dozen files than a thousand. At a previous employer we were up to 1.25mloc by the time I left and we had a different solution: ccache+distcc distributing the build across a room full of computers. Few files took longer than 5m to build all by themselves, and the link phase also took about 5m. (Splitting up large C++ projects that have grown organically is really hard. It requires re-architecting and commitment to spend time on this technical debt.)
tyingq 10y ago
Do you experience any real consequence from this, as the author suggests? I don't deny that can happen, but I have seen teams with build times that exceed 10 minutes that don't have issues. People aren't generally dumb, so they find a way to work that doesn't involve constantly sitting around waiting on the build.
- Macha 10y ago
  Moved from a team that had build times of 15 minutes to a team that had build times of hours, it felt crippling :(
  - tyingq 10y ago
    Hours, sure. But your original 15 minute build time exceeds the author's 10 minute cutoff. I'm questioning whether that seemed to be a huge issue.
- verytrivial 10y ago
  We do, yes. The main one being developers will often have to make a judgement call regarding only doing partial builds and/or coverage so that the commit is not always certain to work during CI (despite what they pwomise to do regarding pre-commit testing). It also means that they have a chunk of mental load that lingers after the commit, the bot might DING them and they'll need to switch back. I don't think the distinction between 10 and 40 minutes is as much as the 40 to 2.5 hours difference -- any problem worth solving is going to take you half and hour IMHO (but let's not bike-shed that), but 2.5 hours you can start a whole investigation from scratch etc...
LegNeato 10y ago
If you switch to a modern build system like Bazel or Buck with remote caching, you will likely get that down to < 5 mins per change...
- verytrivial 10y ago
  They look interesting! I keep an eye on these re: Windows support (we already use IncrediBuild there, and distcc and ccache on Linux ...) We have some monster links that can take upwards of 8 minutes on our fastest machine with SSDs etc, so the 5 mins seems keen.

marcv81 10y ago

This is an okay and mostly correct introduction for anyone who has never heard of CI, but lacks any sort of depth. The author sounds like he just discovered the concept and wants to share it, but lacks practical experience.

2T1Qka0rEiPr 10y ago
And yet is a blog post from a CI platform!
- marcv81 10y ago
  Yes, I realized after commenting. It's not a bad post, but I'm not sure why they tried their luck here.
  - Traubenfuchs 10y ago
    Hackernews provides a relevantly sized audience. "Roughly 2.6M views a day, 300K daily uniques, 3 to 3.5M monthly uniques. It depends on how you count, of course." And that was two years ago... https://news.ycombinator.com/item?id=9219581 https://news.ycombinator.com/item?id=9219581
    - marcv81 10y ago
      Yes, I suppose I am aware of their product now...
Traubenfuchs 10y ago
It's an advertisement for their CI software in the form of an informative blog post to hide that it is an ad so they can post it on websites like hacker news. The original poster is a cofounder of Semaphore.
- pc86 10y ago
  The domain is "semaphoreci.com" and the service is linked very early in the article, I don't think they're trying to hide anything. It's content marketing, just like 80% of the blog posts that end up here, except for Medium which seems to be exclusively marketing.

xyzzy123 10y ago

Well a major problem in many organisations is perceived sensitivity of source code, and therefore a reluctance to outsource many tasks which rely on access to such. I like to think "we'll get there".

SideburnsOfDoom 10y ago

> As with all ideas, everybody does their own version of it in practice. Yes. > Continuous integration (CI) is confusing. At core, no. Not really. See the 1st paragraph of https://en.wikipedia.org/wiki/Continuous_integration https://en.wikipedia.org/wiki/Continuous_integration : > In software engineering, continuous integration (CI) is the practice of merging all developer working copies to a shared mainline several times a day. So the 2 words are defined as follows: Continuous: at least several times a day. Integration: merge changes to master. It's not a tool: it's a practice, a workflow that builds, tests and tools can help you do safely. Some of these "CI Server" tools also support "build and test the branches" which is useful but is not CI. This is a common mistake. See also: https://trunkbaseddevelopment.com/ https://trunkbaseddevelopment.com/

leblancfg 10y ago
Really does sound like "Let me shame your practices by redefining what this means in order to sell this service."
UK-AL 10y ago
As long as those branches are merged in daily, it's still CI. Your working copy is just an informal branch. Sticking it in a feature branch while your working on it is just convince.
- SideburnsOfDoom 10y ago
  Yes. I meant that "build and test the branches" is not itself CI. But it can be a useful practice before merging in a CI workflow. Or it can be a painful set of long-lived branches. But some people have the idea that the build and test of any branch, i.e the automated production of a binary that passes checks, is the "integration" itself. Look out for statements like "We do CI of branches on our CI server".

TurboHaskal 10y ago

Someone introduces $buzzword. As software is a popularity contest industry driven by hype and career building blogposts, now everyone needs to do $buzzword so that they can play SEO with their resumes. This is a highly competitive market after all and you gotta put food on the table. smug-faced guru enters the scene: "Oh it looks like you're not actually doing proper $buzzword. Shame on you for raising the hand!" Now you're trapped. You either a) do what the guru says or b) remove $buzzword from your resume before you kiss your children goodbye and prepare for your new homeless life.

deleted 10y ago

[deleted]

penetrarthur 10y ago

Well, what if the CI is first building xcode project and then deploying it on device and then running tests? 10 minutes?

__mp 10y ago

I think this really depends on the code base. If I had a website where a change would trigger more than 5 minutes of integrating and testing I would go mad. I'm working with a weather model and it takes around 1 hour to build and test everything: We build against 2 super computers, 3 different compilers, 2 different architectures (CPU, GPU) and single and double precision. Building alone takes 30 minutes (more in some cases). For testing we reserve 1x node with 8 GPUs and 9 CPU cores on one machine and 2x nodes with 1 GPUs in a 30 minutes debug slot. With a pull-request based workflow we are able to push to the master multiple times a day. 40 minutes might be achievable by massively revamping our build mechanism and getting jenkins to store gigabytes in build artifacts for each run. However, I do not think it is worth it, because changes can take multiple days, or in some rare cases months to implement. If people need to wait for an hour for their tests to validate that's not so bad.

maxxxxx 10y ago
Our tests can take hours to days and need multiple hardware configurations. I am already happy if all tests run once a week or even once a month. In the past a lot of tests would be run maybe once or twice during a year long release cycle so once a month is already huge progress.
- __mp 10y ago
  I think this would be too expensive for us. Why do they take so long? What area are you working in? Would it make sense to scale down the tests to a sensible size and do the longer tests over the weekends?

pokemongoaway 10y ago

Quite a lot of words in the comments and article, but not the word "automation" for some reason :P

lucaspiller 10y ago

For those using Selenium or something similar, do you have any tips for making test suites fast? Usually I just test the main flows with Selenium, then go more indepth with unit-tests where parts of the application are mocked out - but it's a trade off between accuracy and speed.

vojant 10y ago
Run tests in parallel, only way too make it fast.
cstejerean 10y ago
That's a good trade off though, and not just for speed. Selenium tests are more difficult to write and take more effort to maintain. You should definitely test as much of your application as you can with unit tests and integration tests. The end to end tests with something like Selenium should then focus on the type of bugs that you couldn't catch with the lower level tests. I've seen situations where people had Selenium tests for every possible validation error on a form. A lower level test would have been sufficient to test all validation permutations and making sure they return the right error message, and then a single high level test can ensure that when an error message is returned it is properly rendered on the screen.
markoa 10y ago
I'd advise definitely covering all usage scenarios with Selenium. And then parallelize. Overall unit tests should greatly outnumber UI/integration/acceptance tests. And when you catch a bug with Selenium test, replicate it at the right place with a unit test. Look up "test pyramid" if you're perhaps not familiar with it.
dhpe 10y ago
- Test cases should be designed to allow parallel execution (they should not interfere with each other) - Do not use sleep() for synchronisation - use explicit waits (e.g. wait until element is visible, then proceed immediately)
blktiger 10y ago
If you haven't encountered the concept before, I recommend looking at the testing pyramid[1]. Basically the idea is that UI/Functional tests take a long time to run so you want just enough to test the main, important flow in the application while you want a ton of unit tests because they run quickly and give you very targeted feedback about what might be broken. [1] https://martinfowler.com/bliki/TestPyramid.html https://martinfowler.com/bliki/TestPyramid.html

draw_down 10y ago

What exactly is the significance of 10 minutes? Who came up with that part? These kind of purity tests can get a little silly, I think.

jdlshore 10y ago
The "ten-minute build" is an idea from Extreme Programming, and you can find it in Martin Fowler's article on continuous integration[1]. (That's probably where OP got it from.) The reason, as explained in Fowler's article, is to provide rapid feedback. So the person who came up with it is Kent Beck. [1] https://www.martinfowler.com/articles/continuousIntegration.html https://www.martinfowler.com/articles/continuousIntegration....

jpalomaki 10y ago

More important than build speed is the confidence that code which passed to build really works. I think one big reason why organizations don't integrate and deploy often is that people are afraid they will break something when they do this. Then you end up pushing this big and scary thing forward and forward. This of course does not take away any of the risks, but at least you need to close your eyes and wish for the best only once a month.

mazlix 10y ago

I really don't think build time is very important. With a separate CI server it doesn't much matter if takes an hour or <10 minutes for a build to complete, the gain for me is that it's no longer "your" time. It's some other server that's doing the work with a saved state (master). I'll typically just merge into master on GH, let the CI server take it from there and not bother checking the site manually when the build is complete, it's not necessary. Our QA and CI process handles that. I consider it a major advantage (benefits far outweigh risk) that I can typically merge my ticket into master, close my laptop and head home.

skiplecariboo 10y ago

Is that now a thing to put your css at the end of the page?